| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-24 | |||
| 16:14:25 | mriedem | BjoernT: i think i've sorted out the issue though so the gmr isn't really necessary | |
| 16:14:45 | BjoernT | ok I'm still curious what it looks like | |
| 16:14:48 | mriedem | you're just timing out trying to pull bdms for 705 instances from the conductor within 60 seconds (default rpc_response_timeout) | |
| 16:15:02 | BjoernT | took me a day to actually make that association with failing service checkins and the image cache manager | |
| 16:15:20 | BjoernT | we run rpc_response timeout upwards of 300 already | |
| 16:15:30 | BjoernT | now even larger | |
| 16:15:35 | mriedem | rpc_response_timeout is used mostly globally so tweaking it for one thing does not surprise me that it causes side effects elsehwere | |
| 16:15:50 | BjoernT | yeah I know | |
| 16:16:01 | mriedem | we also have a long_rpc_timeout since rocky, but you're on an older release than that, and this code / query isn't using that timeout | |
| 16:16:27 | BjoernT | I had one workaround in mind to shutoff image cache manger globally except one service node, which would not be used for scheduling | |
| 16:16:43 | BjoernT | ok, I'm still stuck on pike but good to know | |
| 16:17:07 | mriedem | i could maybe hack something up here and you can take a look at it | |
| 16:17:38 | BjoernT | yes I should be able to test that in that environment once I get signoff | |
| 16:22:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078 | |
| 16:22:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Create RequestGroup from neutron port https://review.openstack.org/625941 | |
| 16:22:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079 | |
| 16:22:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Read port resource request from Neutron https://review.openstack.org/630720 | |
| 16:22:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Include requested_resources to allocation candidate query https://review.openstack.org/625942 | |
| 16:22:04 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject resize with port having resource request https://review.openstack.org/630722 | |
| 16:22:04 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject server create with port having resource request https://review.openstack.org/630721 | |
| 16:22:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject evacuate with port having resource request https://review.openstack.org/630724 | |
| 16:22:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migrate with port having resource request https://review.openstack.org/630723 | |
| 16:22:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 16:22:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject unshelve with port having resource request https://review.openstack.org/630725 | |
| 16:22:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | |
| 16:22:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527 | |
| 16:22:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | |
| 16:22:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | |
| 16:22:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 16:22:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | |
| 16:22:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:22:15 | gibi | I'm sorry | |
| 16:22:22 | sean-k-mooney | :) | |
| 16:26:12 | gibi | mriedem: I've fixed your comments in https://review.openstack.org/#/c/570078 | |
| 16:27:17 | melwitt | mriedem, efried_2dmtg: would one of y'all be able to run the nova meeting next week? I will be traveling for work and unable to run it | |
| 16:29:24 | mriedem | yes i can | |
| 16:31:56 | mriedem | gibi: lgtm, +2 | |
| 16:33:42 | gibi | mriedem: thanks | |
| 16:33:52 | gibi | the next in that series is pretty similar | |
| 16:34:27 | melwitt | mriedem: great, thanks | |
| 16:34:32 | mriedem | ok in the tab queue | |
| 16:34:49 | gibi | coll | |
| 16:34:52 | gibi | cool | |
| 17:03:47 | cfriesen | artom: stephenfin: any news on the live migration stuff? I'm happy to help test experimental code. | |
| 17:04:37 | artom | cfriesen, heya, afraid not, was mostly downstream this week. I need to get back to it and put up stuff, even WIP | |
| 17:15:44 | cfriesen | I totally get the downstream stuff. :) Story of my life. | |
| 17:22:36 | mriedem | BjoernT_: i should have a patch up shortly, just running final tests | |
| 17:32:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042 | |
| 17:32:20 | mriedem | BjoernT_: ^ | |
| 17:37:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042 | |
| 17:37:28 | openstackgerrit | melanie witt proposed openstack/nova master: Raise 403 instead of 500 error from attach volume API https://review.openstack.org/632904 | |
| 17:37:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 17:40:48 | mriedem | kashyap: seeing a weird issue in the gate with libvirt 4.0.0 where sometimes an instance delete times out in libvirt and it looks like there is a loop in libvirt that is busted https://bugs.launchpad.net/nova/+bug/1813192 | |
| 17:40:49 | openstack | Launchpad bug 1813192 in OpenStack Compute (nova) "libvirt: instance delete fails with "Cannot destroy instance, operation time out: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)" in bionic nodes (libvirt 4.0.0, qemu 2.11)" [Medium,Confirmed] | |
| 17:40:58 | kashyap | mriedem: Haha, eerie you ping me! | |
| 17:41:10 | kashyap | mriedem: I was _just_ writing a nice message for you to ask for help w/ some unit test debugging | |
| 17:41:14 | mriedem | only seeing this on ubuntu bionic | |
| 17:41:32 | kashyap | Oh, the "cannot acquire state change lock" error. | |
| 17:41:50 | kashyap | Hmm, it is unofficially the most difficult one to debug. | |
| 17:42:14 | mriedem | only 3 hits in 7 days so not a huge issue | |
| 17:42:26 | kashyap | Will check the logs once I have dinner. I was in back-to-back meeting hell | |
| 17:42:42 | kashyap | Thanks for the heads-up | |
| 17:43:28 | kashyap | I was about to ping you to borrow some eyes on these unit test failures on the version bump WIP patch: https://review.openstack.org/#/c/632507/ | |
| 17:43:47 | kashyap | But the patch is not fully done; so I don't want to waste your time clicking on it. But I at least reduced the failures from 136 to 5, though. | |
| 17:44:10 | kashyap | And I spent a crap load of time w/ PDB yesterday & some time today. Was still duking around. | |
| 17:45:31 | kashyap | On the off chance you have a minute, any comments / snide remarks on the review are welcome. | |
| 17:45:47 | mriedem | if tempest is failing you've got bigger problems than just unit tests | |
| 17:46:42 | mriedem | something is busted with console logs http://logs.openstack.org/07/632507/3/check/tempest-full/73a00be/controller/logs/screen-n-cpu.txt.gz?level=TRACE | |
| 17:49:45 | kashyap | Yeah, it is. I know partly the fault from this morning's debugging. And I have a half-baked local patch. | |
| 17:50:05 | kashyap | I'll keep digging tomm / this week. And get it locally passing and only _then_ use Gate resources. | |
| 17:50:15 | stephenfin | cfriesen: What artom said, though the SR-IOV live migration patches are progressing quite nicely | |
| 17:50:37 | stephenfin | adrianc: You should probably stick that in the queue for a runway, if you haven't done so already | |
| 17:51:11 | sean-k-mooney | i need to adress jays comment on the base patch but they should be ready for a runway next week or so | |
| 17:51:50 | sean-k-mooney | the neutron patch need to be ready first | |
| 17:52:39 | sean-k-mooney | this one https://review.openstack.org/#/c/620123/ it work but it needs tests | |
| 17:53:50 | cfriesen | I think https://review.openstack.org/#/c/621646 is ready for wider review. This is the code to preserve UEFI nvram appropriately. | |
| 17:55:39 | sean-k-mooney | oh cool. is the nvram used to store bios/uefi setting or dos it have some other use? | |
| 17:57:30 | sean-k-mooney | cfriesen: why are ye adding keep_nvram=false in all the drivers when that is the default in the base dirver they all inherit form | |
| 17:58:02 | sean-k-mooney | oh never mind | |
| 17:58:32 | sean-k-mooney | you are over rideing the function not over loading them that makes sense | |
| 18:00:33 | cfriesen | sean-k-mooney: I think the nvram is for bios stuff. I think it's also possible to do things like stick crash messages in the NVRAM. | |
| 18:30:03 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 18:36:40 | sean-k-mooney | jackding: fyi if the only thing the operator saw was the release not for ^ it should provide enough info fro them to either make the change or tell them where to find the infor required to make the feature work | |
| 19:00:13 | kashyap | Yikes, even with log filters, the libvirtd log on the above bug (1813192) is 250MB, it crashed my browser | |
| 19:00:49 | kashyap | Ah, Matt is AFK; I'll post a comment on the bug in the morning. I'll sign off for now. | |
| 19:01:53 | sean-k-mooney | kashyap: i have heard good things about http://lnav.org/ for what it is worth | |
| 19:02:48 | kashyap | sean-k-mooney: Hi, what is it? /me checks out | |
| 19:03:39 | sean-k-mooney | its a termial bases log navigato that transparent detect the format,compress and support live viewing with filter and even sql queries | |
| 19:03:40 | adrianc | stephenfin: planning to do so next week. | |
| 19:04:55 | adrianc | once i verify nothing is broken on latest PS :) | |
| 19:07:52 | kashyap | sean-k-mooney: Oh, nice | |
| 19:08:57 | kashyap | sean-k-mooney: Seems like lnav still hasn't hit 1.0 (it's on 0.8.4) | |
| 19:09:05 | kashyap | But thanks for the pointer | |
| 19:09:17 | jackding | sean-k-mooney: noted. will add more info. thanks | |
| 19:11:10 | sean-k-mooney | kashyap: i dont know if they are using semver or not but its packaged on a bunch of distors so i assume this at least somewhat mature | |
| 19:11:19 | kashyap | I see | |
| 19:12:44 | sean-k-mooney | it apparently celebarad 4 years on github in 2013 so its been around a while | |
| 19:14:01 | kashyap | Ah-ha, alright. My eyes are aching, will head out now | |
| 19:14:14 | sean-k-mooney | kashyap: o/ night | |
| 19:27:30 | mriedem | gibi: there is something funny going on with the functional test for server create with a qos network https://review.openstack.org/#/c/570079/ | |