| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-24 | |||
| 15:36:28 | stephenfin | sean-k-mooney++ | |
| 15:49:17 | jangutter | stephenfin: allocating everything to hugepages is the quickest way to get a system unbootable by hitting the OOM killer. | |
| 15:49:57 | sean-k-mooney | jangutter: yep the kernel cant allocate its own memory form hugepages so it gets grumpy | |
| 15:51:14 | stephenfin | sean-k-mooney: When we move hugepage tracking to placement, we should get rid of this thing https://review.openstack.org/#/c/631053/1/nova/virt/hardware.py@1059 | |
| 15:51:45 | stephenfin | (the second part of your comment, that is | |
| 15:51:46 | stephenfin | ) | |
| 15:52:06 | sean-k-mooney | stephenfin we can only do that if we track all memory as meory of a set size | |
| 15:52:40 | sean-k-mooney | so we would need to get rid of memory_mb from the compute node and always afine guest to numa nodes | |
| 15:54:14 | sean-k-mooney | i personally woudl prefer to do that but i dont think we can just remove it simply | |
| 15:54:22 | sean-k-mooney | maybe we can | |
| 15:54:25 | stephenfin | Hmm that's a good point | |
| 15:55:00 | sean-k-mooney | honestly i would prefer to have a singel code path used for dedicate/shared cpus and all mempages sizes | |
| 15:55:15 | sean-k-mooney | instead of the 2 we have now for numa guests and non numa guests | |
| 16:07:28 | bauzas | folks, FWIW, I'm on a conference today and tomorow named 'SnowCamp' where I'm an organizer | |
| 16:07:31 | bauzas | mriedem: ^ | |
| 16:12:07 | mriedem | mdbooth: fyi image cache manager perf bug https://bugs.launchpad.net/nova/+bug/1804262/ - i put some thoughts on how to optimize where they are hitting timeouts | |
| 16:12:08 | openstack | Launchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,Triaged] | |
| 16:13:17 | BjoernT | Thanks for the feedback, I did not know about the GMR report (good old amiga times reflect back on me, lol) | |
| 16:14:25 | mriedem | BjoernT: i think i've sorted out the issue though so the gmr isn't really necessary | |
| 16:14:45 | BjoernT | ok I'm still curious what it looks like | |
| 16:14:48 | mriedem | you're just timing out trying to pull bdms for 705 instances from the conductor within 60 seconds (default rpc_response_timeout) | |
| 16:15:02 | BjoernT | took me a day to actually make that association with failing service checkins and the image cache manager | |
| 16:15:20 | BjoernT | we run rpc_response timeout upwards of 300 already | |
| 16:15:30 | BjoernT | now even larger | |
| 16:15:35 | mriedem | rpc_response_timeout is used mostly globally so tweaking it for one thing does not surprise me that it causes side effects elsehwere | |
| 16:15:50 | BjoernT | yeah I know | |
| 16:16:01 | mriedem | we also have a long_rpc_timeout since rocky, but you're on an older release than that, and this code / query isn't using that timeout | |
| 16:16:27 | BjoernT | I had one workaround in mind to shutoff image cache manger globally except one service node, which would not be used for scheduling | |
| 16:16:43 | BjoernT | ok, I'm still stuck on pike but good to know | |
| 16:17:07 | mriedem | i could maybe hack something up here and you can take a look at it | |
| 16:17:38 | BjoernT | yes I should be able to test that in that environment once I get signoff | |
| 16:22:01 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078 | |
| 16:22:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Create RequestGroup from neutron port https://review.openstack.org/625941 | |
| 16:22:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079 | |
| 16:22:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Read port resource request from Neutron https://review.openstack.org/630720 | |
| 16:22:03 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Include requested_resources to allocation candidate query https://review.openstack.org/625942 | |
| 16:22:04 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject resize with port having resource request https://review.openstack.org/630722 | |
| 16:22:04 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject server create with port having resource request https://review.openstack.org/630721 | |
| 16:22:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject evacuate with port having resource request https://review.openstack.org/630724 | |
| 16:22:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migrate with port having resource request https://review.openstack.org/630723 | |
| 16:22:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transfer port.resource_request to the scheduler https://review.openstack.org/567268 | |
| 16:22:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject unshelve with port having resource request https://review.openstack.org/630725 | |
| 16:22:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Calculate RequestGroup resource provider mapping https://review.openstack.org/616239 | |
| 16:22:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend RequestGroup object for mapping https://review.openstack.org/619527 | |
| 16:22:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass resource provider mapping to neutronv2 api https://review.openstack.org/616240 | |
| 16:22:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fill the RequestGroup mapping during schedule https://review.openstack.org/619528 | |
| 16:22:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send RP uuid in the port binding https://review.openstack.org/569459 | |
| 16:22:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Recalculate request group - RP mapping during re-schedule https://review.openstack.org/619529 | |
| 16:22:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test boot with more ports with bandwidth request https://review.openstack.org/573317 | |
| 16:22:15 | gibi | I'm sorry | |
| 16:22:22 | sean-k-mooney | :) | |
| 16:26:12 | gibi | mriedem: I've fixed your comments in https://review.openstack.org/#/c/570078 | |
| 16:27:17 | melwitt | mriedem, efried_2dmtg: would one of y'all be able to run the nova meeting next week? I will be traveling for work and unable to run it | |
| 16:29:24 | mriedem | yes i can | |
| 16:31:56 | mriedem | gibi: lgtm, +2 | |
| 16:33:42 | gibi | mriedem: thanks | |
| 16:33:52 | gibi | the next in that series is pretty similar | |
| 16:34:27 | melwitt | mriedem: great, thanks | |
| 16:34:32 | mriedem | ok in the tab queue | |
| 16:34:49 | gibi | coll | |
| 16:34:52 | gibi | cool | |
| 17:03:47 | cfriesen | artom: stephenfin: any news on the live migration stuff? I'm happy to help test experimental code. | |
| 17:04:37 | artom | cfriesen, heya, afraid not, was mostly downstream this week. I need to get back to it and put up stuff, even WIP | |
| 17:15:44 | cfriesen | I totally get the downstream stuff. :) Story of my life. | |
| 17:22:36 | mriedem | BjoernT_: i should have a patch up shortly, just running final tests | |
| 17:32:19 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042 | |
| 17:32:20 | mriedem | BjoernT_: ^ | |
| 17:37:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Chunk BDM queries from ImageCacheManager._list_running_instances() https://review.openstack.org/633042 | |
| 17:37:28 | openstackgerrit | melanie witt proposed openstack/nova master: Raise 403 instead of 500 error from attach volume API https://review.openstack.org/632904 | |
| 17:37:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 17:40:48 | mriedem | kashyap: seeing a weird issue in the gate with libvirt 4.0.0 where sometimes an instance delete times out in libvirt and it looks like there is a loop in libvirt that is busted https://bugs.launchpad.net/nova/+bug/1813192 | |
| 17:40:49 | openstack | Launchpad bug 1813192 in OpenStack Compute (nova) "libvirt: instance delete fails with "Cannot destroy instance, operation time out: libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)" in bionic nodes (libvirt 4.0.0, qemu 2.11)" [Medium,Confirmed] | |
| 17:40:58 | kashyap | mriedem: Haha, eerie you ping me! | |
| 17:41:10 | kashyap | mriedem: I was _just_ writing a nice message for you to ask for help w/ some unit test debugging | |
| 17:41:14 | mriedem | only seeing this on ubuntu bionic | |
| 17:41:32 | kashyap | Oh, the "cannot acquire state change lock" error. | |
| 17:41:50 | kashyap | Hmm, it is unofficially the most difficult one to debug. | |
| 17:42:14 | mriedem | only 3 hits in 7 days so not a huge issue | |
| 17:42:26 | kashyap | Will check the logs once I have dinner. I was in back-to-back meeting hell | |
| 17:42:42 | kashyap | Thanks for the heads-up | |
| 17:43:28 | kashyap | I was about to ping you to borrow some eyes on these unit test failures on the version bump WIP patch: https://review.openstack.org/#/c/632507/ | |
| 17:43:47 | kashyap | But the patch is not fully done; so I don't want to waste your time clicking on it. But I at least reduced the failures from 136 to 5, though. | |
| 17:44:10 | kashyap | And I spent a crap load of time w/ PDB yesterday & some time today. Was still duking around. | |
| 17:45:31 | kashyap | On the off chance you have a minute, any comments / snide remarks on the review are welcome. | |
| 17:45:47 | mriedem | if tempest is failing you've got bigger problems than just unit tests | |
| 17:46:42 | mriedem | something is busted with console logs http://logs.openstack.org/07/632507/3/check/tempest-full/73a00be/controller/logs/screen-n-cpu.txt.gz?level=TRACE | |
| 17:49:45 | kashyap | Yeah, it is. I know partly the fault from this morning's debugging. And I have a half-baked local patch. | |
| 17:50:05 | kashyap | I'll keep digging tomm / this week. And get it locally passing and only _then_ use Gate resources. | |
| 17:50:15 | stephenfin | cfriesen: What artom said, though the SR-IOV live migration patches are progressing quite nicely | |
| 17:50:37 | stephenfin | adrianc: You should probably stick that in the queue for a runway, if you haven't done so already | |
| 17:51:11 | sean-k-mooney | i need to adress jays comment on the base patch but they should be ready for a runway next week or so | |
| 17:51:50 | sean-k-mooney | the neutron patch need to be ready first | |
| 17:52:39 | sean-k-mooney | this one https://review.openstack.org/#/c/620123/ it work but it needs tests | |
| 17:53:50 | cfriesen | I think https://review.openstack.org/#/c/621646 is ready for wider review. This is the code to preserve UEFI nvram appropriately. | |
| 17:55:39 | sean-k-mooney | oh cool. is the nvram used to store bios/uefi setting or dos it have some other use? | |
| 17:57:30 | sean-k-mooney | cfriesen: why are ye adding keep_nvram=false in all the drivers when that is the default in the base dirver they all inherit form | |
| 17:58:02 | sean-k-mooney | oh never mind | |
| 17:58:32 | sean-k-mooney | you are over rideing the function not over loading them that makes sense | |
| 18:00:33 | cfriesen | sean-k-mooney: I think the nvram is for bios stuff. I think it's also possible to do things like stick crash messages in the NVRAM. | |
| 18:30:03 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |