| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-05 | |||
| 14:27:44 | mriedem | xenial less for the python thing but more b/c of native packages like postgresql | |
| 15:04:53 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove nova-consoleauth https://review.opendev.org/652969 | |
| 15:07:27 | stephenfin | lyarwood: The stable/stein variant of this has merged now. Could you tackle this one? https://review.opendev.org/#/c/667355/ | |
| 15:20:58 | dansmith | sean-k-mooney: mriedem: Artom made several changes to the revert resize patch yesterday but didn't respond to comments and also rebased it so there's a lot of change.. did I miss any discussion about that? | |
| 15:21:28 | dansmith | looks like he removed the try..except around the signature change which was my deal, but otherwise I either need to eye-diff it or pull it down | |
| 15:21:46 | sean-k-mooney | dansmith: he change it to the db lookup | |
| 15:21:53 | sean-k-mooney | right | |
| 15:22:02 | sean-k-mooney | i havent looked at it this mornig | |
| 15:22:09 | dansmith | alright | |
| 15:23:58 | mriedem | i was going to have to eye-diff anyway | |
| 15:34:07 | sean-k-mooney | artom is on pto while he is flying home to canada today but if there are small things like mising test exctra ill be around for an hour or two to fix them | |
| 15:34:38 | sean-k-mooney | i think artom is back on monday but if not ill continue looking at this until he is back. | |
| 15:55:03 | dansmith | sean-k-mooney: I don't see the "will wait for bind time events" string in the logs | |
| 15:55:21 | dansmith | on the hybrid plug test in experimental | |
| 15:55:51 | dansmith | maybe that test is single-host which means we're not running any of this new stuff anyway? | |
| 15:56:12 | sean-k-mooney | it is | |
| 15:56:22 | sean-k-mooney | it is testign the resize to same host case | |
| 15:56:42 | dansmith | ah yeah I see the same host debug message instead | |
| 15:56:44 | sean-k-mooney | https://review.opendev.org/#/c/664442/14 is testing resize to different host | |
| 15:57:28 | dansmith | ah, right forgot that was on top and was thinking we'd get testing from mriedem's patch, I see | |
| 15:58:24 | sean-k-mooney | ya this is the message form the dnm change http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_49_57_371761 | |
| 15:59:02 | dansmith | yeah, but lower we receive and discard that event: http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_49_58_003384 | |
| 15:59:22 | sean-k-mooney | ya i think this might be the souce node i need to check the other node | |
| 15:59:22 | dansmith | see the "no event matching" above that | |
| 15:59:29 | sean-k-mooney | and confim this is the right vm | |
| 15:59:41 | dansmith | we shouldn't emit the "will wait..." message on the wrong node right? | |
| 16:00:13 | sean-k-mooney | we should not e might get thos events on both nodes | |
| 16:00:23 | dansmith | receive, but not waiting | |
| 16:00:41 | dansmith | actually, that's the unplug event | |
| 16:00:48 | dansmith | we get the plug event shortly after that: | |
| 16:01:02 | dansmith | http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_50_00_983164 | |
| 16:02:55 | sean-k-mooney | i dont kno if we ever actully wait for unplugged events do we | |
| 16:08:00 | sean-k-mooney | it looks like libvirt died oh that vm breilfy http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/screen-n-cpu.txt.gz#_Jul_04_19_59_12_862599 | |
| 16:08:16 | sean-k-mooney | which is unrelated but annoying | |
| 16:08:48 | sean-k-mooney | actully that is part of the testing i think | |
| 16:09:03 | sean-k-mooney | i think we are killing it so we can do evacuate testing | |
| 16:15:52 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: WIP: Drop source node allocations if finish_resize fails https://review.opendev.org/669393 | |
| 16:17:03 | mriedem | the nova-live-migration job, renamed to nova-multinode in 664442, yes does evacuate testing and kills libvirtd on the controller node | |
| 16:19:03 | sean-k-mooney | in any case we start waiting for the event here http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_48_754192 and we recive it here http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_51_905792 | |
| 16:20:00 | sean-k-mooney | there is a vif unplugged and plugged event recrved just before we wait that im not sure where they came form | |
| 16:20:25 | sean-k-mooney | http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_40_441961 and http://logs.openstack.org/42/664442/14/check/nova-multinode/7c532c4/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_04_19_49_45_258629 | |
| 16:21:05 | sean-k-mooney | but it does look like we do wait as expected | |
| 16:31:08 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Drop source node allocations if finish_resize fails https://review.opendev.org/669393 | |
| 16:31:24 | mriedem | lyarwood: dansmith: ^ this backport got a bit hairy because before stein we had to deal with non-migration-based allocations, | |
| 16:31:44 | mriedem | so i've documentd the difference in the commit message and code, and added a functional test for the non-migration-based allocations legacy wrinkle | |
| 16:31:55 | mriedem | just fyi so it doesn't get fast-approved as a stable core | |
| 16:35:24 | dansmith | ugh | |
| 17:18:37 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: [WIP] Allow to spawn VMs with port without IP address https://review.opendev.org/669411 | |
| 17:22:09 | redkrieg | I'm having an interesting issue with a large flavor not being deployable on a compute host but two flavors of half the size working fine. the host has just enough disk and ram for the flavor, plus disk_allocation_ratio and ram_allocation_ratio are > 1. Nova scheduler logs with debug enabled don't even show the host in question during filtering, so I think it's being removed earlier somehow. I | |
| 17:22:15 | redkrieg | see accurate information in the "Update host state from compute node" ComputeNode output just minutes before launching an instance. Can anyone suggest a reason it wouldn't even be considered for filtering? | |
| 17:37:24 | redkrieg | I've narrowed it down to a ram issue by dropping my flavor's ram requirement to 30000 from 32768. the logs show that the hypervisor is reporting 32141 but even 32000 doesn't work. Is there some hard limit I don't know about that bypasses the allocation ratio settings? | |
| 17:58:40 | openstackgerrit | Corey Bryant proposed openstack/nova master: Add Python 3 Train unit tests https://review.opendev.org/669437 | |
| 18:00:20 | openstackgerrit | Corey Bryant proposed openstack/os-vif master: Add Python 3 Train unit tests https://review.opendev.org/669438 | |
| 18:01:06 | openstackgerrit | Corey Bryant proposed openstack/python-novaclient master: Add Python 3 Train unit tests https://review.opendev.org/669439 | |
| 18:08:50 | mriedem | redkrieg: which release? placement could be filtering it out - enable debug logging in the placement logs and you should see the allocation candidate filtering messages (on newer releases), there is the reserved_host_ram_mb config option on the compute that will be reporting reserved MEMORY_MB to placement | |
| 18:08:57 | mriedem | which is part of the overall capacity calculation | |
| 18:12:20 | Nick_A | mriedem - rocky, phys_ram=32141MB used_ram=0MB, reserved_host_memory_mb = 0 in nova.conf | |
| 18:12:44 | Nick_A | we found https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html which explains why 32768 doesn't work, but we're not seeing why 32000 is also failing | |
| 18:17:27 | mriedem | are you using the RamFilter or AggregateRamFilter? | |
| 18:21:55 | redkrieg | yes, we do have the AggregateRamFilter active. I don't see this host being filtered by it though. | |
| 18:25:07 | openstackgerrit | Merged openstack/nova master: libvirt: Remove MIN_{QEMU,LIBVIRT}_LUKS_VERSION https://review.opendev.org/668924 | |
| 18:31:18 | openstackgerrit | Merged openstack/nova-specs master: Fix the cross_az_attach value in AZ-unshelved spec https://review.opendev.org/669261 | |
| 18:34:48 | mriedem | the reason i ask is because those filters return a "limit" from the scheduler which gets passed down to the compute which is part of the resource claim, | |
| 18:35:00 | mriedem | and if the scheduler is passing down a limit for something that you don't expect, that might be the cause | |
| 18:35:39 | mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios | |
| 18:37:17 | mriedem | iow, what you think and what placement is using might be different allocation ratios | |
| 18:37:28 | mriedem | hopefully that doc gives some things to look at | |
| 18:48:34 | Nick_A | It actually looks like we're dealing with some sort of qemu/kernel bug... It successfully attempts a claim on compute if we set the flavor RAM less than the physical RAM, but libvirt errors at certain amounts with "qemu-system-x86_64: cannot set up guest memory 'pc.ram': Cannot allocate memory." | |
| 19:15:56 | dansmith | Nick_A: using anything that requires pinned memory? cpu pinning, huge pages, etc? | |
| 19:16:36 | dansmith | also, even if you have 32g of memory, you can't necessarily expect to land a 30g instance there if you have no swap and have a large enough footprint of other stuff | |
| 19:17:04 | dansmith | but, if you're getting to libvirt, then you're out of the realm of it being a nova thing | |
| 19:17:21 | Nick_A | we are doing cpu pinning on here - yeah, I think the only nova issue was the hard limit based on physical ram | |
| 19:17:46 | Nick_A | I am curious why that's the case if you happen to know | |
| 19:18:17 | Nick_A | we have non-openstack centos libvirt nodes that are happy to create KVMs with over 32GB on the same hardware | |
| 19:18:19 | dansmith | do you also have a numa topology specified? | |
| 19:18:34 | dansmith | there's no 32g limit specifically of course | |
| 19:19:16 | Nick_A | "Regardless of the overcommit ratio, an instance can not be placed on any physical node with fewer raw (pre-overcommit) resources than the instance flavor requires." Am I misunderstanding that? | |
| 19:19:40 | Nick_A | re: numa - we're using openstack-ansible defaults, not sure | |
| 19:21:14 | Nick_A | I got that from https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html | |
| 19:21:29 | dansmith | no, I'm just saying there's no specific 32g limit. nova's overcommit is for landing multiple instances on a node, however ill-advised | |
| 19:21:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove old TODO about forced_host policy check https://review.opendev.org/669474 | |
| 19:22:08 | dansmith | a single instance larger than physical memory is silly, so yeah, no support for it | |
| 19:22:41 | Nick_A | right, but there is a limit based on the physical RAM preventing a single instance from being allocated above that amount. I'm curious why it allows multiple instances to do so but not a single one. | |
| 19:23:50 | dansmith | it allows it because it was designed that way for earlier cloud providers that did it for cost scale, but it's never a good idea for performance, obviously, as you're just using host disk for the overage | |
| 19:24:13 | dansmith | a single instance that is larger than the host doesn't make sense, you might as well run a smaller guest and let it do its own swapping | |
| 19:25:10 | dansmith | anyway, if you're doing cpu pinning you might be running up against something that requires page pinning which is getting you to the "cannot allocate memory" thing | |
| 19:25:18 | dansmith | I doubt it's a kernel/qemu bug | |
| 19:25:36 | dansmith | best bet would be to ask around monday morning when more of the folks that are familiar with that stuff are around | |
| 19:25:42 | Nick_A | thank you | |
| 19:25:43 | dansmith | sean, sfinucan, etc | |
| 19:43:47 | openstackgerrit | Corey Bryant proposed openstack/os-resource-classes master: Add Python 3 Train unit tests https://review.opendev.org/669479 | |
| 19:44:20 | openstackgerrit | Corey Bryant proposed openstack/os-traits master: Add Python 3 Train unit tests https://review.opendev.org/669480 | |
| 20:15:16 | mriedem | bauzas: how is your audit command coming? it's been a couple of weeks and this came up in the ML today http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007534.html | |
| 20:35:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Newton-era min compute checks for server create with device tags https://review.opendev.org/669523 | |
| 20:37:36 | mriedem | dansmith: is https://review.opendev.org/#/c/667177/ just waiting for my eye-ball diff and if we're good, fix the thing you noted and then we can move forward? | |
| 20:37:48 | dansmith | mriedem: yeah | |
| 21:16:46 | mriedem | ok comments in there now | |
| 21:17:24 | mriedem | also - https://review.opendev.org/#/c/668631/ - it's a good thing we didn't do the interface change in the same patch... | |
| 21:31:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Init HostState.failed_builds https://review.opendev.org/669538 | |
| 21:37:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Rocky-era min compute trusted certs compat check https://review.opendev.org/669539 | |