| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-31 | |||
| 16:37:24 | sean-k-mooney | and asked them to file a bz which you triaged but i have filed an upstream bug | |
| 16:37:34 | stephenfin | Got a link? | |
| 16:37:35 | sean-k-mooney | if you want to fix it feel free | |
| 16:37:48 | sean-k-mooney | *have not | |
| 16:38:50 | stephenfin | sean-k-mooney: Want to give the tl;dr? | |
| 16:39:20 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1038 | |
| 16:39:37 | sean-k-mooney | we use the pagesize form the embeded instance numa toplogy object | |
| 16:40:22 | sean-k-mooney | so in a resize when we do https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L98 | |
| 16:40:33 | stephenfin | sean-k-mooney: Right, which we have one of already https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L74 | |
| 16:40:35 | stephenfin | yeah | |
| 16:40:51 | stephenfin | Why doesn't this hit rebuild though? | |
| 16:41:13 | sean-k-mooney | because its a resize | |
| 16:41:16 | stephenfin | Oh, because the size hasn't changed | |
| 16:41:18 | stephenfin | Yah | |
| 16:41:20 | stephenfin | *yeah | |
| 16:41:29 | stephenfin | *mempage size | |
| 16:42:16 | sean-k-mooney | anyway for a resize we need to update the instance toplogy object with the mem_page_size form the new flavor | |
| 16:42:43 | sean-k-mooney | so we can either make a copy of the toplogy and pass that in our maybe use the tempory_mutation decorator | |
| 16:42:56 | stephenfin | I'm trying to figure out what sets it for the requested_topology the first day (i.e. on a server create) | |
| 16:43:34 | sean-k-mooney | i belive we generate it form the flavor but dont fill in all the fields | |
| 16:43:45 | sean-k-mooney | but i didnt look | |
| 16:44:13 | sean-k-mooney | its foming from the sepc_object whic is the request_spec | |
| 16:44:48 | stephenfin | Must be https://github.com/openstack/nova/blob/master/nova/compute/api.py#L978 | |
| 16:45:50 | stephenfin | which is set by https://github.com/openstack/nova/blob/master/nova/compute/api.py#L876-L877 | |
| 16:50:37 | stephenfin | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1831269 | |
| 16:50:38 | openstack | Launchpad bug 1831269 in OpenStack Compute (nova) "Resize ignores mem_page_size in new flavor" [Undecided,New] | |
| 16:50:48 | sean-k-mooney | we should proably be updating the request spec here https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3558-L3560 | |
| 16:51:48 | sean-k-mooney | which is just before we call resize instnace | |
| 16:52:08 | sean-k-mooney | that or self.compute_task_api.resize_instance should do it | |
| 17:01:41 | sean-k-mooney | stephenfin: i have triage it upstream and commented on the issue https://bugs.launchpad.net/nova/+bug/1831269 | |
| 17:01:42 | openstack | Launchpad bug 1831269 in OpenStack Compute (nova) "Resize ignores mem_page_size in new flavor" [Medium,Confirmed] | |
| 17:02:17 | sean-k-mooney | stephenfin: it should in theory be a fairly simple fix | |
| 17:02:45 | stephenfin | Yup, it should | |
| 17:03:08 | sean-k-mooney | i think it should be backportable too | |
| 17:03:18 | sean-k-mooney | rebuild is harder to fix | |
| 17:03:45 | stephenfin | I'm not sure why this is passing though https://review.rdoproject.org/r/#/c/11297/11/whitebox_tempest_plugin/api/compute/test_cpu_pinning.py@108 | |
| 17:03:45 | sean-k-mooney | or rather inplace rebuild without the double allocations | |
| 17:04:24 | sean-k-mooney | the downstream bug only happens because the destination host did not have any 2mb hugepages only 1G | |
| 17:04:27 | stephenfin | Actually, maybe it is. But we have a host that actually has enough free CPUs | |
| 17:04:40 | kashyap | efried: aspiers: Thanks for the reviews. | |
| 17:04:52 | stephenfin | So we'd start seeing this more when cpu-resources is done, because we might only have limited PCPU inventory | |
| 17:04:55 | aspiers | kashyap: we still need one more +2 | |
| 17:05:09 | aspiers | I guess we should wait for cdent | |
| 17:05:10 | kashyap | aspiers: Right, not sure who can give it. Chris Dent? | |
| 17:05:15 | kashyap | Bingo | |
| 17:05:27 | stephenfin | oooh, review.rdoproject.org supports polygerrit now | |
| 17:05:28 | sean-k-mooney | stephenfin we likely are looking up the cpu policy differently | |
| 17:05:30 | stephenfin | pretty | |
| 17:05:52 | aspiers | stephenfin: still waiting for OpenStack Gerrit upgrade | |
| 17:06:27 | stephenfin | yup, but I'm assured they're on it. Looking forward to a few of those features | |
| 17:06:32 | sean-k-mooney | nope https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L1100 | |
| 17:06:38 | aspiers | stephenfin: but even that upgrade is fairly old: 2.14.7, and 3.0 is now out | |
| 17:06:55 | aspiers | there are lots of really nice improvements since 2.14 | |
| 17:07:13 | stephenfin | Yeah, I think we discussed before and the strategy was to go to 2.16, which deprecates a load of things but would provide time to prepare for 3.0 | |
| 17:07:28 | aspiers | stephenfin: skip upgrades are not supported anyway | |
| 17:07:42 | aspiers | you have to go to 2.16 to migrate to notedb before moving to 3.0 | |
| 17:07:48 | stephenfin | yup | |
| 17:08:07 | efried | stephenfin: Not sure how much mikal is monitoring his reviews these days; if I do those two fixups (remove redundant lines) would you fast approve? | |
| 17:08:34 | sean-k-mooney | stephenfin: so ya https://review.rdoproject.org/r/#/c/11297/11/whitebox_tempest_plugin/api/compute/test_cpu_pinning.py@108 should fail | |
| 17:08:47 | stephenfin | efried: I would | |
| 17:09:01 | stephenfin | sean-k-mooney: I'm thinking it would fail if the other had no free cores | |
| 17:09:04 | stephenfin | *other host | |
| 17:09:33 | sean-k-mooney | stephenfin: the self.get_server_cpu_pinning call should be checking the xml | |
| 17:09:38 | stephenfin | but because it does, the NUMATopologyFilter is passing and then things get recalculated properly when it starts building the instance on the host | |
| 17:09:56 | sean-k-mooney | unless we actully update the request spec later | |
| 17:10:06 | sean-k-mooney | e.g. on the compute node | |
| 17:10:19 | sean-k-mooney | right | |
| 17:10:36 | sean-k-mooney | ok so ya that is passing because there is enough free cpus | |
| 17:11:05 | sean-k-mooney | ok so resize with numa is not totally broken | |
| 17:11:15 | sean-k-mooney | just the sechduler bit | |
| 17:11:37 | sean-k-mooney | the compute node will still validate it with the new flavor when it trys to build the instance xml | |
| 17:12:28 | sean-k-mooney | or just before it when it generates the new numa toplogy | |
| 17:14:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/662522 | |
| 17:14:56 | stephenfin | sean-k-mooney: ^ untested (completely) but something like that should do the trick, I imagine? | |
| 17:15:22 | sean-k-mooney | yes that is exactly what i was thinking too | |
| 17:15:44 | stephenfin | Nice | |
| 17:15:57 | sean-k-mooney | the one thing im not sure about however is should we but using the tempory mutation thing | |
| 17:16:12 | stephenfin | I'll try whip up a functional test for the on Monday^H^H^H Tuesday and see if I can reproduce it there | |
| 17:16:15 | stephenfin | Possibly | |
| 17:16:15 | sean-k-mooney | e.g. if we revert we need to make sure we go back to the old numa toplogy | |
| 17:16:45 | stephenfin | The fact that we're overwriting some RequestSpec fields already leads me to think we might be rolling back somewhere, but I haven't investigated | |
| 17:17:15 | sean-k-mooney | i think we nolonger persist the ignore_hosts field to the db | |
| 17:17:36 | sean-k-mooney | anyway that is at least a step in the right direction | |
| 17:23:46 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils from virt driver tests. https://review.opendev.org/643894 | |
| 17:23:47 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils from libvirt imagebackend tests. https://review.opendev.org/643895 | |
| 17:23:47 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove remaining vestiges of fake_libvirt_utils from unit tests. https://review.opendev.org/643896 | |
| 17:23:48 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove fake_libvirt_utils users in functional testing. https://review.opendev.org/644793 | |
| 17:23:54 | efried | stephenfin: done | |
| 18:08:56 | openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.opendev.org/638324 | |
| 18:08:56 | openstackgerrit | melanie witt proposed openstack/nova master: Add documentation for counting quota usage from placement https://review.opendev.org/653845 | |
| 18:08:57 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up for counting quota usage from placement https://review.opendev.org/662056 | |
| 18:52:23 | openstackgerrit | Merged openstack/nova stable/stein: Fix live-migration when glance image deleted https://review.opendev.org/662152 | |
| 18:53:18 | mriedem | dansmith: thanks, replied | |
| 18:53:39 | mriedem | dansmith: sounds like you're prefer me to go back to PS3 for the backports and leave the group_uuid stuff separately on master - if so, that's fine and i can do that | |
| 18:53:45 | mriedem | *you'd | |
| 18:59:00 | dansmith | mriedem: ah I didn't see that it was that way in a previous set.. I'm not super opposed, but just feels like the right thing to do | |
| 19:00:03 | mriedem | ok i'll split that one apart and make the func test change to pop request_spec from kwargs | |
| 19:01:18 | artom | dansmith, welcome back from PTO. I left you a steaming homecoming present: https://review.opendev.org/#/c/653498/ | |
| 19:02:05 | artom | (It's the revert resize thing we had a hangouts about) | |
| 19:03:13 | mriedem | ....except much different | |