| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-03-19 | |||
| 16:35:42 | bauzas | nope | |
| 16:35:51 | bauzas | wow | |
| 16:36:27 | fwiesel | bauzas: Back to you then | |
| 16:36:43 | bauzas | cool, that will be short | |
| 16:36:51 | bauzas | #topic Open discussion | |
| 16:36:55 | bauzas | anyuthing anyone ? | |
| 16:37:04 | dansmith | a tempest run takes 20 minutes? that would be shockingly fasty | |
| 16:37:52 | fwiesel | Well, it is a hefty machine, and I run it as parallel as I can. | |
| 16:38:19 | bauzas | ok guys, I think I can close the meeting | |
| 16:38:21 | bauzas | thanks all | |
| 16:38:23 | bauzas | #endmeeting | |
| 16:38:23 | opendevmeet | Meeting ended Tue Mar 19 16:38:23 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:38:23 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.html | |
| 16:38:23 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.txt | |
| 16:38:23 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.log.html | |
| 16:38:28 | fwiesel | Thank you too | |
| 16:38:37 | sean-k-mooney | well i guess it depend on how many tests you run. devstack can stack in about 8 mins if its not io bound | |
| 16:39:03 | elodilles | thanks o/ | |
| 16:39:40 | fwiesel | The tempest run takes 10 minutes, by the way :) | |
| 16:40:19 | sean-k-mooney | yep but your also unly runnign 525 ish tests | |
| 16:40:31 | sean-k-mooney | at least based on | |
| 16:40:33 | fwiesel | Sure, are there tests I am missing? | |
| 16:40:33 | sean-k-mooney | http://openstack-ci-logs.global.cloud.sap/openstack-nova-909474-f54wv/tempest.html | |
| 16:40:57 | sean-k-mooney | fwiesel: well you only runing the compute api tests | |
| 16:41:16 | sean-k-mooney | your not runng the senario tests ro any fo teh test fo other services like cinder/nuetorn extra | |
| 16:41:25 | sean-k-mooney | we dont need you to run all of those | |
| 16:41:40 | sean-k-mooney | but i was just providing context for others | |
| 16:41:47 | sean-k-mooney | i.e. its not tempest-full | |
| 16:42:45 | fwiesel | No, indeed it isn't. | |
| 16:43:26 | fwiesel | I am not even running all compute.api-tests, only those that are implemented for the vmware api driver. | |
| 16:43:51 | sean-k-mooney | ack that kind of makes sense | |
| 16:44:11 | sean-k-mooney | for the hardware based cis they took a simialr approch | |
| 16:44:38 | sean-k-mooney | i.e. the old intel ci just ran the cpu pinning/sriov/numa related subset | |
| 16:46:22 | fwiesel | Possibly it would make sense to check what code-coverage the tempest test generates on the driver implementation, and adjust the tests accordingly. | |
| 17:55:03 | sean-k-mooney | dansmith: correct me if im wrong but we do not have the ablity to ask placement for a host that does not have a resouce type right like a forbiden trait, we would have to do that in a scheduler filter | |
| 17:56:42 | sean-k-mooney | related to our converation about non numa instances | |
| 17:56:49 | dansmith | sean-k-mooney: I really don't know currently, but I'm pretty sure that was one of the early limitations.. we could ask for inclusion but not exclusion | |
| 17:57:27 | sean-k-mooney | ack i think its still that way. i belive we coudl have a new filter that does the following check | |
| 17:58:26 | sean-k-mooney | for a given instance look at the guest falvor/image to see if it is requesting numa aware memory i.e. hw:mem_page_size | |
| 17:58:54 | sean-k-mooney | if it is not check the allcoation candates for the host to see if it has either the file backed memory trait or an inventory of PCPU | |
| 17:59:02 | sean-k-mooney | if it has either reject the host | |
| 18:00:05 | sean-k-mooney | that would allow operators to opt in/out by not using the filter | |
| 18:00:39 | sean-k-mooney | we can add it to the default filters to enforce the correct behavior. | |
| 18:00:54 | dansmith | well, that feels more formal of a feature opt-out than a workaround named I_like_to_oom_my_instances=True | |
| 18:01:12 | dansmith | but it would mean an abort in scheduling instead of compute, which would be better | |
| 18:02:07 | sean-k-mooney | well we cloud perhaps not make it a filter but a config option that woudl work like a filter | |
| 18:02:19 | sean-k-mooney | i.e. have a workaround to remove this mandatory filter | |
| 18:02:21 | dansmith | sean-k-mooney: btw, I am still interested in the answer to my question here: https://review.opendev.org/c/openstack/nova/+/877773/9/nova/virt/libvirt/migration.py | |
| 18:02:44 | dansmith | because it seems like it should always fail and I don't know why it's not | |
| 18:03:41 | sean-k-mooney | reading that now ill need to check but if i rememebr correctly we pop the value form the dict before it ever get there | |
| 18:04:00 | dansmith | that's a bad solution IMHO if so because it would break computes that don't have this code yet, | |
| 18:04:05 | dansmith | but I also couldn't find any such pop | |
| 18:04:14 | sean-k-mooney | your asertion is calling int(a string containing a list of ints) | |
| 18:04:35 | dansmith | sean-k-mooney: no it's calling int() on the fake key | |
| 18:04:36 | sean-k-mooney | should always explode so either we are not hitting that path in the test or there is somethign else going on here | |
| 18:04:58 | dansmith | that dict is supposed to be keys of '1', '2', etc and so it int()'s the keys, | |
| 18:05:08 | dansmith | but now there's a key of "dst_cpu_whatever" in the dict | |
| 18:05:20 | dansmith | and I don't know how it's not failing to int() that when it iterates the items | |
| 18:05:35 | dansmith | and I'm wondering if it's because we don't have test coverage that was hitting that code path with the three conditions being true | |
| 18:07:58 | sean-k-mooney | oh right "guest_id, host_ids in info.cpu_pins.items()" so guest_id would be the key | |
| 18:08:04 | dansmith | right | |
| 18:08:09 | dansmith | so I don't know how that's not breaking | |
| 19:23:13 | sean-k-mooney | artom: https://review.opendev.org/c/openstack/nova/+/631053 and https://bugs.launchpad.net/nova/+bug/1811886 are related to our converstaion about the issue relating to not requesting a page size. | |
| 19:23:52 | sean-k-mooney | its not the example i was giving when we talks but its a similar equally wrong case. | |
| 19:42:02 | opendevreview | Merged openstack/nova master: Remove nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902686 | |
| 19:42:13 | opendevreview | Merged openstack/nova master: Add new nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902687 | |
| 19:48:25 | sean-k-mooney | dansmith: i looked brifly but i dont know why that is not brekaing things but my best guess is we are somehow mocking this and we dont have cpu_shared_set defiend in any of the jobs | |
| 19:49:05 | sean-k-mooney | there is no reason why we cant have cpu share set in one of our live migration jobs so i might try and enable that | |
| 19:50:27 | sean-k-mooney | i.e. 0-5 on one node and 2-7 on the other | |
| 19:51:00 | sean-k-mooney | that pretty trivial to do so ill add a patch on top or renes serise to do that | |
| 19:52:13 | sean-k-mooney | in theory rene's code should be runnign even if that is not devied but we should be able to test this in the gate anyway | |
| 19:59:42 | opendevreview | Stephen Finucane proposed openstack/nova master: Make overcommit check for pinned instance pagesize aware https://review.opendev.org/c/openstack/nova/+/631053 | |
| 20:01:56 | dansmith | sean-k-mooney: yeah, I'm unsure either, but I'm positive that means we're just not running the code we think we're running | |
| 20:02:02 | opendevreview | ribaudr proposed openstack/nova master: Fix [backport]: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/877773 | |
| 20:02:02 | opendevreview | ribaudr proposed openstack/nova master: Fix: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/903706 | |
| 20:07:51 | sean-k-mooney | dansmith: my guess right now is we are never populating the numa toplogy info in the migrate_data | |
| 20:08:18 | dansmith | sean-k-mooney: you mean not running a test that causes it to be populated right? | |
| 20:08:46 | dansmith | I mean...not running a test _case_ where it would be populated and that code path exercised | |
| 20:10:40 | sean-k-mooney | no i ment not seting the object that had the new data at all but we appear to be | |
| 20:10:54 | artom | sean-k-mooney, dansmith, hrmm, I wonder if we're fine because... let my type this out | |
| 20:11:08 | artom | The NUMA live migration code kicks in only if the instance has a NUMA topology | |
| 20:11:39 | artom | Uggla's patch _always_ adds that hidden item in the dst_numa_info | |
| 20:12:19 | sean-k-mooney | so it alwasy uses the numa migration code? | |
| 20:12:42 | artom | Hold on | |
| 20:13:50 | artom | No, I thought I had it... | |
| 20:16:11 | sean-k-mooney | the fact we have to have this converstaion and its not obvious how its working after a few of us lokking at it is a code smell to me in either case | |
| 20:16:31 | sean-k-mooney | the code might be correct but its to complex to debug | |
| 20:18:31 | artom | So to trigger it, all that should be required in live migration a NUMA VM to a dest with cpu_shared_set, right? | |
| 20:19:10 | artom | That would cause Uggla's patch to stash the cpu_shared_set in the "hidden" key in dst_numa_info, and *should* cause the update XML code to trip over and explode? | |
| 20:19:26 | artom | We've _definitely_ done that ^^ in whitebox, and nothing has exploded | |
| 20:19:42 | sean-k-mooney | artom: no | |
| 20:19:56 | sean-k-mooney | to trigger the bug yuou need to migate a non numa vm | |
| 20:20:06 | sean-k-mooney | oh | |
| 20:20:17 | sean-k-mooney | you mena the the driver issue | |
| 20:20:37 | artom | Yeah, the thing that we're trying to understand how it even works | |
| 20:20:39 | sean-k-mooney | yes if the loop over the keys was broken then you woudl have to migrate a pinned instance | |
| 20:22:06 | sean-k-mooney | right but since this is alwasy set https://review.opendev.org/c/openstack/nova/+/877773/10/nova/virt/libvirt/driver.py#10028 | |
| 20:22:32 | sean-k-mooney | any numa instance should triger this code path | |
| 20:22:38 | artom | Exactly | |
| 20:23:05 | artom | I'm expecting migrate_data.dst_numa_info.cpu_pins["dst_cpu_shared_set_info"] to always be there | |