Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-03-19
16:32:55 bauzas fwiesel: awesome about where you are
16:33:32 fwiesel Missing now are hopefully only cleaning up the rough edges... A nicer UI with linkable lines we still are missing.
16:34:45 bauzas okay, anything else ?
16:34:52 fwiesel No, not from my side.
16:35:00 fwiesel Questions?
16:35:41 fwiesel One remark: A test run with fixes in place takes roughly 20minutes in total.
16:35:42 bauzas nope
16:35:51 bauzas wow
16:36:27 fwiesel bauzas: Back to you then
16:36:43 bauzas cool, that will be short
16:36:51 bauzas #topic Open discussion
16:36:55 bauzas anyuthing anyone ?
16:37:04 dansmith a tempest run takes 20 minutes? that would be shockingly fasty
16:37:52 fwiesel Well, it is a hefty machine, and I run it as parallel as I can.
16:38:19 bauzas ok guys, I think I can close the meeting
16:38:21 bauzas thanks all
16:38:23 bauzas #endmeeting
16:38:23 opendevmeet Meeting ended Tue Mar 19 16:38:23 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:38:23 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.html
16:38:23 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.txt
16:38:23 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-03-19-16.00.log.html
16:38:28 fwiesel Thank you too
16:38:37 sean-k-mooney well i guess it depend on how many tests you run. devstack can stack in about 8 mins if its not io bound
16:39:03 elodilles thanks o/
16:39:40 fwiesel The tempest run takes 10 minutes, by the way :)
16:40:19 sean-k-mooney yep but your also unly runnign 525 ish tests
16:40:31 sean-k-mooney at least based on
16:40:33 fwiesel Sure, are there tests I am missing?
16:40:33 sean-k-mooney http://openstack-ci-logs.global.cloud.sap/openstack-nova-909474-f54wv/tempest.html
16:40:57 sean-k-mooney fwiesel: well you only runing the compute api tests
16:41:16 sean-k-mooney your not runng the senario tests ro any fo teh test fo other services like cinder/nuetorn extra
16:41:25 sean-k-mooney we dont need you to run all of those
16:41:40 sean-k-mooney but i was just providing context for others
16:41:47 sean-k-mooney i.e. its not tempest-full
16:42:45 fwiesel No, indeed it isn't.
16:43:26 fwiesel I am not even running all compute.api-tests, only those that are implemented for the vmware api driver.
16:43:51 sean-k-mooney ack that kind of makes sense
16:44:11 sean-k-mooney for the hardware based cis they took a simialr approch
16:44:38 sean-k-mooney i.e. the old intel ci just ran the cpu pinning/sriov/numa related subset
16:46:22 fwiesel Possibly it would make sense to check what code-coverage the tempest test generates on the driver implementation, and adjust the tests accordingly.
17:55:03 sean-k-mooney dansmith: correct me if im wrong but we do not have the ablity to ask placement for a host that does not have a resouce type right like a forbiden trait, we would have to do that in a scheduler filter
17:56:42 sean-k-mooney related to our converation about non numa instances
17:56:49 dansmith sean-k-mooney: I really don't know currently, but I'm pretty sure that was one of the early limitations.. we could ask for inclusion but not exclusion
17:57:27 sean-k-mooney ack i think its still that way. i belive we coudl have a new filter that does the following check
17:58:26 sean-k-mooney for a given instance look at the guest falvor/image to see if it is requesting numa aware memory i.e. hw:mem_page_size
17:58:54 sean-k-mooney if it is not check the allcoation candates for the host to see if it has either the file backed memory trait or an inventory of PCPU
17:59:02 sean-k-mooney if it has either reject the host
18:00:05 sean-k-mooney that would allow operators to opt in/out by not using the filter
18:00:39 sean-k-mooney we can add it to the default filters to enforce the correct behavior.
18:00:54 dansmith well, that feels more formal of a feature opt-out than a workaround named I_like_to_oom_my_instances=True
18:01:12 dansmith but it would mean an abort in scheduling instead of compute, which would be better
18:02:07 sean-k-mooney well we cloud perhaps not make it a filter but a config option that woudl work like a filter
18:02:19 sean-k-mooney i.e. have a workaround to remove this mandatory filter
18:02:21 dansmith sean-k-mooney: btw, I am still interested in the answer to my question here: https://review.opendev.org/c/openstack/nova/+/877773/9/nova/virt/libvirt/migration.py
18:02:44 dansmith because it seems like it should always fail and I don't know why it's not
18:03:41 sean-k-mooney reading that now ill need to check but if i rememebr correctly we pop the value form the dict before it ever get there
18:04:00 dansmith that's a bad solution IMHO if so because it would break computes that don't have this code yet,
18:04:05 dansmith but I also couldn't find any such pop
18:04:14 sean-k-mooney your asertion is calling int(a string containing a list of ints)
18:04:35 dansmith sean-k-mooney: no it's calling int() on the fake key
18:04:36 sean-k-mooney should always explode so either we are not hitting that path in the test or there is somethign else going on here
18:04:58 dansmith that dict is supposed to be keys of '1', '2', etc and so it int()'s the keys,
18:05:08 dansmith but now there's a key of "dst_cpu_whatever" in the dict
18:05:20 dansmith and I don't know how it's not failing to int() that when it iterates the items
18:05:35 dansmith and I'm wondering if it's because we don't have test coverage that was hitting that code path with the three conditions being true
18:07:58 sean-k-mooney oh right "guest_id, host_ids in info.cpu_pins.items()" so guest_id would be the key
18:08:04 dansmith right
18:08:09 dansmith so I don't know how that's not breaking
19:23:13 sean-k-mooney artom: https://review.opendev.org/c/openstack/nova/+/631053 and https://bugs.launchpad.net/nova/+bug/1811886 are related to our converstaion about the issue relating to not requesting a page size.
19:23:52 sean-k-mooney its not the example i was giving when we talks but its a similar equally wrong case.
19:42:02 opendevreview Merged openstack/nova master: Remove nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902686
19:42:13 opendevreview Merged openstack/nova master: Add new nova.wsgi module https://review.opendev.org/c/openstack/nova/+/902687
19:48:25 sean-k-mooney dansmith: i looked brifly but i dont know why that is not brekaing things but my best guess is we are somehow mocking this and we dont have cpu_shared_set defiend in any of the jobs
19:49:05 sean-k-mooney there is no reason why we cant have cpu share set in one of our live migration jobs so i might try and enable that
19:50:27 sean-k-mooney i.e. 0-5 on one node and 2-7 on the other
19:51:00 sean-k-mooney that pretty trivial to do so ill add a patch on top or renes serise to do that
19:52:13 sean-k-mooney in theory rene's code should be runnign even if that is not devied but we should be able to test this in the gate anyway
19:59:42 opendevreview Stephen Finucane proposed openstack/nova master: Make overcommit check for pinned instance pagesize aware https://review.opendev.org/c/openstack/nova/+/631053
20:01:56 dansmith sean-k-mooney: yeah, I'm unsure either, but I'm positive that means we're just not running the code we think we're running
20:02:02 opendevreview ribaudr proposed openstack/nova master: Fix [backport]: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/877773
20:02:02 opendevreview ribaudr proposed openstack/nova master: Fix: Live migrating to a host with cpu_shared_set configured will now update the VM's configuration accordingly. https://review.opendev.org/c/openstack/nova/+/903706
20:07:51 sean-k-mooney dansmith: my guess right now is we are never populating the numa toplogy info in the migrate_data
20:08:18 dansmith sean-k-mooney: you mean not running a test that causes it to be populated right?
20:08:46 dansmith I mean...not running a test _case_ where it would be populated and that code path exercised
20:10:40 sean-k-mooney no i ment not seting the object that had the new data at all but we appear to be
20:10:54 artom sean-k-mooney, dansmith, hrmm, I wonder if we're fine because... let my type this out
20:11:08 artom The NUMA live migration code kicks in only if the instance has a NUMA topology
20:11:39 artom Uggla's patch _always_ adds that hidden item in the dst_numa_info
20:12:19 sean-k-mooney so it alwasy uses the numa migration code?
20:12:42 artom Hold on
20:13:50 artom No, I thought I had it...
20:16:11 sean-k-mooney the fact we have to have this converstaion and its not obvious how its working after a few of us lokking at it is a code smell to me in either case
20:16:31 sean-k-mooney the code might be correct but its to complex to debug
20:18:31 artom So to trigger it, all that should be required in live migration a NUMA VM to a dest with cpu_shared_set, right?
20:19:10 artom That would cause Uggla's patch to stash the cpu_shared_set in the "hidden" key in dst_numa_info, and *should* cause the update XML code to trip over and explode?
20:19:26 artom We've _definitely_ done that ^^ in whitebox, and nothing has exploded
20:19:42 sean-k-mooney artom: no
20:19:56 sean-k-mooney to trigger the bug yuou need to migate a non numa vm
20:20:06 sean-k-mooney oh
20:20:17 sean-k-mooney you mena the the driver issue

Earlier   Later