| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-06-11 | |||
| 13:15:02 | WJeffs | Hey, I might be being blind, but is there any options for Numa placement when allocating them to a VM. Say we have a HV with 8 numa groups across 2 cpus, and I want to allocate all the cores from 2 groups, thats no issue, but it gives me 1+6, but is there a way we can make it always use numa on same physical cpu at all times? | |
| 13:59:06 | sean-k-mooney | WJeffs: form the same cpu package no. if you have a numa affined guest and ask for 2 numa nodes we will either pack or balance the vm across the numa node depenign on a config option | |
| 13:59:39 | sean-k-mooney | WJeffs: in the past the beahvior was to pack leading to poor performance we now default to spreading | |
| 14:00:12 | sean-k-mooney | WJeffs: we did have a request ot supprot socket level optimiations for amd cpus in the past | |
| 14:00:21 | sean-k-mooney | but we have not had time to implemtn that | |
| 15:00:51 | WJeffs | sean-k-mooney: ah it is for AMD Genoa, I guess that is the same part, we are already creating the numa configuration inside the VM and that works great and gives us good performance, now we are trying to test some extra configurations and found that the numas are spread across sockets | |
| 15:01:20 | WJeffs | I'll have a look for the request/blueprint and see if there is anything we can do to help on this. | |
| 15:05:22 | sean-k-mooney | WJeffs: this is basicaly all contolled by https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2449-L2511 currently | |
| 15:05:47 | sean-k-mooney | to do what you want we would need to add the socket info int the numa toplogy | |
| 15:06:10 | sean-k-mooney | and then add an addtional sort here https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2481 | |
| 15:06:18 | sean-k-mooney | to sort the nodes by socket | |
| 15:07:37 | sean-k-mooney | sorry not there just below it https://github.com/openstack/nova/blob/master/nova/virt/hardware.py#L2488 | |
| 15:07:43 | sean-k-mooney | the les is for pinned cpus | |
| 15:38:27 | WJeffs | sean-k-mooney: Thanks, I'll take a look and see if there is something can do to help | |
| 15:40:00 | sean-k-mooney | the main gap right now is we dont track the socket id in the host numa topogy data stucture | |
| 15:40:07 | sean-k-mooney | or the numa distance btween numa ndoes | |
| 15:40:13 | sean-k-mooney | since that really waht you want to minimise | |
| 15:40:34 | sean-k-mooney | libvirt provide both we jsut dont collect it right now | |
| 15:41:00 | WJeffs | yea, thats the main thing we are trying to do. So its collect from libvirt -> use that data that needs to be added. | |
| 15:41:29 | sean-k-mooney | ya so artom looked at this brifly a year or two ago | |
| 15:42:00 | sean-k-mooney | we have a downstream tracker in our backlog to eventually look at this https://issues.redhat.com/browse/OSPRH-36 | |
| 15:42:15 | sean-k-mooney | but we jsut didnt have time or custoemr ask to priories it | |
| 15:42:21 | artom | I even have/had WIP patches up, at some point | |
| 15:42:35 | artom | Just the basics to collect those distance from libvirt | |
| 15:42:35 | sean-k-mooney | https://review.opendev.org/q/topic:%22bp/libvirt-smarter-cpu-placement%22 | |
| 15:42:39 | sean-k-mooney | yep ^ | |
| 15:43:06 | sean-k-mooney | there in merge conflict but its proably pretty trivial | |
| 15:43:28 | sean-k-mooney | this was the bluepritn https://blueprints.launchpad.net/nova/+spec/libvirt-smarter-cpu-placement | |
| 15:43:34 | sean-k-mooney | btu not much detail there | |
| 15:44:53 | WJeffs | ah wow, it was pretty close then, just come work to tidy up. | |
| 15:47:06 | sean-k-mooney | tests, ensuring this works for live mirgation that sort of thing | |
| 15:47:12 | WJeffs | appreciate all the hard work guys | |
| 15:47:36 | sean-k-mooney | i dont recal exactly how far artom got but its a good starting point | |
| 15:47:48 | artom | Those two patches are literally all I had. | |
| 15:48:13 | artom | The hard part is integrating that into the placement of VMs | |
| 15:55:30 | bauzas | nova meeting in 5 mins here | |
| 15:55:41 | bauzas | I could be a bit lagging, don't be afraid | |
| 16:00:22 | bauzas | #startmeeting nova | |
| 16:00:22 | opendevmeet | Meeting started Tue Jun 11 16:00:22 2024 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:22 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:22 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:35 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:00:35 | gibi | o/ | |
| 16:00:56 | elodilles | o/ | |
| 16:01:00 | fwiesel | \o | |
| 16:01:11 | dansmith | o/ | |
| 16:01:27 | auniyal | o/ | |
| 16:01:34 | bauzas | my current time : | |
| 16:01:37 | bauzas | 64 bytes from par21s19-in-f4.1e100.net (142.250.179.68): icmp_seq=27972 ttl=112 time=193 ms | |
| 16:01:49 | bauzas | anyway, let's start | |
| 16:02:05 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:10 | bauzas | #info No Critical bug | |
| 16:02:15 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:02:22 | bauzas | anything to discuss ? | |
| 16:03:09 | bauzas | looks not | |
| 16:03:19 | bauzas | #topic Gate status | |
| 16:03:25 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:03:30 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures-minimal | |
| 16:03:36 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:03:56 | bauzas | we had a nova-emulation master job failure https://zuul.openstack.org/build/2f26196745fe4cc8a93b5453317710ee | |
| 16:04:09 | sean-k-mooney | the fix for that job is still not merged | |
| 16:04:23 | sean-k-mooney | it keeps hitting job timeout or unrelated failures in other jobs | |
| 16:04:36 | bauzas | because of when deleting some instance, it was not found | |
| 16:04:53 | sean-k-mooney | i have started to look in to how to gather some data to deteimn if we should bump the timeouts in the integrated-compute jobs and its decendent jobs | |
| 16:05:00 | bauzas | ok | |
| 16:05:56 | sean-k-mooney | the simple solution woudl be to temporally add 30mins to the integrated comptue job while i try and figure out what a more reasonable value would be | |
| 16:06:20 | bauzas | but it was not a timeout issue | |
| 16:06:28 | sean-k-mooney | not in this case | |
| 16:06:36 | sean-k-mooney | but the patch that fixes the OOM issue in that job | |
| 16:06:44 | sean-k-mooney | is being blocked by timouts in other jobs | |
| 16:06:56 | bauzas | ah ok | |
| 16:07:23 | sean-k-mooney | anyway i think we can move on for now | |
| 16:07:27 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:07:31 | bauzas | #info Please try to provide meaningful comment when you recheck | |
| 16:07:36 | bauzas | #topic Release Planning | |
| 16:07:40 | bauzas | #link https://releases.openstack.org/dalmatian/schedule.html | |
| 16:07:45 | bauzas | #info Dalmatian-2 in 3 weeks | |
| 16:08:25 | bauzas | #info we'll have a spec review day on July 3rd | |
| 16:08:34 | bauzas | (which is a Wednesday) | |
| 16:08:55 | bauzas | #action bauzas to tell this in the mailing-list | |
| 16:09:08 | bauzas | (on July 2nd I'll be off) | |
| 16:09:50 | bauzas | would people prefer to have this day on Monday, as July 4th is some US holiday | |
| 16:09:55 | bauzas | ? | |
| 16:11:31 | bauzas | is it me lagging or nobody replied ? | |
| 16:11:51 | dansmith | nobody replied | |
| 16:11:57 | bauzas | thoughts then ? | |
| 16:12:10 | dansmith | it's right around the american holiday | |
| 16:12:20 | dansmith | most of us will be plotting to blow up our neighbors on the following day | |
| 16:13:00 | sean-k-mooney | i have no real prefernce os i think tis also fine if we do it semi async | |
| 16:13:26 | sean-k-mooney | i.e. if us folks wont be around that day and feel like doing reviews on monday then the rest of us will already have that feedback | |
| 16:13:33 | sean-k-mooney | on wednesday | |
| 16:15:37 | bauzas | ok, let's do this on monday and I'll explain that it can be async for the week | |
| 16:15:44 | bauzas | moving on | |
| 16:15:49 | bauzas | #topic Review priorities | |
| 16:15:55 | bauzas | #link https://etherpad.opendev.org/p/nova-dalmatian-status | |
| 16:16:15 | bauzas | nothing to report, I'm planning to review specs but my lag doesn't help much today | |
| 16:16:26 | bauzas | hopefully my connection will come back tomorrow | |
| 16:17:02 | bauzas | moving on | |
| 16:17:04 | bauzas | #topic Stable Branches | |
| 16:17:13 | bauzas | elodilles: please | |