| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-14 | |||
| 12:06:10 | gibi | so it might optimize for the current csae | |
| 12:06:11 | gibi | case | |
| 12:06:31 | sean-k-mooney | the worst case performace woudl still be the same but i think it would imporve best and average case | |
| 12:11:03 | gibi | so we optimize based on some heuristics that the first numa nodes are more likely to be filled than the later numa nodes | |
| 12:11:24 | sean-k-mooney | well until master it did a liniar search | |
| 12:11:37 | sean-k-mooney | so yes the first numa nodes were always filled first deterministically | |
| 12:11:51 | sean-k-mooney | we recently added numa node blancing | |
| 12:12:22 | gibi | true, so using the old linear fill, it make sense to add a heuristics to the search to | |
| 12:12:25 | gibi | o | |
| 12:12:44 | sean-k-mooney | also my entire system just hard locked up while i was debuging that even with it paused | |
| 12:12:51 | gibi | wondering if just simply using permutations(reversed(host_cells)) would be enough to optimize too | |
| 12:12:53 | sean-k-mooney | perhaps i should close some broser tabs | |
| 12:13:10 | sean-k-mooney | well we are not sorting the host_cells | |
| 12:13:25 | gibi | the other day I run out of memory on my laptop, so now I added some swap | |
| 12:13:29 | sean-k-mooney | based on aviable ram disk pci devices and if you asked for them | |
| 12:14:00 | sean-k-mooney | i ran out of swap but still had 8GB of ram free | |
| 12:14:31 | sean-k-mooney | i do have a 2 node devstack and openshift running in 3 8G vms currently too | |
| 12:16:17 | sean-k-mooney | oh my email client is only using 4G of ram today that is nice of it. it was using 15 last week... | |
| 12:17:34 | sean-k-mooney | 1910527.379325] Out of memory: Killed process 1338765 (.qemu-system-x8) total-vm:13382508kB, anon-rss:7832544kB, file-rss:0kB, shmem-rss:4kB, UID:0 pgtables:17324kB oom_score_adj:0 | |
| 12:17:42 | sean-k-mooney | yep i ran out of memory ill stop the vms for now | |
| 12:18:44 | sean-k-mooney | thats totally a good sign for the effiency fo this code right | |
| 12:29:30 | sean-k-mooney | gibi: :) | |
| 12:29:33 | sean-k-mooney | import nova.conf | |
| 12:29:35 | sean-k-mooney | CONF = nova.conf.CONF | |
| 12:29:37 | sean-k-mooney | CONF.compute.packing_host_numa_cells_allocation_strategy = False | |
| 12:29:39 | sean-k-mooney | that fixes the issue | |
| 12:29:58 | sean-k-mooney | we disable the numa blancing by defualt for "backwards compatiblity" | |
| 12:30:31 | sean-k-mooney | gibi: if you enable it when we sort by free memory all the used nodes go to the end | |
| 12:30:41 | sean-k-mooney | so the first permuation fits | |
| 12:31:17 | sean-k-mooney | [13:30:55]➜ time python t.py | |
| 12:31:18 | sean-k-mooney | InstanceNUMATopology(cells=[InstanceNUMACell(8),InstanceNUMACell(9),InstanceNUMACell(10),InstanceNUMACell(11),InstanceNUMACell(12),InstanceNUMACell(13),InstanceNUMACell(14)],emulator_threads_policy=None,id=<?>,instance_uuid=<?>) | |
| 12:31:20 | sean-k-mooney | real 0m1.489s | |
| 12:31:22 | sean-k-mooney | user 0m1.373s | |
| 12:31:24 | sean-k-mooney | sys 0m0.095s | |
| 12:31:44 | sean-k-mooney | gibi: not as fast as my out of tree version but pretty close | |
| 12:31:51 | gibi | ahh | |
| 12:31:53 | sean-k-mooney | and fully feature complete | |
| 12:32:02 | gibi | that is an easy workaround for the particular case | |
| 12:32:43 | sean-k-mooney | well the spread approch is generaly beter provided you dont need really large vms | |
| 12:32:56 | sean-k-mooney | that depend on fully filling the numa nodes to spawn | |
| 12:32:58 | sean-k-mooney | but yes | |
| 12:38:31 | sean-k-mooney | gibi: ok added a comment https://bugs.launchpad.net/nova/+bug/1978372/comments/6 | |
| 12:38:59 | sean-k-mooney | gibi: we have already started backporting the pack/spread behavior to xena they reported it on wallaby | |
| 12:39:22 | sean-k-mooney | so if we just continue the backport of the sorting behavior we could close it as a dupe i guess | |
| 12:39:56 | gibi | sean-k-mooney: thanks | |
| 12:39:56 | sean-k-mooney | that does raise the question of if we should change the default for pack vs spread on master and or still look at the other implemation in the future | |
| 12:40:29 | gibi | I think we can chance default on master now | |
| 12:40:33 | gibi | if we want | |
| 12:40:45 | gibi | probably not on stable | |
| 12:44:09 | sean-k-mooney | right not stabel but i can propose a patch to change the default and add a release note for people to review | |
| 12:47:40 | gibi | yeah, lets try | |
| 12:48:00 | gibi | I have to refresh myself about the trade off though | |
| 12:49:42 | sean-k-mooney | spread will try to put vms on empty numa nodes first pack does the reverse trying to use all aviable space on numa ndoes before using the next one | |
| 12:50:06 | sean-k-mooney | if you have 2 numa nodes and but 3 vms 1 that need a full numa node and 2 that each need a half numa node | |
| 12:50:26 | sean-k-mooney | then with pack all 3 will in any order | |
| 12:50:45 | sean-k-mooney | with spread unless the big vms is booted first only 2 of the 2 will schdule | |
| 12:50:54 | sean-k-mooney | gibi: ^ that the main trade off | |
| 12:51:28 | gibi | thanks | |
| 12:51:41 | sean-k-mooney | but if you spread you get better cpu/memory performace in the guest until the node is full and then its the same | |
| 12:52:08 | gibi | I think both way is valid, but if we know that the scheduling performance is better in the spread case then that might be enough reasoning to switch | |
| 12:52:29 | gibi | the default | |
| 12:52:41 | sean-k-mooney | yep both are valid but i personaly prefer spread as the default | |
| 12:53:02 | sean-k-mooney | it does need to be configurable | |
| 12:53:09 | sean-k-mooney | so that operators can choose | |
| 12:53:38 | sean-k-mooney | you could technially have differnt values on scheduler vs compute too | |
| 12:53:52 | sean-k-mooney | you could use spread in schduler and pack on computes | |
| 12:54:55 | bauzas | gibi: good news, I was able to ask for multiple vGPUs | |
| 12:54:59 | sean-k-mooney | currently we recompute the pinning on the compute anyway since we throwaway the schduler info | |
| 12:55:20 | bauzas | but yeah, we hit the warning, so we only have one VGPU | |
| 12:55:27 | sean-k-mooney | bauzas: using generic mdevs to use differnt device classees for differnt host gpus | |
| 12:55:40 | sean-k-mooney | or via multi create | |
| 12:55:46 | bauzas | sean-k-mooney: lemme upload the functest | |
| 12:55:52 | sean-k-mooney | ack | |
| 13:09:36 | opendevreview | Sylvain Bauza proposed openstack/nova master: Add a functest for verifying multiple VGPU allocations https://review.opendev.org/c/openstack/nova/+/845747 | |
| 13:09:42 | bauzas | sean-k-mooney: ^ | |
| 13:10:34 | bauzas | sean-k-mooney: tl,dr: given of the nvidia driver issue (you can't ask for more than one VGPU per pGPU per instance), operators would then want to spread the vGPUs between multiple pGPUs | |
| 13:12:49 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/845748 | |
| 13:26:14 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/845753 | |
| 13:29:29 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | |
| 13:42:05 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP : Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 13:42:11 | bauzas | gibi: ^ | |
| 14:15:50 | gibi | bauzas: left feedback :) | |
| 14:16:05 | bauzas | cool, this is just a WIP tho | |
| 14:17:19 | bauzas | gibi: thanks for commenting, those are good thoughts | |
| 14:18:25 | gibi | I didn't even relaized that it is a WIP, it has functional test :D | |
| 14:19:06 | bauzas | Zuul will hit my face as I think I'm closing some gaps | |
| 14:19:15 | bauzas | and I'll need to change some UTs | |
| 14:27:36 | ygk_12345 | can someone look into this https://bugs.launchpad.net/oslo.messaging/+bug/1978562 please | |
| 14:32:52 | opendevreview | Balazs Gibizer proposed openstack/nova master: Clean up mapping input to address spec types https://review.opendev.org/c/openstack/nova/+/845765 | |
| 14:36:29 | opendevreview | Balazs Gibizer proposed openstack/nova master: Clean up mapping input to address spec types https://review.opendev.org/c/openstack/nova/+/845765 | |
| 15:02:10 | opendevreview | Elod Illes proposed openstack/placement stable/victoria: Add periodic-stable-jobs template https://review.opendev.org/c/openstack/placement/+/845770 | |
| 15:03:28 | opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: remove default cputune shares value https://review.opendev.org/c/openstack/nova/+/824048 | |
| 15:09:15 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/845748 | |
| 15:10:56 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/845753 | |
| 15:11:18 | opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | |
| 15:21:59 | opendevreview | Balazs Gibizer proposed openstack/nova master: Remove unused PF checking from get_function_by_ifname https://review.opendev.org/c/openstack/nova/+/845775 | |
| 15:30:50 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix type annotation of pci.Whitelist class https://review.opendev.org/c/openstack/nova/+/845780 | |
| 15:35:30 | ygk_12345 | can someone look into this https://bugs.launchpad.net/oslo.messaging/+bug/1978562 please | |
| 15:40:23 | bauzas | reminder: nova meeting in 20 mins | |
| 15:50:57 | opendevreview | Balazs Gibizer proposed openstack/nova master: Move __str__ to the PciAddressSpec base class https://review.opendev.org/c/openstack/nova/+/845781 | |