| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-21 | |||
| 20:17:39 | sean-k-mooney | for now can you file a bug for isolate | |
| 20:18:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/877773 | |
| 20:18:37 | sean-k-mooney | so ya still open | |
| 20:18:53 | artom | Yeah, I guess he's concentrating on the manila stuff | |
| 20:19:05 | sean-k-mooney | which is fair | |
| 20:20:24 | sean-k-mooney | but that why that is broken | |
| 20:20:47 | sean-k-mooney | it should assumign its not in mergee confict | |
| 20:21:15 | artom | I'll see you in two hours ;) | |
| 20:21:43 | sean-k-mooney | i was going to say i wont be around then.... | |
| 20:21:53 | artom | Which is entirely legitimate | |
| 20:21:56 | sean-k-mooney | but when has that happend when i planed | |
| 20:22:26 | artom | That's between you and your sense of disipline ;) | |
| 20:23:36 | sean-k-mooney | and my stomach | |
| 20:23:50 | sean-k-mooney | for some reasons it has notions that i should feed it at least once a day | |
| 20:24:22 | artom | Pfft, what does it think it is | |
| 20:25:23 | sean-k-mooney | so the bug is here by the way https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/api.py#L105 | |
| 20:25:52 | sean-k-mooney | i could have sworn instance.numa_topology.cpu_pinning also had the isolated core for emulator threads | |
| 20:26:03 | sean-k-mooney | when using that policy | |
| 20:26:49 | artom | More in power_up(), but yeah | |
| 20:26:54 | artom | Both, really | |
| 20:28:50 | sean-k-mooney | so based on the docs string https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L291-L296 | |
| 20:29:07 | sean-k-mooney | it should be all the cores for teh instance | |
| 20:29:18 | artom | When has a docstring ever been wrong ;) | |
| 20:39:06 | sean-k-mooney | i said this hsould be easy but this behviaor is berried deep in the cpu pinning code | |
| 20:39:57 | sean-k-mooney | looking at it quickly https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L2693-L2709 | |
| 20:40:05 | sean-k-mooney | i think it shold be working... | |
| 20:40:13 | artom | Yeah, same, the more I look, the more I'm lost | |
| 20:40:45 | sean-k-mooney | as far as i can tell we are correctly reserving 1 addtional core in the vms numa node 0 for the isolated thread | |
| 20:41:09 | sean-k-mooney | and then storign it in the same set/dict as all the rest | |
| 20:41:25 | sean-k-mooney | that what if instance_cell.cpuset_reserved: | |
| 20:41:27 | sean-k-mooney | pinned_cpus |= instance_cell.cpuset_reserved | |
| 20:41:31 | sean-k-mooney | is doing | |
| 20:42:01 | sean-k-mooney | anyway first stpe file bug | |
| 20:42:18 | sean-k-mooney | second step create repoducer | |
| 20:43:50 | artom | So at least for that same instance | |
| 20:43:57 | artom | With the emulator thread on host CPU 4 | |
| 20:44:01 | artom | It logs this: | |
| 20:44:06 | artom | Feb 21 19:15:30.167111 [compute/logs/screen-n-cpu.txt] np0036828693 nova-compute[42254]: DEBUG nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Selected cores for pinning: [(0, 6), (1, 5)], in cell 0 {{(pid=42254) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:905}} | |
| 20:44:21 | sean-k-mooney | so im sus that https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L2689-L2694 | |
| 20:44:39 | sean-k-mooney | does not have a bug | |
| 20:44:54 | sean-k-mooney | but i cant fully say why i felll like that | |
| 20:45:56 | artom | So it doesn't log the "extra" emulator thread core | |
| 20:46:03 | sean-k-mooney | right | |
| 20:46:08 | sean-k-mooney | just doing an or | |
| 20:46:08 | artom | And doesn't pin it at https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L908 | |
| 20:46:10 | sean-k-mooney | pinned_cpus |= instance_cell.cpuset_reserved | |
| 20:46:48 | sean-k-mooney | would work if cpuset_reserved is disjoit from instance_cell.cpu_pinning.values() | |
| 20:46:52 | sean-k-mooney | which it should be | |
| 20:47:03 | sean-k-mooney | but... | |
| 20:48:37 | sean-k-mooney | this do you see this debug message https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L663-L673 | |
| 20:48:55 | sean-k-mooney | Packing an instance onto a set of sibling ... | |
| 20:49:33 | artom | cated',cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([]),cpuset_reserved=None,id=0,memory=64,pagesize=None,pcpuset=set([0,1])) host_cell_id: 0 threads_per_core: 1 num_cpu_reserved: 1 {{(pid=42254) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:663}} | |
| 20:49:33 | artom | Feb 21 19:15:30.165439 [compute/logs/screen-n-cpu.txt] np0036828693 nova-compute[42254]: DEBUG nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Packing an instance onto a set of siblings: host_cell_free_siblings: [{6}, set(), {5}, {4}, set()] instance_cell: InstanceNUMACell(cpu_pinning_raw=None,cpu_policy='dedi | |
| 20:50:07 | sean-k-mooney | ok so ya its asking for the the isolated core | |
| 20:50:22 | artom | Well, we know the policy itself works | |
| 20:50:37 | artom | But is it recording the extra pinned core in a way that the CPU power management code can then power on/off | |
| 20:50:52 | sean-k-mooney | so we should run https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L754-L782 | |
| 20:51:18 | sean-k-mooney | its recordigin it in cpuset_reserved | |
| 20:51:34 | sean-k-mooney | which is then |= with the other cores | |
| 20:51:35 | artom | The comment on that says that it's reserved for the hypervisor | |
| 20:51:46 | artom | In the object | |
| 20:52:12 | sean-k-mooney | in this case hypervior means qemu | |
| 20:52:22 | sean-k-mooney | its reserved for the qemu emulator thread | |
| 20:52:28 | artom | Hah | |
| 20:52:58 | artom | So should the power management code be powering down/up cpuset_reserved as well? | |
| 20:53:25 | artom | Without going through the whole paperwork, we can do a quick POC and Depends-on in whitebox to see | |
| 20:53:29 | sean-k-mooney | it should already be incldued into pinned_cpus | |
| 20:53:33 | sean-k-mooney | but yes | |
| 20:53:44 | sean-k-mooney | it should be powering up the emulator thread core | |
| 20:53:59 | sean-k-mooney | do you see this message https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L776-L777 | |
| 20:54:55 | artom | It just checks cpu_pinning... | |
| 20:54:58 | sean-k-mooney | yep | |
| 20:55:00 | sean-k-mooney | Feb 21 19:15:30.166825 np0036828693 nova-compute[42254]: INFO nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Computed NUMA topology reserved pCPUs: usable pCPUs: [[], [], [4]], reserved pCPUs: {4} | |
| 20:55:13 | sean-k-mooney | but notice anythying odd | |
| 20:55:36 | sean-k-mooney | we have these two logs back to back | |
| 20:55:39 | sean-k-mooney | Feb 21 19:15:30.166472 np0036828693 nova-compute[42254]: INFO nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Computed NUMA topology CPU pinning: usable pCPUs: [[6], [5], [4]], vCPUs mapping: [(0, 6), (1, 5)] | |
| 20:55:40 | sean-k-mooney | Feb 21 19:15:30.166825 np0036828693 nova-compute[42254]: INFO nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Computed NUMA topology reserved pCPUs: usable pCPUs: [[], [], [4]], reserved pCPUs: {4} | |
| 20:56:29 | sean-k-mooney | it calulated both | |
| 20:56:34 | sean-k-mooney | and we shoudl be merging them | |
| 20:57:22 | sean-k-mooney | but then it prints | |
| 20:57:24 | sean-k-mooney | Selected cores for pinning: [(0, 6), (1, 5)], in cell 0 {{(pid=42254) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:905}} | |
| 20:58:29 | sean-k-mooney | which is form here https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L905-L906 | |
| 20:59:44 | sean-k-mooney | artom: tldr adding cpuset_reserved to power on and power off would fix it | |
| 20:59:56 | artom | Yeah, lemme try a quick hax of that | |
| 21:00:16 | sean-k-mooney | but but i thin kbased on the code we ere expiectin that not to be required | |
| 21:00:50 | sean-k-mooney | so maybe we need to update the doc string or the propery to include cpuset_reserved | |
| 21:02:00 | sean-k-mooney | i.e. if you made https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/objects/instance_numa.py#L291-L296 also include it | |
| 21:02:26 | sean-k-mooney | ok im going to finish there let me know how your hacking goes | |
| 21:02:45 | artom | It's done, just need to push and retry whitebox | |
| 21:03:34 | opendevreview | Artom Lifshitz proposed openstack/nova master: POC: online dedicated emulator threads as well https://review.opendev.org/c/openstack/nova/+/909795 | |
| 21:03:57 | artom | Hrmm, I can't Depends-on different Nova changes, can I? | |
| 21:04:49 | sean-k-mooney | you can | |
| 21:05:00 | sean-k-mooney | you can depends on 2 diffent patches in the same repo | |
| 21:05:07 | sean-k-mooney | you normally dont want too | |
| 21:05:09 | artom | Really? And Zuul does like a magic merge of them? | |
| 21:05:14 | sean-k-mooney | yep | |
| 21:05:32 | sean-k-mooney | so as long as they dont have merge conflcit between each other it will work | |
| 21:05:59 | artom | Dayum, I'm impressed | |
| 21:06:06 | artom | OK, https://review.opendev.org/c/openstack/nova/+/909795 can run while I pick up the kids from school | |
| 21:06:15 | sean-k-mooney | we normally done do tha because its way more fragile but it works i limited caes | |