| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-21 | |||
| 14:29:18 | sean-k-mooney | yes, although i do not think that is relvent here | |
| 14:29:28 | bauzas | I think the intent was to let the user restart its instance if it was erroring due to transient issues | |
| 14:29:48 | sean-k-mooney | yes we will need an rpc but that rpc would only be successful if the share was both mounted and attached to the vm | |
| 14:30:06 | sean-k-mooney | it shoudl likely revert the share mounting on the host if the libvirt attach failed | |
| 14:30:13 | sean-k-mooney | so you can try attach again | |
| 14:30:13 | bauzas | that said, there are billing implications on a stop action that would fail due to some internal problem | |
| 14:30:34 | sean-k-mooney | we explcitly allow detach in ERROR | |
| 14:30:40 | sean-k-mooney | to make sure that is not a porblem | |
| 14:30:48 | bauzas | I don't think our public cloud providers woud like their users to be continued to be charged because they have some network issues | |
| 14:30:58 | gibi | I think bauzas refer to stop failing not detach failing | |
| 14:30:58 | sean-k-mooney | if stop failes you can detach and then stop again | |
| 14:31:04 | bauzas | gibi: correct | |
| 14:31:05 | gibi | ahh | |
| 14:31:24 | gibi | you cannot detach if it is not stopped | |
| 14:31:39 | sean-k-mooney | the spec say you can do it in the error state | |
| 14:31:58 | sean-k-mooney | also from the ordering the vm would be stopped then the detach woudl fail | |
| 14:32:09 | sean-k-mooney | *then the unount woudl fail | |
| 14:32:42 | gibi | so stop fails, puts the VM in ERROR, we allow detach in ERROR, so user does the detach, detach deletes the mapping, we lost the mapping while the unmount never happened | |
| 14:32:44 | sean-k-mooney | so it should be safe to allow the detach to update the db. | |
| 14:33:05 | sean-k-mooney | yep and that where a perodic would come in | |
| 14:33:10 | sean-k-mooney | to allow that to be healed | |
| 14:34:07 | sean-k-mooney | same | |
| 14:34:16 | gibi | then lets park this for after the meeting | |
| 15:05:05 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Drop wrong stacklevel https://review.opendev.org/c/openstack/os-vif/+/909682 | |
| 15:23:25 | tkajinam | ^^^ I noticed I've made this mistake in a few places... it'd be nice if this can be merged and included in the final release for 2024.1 ... | |
| 15:29:20 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 18:06:22 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448 | |
| 20:05:36 | artom | So, I tried running with CPU power management defaulted to on, and got this: https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785/1 | |
| 20:05:52 | artom | Interesting libvirt failures launching, for example, with an emulator thread policy | |
| 20:06:08 | artom | I wonder if the job is doing something wrong, or are there bugs... | |
| 20:09:05 | sean-k-mooney | is that using isolate | |
| 20:10:42 | artom | <cputune> | |
| 20:10:42 | artom | <vcpupin vcpu='0' cpuset='6'/> | |
| 20:10:42 | artom | <vcpupin vcpu='1' cpuset='5'/> | |
| 20:10:42 | artom | <emulatorpin cpuset='4'/> | |
| 20:10:42 | artom | </cputune> | |
| 20:10:47 | artom | That looks like `share` | |
| 20:11:24 | sean-k-mooney | its only share if cpu_shared_set=4 | |
| 20:11:54 | sean-k-mooney | cpu_dedicated_set = 4-6 | |
| 20:11:57 | sean-k-mooney | that is isolate | |
| 20:12:00 | sean-k-mooney | https://763b7c31d44017973024-0677f910aec9b59957afc606ff2e533f.ssl.cf2.rackcdn.com/909785/1/check/whitebox-devstack-multinode/532b307/compute/logs/etc/nova/nova-cpu_conf.txt | |
| 20:12:18 | sean-k-mooney | su yes there is a bug its not onlinging the isolat cores | |
| 20:12:34 | sean-k-mooney | there will be 1 and its included in teh instnace numa toplogy | |
| 20:12:35 | artom | That's on the compute | |
| 20:12:40 | artom | cpu_dedicated_set = 4-6 | |
| 20:12:48 | artom | cpu_shared_set = 2,3 | |
| 20:13:03 | sean-k-mooney | oh this was on the contoler | |
| 20:13:15 | artom | No you got it right, it's the compute | |
| 20:13:42 | sean-k-mooney | so ya sound like a trivial bug with isolate | |
| 20:13:54 | sean-k-mooney | that should eb simple to reproduce and fix | |
| 20:14:57 | artom | I think there are others | |
| 20:15:10 | artom | It's failing with a similar error on NUMA live migration... | |
| 20:15:21 | sean-k-mooney | that is the only one that can use a core form the cpu dedicated set that is not for a vcpu | |
| 20:15:22 | artom | But I need to track down the exact instance that it's live migrating... | |
| 20:16:02 | sean-k-mooney | so isolate is the only way we have a non vcpu pinned to the cpu_dedicated_set | |
| 20:16:42 | sean-k-mooney | for numa live migration this would cause issues if we dont have renes fix for the cpu pinning | |
| 20:17:16 | artom | Which we don't | |
| 20:17:33 | sean-k-mooney | well we should merge and backport that fix | |
| 20:17:39 | sean-k-mooney | for now can you file a bug for isolate | |
| 20:18:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/877773 | |
| 20:18:37 | sean-k-mooney | so ya still open | |
| 20:18:53 | artom | Yeah, I guess he's concentrating on the manila stuff | |
| 20:19:05 | sean-k-mooney | which is fair | |
| 20:20:24 | sean-k-mooney | but that why that is broken | |
| 20:20:47 | sean-k-mooney | it should assumign its not in mergee confict | |
| 20:21:15 | artom | I'll see you in two hours ;) | |
| 20:21:43 | sean-k-mooney | i was going to say i wont be around then.... | |
| 20:21:53 | artom | Which is entirely legitimate | |
| 20:21:56 | sean-k-mooney | but when has that happend when i planed | |
| 20:22:26 | artom | That's between you and your sense of disipline ;) | |
| 20:23:36 | sean-k-mooney | and my stomach | |
| 20:23:50 | sean-k-mooney | for some reasons it has notions that i should feed it at least once a day | |
| 20:24:22 | artom | Pfft, what does it think it is | |
| 20:25:23 | sean-k-mooney | so the bug is here by the way https://github.com/openstack/nova/blob/master/nova/virt/libvirt/cpu/api.py#L105 | |
| 20:25:52 | sean-k-mooney | i could have sworn instance.numa_topology.cpu_pinning also had the isolated core for emulator threads | |
| 20:26:03 | sean-k-mooney | when using that policy | |
| 20:26:49 | artom | More in power_up(), but yeah | |
| 20:26:54 | artom | Both, really | |
| 20:28:50 | sean-k-mooney | so based on the docs string https://github.com/openstack/nova/blob/master/nova/objects/instance_numa.py#L291-L296 | |
| 20:29:07 | sean-k-mooney | it should be all the cores for teh instance | |
| 20:29:18 | artom | When has a docstring ever been wrong ;) | |
| 20:39:06 | sean-k-mooney | i said this hsould be easy but this behviaor is berried deep in the cpu pinning code | |
| 20:39:57 | sean-k-mooney | looking at it quickly https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L2693-L2709 | |
| 20:40:05 | sean-k-mooney | i think it shold be working... | |
| 20:40:13 | artom | Yeah, same, the more I look, the more I'm lost | |
| 20:40:45 | sean-k-mooney | as far as i can tell we are correctly reserving 1 addtional core in the vms numa node 0 for the isolated thread | |
| 20:41:09 | sean-k-mooney | and then storign it in the same set/dict as all the rest | |
| 20:41:25 | sean-k-mooney | that what if instance_cell.cpuset_reserved: | |
| 20:41:27 | sean-k-mooney | pinned_cpus |= instance_cell.cpuset_reserved | |
| 20:41:31 | sean-k-mooney | is doing | |
| 20:42:01 | sean-k-mooney | anyway first stpe file bug | |
| 20:42:18 | sean-k-mooney | second step create repoducer | |
| 20:43:50 | artom | So at least for that same instance | |
| 20:43:57 | artom | With the emulator thread on host CPU 4 | |
| 20:44:01 | artom | It logs this: | |
| 20:44:06 | artom | Feb 21 19:15:30.167111 [compute/logs/screen-n-cpu.txt] np0036828693 nova-compute[42254]: DEBUG nova.virt.hardware [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] Selected cores for pinning: [(0, 6), (1, 5)], in cell 0 {{(pid=42254) _pack_instance_onto_cores /opt/stack/nova/nova/virt/hardware.py:905}} | |
| 20:44:21 | sean-k-mooney | so im sus that https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/hardware.py#L2689-L2694 | |
| 20:44:39 | sean-k-mooney | does not have a bug | |
| 20:44:54 | sean-k-mooney | but i cant fully say why i felll like that | |
| 20:45:56 | artom | So it doesn't log the "extra" emulator thread core | |