| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-02-21 | |||
| 14:18:59 | bauzas | umount is not strictly required if we still have a share mapping, right ? | |
| 14:19:21 | gibi | bauzas: if you mean we have a record of the mount so we can do the unmount later, then yes | |
| 14:19:21 | sean-k-mooney | as far as im concerned the attach and detach rps shoudl not try to mount/unmount the share | |
| 14:19:55 | bauzas | gibi: I'm going back to my original statement : I don't want to have share leaks | |
| 14:20:26 | bauzas | gibi: if we're not able to guarantee that some share mapping state will continue to handle shares that are sitll residues on the compute, then we need to error the detach call | |
| 14:20:27 | sean-k-mooney | form the spec | |
| 14:20:30 | sean-k-mooney | """Share attachment/detachment can only be done if the VM state is STOPPED or ERROR. These are operations only on the database, and no RPC calls will be required to the compute API. This is an intentional design for this spec. As a result, this could lead to situation where the VM start operation fails as an underlying share attach fails.""" | |
| 14:20:41 | sean-k-mooney | that ^ is what the code should be doing | |
| 14:20:41 | gibi | bauzas: I agree | |
| 14:21:21 | bauzas | the phi-state in the diagram actually needs to be explained more | |
| 14:21:27 | gibi | OK. so If we follow that statement from the spec then both the mount and unmount RPC is unnecessary | |
| 14:21:45 | sean-k-mooney | correct | |
| 14:21:51 | sean-k-mooney | its just a cell db update | |
| 14:21:58 | bauzas | I have a vgpu meeting in 8 mins so I'll need to bail out shortly | |
| 14:21:59 | sean-k-mooney | to create a row or delete it | |
| 14:22:04 | gibi | this also means we will host mount during start and unmount during stop / delete | |
| 14:22:15 | sean-k-mooney | yep | |
| 14:22:35 | sean-k-mooney | if start fails to mount we go to error wheere you can either retry or detach the share | |
| 14:22:38 | gibi | so if the mount / umount fails start or stop action can fail and put the VM state in ERROR | |
| 14:22:54 | sean-k-mooney | yep | |
| 14:22:58 | sean-k-mooney | thats what we approved | |
| 14:23:22 | sean-k-mooney | anb all the later assumtiosn of the spec are based on that | |
| 14:23:36 | gibi | Uggla: do you have a reasoning why we diverged from the spec in this regard? Was there any late findings during implementation? | |
| 14:25:11 | gibi | or in other words, what issues do you forsee if we do what the spec says and move the host mount / unmount? | |
| 14:27:46 | Uggla | gibi, yes but I do not remember well. I guess the reason was simpler error mgmt and simplify later online attach. | |
| 14:28:18 | sean-k-mooney | well this discussion shows that the error magament is more complex | |
| 14:28:41 | gibi | online attach is a good topic. When we have that in libvirt we will need an RPC during the attach REST API call. | |
| 14:29:18 | sean-k-mooney | yes, although i do not think that is relvent here | |
| 14:29:28 | bauzas | I think the intent was to let the user restart its instance if it was erroring due to transient issues | |
| 14:29:48 | sean-k-mooney | yes we will need an rpc but that rpc would only be successful if the share was both mounted and attached to the vm | |
| 14:30:06 | sean-k-mooney | it shoudl likely revert the share mounting on the host if the libvirt attach failed | |
| 14:30:13 | sean-k-mooney | so you can try attach again | |
| 14:30:13 | bauzas | that said, there are billing implications on a stop action that would fail due to some internal problem | |
| 14:30:34 | sean-k-mooney | we explcitly allow detach in ERROR | |
| 14:30:40 | sean-k-mooney | to make sure that is not a porblem | |
| 14:30:48 | bauzas | I don't think our public cloud providers woud like their users to be continued to be charged because they have some network issues | |
| 14:30:58 | gibi | I think bauzas refer to stop failing not detach failing | |
| 14:30:58 | sean-k-mooney | if stop failes you can detach and then stop again | |
| 14:31:04 | bauzas | gibi: correct | |
| 14:31:05 | gibi | ahh | |
| 14:31:24 | gibi | you cannot detach if it is not stopped | |
| 14:31:39 | sean-k-mooney | the spec say you can do it in the error state | |
| 14:31:58 | sean-k-mooney | also from the ordering the vm would be stopped then the detach woudl fail | |
| 14:32:09 | sean-k-mooney | *then the unount woudl fail | |
| 14:32:42 | gibi | so stop fails, puts the VM in ERROR, we allow detach in ERROR, so user does the detach, detach deletes the mapping, we lost the mapping while the unmount never happened | |
| 14:32:44 | sean-k-mooney | so it should be safe to allow the detach to update the db. | |
| 14:33:05 | sean-k-mooney | yep and that where a perodic would come in | |
| 14:33:10 | sean-k-mooney | to allow that to be healed | |
| 14:34:07 | sean-k-mooney | same | |
| 14:34:16 | gibi | then lets park this for after the meeting | |
| 15:05:05 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Drop wrong stacklevel https://review.opendev.org/c/openstack/os-vif/+/909682 | |
| 15:23:25 | tkajinam | ^^^ I noticed I've made this mistake in a few places... it'd be nice if this can be merged and included in the final release for 2024.1 ... | |
| 15:29:20 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing requested az in output https://review.opendev.org/c/openstack/nova/+/904568 | |
| 18:06:22 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Add openapi spec https://review.opendev.org/c/openstack/nova-specs/+/909448 | |
| 20:05:36 | artom | So, I tried running with CPU power management defaulted to on, and got this: https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/909785/1 | |
| 20:05:52 | artom | Interesting libvirt failures launching, for example, with an emulator thread policy | |
| 20:06:08 | artom | I wonder if the job is doing something wrong, or are there bugs... | |
| 20:09:05 | sean-k-mooney | is that using isolate | |
| 20:10:42 | artom | <cputune> | |
| 20:10:42 | artom | <vcpupin vcpu='0' cpuset='6'/> | |
| 20:10:42 | artom | <vcpupin vcpu='1' cpuset='5'/> | |
| 20:10:42 | artom | <emulatorpin cpuset='4'/> | |
| 20:10:42 | artom | </cputune> | |
| 20:10:47 | artom | That looks like `share` | |
| 20:11:24 | sean-k-mooney | its only share if cpu_shared_set=4 | |
| 20:11:54 | sean-k-mooney | cpu_dedicated_set = 4-6 | |
| 20:11:57 | sean-k-mooney | that is isolate | |
| 20:12:00 | sean-k-mooney | https://763b7c31d44017973024-0677f910aec9b59957afc606ff2e533f.ssl.cf2.rackcdn.com/909785/1/check/whitebox-devstack-multinode/532b307/compute/logs/etc/nova/nova-cpu_conf.txt | |
| 20:12:18 | sean-k-mooney | su yes there is a bug its not onlinging the isolat cores | |
| 20:12:34 | sean-k-mooney | there will be 1 and its included in teh instnace numa toplogy | |
| 20:12:35 | artom | That's on the compute | |
| 20:12:40 | artom | cpu_dedicated_set = 4-6 | |
| 20:12:48 | artom | cpu_shared_set = 2,3 | |
| 20:13:03 | sean-k-mooney | oh this was on the contoler | |
| 20:13:15 | artom | No you got it right, it's the compute | |
| 20:13:42 | sean-k-mooney | so ya sound like a trivial bug with isolate | |
| 20:13:54 | sean-k-mooney | that should eb simple to reproduce and fix | |
| 20:14:57 | artom | I think there are others | |
| 20:15:10 | artom | It's failing with a similar error on NUMA live migration... | |
| 20:15:21 | sean-k-mooney | that is the only one that can use a core form the cpu dedicated set that is not for a vcpu | |
| 20:15:22 | artom | But I need to track down the exact instance that it's live migrating... | |
| 20:16:02 | sean-k-mooney | so isolate is the only way we have a non vcpu pinned to the cpu_dedicated_set | |
| 20:16:42 | sean-k-mooney | for numa live migration this would cause issues if we dont have renes fix for the cpu pinning | |
| 20:17:16 | artom | Which we don't | |
| 20:17:33 | sean-k-mooney | well we should merge and backport that fix | |
| 20:17:39 | sean-k-mooney | for now can you file a bug for isolate | |
| 20:18:23 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/877773 | |
| 20:18:37 | sean-k-mooney | so ya still open | |
| 20:18:53 | artom | Yeah, I guess he's concentrating on the manila stuff | |
| 20:19:05 | sean-k-mooney | which is fair | |
| 20:20:24 | sean-k-mooney | but that why that is broken | |
| 20:20:47 | sean-k-mooney | it should assumign its not in mergee confict | |
| 20:21:15 | artom | I'll see you in two hours ;) | |
| 20:21:43 | sean-k-mooney | i was going to say i wont be around then.... | |
| 20:21:53 | artom | Which is entirely legitimate | |
| 20:21:56 | sean-k-mooney | but when has that happend when i planed | |
| 20:22:26 | artom | That's between you and your sense of disipline ;) | |
| 20:23:36 | sean-k-mooney | and my stomach | |
| 20:23:50 | sean-k-mooney | for some reasons it has notions that i should feed it at least once a day | |
| 20:24:22 | artom | Pfft, what does it think it is | |