| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-29 | |||
| 12:14:46 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose remove-os-volumes_boot-api https://review.opendev.org/c/openstack/nova-specs/+/958865 | |
| 12:14:46 | opendevreview | Stephen Finucane proposed openstack/nova master: api: Deprecate os-volumes_boot API https://review.opendev.org/c/openstack/nova/+/958864 | |
| 12:14:57 | gibi | sean-k-mooney: sorry for that. This is not urgent so feel free to punt it for a while | |
| 12:15:57 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose flavor-search-by-name spec https://review.opendev.org/c/openstack/nova-specs/+/958866 | |
| 12:16:22 | sean-k-mooney | im not complaining but im going to spend 15 mins considerign the patch then move on. ill come back to it on monday | |
| 12:17:39 | sean-k-mooney | i think the change they are propsoing is safe to proceed with | |
| 12:18:58 | sean-k-mooney | i am not conviced it will fix all 2-3 of the related cases i am vagely aware of but i dont think it will make any of those harder to fix later | |
| 12:19:08 | opendevreview | Merged openstack/nova-specs master: Create specs directory for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/956166 | |
| 12:19:20 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Disable VMCoreInfo device for SEV-encrypted instances https://review.opendev.org/c/openstack/nova/+/958868 | |
| 12:20:56 | sean-k-mooney | gibi: i feel like it need a bit more testing but im tempeted to pull this into one of my devstacks and try it and maybe jsut take a good enough is better then perfect approch in this case | |
| 12:21:47 | sean-k-mooney | gibi: in other words we can proably proceed with this before rc1 but i want to poke at it a bit before then | |
| 12:22:10 | opendevreview | Stephen Finucane proposed openstack/nova-specs master: Repropose openapi spec for Gazpacho https://review.opendev.org/c/openstack/nova-specs/+/958869 | |
| 12:33:21 | gibi | sean-k-mooney: OK. Thanks | |
| 12:35:22 | stephenfin | gmaan: For next cycle, but you'll probably like https://review.opendev.org/c/openstack/nova/+/958856/ and https://review.opendev.org/c/openstack/nova/+/958858/ | |
| 12:35:42 | stephenfin | one less thing for you to worry about during reviews 😄 | |
| 12:37:40 | sean-k-mooney | gibi: i basiclly want to see if i can replciate this with 2 ovs network with differnt mtu or if it actully need ovs + sriov to see if we can maybe test this in tempest and if it obviously fixes the issue for me locally | |
| 12:39:05 | sean-k-mooney | gibi: i dont curertnly have sriov hardware avaibel to thise this simiply so i need to create 2 devstack vms using igb and set it up for live migration ectra | |
| 12:39:12 | sean-k-mooney | so i wont get to thtat today | |
| 12:39:55 | opendevreview | Takashi Kajinami proposed openstack/nova stable/2025.1: libvirt: Disable VMCoreInfo device for SEV-encrypted instances https://review.opendev.org/c/openstack/nova/+/958874 | |
| 12:44:52 | gibi | sean-k-mooney: sync with Arnau from neutron they migh have envs as this was reported from running neutron's whitebox against antelope | |
| 13:03:37 | sean-k-mooney | ack i mostly want to see if we can test this edge case in the 1st party ci going forward via tempest without the sriov depency described in the bug | |
| 13:04:11 | sean-k-mooney | if it can happen anytime you have a vm with 2 networks with diffent mtus for exampel we can easilly add a senario test in neutron to do that | |
| 13:05:45 | sean-k-mooney | like in theory if usign the neutron whitebox tempest plugin found it downstream that same test should have repodcuded it on master right | |
| 13:08:58 | gibi | yepp that theory make sense | |
| 13:09:33 | gibi | there was also recent changes in that repo about the exact same test case so it might pass on master but not on older whitebox versions | |
| 13:42:06 | dansmith | gibi: woohoo, thanks | |
| 13:43:00 | amorin | hey team, the call to attach a volume on nova is done through rpc. During this, the wsgi worker keep the connection open (it's a synchronous call). Do you know why it's a sync call and not an async one? | |
| 13:43:10 | amorin | see here https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5127 | |
| 13:43:19 | amorin | and there https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5150 | |
| 13:45:58 | dansmith | amorin: probably so that we delete the BDM if the attach fails | |
| 13:47:54 | amorin | hum, damn | |
| 13:49:18 | amorin | what if we send the attach action to the compute, answer to the API request, and then delegate all the hard work to conductor / compute? | |
| 13:54:31 | dansmith | I mean, anything is possible, but this used to be a situation where you got the device name back when you attached, which is not possible otherwise | |
| 13:54:47 | dansmith | why are you asking? I'm guessing you've got attach requests stacked up or something? | |
| 13:54:54 | amorin | exactly :) | |
| 13:54:59 | amorin | exhausting my apache2 workers | |
| 13:55:07 | amorin | and finally nova was answering 503 | |
| 13:55:20 | amorin | due to the fact that one of my ceph cluster was not healthy | |
| 13:55:37 | amorin | I am trying to figure out a way to avoid such behavior | |
| 13:56:39 | dansmith | in that case the attach is going to fail regardless right? | |
| 13:56:57 | dansmith | is the problem that compute gets totally hung on ceph, or that it waits too long for ceph? | |
| 13:57:02 | amorin | yes, it fails after the RPC timeout | |
| 13:57:29 | dansmith | because part of the eventlet stuff is hopefully going to make us less stupid about talking to (and specifically timing out on) ceph in the compute node | |
| 13:57:46 | dansmith | so having a short recoverable timeout there would help your situation I think, yeah? | |
| 13:58:55 | amorin | I think so | |
| 13:59:00 | amorin | something under 10s | |
| 13:59:13 | amorin | at least that would help | |
| 14:00:20 | dansmith | gibi: are you seeing this? ^ | |
| 14:01:32 | amorin | otherwize, I was thinking to have dedicated apache workers for attach volume requests, so at least 503 answers are given only to people trying to attach, but other calls succeed, do you know if some other operators are already doing this? | |
| 14:01:35 | dansmith | amorin: speaking of that in general, are you positioned to provide some pre-prod feedback for us on 2025.2 with scheduler and api in non-eventlet mode? we really need some real-world testing of those as we work on the other services | |
| 14:02:03 | dansmith | amorin: no, I'm not sure, but that's definitely something I'd consider yeah | |
| 14:02:03 | gibi | dansmith: not hanging on ceph for ever, sure. :) | |
| 14:02:11 | dansmith | gibi: :P | |
| 14:03:12 | amorin | dansmith: about preprod, maybe I can talk to the team about this | |
| 14:04:23 | dansmith | amorin: that would be super helpful (hopefully for both of us :) ) | |
| 14:06:53 | gibi | amorin: if you hit some walls with non-eventlet services then feel free to ping me as well with the issue I was the one that commites most of the crimes there | |
| 14:07:23 | dansmith | amorin: unless everything goes well, in which case it was a team effort :D | |
| 14:08:07 | jkulik | attachments and synchronous calls reminds me of the bug report: https://bugs.launchpad.net/nova/+bug/1930406 | |
| 14:08:13 | amorin | ack gibi | |
| 14:08:22 | gibi | dansmith: btw, just verified that with the futurist fix https://review.opendev.org/c/openstack/futurist/+/958689 aggregate image caching works well in threding mode now. The futurist fix will not be in Flamingo due to req freeze, but at least we can land nova-conductor change early in G | |
| 14:08:52 | dansmith | gibi: excellent | |
| 14:08:58 | gibi | dansmith: yepp yepp, failure is on my, success is on the team :) | |
| 14:09:06 | gibi | s/on my/on me/ | |
| 14:09:12 | amorin | jkulik: sounds like my exact issue | |
| 14:09:40 | dansmith | jkulik: yep | |
| 14:09:58 | jkulik | maybe we can get that mentioned spec back on track? :D | |
| 14:11:07 | jkulik | since eventlet will go away, we also need a solution for this problem. currently, we're still on the eventlet-based webserver for only this reason :/ | |
| 14:11:19 | dansmith | we could just make this an async operation in a later microversion | |
| 14:11:41 | amorin | +1 | |
| 14:11:42 | dansmith | easiest would be to delegate to conductor and keep compute the same | |
| 14:11:53 | dansmith | that's the least upgrade-impact too | |
| 14:12:02 | jkulik | > simply an approximation by most virt drivers as their underlying hypervisors cannot guarantee how the device will eventually be presented. | |
| 14:12:17 | jkulik | my understanding was, that we cannot guarantee the device name anyways | |
| 14:12:19 | dansmith | but it might require blocking earlier microversions to prevent people from triggering it | |
| 14:12:25 | jkulik | so the whole call is ... senseless | |
| 14:12:35 | dansmith | jkulik: on libvirt it's fairly pointless, but it was not on other hypervisors | |
| 14:12:52 | jkulik | I can say it's also pointless on vmware | |
| 14:15:03 | dansmith | gibi: so maybe we should make this under the eventlet removal umbrella.. moving this to an async task on conductor | |
| 14:15:19 | jkulik | https://github.com/sapcc/nova/commit/e4e8e6276ab0324704f31d6a3cc4e47f061d25ca this is how we currently try to get relieve on the wait time for the lock, FYI | |
| 14:15:27 | dansmith | the spec (as skimmed) looks far more complex than it needs to be if we just kick this to conductor in a newer microversion | |
| 14:15:39 | gibi | dansmith: maybe. I'm a bit affraid of increasing the scope. But I understand that it is related | |
| 14:16:24 | dansmith | well, if jkulik is sticking on eventlet because of this that tells me that (a) people might be less likely to try threaded until they're forced and (b) a lot more people will hit this once they move | |
| 14:16:54 | dansmith | and tbh, it's probably worth a security hardening classification since this is sort of DoSable | |
| 14:17:22 | jkulik | I agree to ^ | |
| 14:17:32 | jkulik | DoS waiting to happen | |
| 14:19:21 | dansmith | since it's friday maybe I'll at least write this up in a spec rough draft to capture it while it is in my head | |
| 14:19:56 | jkulik | please CC me on that. I'm quite interested in the topic :) | |
| 14:20:22 | amorin | CC me as well :) | |
| 14:20:58 | dansmith | are either of you interested in working on the implementation? | |
| 14:21:55 | jkulik | yes, I can probably do that | |
| 14:22:21 | amorin | not sure I would have a lot of time very soon, but we I probably help (or someone from our team) | |
| 14:23:23 | dansmith | cool | |
| 14:23:32 | amorin | but I try to motivate our management, as this is affecting us quite badly | |
| 14:23:37 | gibi | I can be pulled in as reviewer for sure. If we decide that this should be fixed before the n-cpu gets threading support then I can be pulled in as author as well | |
| 14:24:23 | gibi | dansmith: thank for writing up a spec skeleton. My friday brain is already pretty useless anyhow | |
| 14:25:01 | dansmith | I don't think this is going to be very big and I don't think it needs to be serialized against n-cpu work | |
| 14:26:00 | gibi | I just try to limit the number of things open in parallel waiting for me | |
| 14:26:55 | dansmith | ack | |
| 15:10:23 | opendevreview | Dan Smith proposed openstack/nova-specs master: WIP: Spec for async volume attach https://review.opendev.org/c/openstack/nova-specs/+/958900 | |
| 15:10:45 | dansmith | jkulik: amorin ^ .. just rough brain dump of what we talked about | |