Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-08-29
13:43:00 amorin hey team, the call to attach a volume on nova is done through rpc. During this, the wsgi worker keep the connection open (it's a synchronous call). Do you know why it's a sync call and not an async one?
13:43:10 amorin see here https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5127
13:43:19 amorin and there https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5150
13:45:58 dansmith amorin: probably so that we delete the BDM if the attach fails
13:47:54 amorin hum, damn
13:49:18 amorin what if we send the attach action to the compute, answer to the API request, and then delegate all the hard work to conductor / compute?
13:54:31 dansmith I mean, anything is possible, but this used to be a situation where you got the device name back when you attached, which is not possible otherwise
13:54:47 dansmith why are you asking? I'm guessing you've got attach requests stacked up or something?
13:54:54 amorin exactly :)
13:54:59 amorin exhausting my apache2 workers
13:55:07 amorin and finally nova was answering 503
13:55:20 amorin due to the fact that one of my ceph cluster was not healthy
13:55:37 amorin I am trying to figure out a way to avoid such behavior
13:56:39 dansmith in that case the attach is going to fail regardless right?
13:56:57 dansmith is the problem that compute gets totally hung on ceph, or that it waits too long for ceph?
13:57:02 amorin yes, it fails after the RPC timeout
13:57:29 dansmith because part of the eventlet stuff is hopefully going to make us less stupid about talking to (and specifically timing out on) ceph in the compute node
13:57:46 dansmith so having a short recoverable timeout there would help your situation I think, yeah?
13:58:55 amorin I think so
13:59:00 amorin something under 10s
13:59:13 amorin at least that would help
14:00:20 dansmith gibi: are you seeing this? ^
14:01:32 amorin otherwize, I was thinking to have dedicated apache workers for attach volume requests, so at least 503 answers are given only to people trying to attach, but other calls succeed, do you know if some other operators are already doing this?
14:01:35 dansmith amorin: speaking of that in general, are you positioned to provide some pre-prod feedback for us on 2025.2 with scheduler and api in non-eventlet mode? we really need some real-world testing of those as we work on the other services
14:02:03 gibi dansmith: not hanging on ceph for ever, sure. :)
14:02:03 dansmith amorin: no, I'm not sure, but that's definitely something I'd consider yeah
14:02:11 dansmith gibi: :P
14:03:12 amorin dansmith: about preprod, maybe I can talk to the team about this
14:04:23 dansmith amorin: that would be super helpful (hopefully for both of us :) )
14:06:53 gibi amorin: if you hit some walls with non-eventlet services then feel free to ping me as well with the issue I was the one that commites most of the crimes there
14:07:23 dansmith amorin: unless everything goes well, in which case it was a team effort :D
14:08:07 jkulik attachments and synchronous calls reminds me of the bug report: https://bugs.launchpad.net/nova/+bug/1930406
14:08:13 amorin ack gibi
14:08:22 gibi dansmith: btw, just verified that with the futurist fix https://review.opendev.org/c/openstack/futurist/+/958689 aggregate image caching works well in threding mode now. The futurist fix will not be in Flamingo due to req freeze, but at least we can land nova-conductor change early in G
14:08:52 dansmith gibi: excellent
14:08:58 gibi dansmith: yepp yepp, failure is on my, success is on the team :)
14:09:06 gibi s/on my/on me/
14:09:12 amorin jkulik: sounds like my exact issue
14:09:40 dansmith jkulik: yep
14:09:58 jkulik maybe we can get that mentioned spec back on track? :D
14:11:07 jkulik since eventlet will go away, we also need a solution for this problem. currently, we're still on the eventlet-based webserver for only this reason :/
14:11:19 dansmith we could just make this an async operation in a later microversion
14:11:41 amorin +1
14:11:42 dansmith easiest would be to delegate to conductor and keep compute the same
14:11:53 dansmith that's the least upgrade-impact too
14:12:02 jkulik > simply an approximation by most virt drivers as their underlying hypervisors cannot guarantee how the device will eventually be presented.
14:12:17 jkulik my understanding was, that we cannot guarantee the device name anyways
14:12:19 dansmith but it might require blocking earlier microversions to prevent people from triggering it
14:12:25 jkulik so the whole call is ... senseless
14:12:35 dansmith jkulik: on libvirt it's fairly pointless, but it was not on other hypervisors
14:12:52 jkulik I can say it's also pointless on vmware
14:15:03 dansmith gibi: so maybe we should make this under the eventlet removal umbrella.. moving this to an async task on conductor
14:15:19 jkulik https://github.com/sapcc/nova/commit/e4e8e6276ab0324704f31d6a3cc4e47f061d25ca this is how we currently try to get relieve on the wait time for the lock, FYI
14:15:27 dansmith the spec (as skimmed) looks far more complex than it needs to be if we just kick this to conductor in a newer microversion
14:15:39 gibi dansmith: maybe. I'm a bit affraid of increasing the scope. But I understand that it is related
14:16:24 dansmith well, if jkulik is sticking on eventlet because of this that tells me that (a) people might be less likely to try threaded until they're forced and (b) a lot more people will hit this once they move
14:16:54 dansmith and tbh, it's probably worth a security hardening classification since this is sort of DoSable
14:17:22 jkulik I agree to ^
14:17:32 jkulik DoS waiting to happen
14:19:21 dansmith since it's friday maybe I'll at least write this up in a spec rough draft to capture it while it is in my head
14:19:56 jkulik please CC me on that. I'm quite interested in the topic :)
14:20:22 amorin CC me as well :)
14:20:58 dansmith are either of you interested in working on the implementation?
14:21:55 jkulik yes, I can probably do that
14:22:21 amorin not sure I would have a lot of time very soon, but we I probably help (or someone from our team)
14:23:23 dansmith cool
14:23:32 amorin but I try to motivate our management, as this is affecting us quite badly
14:23:37 gibi I can be pulled in as reviewer for sure. If we decide that this should be fixed before the n-cpu gets threading support then I can be pulled in as author as well
14:24:23 gibi dansmith: thank for writing up a spec skeleton. My friday brain is already pretty useless anyhow
14:25:01 dansmith I don't think this is going to be very big and I don't think it needs to be serialized against n-cpu work
14:26:00 gibi I just try to limit the number of things open in parallel waiting for me
14:26:55 dansmith ack
15:10:23 opendevreview Dan Smith proposed openstack/nova-specs master: WIP: Spec for async volume attach https://review.opendev.org/c/openstack/nova-specs/+/958900
15:10:45 dansmith jkulik: amorin ^ .. just rough brain dump of what we talked about
15:11:05 amorin yay!
15:11:50 ratailor sean-k-mooney, gibi could you please review this https://review.opendev.org/q/Ifc20894801f723627726e3c9bed7076144542660 and provide your suggestion on approach here https://review.opendev.org/c/openstack/nova/+/954460
15:12:35 dansmith I'm sure I saw the bug years ago when it was filed, but this is the sort of latent thing I don't want to lose track of as we push for eventlet deprecation,
15:13:03 dansmith where we think "yeah everything works in devstack" but we have lots of smaller reasons like this that prevent people from actually being able to move
15:13:43 dansmith this isn't even the class of things that are highest concern (i.e. general threading problems, deadlocks, etc) but they are just as much of a problem for getting people transitioned
15:19:29 ratailor stephenfin, submitted fix for microversion v2.100 as well https://review.opendev.org/c/openstack/python-openstackclient/+/958829 and backported v2.96 to relevant stable/releases https://review.opendev.org/q/I1e398bb3379fa6443b0a44db76baaf6241a945e7
15:43:53 gmaan stephenfin: perfect, that is really helpful. thanks. ack about targeting for next cycle
19:23:37 gmaan sean-k-mooney: glancer service role fix is ready for review now https://review.opendev.org/q/topic:%22glance-service-api%22
21:12:31 opendevreview Merged openstack/nova master: api: Correct expected errors https://review.opendev.org/c/openstack/nova/+/951640
#openstack-nova - 2025-08-31
07:36:58 opendevreview Takashi Kajinami proposed openstack/nova master: Replace remaining usage of Ubuntu Jammy https://review.opendev.org/c/openstack/nova/+/958989
18:12:47 opendevreview Merged openstack/nova master: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957732
18:13:08 opendevreview Merged openstack/nova master: Update min support for Flamingo https://review.opendev.org/c/openstack/nova/+/944018
#openstack-nova - 2025-09-01
07:17:19 gibi bauzas: when you are up, we have 3 test only (unit test enabling) commits left from eventlet removal series. Each has +2 from Sean. Neither of them is complicated. Could you please take a look. This is the first https://review.opendev.org/c/openstack/nova/+/953475 and the next two are in the same bucket.
07:44:43 bauzas gibi: sure I can take a look
07:53:54 gibi bauzas: thanks!
08:05:50 bauzas gibi: sean-k-mooney: +Wd two of them but I have an open question about when we could have the vote for the job https://review.opendev.org/c/openstack/nova/+/955791
08:12:29 gibi bauzas: I think we can make it voting now. See my answer with data in https://review.opendev.org/c/openstack/nova/+/955791/12#message-22f01d1fcd5abdf549b2dd3e206d1292eebcae8b
08:13:29 bauzas gibi: I understand, let's just give a bit of time, that's what I'm just saying
08:13:50 gibi do you have a limit how many successful run you need before you accept it?
08:14:38 gibi (some of our voting jobs has a lot worse than 50:0 success:pass ratio)
08:14:54 gibi s/success:pass/success:fail/
08:31:07 bauzas sorry was otp
08:32:05 bauzas honestly, I don't have any number, if the job works up to tomorrow, then if so, let's leave it to be voting :)
08:53:16 gibi I don't think there will be much run of it up until tomorrow. But I can do some rechecks
09:06:56 tkajinam I noticed one job still running on Jammy and proposed a change to migrate it to Noble. Can we merge this for 2025.2 ? (because jammy is no longer supported) https://review.opendev.org/c/openstack/nova/+/958989
09:10:15 tkajinam sean-k-mooney, could you review https://review.opendev.org/c/openstack/nova/+/958822 when you have time ? This is a follow-up of SEV-ES support which I'm hoping to include in 2025.2

Earlier   Later