Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-08-29
12:35:42 stephenfin one less thing for you to worry about during reviews 😄
12:37:40 sean-k-mooney gibi: i basiclly want to see if i can replciate this with 2 ovs network with differnt mtu or if it actully need ovs + sriov to see if we can maybe test this in tempest and if it obviously fixes the issue for me locally
12:39:05 sean-k-mooney gibi: i dont curertnly have sriov hardware avaibel to thise this simiply so i need to create 2 devstack vms using igb and set it up for live migration ectra
12:39:12 sean-k-mooney so i wont get to thtat today
12:39:55 opendevreview Takashi Kajinami proposed openstack/nova stable/2025.1: libvirt: Disable VMCoreInfo device for SEV-encrypted instances https://review.opendev.org/c/openstack/nova/+/958874
12:44:52 gibi sean-k-mooney: sync with Arnau from neutron they migh have envs as this was reported from running neutron's whitebox against antelope
13:03:37 sean-k-mooney ack i mostly want to see if we can test this edge case in the 1st party ci going forward via tempest without the sriov depency described in the bug
13:04:11 sean-k-mooney if it can happen anytime you have a vm with 2 networks with diffent mtus for exampel we can easilly add a senario test in neutron to do that
13:05:45 sean-k-mooney like in theory if usign the neutron whitebox tempest plugin found it downstream that same test should have repodcuded it on master right
13:08:58 gibi yepp that theory make sense
13:09:33 gibi there was also recent changes in that repo about the exact same test case so it might pass on master but not on older whitebox versions
13:42:06 dansmith gibi: woohoo, thanks
13:43:00 amorin hey team, the call to attach a volume on nova is done through rpc. During this, the wsgi worker keep the connection open (it's a synchronous call). Do you know why it's a sync call and not an async one?
13:43:10 amorin see here https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5127
13:43:19 amorin and there https://opendev.org/openstack/nova/src/commit/07ab08aa69ae39644c276a0907759175f8936239/nova/compute/api.py#L5150
13:45:58 dansmith amorin: probably so that we delete the BDM if the attach fails
13:47:54 amorin hum, damn
13:49:18 amorin what if we send the attach action to the compute, answer to the API request, and then delegate all the hard work to conductor / compute?
13:54:31 dansmith I mean, anything is possible, but this used to be a situation where you got the device name back when you attached, which is not possible otherwise
13:54:47 dansmith why are you asking? I'm guessing you've got attach requests stacked up or something?
13:54:54 amorin exactly :)
13:54:59 amorin exhausting my apache2 workers
13:55:07 amorin and finally nova was answering 503
13:55:20 amorin due to the fact that one of my ceph cluster was not healthy
13:55:37 amorin I am trying to figure out a way to avoid such behavior
13:56:39 dansmith in that case the attach is going to fail regardless right?
13:56:57 dansmith is the problem that compute gets totally hung on ceph, or that it waits too long for ceph?
13:57:02 amorin yes, it fails after the RPC timeout
13:57:29 dansmith because part of the eventlet stuff is hopefully going to make us less stupid about talking to (and specifically timing out on) ceph in the compute node
13:57:46 dansmith so having a short recoverable timeout there would help your situation I think, yeah?
13:58:55 amorin I think so
13:59:00 amorin something under 10s
13:59:13 amorin at least that would help
14:00:20 dansmith gibi: are you seeing this? ^
14:01:32 amorin otherwize, I was thinking to have dedicated apache workers for attach volume requests, so at least 503 answers are given only to people trying to attach, but other calls succeed, do you know if some other operators are already doing this?
14:01:35 dansmith amorin: speaking of that in general, are you positioned to provide some pre-prod feedback for us on 2025.2 with scheduler and api in non-eventlet mode? we really need some real-world testing of those as we work on the other services
14:02:03 dansmith amorin: no, I'm not sure, but that's definitely something I'd consider yeah
14:02:03 gibi dansmith: not hanging on ceph for ever, sure. :)
14:02:11 dansmith gibi: :P
14:03:12 amorin dansmith: about preprod, maybe I can talk to the team about this
14:04:23 dansmith amorin: that would be super helpful (hopefully for both of us :) )
14:06:53 gibi amorin: if you hit some walls with non-eventlet services then feel free to ping me as well with the issue I was the one that commites most of the crimes there
14:07:23 dansmith amorin: unless everything goes well, in which case it was a team effort :D
14:08:07 jkulik attachments and synchronous calls reminds me of the bug report: https://bugs.launchpad.net/nova/+bug/1930406
14:08:13 amorin ack gibi
14:08:22 gibi dansmith: btw, just verified that with the futurist fix https://review.opendev.org/c/openstack/futurist/+/958689 aggregate image caching works well in threding mode now. The futurist fix will not be in Flamingo due to req freeze, but at least we can land nova-conductor change early in G
14:08:52 dansmith gibi: excellent
14:08:58 gibi dansmith: yepp yepp, failure is on my, success is on the team :)
14:09:06 gibi s/on my/on me/
14:09:12 amorin jkulik: sounds like my exact issue
14:09:40 dansmith jkulik: yep
14:09:58 jkulik maybe we can get that mentioned spec back on track? :D
14:11:07 jkulik since eventlet will go away, we also need a solution for this problem. currently, we're still on the eventlet-based webserver for only this reason :/
14:11:19 dansmith we could just make this an async operation in a later microversion
14:11:41 amorin +1
14:11:42 dansmith easiest would be to delegate to conductor and keep compute the same
14:11:53 dansmith that's the least upgrade-impact too
14:12:02 jkulik > simply an approximation by most virt drivers as their underlying hypervisors cannot guarantee how the device will eventually be presented.
14:12:17 jkulik my understanding was, that we cannot guarantee the device name anyways
14:12:19 dansmith but it might require blocking earlier microversions to prevent people from triggering it
14:12:25 jkulik so the whole call is ... senseless
14:12:35 dansmith jkulik: on libvirt it's fairly pointless, but it was not on other hypervisors
14:12:52 jkulik I can say it's also pointless on vmware
14:15:03 dansmith gibi: so maybe we should make this under the eventlet removal umbrella.. moving this to an async task on conductor
14:15:19 jkulik https://github.com/sapcc/nova/commit/e4e8e6276ab0324704f31d6a3cc4e47f061d25ca this is how we currently try to get relieve on the wait time for the lock, FYI
14:15:27 dansmith the spec (as skimmed) looks far more complex than it needs to be if we just kick this to conductor in a newer microversion
14:15:39 gibi dansmith: maybe. I'm a bit affraid of increasing the scope. But I understand that it is related
14:16:24 dansmith well, if jkulik is sticking on eventlet because of this that tells me that (a) people might be less likely to try threaded until they're forced and (b) a lot more people will hit this once they move
14:16:54 dansmith and tbh, it's probably worth a security hardening classification since this is sort of DoSable
14:17:22 jkulik I agree to ^
14:17:32 jkulik DoS waiting to happen
14:19:21 dansmith since it's friday maybe I'll at least write this up in a spec rough draft to capture it while it is in my head
14:19:56 jkulik please CC me on that. I'm quite interested in the topic :)
14:20:22 amorin CC me as well :)
14:20:58 dansmith are either of you interested in working on the implementation?
14:21:55 jkulik yes, I can probably do that
14:22:21 amorin not sure I would have a lot of time very soon, but we I probably help (or someone from our team)
14:23:23 dansmith cool
14:23:32 amorin but I try to motivate our management, as this is affecting us quite badly
14:23:37 gibi I can be pulled in as reviewer for sure. If we decide that this should be fixed before the n-cpu gets threading support then I can be pulled in as author as well
14:24:23 gibi dansmith: thank for writing up a spec skeleton. My friday brain is already pretty useless anyhow
14:25:01 dansmith I don't think this is going to be very big and I don't think it needs to be serialized against n-cpu work
14:26:00 gibi I just try to limit the number of things open in parallel waiting for me
14:26:55 dansmith ack
15:10:23 opendevreview Dan Smith proposed openstack/nova-specs master: WIP: Spec for async volume attach https://review.opendev.org/c/openstack/nova-specs/+/958900
15:10:45 dansmith jkulik: amorin ^ .. just rough brain dump of what we talked about
15:11:05 amorin yay!
15:11:50 ratailor sean-k-mooney, gibi could you please review this https://review.opendev.org/q/Ifc20894801f723627726e3c9bed7076144542660 and provide your suggestion on approach here https://review.opendev.org/c/openstack/nova/+/954460
15:12:35 dansmith I'm sure I saw the bug years ago when it was filed, but this is the sort of latent thing I don't want to lose track of as we push for eventlet deprecation,
15:13:03 dansmith where we think "yeah everything works in devstack" but we have lots of smaller reasons like this that prevent people from actually being able to move
15:13:43 dansmith this isn't even the class of things that are highest concern (i.e. general threading problems, deadlocks, etc) but they are just as much of a problem for getting people transitioned
15:19:29 ratailor stephenfin, submitted fix for microversion v2.100 as well https://review.opendev.org/c/openstack/python-openstackclient/+/958829 and backported v2.96 to relevant stable/releases https://review.opendev.org/q/I1e398bb3379fa6443b0a44db76baaf6241a945e7
15:43:53 gmaan stephenfin: perfect, that is really helpful. thanks. ack about targeting for next cycle
19:23:37 gmaan sean-k-mooney: glancer service role fix is ready for review now https://review.opendev.org/q/topic:%22glance-service-api%22
21:12:31 opendevreview Merged openstack/nova master: api: Correct expected errors https://review.opendev.org/c/openstack/nova/+/951640
#openstack-nova - 2025-08-31
07:36:58 opendevreview Takashi Kajinami proposed openstack/nova master: Replace remaining usage of Ubuntu Jammy https://review.opendev.org/c/openstack/nova/+/958989
18:12:47 opendevreview Merged openstack/nova master: api: Fix validators for hw:cpu_max_* extra specs https://review.opendev.org/c/openstack/nova/+/957732
18:13:08 opendevreview Merged openstack/nova master: Update min support for Flamingo https://review.opendev.org/c/openstack/nova/+/944018
#openstack-nova - 2025-09-01
07:17:19 gibi bauzas: when you are up, we have 3 test only (unit test enabling) commits left from eventlet removal series. Each has +2 from Sean. Neither of them is complicated. Could you please take a look. This is the first https://review.opendev.org/c/openstack/nova/+/953475 and the next two are in the same bucket.
07:44:43 bauzas gibi: sure I can take a look

Earlier   Later