Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-02-03
16:07:42 dansmith I haven't looked at either in detail, and not at all for months so I don't even know, just saying
16:07:51 melwitt dansmith: that was my main concern heh. yeah that makes sense
16:13:15 dansmith melwitt: yeah, because I think host is the most useful of these modes so I definitely don't want to hold it hostage
16:13:56 melwitt ++
16:38:26 raineszm sean-k-mooney: melwitt: sorry about that. I know you're looking at the issue on the Nova side, but I put up a patch to set the default behavior of disk_geometry to unset https://review.opendev.org/c/openstack/cinder/+/975540
16:54:22 sean-k-mooney raineszm: i have had ai crete a fix on the nova side while i was on calls ill push it soon and iteragte on it
16:55:09 raineszm sean-k-mooney: ack. ping me if I can help with anything on my end.
16:55:42 opendevreview sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542
16:55:59 sean-k-mooney raineszm: its failing 100% on the nova-next job so if ^ passes i think we are good
16:56:50 sean-k-mooney ill see if i can pull that into a devstack env and test it too seperately
17:07:58 sean-k-mooney that actully pretty good form a 1 shot promt https://paste.opendev.org/show/bJ8ytZ1Kb6LMmv9CCLe0/ ia sed for 2 commit but it was over egere and did it in 1 but ignoreing that it created a resonable plan and created a potially viable fix
17:13:04 melwitt thanks raineszm and sean-k-mooney. I'm sending a heads up email to openstack-discuss@
17:13:48 sean-k-mooney oh ya thats a good idea to avoid rechecks
19:34:55 sean-k-mooney melwitt: https://review.opendev.org/c/openstack/nova/+/975542 passed nova-next if you have time to review
19:36:43 melwitt sean-k-mooney: will look, thanks
19:37:44 sean-k-mooney im going to finish for today shortly so if you or others want to update it go ahead otherwise ill take a look at any comments tomorrow
20:15:21 opendevreview Merged openstack/nova stable/2025.1: libvirt: Get info with abs path, rebase with rel path https://review.opendev.org/c/openstack/nova/+/958674
22:07:34 opendevreview sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542
22:15:32 opendevreview sean mooney proposed openstack/nova master: Fix blockio generation for LUN volumes https://review.opendev.org/c/openstack/nova/+/975542
22:17:34 sean-k-mooney melwitt: i think ^ should now verify the regression and fix the issue. im going to head off again but it looks like the libvirt fixutre was not preserving the blockio elements
#openstack-nova - 2026-02-04
02:00:41 opendevreview Ghanshyam proposed openstack/nova master: Add manager graceful shutdown timeout and wait https://review.opendev.org/c/openstack/nova/+/975586
02:01:43 gmaan bauzas: this is ready, gate is failed due to known broken nova-next job otherwise it pass the gate https://review.opendev.org/c/openstack/nova/+/975242/4
02:02:29 opendevreview Ghanshyam proposed openstack/nova master: Add manager graceful shutdown timeout and wait https://review.opendev.org/c/openstack/nova/+/975586
02:18:46 melwitt sean-k-mooney: thanks, +2
04:16:11 opendevreview Ghanshyam proposed openstack/nova master: WIP: Use 2nd RPC server in compute operations https://review.opendev.org/c/openstack/nova/+/975588
08:06:36 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: refactor memory encryption trait logic for extensiblity https://review.opendev.org/c/openstack/nova/+/967971
08:06:36 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: stop using _get_mem_encryption_config() for SEV checks https://review.opendev.org/c/openstack/nova/+/967970
08:06:37 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: refactor _guest_configure_mem_encryption() for extensibility https://review.opendev.org/c/openstack/nova/+/967973
08:06:37 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: make RP creation independent of specific encryption models https://review.opendev.org/c/openstack/nova/+/967972
08:06:38 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: introduce a check between mem_encryption and locked_memory https://review.opendev.org/c/openstack/nova/+/971300
08:06:38 opendevreview Taketani Ryo proposed openstack/nova master: mem-enc: adjust requirement checks for mem_encryption guests https://review.opendev.org/c/openstack/nova/+/967974
10:10:39 sean-k-mooney gmaan: gibi speakign of the nova-next jobs if i can get a second core to review https://review.opendev.org/c/openstack/nova/+/975542 that will fix it
10:19:49 sean-k-mooney actully bauzas ^ if your around you might have time to take a look also
10:20:06 bauzas sure, I saw the gate failing
10:25:54 sean-k-mooney its basically the same issue we had with the disk serial for device_type=lun in lun mode the phsical aspect fo the backing lun like the serial and in this case sector size cannot be overriden by qemu/libvirt the are presented as is form the storage array to the vm so the fix is to just not set those parmaters in the xml for lun type volumes
10:26:04 bauzas yup, I reviewed it
10:26:15 bauzas thanks for amending the fixture, as I said
10:26:51 sean-k-mooney i spoke to the virt folks about it yesterday and they have created a rhel/libvirt bug to actully have libvirt validate this in the future instead of having ti fail at the qemu level
10:27:40 sean-k-mooney bauzas ++ thanks once thats merged ill reply to mel's email to let folk know its safe to recheck
11:15:27 opendevreview Balazs Gibizer proposed openstack/nova master: Remove spawn_after https://review.opendev.org/c/openstack/nova/+/975396
11:15:27 opendevreview Balazs Gibizer proposed openstack/nova master: Use an executor to delay STOPPED events https://review.opendev.org/c/openstack/nova/+/974445
11:15:28 opendevreview Balazs Gibizer proposed openstack/nova master: Fix full executor warning on noname executor https://review.opendev.org/c/openstack/nova/+/975172
11:15:28 opendevreview Balazs Gibizer proposed openstack/nova master: Cleanup libvirt driver at service stop https://review.opendev.org/c/openstack/nova/+/975128
11:15:29 opendevreview Balazs Gibizer proposed openstack/nova master: Run nova-compute in native threading mode https://review.opendev.org/c/openstack/nova/+/965467
11:15:55 gibi bauzas: ^^ removed the print() call
12:32:46 opendevreview Balazs Gibizer proposed openstack/nova master: DNM: Troubleshoot deadlocks in threading mode https://review.opendev.org/c/openstack/nova/+/975515
13:27:51 opendevreview sean mooney proposed openstack/nova-specs master: add pci passthrough groups spec for 2026.2 https://review.opendev.org/c/openstack/nova-specs/+/975635
13:36:42 opendevreview sean mooney proposed openstack/nova-specs master: add pci passthrough groups spec for 2026.2 https://review.opendev.org/c/openstack/nova-specs/+/975635
16:04:08 opendevreview Max proposed openstack/nova master: fix: device_by_alias should respect config type https://review.opendev.org/c/openstack/nova/+/975651
16:14:35 sean-k-mooney gibi: ya so that was the orginal comemnt on the delay behavior. which is being adressed by https://review.opendev.org/c/openstack/nova/+/974445/14 so ill review it properly after i grab somethign to drink but we can likely proceed.
16:14:58 sean-k-mooney note i need to recheck the gate fix again because it had a network timeout downloaing pip packages form teh mirror
16:39:58 sean-k-mooney gibi: i was thinking of https://review.opendev.org/c/openstack/nova/+/922497 and https://review.opendev.org/c/openstack/nova/+/905287/9 which on master is now https://github.com/openstack/nova/blob/master/nova/image/glance.py#L580-L586 so it looks like in threaded mode we jsut execute it inline and block on the upload
16:40:40 sean-k-mooney so unless we move snapshots to the long runting thread pool that will tie up an rpc thread for the entirty fo the galnce snapshot upload
16:42:33 sean-k-mooney gibi: when you took over the eventlet work you didnt add a dedicated io thread pool but rather a defult thread pool but we didnt update that code to dispatch the upload to the thread pool.
17:18:50 gibi sean-k-mooney: thanks for the pointers. I don't see how that io executor helps, the usage in RBD is spawning a task to the executor and immediately blocking on waiting for the result. Logically that is the same as executing the function on the caller thread in my eyes (in native threading)
17:19:14 gibi https://review.opendev.org/c/openstack/nova/+/917962/8/nova/storage/rbd_utils.py#56
17:19:46 sean-k-mooney i wrote it before dan told me that the rbd client was doing that internally
17:20:04 sean-k-mooney so its not useful for that
17:20:44 gibi ack
17:20:53 sean-k-mooney so the up shot is we have 1 less executor then i tought we had (i.e. we dont have a dedicated io executor for long running file and api calls
17:21:01 sean-k-mooney )
17:22:25 sean-k-mooney it was orgilly suggestign puting the neutron netrokign call on that but i guess the only thing that impoant to consider is the _upload_data function
17:22:40 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/image/glance.py#L580-L586
17:22:42 gibi OK, I will check the _upload_data codepath
17:23:09 sean-k-mooney im not sure we want to block the RPC thread with doing the file upload for snapshots
17:24:04 sean-k-mooney but as dan also pointed out unless we have a futre we can pass in or simialr i dont know hwe we are going to deal with returning the image uuid that is created form this call
17:24:45 sean-k-mooney when you do openstack server image create we need to return teh glance image id for the snapshot form the api if i recall correctly
17:24:49 gibi yepp, I need to think about all these.
17:26:05 sean-k-mooney doign it inline is fine if we move thie operation entrily to teh long runing thread pool but th ereply path is still an unknow i guess
17:26:26 gibi sean-k-mooney: food for thought. How do you feel about https://github.com/openstack/futurist/blob/master/futurist/_futures.py#L232 as a way out of deadlocks due to limited executor sizes?
17:26:37 gibi DynamicThreadPoolExecutor
17:27:07 sean-k-mooney i have not looked at that before but i was debeating about creatin a new executor propsosla that handeled prities canclation and delayed exection
17:27:53 sean-k-mooney gibi: but i dont feel terible about one that cna grow as a way out
17:27:53 gibi for the delayed execution I have this WIP in futurist https://review.opendev.org/c/openstack/futurist/+/975508
17:28:39 gibi yeah we still need to limit our parallel work due to IO and overall memory usage but at least the deadlock becomes something else :)
17:28:52 sean-k-mooney so the lib that watcher and zuul uses has lots of interest feature but the reason im not really suggesting that at the morment is that lib is 4? yes int a v4 rewirte which break backward compatiblity
17:29:02 sean-k-mooney *4 years?
17:30:27 sean-k-mooney gibi: ill try and rominate on it. im not a huge fan of async io. i dont hate it but i was also wonderign if we could/should consider limited use fo an asynio executor
17:32:05 sean-k-mooney if we were to use the DynamicThreadPoolExecutor
17:32:33 sean-k-mooney are you suggesting we woudl not set a max workers or set it relitivly high with the expcation taht in reality it will stay small?
17:33:12 gibi I have to read on what the Dynamic supports regarding sizing but I see that it can grow and shrink
17:33:38 sean-k-mooney """ If threads are not created often in your application, the number of idle
17:33:40 sean-k-mooney threads may stay high for a long time. To avoid it, you can call
17:33:42 sean-k-mooney :py:meth:`.maintain` periodically to keep the number of threads within
17:33:44 sean-k-mooney the thresholds."""
17:34:07 sean-k-mooney we might need to do that ^
17:35:32 sean-k-mooney gibi: i dont hate that idea if im honest. but its proably the rpc pool in oslo service that woudl be the most benifical to use that with right?
17:36:02 sean-k-mooney sorry in oslo.messaging
17:36:56 gibi yeah that too :)
17:37:24 sean-k-mooney i know we can set an executor type when we create the rpc servier im wondering if we can just provide an rpc instance
17:37:30 sean-k-mooney *executor instance
17:38:31 sean-k-mooney ya its just the name https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/rpc/server.py#L226-L240
17:41:15 sean-k-mooney so we woudl have ot update https://github.com/openstack/oslo.messaging/blob/master/pyproject.toml#L53
17:41:24 sean-k-mooney or add a new value
17:41:55 sean-k-mooney might be worth suggeting that to the oslo folks and see what they think
18:03:16 dansmith I'm not sure how the dynamic one really helps
18:03:43 dansmith you still have to set your max to the upper limit of what you're willing tolerate.. it uses less memory in slow times, which is a benefit, but it's still the same problem when you hit the limit
18:06:50 gmaan yeah, it does not solve the limit issue. I think we should be gradually learning about those limit issue (after solving the known one) and keep improving the defaults limit or better separate executor if any specific operations causing problem
18:36:20 gibi https://etherpad.opendev.org/p/eventlet-executor-semaphore-limit this is my brain dump looking at the long running task executor possibilities
18:36:47 gibi we have a list of pros and cons to choose from

Earlier   Later