Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-02
17:20:11 sean-k-mooney https://github.com/openstack/nova/blob/4ca795536521f5e3d65d44b4de63c25294a5e0c4/nova/utils.py#L63
17:20:14 gibi allows hacking timing to reproduce the problem
17:20:30 sean-k-mooney we just need to use threading.Thread right
17:22:04 sean-k-mooney or
17:22:06 gibi basically we need to make sure threading.current_thread points to eventlet.getcurrent if called in a patched thread
17:22:19 sean-k-mooney you could use my unix socket impl
17:23:19 sean-k-mooney ctully maybe not
17:23:38 gibi back to testing, I can probably add sleeps to nova running in devstack to simulat the timing but that is not something we can merge as a test
17:24:40 gibi ahh, actually it is haard. as I need timing right in scheduler and in the compute too
17:24:58 gibi the scheduler needs to run the two evac parallel enough to select the same host
17:24:59 sean-k-mooney ya i dint know if https://review.opendev.org/c/openstack/oslo.messaging/+/841892/4/oslo_messaging/tests/functional/notify/test_unix_socket.py will be able to trigger it
17:25:37 gibi I guess this does not use the rabbit driver for notifications
17:25:50 sean-k-mooney its my unix socket one
17:26:11 sean-k-mooney its using either the eventlet or threadpool executor
17:26:11 gibi yeah so with that I can prove that your unix impl is fixed
17:26:26 sean-k-mooney but i dont know if its broken
17:26:43 gibi true :)
17:26:48 sean-k-mooney https://review.opendev.org/c/openstack/oslo.messaging/+/841892/4/oslo_messaging/notify/_impl_unix_socket.py#100
17:27:02 sean-k-mooney so its dynamicaly getting the exectuor and i think the futurist
17:27:13 sean-k-mooney threadpool executor is using threading.Thread internally
17:27:18 sean-k-mooney but its a streach
17:27:39 gibi I have to drop soon so I will sleep on this
17:27:52 gibi but fixing this is probalby an RC blocker
17:28:09 sean-k-mooney i think the best way to proceed is with a syntetic fix
17:28:18 sean-k-mooney import synconise for nova utiles
17:28:23 sean-k-mooney and tweak it until it works
17:28:34 sean-k-mooney using the simpler repoducer you were creating
17:29:05 sean-k-mooney https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L43
17:29:06 gibi we can test the fix with the simple repro I have, what we cannot test that such fix is applied to every places we need it in nova :)
17:29:39 sean-k-mooney well i was thinkign of fixing it in oslo eventurlly
17:29:43 sean-k-mooney just for that lock
17:29:51 gibi that is better
17:30:03 gibi we can assume everything uses lock from oslo
17:30:10 gibi at least within core openstack
17:30:14 sean-k-mooney yep
17:30:16 sean-k-mooney they should
17:30:43 sean-k-mooney so when my unix socket driver is runing with the trehad execurftor it is using threading.Thread
17:30:44 gibi ack that is a way forward
17:30:49 sean-k-mooney but its not currently locking
17:30:53 sean-k-mooney so i dont think that helps
17:31:15 gibi I will summarise what we have in the bug and target the bug to oslo too
17:31:23 sean-k-mooney but i think we could adapt your fucn test into and oslo.synconisation fucn test
17:31:26 sean-k-mooney for the lock
17:32:15 sean-k-mooney sorry oslo.concurrency func test
17:34:29 gibi yes, I think so too
17:34:54 opendevreview Merged openstack/placement master: Fix typo in schema https://review.opendev.org/c/openstack/placement/+/849348
17:39:28 gibi bauzas: I think this is an RC blocker https://bugs.launchpad.net/oslo.concurrency/+bug/1988311 I tagged it but it seem we don't have official zed-rc-potential tag yet
17:45:16 gibi sean-k-mooney: an alternative fix is to roll back to fasteners < 0.15.0
17:45:30 gibi but I'm not sure that is viable in the whole core openstack
17:53:48 melwitt are yall considering doing sean-k-mooney's patch to monkey patch all of our spawn_n with spawn?
17:55:02 melwitt nvm, reading backscroll and saw mention
17:56:04 gibi melwitt: o/ when you found https://github.com/eventlet/eventlet/issues/731#issuecomment-969891721 where the thread.Thread was called?
17:57:11 melwitt gibi: that isn't what I found directly and tbh I need to trace it again in case I made a mistake, but what I found in oslo.messaging is that *it* uses spawn_n in eventlet mode. but I think sean's patch would solve that right?
17:57:48 gibi we need to try probably
17:58:02 gibi it depends how oslo.messaging actaully calls spawn_n
17:58:17 melwitt doing s/spawn_n/spawn/ in our code wouldn't be enough but I think monkey patching the whole thing should be if I'm not missing something. let me see if I can find a link real quick
18:00:57 gibi melwitt: no need to rush, my brain is already toasted :)
18:00:57 melwitt urgh, I remembering now that it was more complicated than that. I'll try to find where I saw thread.Thread. I wish I had put code links in the comment
18:01:37 melwitt hehe ok
18:04:38 melwitt I'll find whatever it was and add comment on the bug
18:07:33 gibi thank you
18:08:01 gibi your original eventlet issue thread was a really good information source already. thank you for that too
23:29:22 melwitt gibi, sean-k-mooney, bauzas: just fyi monday is a holiday in the US and I will be back tuesday
#openstack-nova - 2022-09-03
01:22:54 opendevreview Takashi Natsume proposed openstack/nova master: Update compute rpc version alias for zed https://review.opendev.org/c/openstack/nova/+/855706
01:44:21 opendevreview Takashi Natsume proposed openstack/nova master: doc: mark the max microversion for zed https://review.opendev.org/c/openstack/nova/+/855707
15:27:20 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
15:31:21 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
15:37:14 gibi OK, so from nova perspective only master(zed), and yoga is affected. On xena we use fasteners 0.14.1 which has the original workaround
16:07:48 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
16:08:31 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
20:33:16 opendevreview Elod Illes proposed openstack/os-vif master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/os-vif/+/855742
20:34:14 opendevreview Elod Illes proposed openstack/osc-placement master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/osc-placement/+/855745
20:47:14 opendevreview Elod Illes proposed openstack/python-novaclient master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/python-novaclient/+/855785
#openstack-nova - 2022-09-04
18:31:07 opendevreview Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume 2.93 https://review.opendev.org/c/openstack/python-novaclient/+/827163
#openstack-nova - 2022-09-05
08:01:47 bauzas good morning Nova
08:02:04 bauzas gibi: ack, saw the Critical bug
08:02:16 bauzas we still have 2 weeks for RC1, hopefully should be OK
08:12:06 gibi good morning
08:13:16 gibi yeah, hopefully we can agree on where to fix it, how to fix it, and if we fix it in oslo then hopefully we can release a new olso.concurrency version
08:25:33 frickler FYI placement seems to have issues with the latest oslo.db release https://zuul.opendev.org/t/openstack/build/d7bcb42a3c4a466cac375badba13b18b not that it will likely matter much with all the projects that are completely broken by it
08:45:34 sean-k-mooney[m] frickler: thats just a deprecation warning not an actual failure. placemient is mostly sqlachemey 2.0 compatiable already
08:46:03 sean-k-mooney[m] looks like we have missed one change but that should not be hard to adress
08:46:22 gibi frickler, sean-k-mooney[m]: I'm on it
08:47:02 frickler sean-k-mooney[m]: yes, just mentioned it because if would block that requirements patch. certainly not a big thing compared to the other blockers there
08:47:12 gibi we intended to store dicts in a caches but we stored row objects instead
08:47:12 frickler s/if/it/
08:48:30 sean-k-mooney[m] frickler: well as it stands its just breaking the test as we treat warnings as errors. i doubt its breaking placment at all
08:48:45 sean-k-mooney[m] but our cache is not working correctly so thats a latent bug
08:54:41 sean-k-mooney[m] gibi https://github.com/openstack/placement/commit/c68d472dca6619055579831ad5464042f745557a i guess we missed this usecase
08:55:09 sean-k-mooney[m] the test is useing the dict interface instead of the field interface
08:55:58 gibi yeah, good point, we can store row objects in our rc caches if we access the columns by attribute acces and not by dict access
08:56:04 gibi let me check that it works
08:56:10 gibi as it provides an easier solutions
08:56:57 sean-k-mooney[m] https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/objects/resource_class.py#L220 we are tryign to use _mappings to convert them to dict on get all
08:57:23 sean-k-mooney[m] well not to convert the via a dict to a resouceclass which should be fine
08:58:04 gibi here we have Row objects in the caches https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L155
08:59:05 gibi which is actually fine
08:59:11 gibi as we use attribute access everywhere
08:59:20 gibi except in that one func test that now fails
09:00:02 gibi hm, no

Earlier   Later