Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-02
17:27:52 gibi but fixing this is probalby an RC blocker
17:28:09 sean-k-mooney i think the best way to proceed is with a syntetic fix
17:28:18 sean-k-mooney import synconise for nova utiles
17:28:23 sean-k-mooney and tweak it until it works
17:28:34 sean-k-mooney using the simpler repoducer you were creating
17:29:05 sean-k-mooney https://github.com/openstack/futurist/blob/master/futurist/_thread.py#L43
17:29:06 gibi we can test the fix with the simple repro I have, what we cannot test that such fix is applied to every places we need it in nova :)
17:29:39 sean-k-mooney well i was thinkign of fixing it in oslo eventurlly
17:29:43 sean-k-mooney just for that lock
17:29:51 gibi that is better
17:30:03 gibi we can assume everything uses lock from oslo
17:30:10 gibi at least within core openstack
17:30:14 sean-k-mooney yep
17:30:16 sean-k-mooney they should
17:30:43 sean-k-mooney so when my unix socket driver is runing with the trehad execurftor it is using threading.Thread
17:30:44 gibi ack that is a way forward
17:30:49 sean-k-mooney but its not currently locking
17:30:53 sean-k-mooney so i dont think that helps
17:31:15 gibi I will summarise what we have in the bug and target the bug to oslo too
17:31:23 sean-k-mooney but i think we could adapt your fucn test into and oslo.synconisation fucn test
17:31:26 sean-k-mooney for the lock
17:32:15 sean-k-mooney sorry oslo.concurrency func test
17:34:29 gibi yes, I think so too
17:34:54 opendevreview Merged openstack/placement master: Fix typo in schema https://review.opendev.org/c/openstack/placement/+/849348
17:39:28 gibi bauzas: I think this is an RC blocker https://bugs.launchpad.net/oslo.concurrency/+bug/1988311 I tagged it but it seem we don't have official zed-rc-potential tag yet
17:45:16 gibi sean-k-mooney: an alternative fix is to roll back to fasteners < 0.15.0
17:45:30 gibi but I'm not sure that is viable in the whole core openstack
17:53:48 melwitt are yall considering doing sean-k-mooney's patch to monkey patch all of our spawn_n with spawn?
17:55:02 melwitt nvm, reading backscroll and saw mention
17:56:04 gibi melwitt: o/ when you found https://github.com/eventlet/eventlet/issues/731#issuecomment-969891721 where the thread.Thread was called?
17:57:11 melwitt gibi: that isn't what I found directly and tbh I need to trace it again in case I made a mistake, but what I found in oslo.messaging is that *it* uses spawn_n in eventlet mode. but I think sean's patch would solve that right?
17:57:48 gibi we need to try probably
17:58:02 gibi it depends how oslo.messaging actaully calls spawn_n
17:58:17 melwitt doing s/spawn_n/spawn/ in our code wouldn't be enough but I think monkey patching the whole thing should be if I'm not missing something. let me see if I can find a link real quick
18:00:57 melwitt urgh, I remembering now that it was more complicated than that. I'll try to find where I saw thread.Thread. I wish I had put code links in the comment
18:00:57 gibi melwitt: no need to rush, my brain is already toasted :)
18:01:37 melwitt hehe ok
18:04:38 melwitt I'll find whatever it was and add comment on the bug
18:07:33 gibi thank you
18:08:01 gibi your original eventlet issue thread was a really good information source already. thank you for that too
23:29:22 melwitt gibi, sean-k-mooney, bauzas: just fyi monday is a holiday in the US and I will be back tuesday
#openstack-nova - 2022-09-03
01:22:54 opendevreview Takashi Natsume proposed openstack/nova master: Update compute rpc version alias for zed https://review.opendev.org/c/openstack/nova/+/855706
01:44:21 opendevreview Takashi Natsume proposed openstack/nova master: doc: mark the max microversion for zed https://review.opendev.org/c/openstack/nova/+/855707
15:27:20 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
15:31:21 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
15:37:14 gibi OK, so from nova perspective only master(zed), and yoga is affected. On xena we use fasteners 0.14.1 which has the original workaround
16:07:48 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
16:08:31 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
20:33:16 opendevreview Elod Illes proposed openstack/os-vif master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/os-vif/+/855742
20:34:14 opendevreview Elod Illes proposed openstack/osc-placement master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/osc-placement/+/855745
20:47:14 opendevreview Elod Illes proposed openstack/python-novaclient master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/python-novaclient/+/855785
#openstack-nova - 2022-09-04
18:31:07 opendevreview Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume 2.93 https://review.opendev.org/c/openstack/python-novaclient/+/827163
#openstack-nova - 2022-09-05
08:01:47 bauzas good morning Nova
08:02:04 bauzas gibi: ack, saw the Critical bug
08:02:16 bauzas we still have 2 weeks for RC1, hopefully should be OK
08:12:06 gibi good morning
08:13:16 gibi yeah, hopefully we can agree on where to fix it, how to fix it, and if we fix it in oslo then hopefully we can release a new olso.concurrency version
08:25:33 frickler FYI placement seems to have issues with the latest oslo.db release https://zuul.opendev.org/t/openstack/build/d7bcb42a3c4a466cac375badba13b18b not that it will likely matter much with all the projects that are completely broken by it
08:45:34 sean-k-mooney[m] frickler: thats just a deprecation warning not an actual failure. placemient is mostly sqlachemey 2.0 compatiable already
08:46:03 sean-k-mooney[m] looks like we have missed one change but that should not be hard to adress
08:46:22 gibi frickler, sean-k-mooney[m]: I'm on it
08:47:02 frickler sean-k-mooney[m]: yes, just mentioned it because if would block that requirements patch. certainly not a big thing compared to the other blockers there
08:47:12 frickler s/if/it/
08:47:12 gibi we intended to store dicts in a caches but we stored row objects instead
08:48:30 sean-k-mooney[m] frickler: well as it stands its just breaking the test as we treat warnings as errors. i doubt its breaking placment at all
08:48:45 sean-k-mooney[m] but our cache is not working correctly so thats a latent bug
08:54:41 sean-k-mooney[m] gibi https://github.com/openstack/placement/commit/c68d472dca6619055579831ad5464042f745557a i guess we missed this usecase
08:55:09 sean-k-mooney[m] the test is useing the dict interface instead of the field interface
08:55:58 gibi yeah, good point, we can store row objects in our rc caches if we access the columns by attribute acces and not by dict access
08:56:04 gibi let me check that it works
08:56:10 gibi as it provides an easier solutions
08:56:57 sean-k-mooney[m] https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/objects/resource_class.py#L220 we are tryign to use _mappings to convert them to dict on get all
08:57:23 sean-k-mooney[m] well not to convert the via a dict to a resouceclass which should be fine
08:58:04 gibi here we have Row objects in the caches https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L155
08:59:05 gibi which is actually fine
08:59:11 gibi as we use attribute access everywhere
08:59:20 gibi except in that one func test that now fails
09:00:02 gibi hm, no
09:00:09 gibi that caches is broken
09:00:14 gibi as sometimes we store dicts
09:00:19 gibi sometimes we store Row objects
09:00:37 gibi https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L163
09:01:16 sean-k-mooney[m] https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/attribute_cache.py#L148-L151
09:01:54 sean-k-mooney[m] we store db rows when we load it form the db
09:02:05 gibi yep so this is a type mess
09:03:05 sean-k-mooney[m] ya stephen was complainging about this in a differnt patch
09:04:55 sean-k-mooney[m] so the all_cache
09:05:00 sean-k-mooney[m] currently had the row
09:05:09 sean-k-mooney[m] but that could jsut be a new tuple firht
09:05:47 sean-k-mooney[m] https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/attribute_cache.py#L151
09:06:38 sean-k-mooney[m] self._all_cache = {r[1]: r for r in res} -> self._all_cache = {r[1]: (r[0], r[1]) for r in res}
09:06:46 gibi so while 'self._all_cache = {r[1]: r._mapping for r in res}' fixed the currently failing test case it breaks a bunch of gabbi tests
09:07:31 sean-k-mooney[m] what elese is in that row beyond the id and string
09:08:05 gibi updated_at and created_at
09:08:31 gibi it is in the select above the fetachall
09:08:46 sean-k-mooney[m] ah base is the time stamped mixin
09:09:35 sean-k-mooney[m] also the value of the cache is ment to be a dict not a tuple so my version is incorrect
09:10:14 gibi yeah I have to figure out while the gabbi tests fails with my change
09:10:32 gibi technically Row._mapping is not a dict just a dict like object
09:10:36 gibi so it might be a problem

Earlier   Later