Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-02
17:55:02 melwitt nvm, reading backscroll and saw mention
17:56:04 gibi melwitt: o/ when you found https://github.com/eventlet/eventlet/issues/731#issuecomment-969891721 where the thread.Thread was called?
17:57:11 melwitt gibi: that isn't what I found directly and tbh I need to trace it again in case I made a mistake, but what I found in oslo.messaging is that *it* uses spawn_n in eventlet mode. but I think sean's patch would solve that right?
17:57:48 gibi we need to try probably
17:58:02 gibi it depends how oslo.messaging actaully calls spawn_n
17:58:17 melwitt doing s/spawn_n/spawn/ in our code wouldn't be enough but I think monkey patching the whole thing should be if I'm not missing something. let me see if I can find a link real quick
18:00:57 gibi melwitt: no need to rush, my brain is already toasted :)
18:00:57 melwitt urgh, I remembering now that it was more complicated than that. I'll try to find where I saw thread.Thread. I wish I had put code links in the comment
18:01:37 melwitt hehe ok
18:04:38 melwitt I'll find whatever it was and add comment on the bug
18:07:33 gibi thank you
18:08:01 gibi your original eventlet issue thread was a really good information source already. thank you for that too
23:29:22 melwitt gibi, sean-k-mooney, bauzas: just fyi monday is a holiday in the US and I will be back tuesday
#openstack-nova - 2022-09-03
01:22:54 opendevreview Takashi Natsume proposed openstack/nova master: Update compute rpc version alias for zed https://review.opendev.org/c/openstack/nova/+/855706
01:44:21 opendevreview Takashi Natsume proposed openstack/nova master: doc: mark the max microversion for zed https://review.opendev.org/c/openstack/nova/+/855707
15:27:20 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
15:31:21 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
15:37:14 gibi OK, so from nova perspective only master(zed), and yoga is affected. On xena we use fasteners 0.14.1 which has the original workaround
16:07:48 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
16:08:31 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
20:33:16 opendevreview Elod Illes proposed openstack/os-vif master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/os-vif/+/855742
20:34:14 opendevreview Elod Illes proposed openstack/osc-placement master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/osc-placement/+/855745
20:47:14 opendevreview Elod Illes proposed openstack/python-novaclient master: DNM: dummy change to test gate health https://review.opendev.org/c/openstack/python-novaclient/+/855785
#openstack-nova - 2022-09-04
18:31:07 opendevreview Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume 2.93 https://review.opendev.org/c/openstack/python-novaclient/+/827163
#openstack-nova - 2022-09-05
08:01:47 bauzas good morning Nova
08:02:04 bauzas gibi: ack, saw the Critical bug
08:02:16 bauzas we still have 2 weeks for RC1, hopefully should be OK
08:12:06 gibi good morning
08:13:16 gibi yeah, hopefully we can agree on where to fix it, how to fix it, and if we fix it in oslo then hopefully we can release a new olso.concurrency version
08:25:33 frickler FYI placement seems to have issues with the latest oslo.db release https://zuul.opendev.org/t/openstack/build/d7bcb42a3c4a466cac375badba13b18b not that it will likely matter much with all the projects that are completely broken by it
08:45:34 sean-k-mooney[m] frickler: thats just a deprecation warning not an actual failure. placemient is mostly sqlachemey 2.0 compatiable already
08:46:03 sean-k-mooney[m] looks like we have missed one change but that should not be hard to adress
08:46:22 gibi frickler, sean-k-mooney[m]: I'm on it
08:47:02 frickler sean-k-mooney[m]: yes, just mentioned it because if would block that requirements patch. certainly not a big thing compared to the other blockers there
08:47:12 gibi we intended to store dicts in a caches but we stored row objects instead
08:47:12 frickler s/if/it/
08:48:30 sean-k-mooney[m] frickler: well as it stands its just breaking the test as we treat warnings as errors. i doubt its breaking placment at all
08:48:45 sean-k-mooney[m] but our cache is not working correctly so thats a latent bug
08:54:41 sean-k-mooney[m] gibi https://github.com/openstack/placement/commit/c68d472dca6619055579831ad5464042f745557a i guess we missed this usecase
08:55:09 sean-k-mooney[m] the test is useing the dict interface instead of the field interface
08:55:58 gibi yeah, good point, we can store row objects in our rc caches if we access the columns by attribute acces and not by dict access
08:56:04 gibi let me check that it works
08:56:10 gibi as it provides an easier solutions
08:56:57 sean-k-mooney[m] https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/objects/resource_class.py#L220 we are tryign to use _mappings to convert them to dict on get all
08:57:23 sean-k-mooney[m] well not to convert the via a dict to a resouceclass which should be fine
08:58:04 gibi here we have Row objects in the caches https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L155
08:59:05 gibi which is actually fine
08:59:11 gibi as we use attribute access everywhere
08:59:20 gibi except in that one func test that now fails
09:00:02 gibi hm, no
09:00:09 gibi that caches is broken
09:00:14 gibi as sometimes we store dicts
09:00:19 gibi sometimes we store Row objects
09:00:37 gibi https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L163
09:01:16 sean-k-mooney[m] https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/attribute_cache.py#L148-L151
09:01:54 sean-k-mooney[m] we store db rows when we load it form the db
09:02:05 gibi yep so this is a type mess
09:03:05 sean-k-mooney[m] ya stephen was complainging about this in a differnt patch
09:04:55 sean-k-mooney[m] so the all_cache
09:05:00 sean-k-mooney[m] currently had the row
09:05:09 sean-k-mooney[m] but that could jsut be a new tuple firht
09:05:47 sean-k-mooney[m] https://github.com/openstack/placement/blob/48f31d446be5dd8743392e6d1e45ed8183a9ce1b/placement/attribute_cache.py#L151
09:06:38 sean-k-mooney[m] self._all_cache = {r[1]: r for r in res} -> self._all_cache = {r[1]: (r[0], r[1]) for r in res}
09:06:46 gibi so while 'self._all_cache = {r[1]: r._mapping for r in res}' fixed the currently failing test case it breaks a bunch of gabbi tests
09:07:31 sean-k-mooney[m] what elese is in that row beyond the id and string
09:08:05 gibi updated_at and created_at
09:08:31 gibi it is in the select above the fetachall
09:08:46 sean-k-mooney[m] ah base is the time stamped mixin
09:09:35 sean-k-mooney[m] also the value of the cache is ment to be a dict not a tuple so my version is incorrect
09:10:14 gibi yeah I have to figure out while the gabbi tests fails with my change
09:10:32 gibi technically Row._mapping is not a dict just a dict like object
09:10:36 gibi so it might be a problem
09:10:55 sean-k-mooney[m] do you want to do dict(**r._mappings)
09:11:12 sean-k-mooney[m] by the way to make this dicts not rows
09:11:13 gibi yeah I can try that
09:11:37 sean-k-mooney[m] although if we are expecting rows and using .id
09:11:48 sean-k-mooney[m] you would need a named tuple instead
09:12:47 gibi that cache stores dicts so if there is .id access now that would fail anyhow
09:12:50 sean-k-mooney[m] named tuple is the only “standard” class that will give you the field and dict style access
09:13:12 sean-k-mooney[m] well its storing row objects currently
09:13:20 sean-k-mooney[m] its ment to ba a dict
09:14:05 gibi namedtuple does not give you dict access
09:14:22 gibi it sometimes store Row sometimes store Dict
09:14:36 gibi https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L184-L189
09:15:07 gibi so it seems we started using that cache also in a mixed mode
09:15:16 gibi at some places we use attribute acces on it
09:15:26 gibi hence the gabbi test failures
09:16:12 sean-k-mooney[m] https://github.com/openstack/placement/blame/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/objects/trait.py#L151 ya stephen added a fixme when they noticed that
09:16:44 sean-k-mooney[m] namedtuple give you indexed acces i.e. r[0]
09:16:58 sean-k-mooney[m] but i guess it wont give you r[‘id’]
09:17:18 gibi https://github.com/openstack/placement/blame/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/objects/resource_class.py#L71-L74 so we assumes Row here
09:18:35 gibi so we need to decide which was we go. a) Store Row (or namedtuple) objects in caches and keep attribute access b) store dict and go with dict access
09:18:37 sean-k-mooney[m] https://github.com/openstack/placement/commit/b3fe04f081a096258468d032560f46cdfe77e144 stpehen tried to remove these assumtions in ^
09:19:07 sean-k-mooney[m] well that will work as a named tuple
09:19:28 gibi Row and namedtuple is pretty much compatible, yes
09:19:48 sean-k-mooney[m] i would avoid random dicts personally
09:19:50 gibi ack
09:20:07 sean-k-mooney[m] and either store the row or named tuple
09:20:14 sean-k-mooney[m] im not sure if there is a reason not to sotre thr row object
09:20:21 sean-k-mooney[m] does it increase memory

Earlier   Later