Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-14
12:21:33 sean-k-mooney in the fixture we can stash that value
12:22:05 sean-k-mooney so the ideay i had was use a dict with set_default with the test_id as a key and a new fake notifyer as the default
12:22:25 sean-k-mooney then return the result
12:22:44 sean-k-mooney then clear it at the end fo a test run
12:23:25 sean-k-mooney if a long runing eventlet sends a notificaiton after the test we will get a new notifyer
12:23:47 sean-k-mooney instead of the current one
12:23:53 gibi the long running eventlet when calls nova.rpc.get_versioned_notifier it does not provide any tc id, same as if the currntly runnig tc calls nova.rpc.get_versioned_notifier
12:24:25 gibi from the fixture prespective both nova.rpc.get_versioned_notifier call are happening at the current tc time
12:24:31 gibi and providing no id
12:25:15 gibi is there a greenlet specific storage space like threadlocal?
12:25:54 sean-k-mooney i think we shoudl be able to make it sticky to the greenlet yes
12:25:56 gibi somehow we need to mark the long running eventlet with a different id than the current eventlets
12:26:02 sean-k-mooney i feel like i have done this before
12:32:12 gibi an we have to store the tc id automatically in each greenlet nova spawns which is /o\
12:32:48 sean-k-mooney i rememebr trying to use with context managers to create funcitonal test where each isntance of nova compute had a different nova.conf in the past
12:33:42 sean-k-mooney we did not merge it but i was able to make each nova-compute have a differfent view of the global config
12:34:36 sean-k-mooney i have no idea where that his however so i think we can spawn the nova-comptue serivce such that the things we have monkey patched are sticky to that instance but i have no idea if that would out live the test
12:34:53 sean-k-mooney i think those patcher would likely get towrn down when the test funciton ends
12:35:04 sean-k-mooney leading to the same problem
12:35:34 sean-k-mooney gibi: basicaly i was hopign we could use functool.partil or something to carry the extra info
12:36:21 sean-k-mooney gibi: there is https://eventlet.net/doc/modules/corolocal.html
12:36:45 gibi for the partial: for that we need to attach the partial to a thing that is specific to the current test case execution
12:37:08 sean-k-mooney gibi: ya and i dont really know how to do that
12:37:10 gibi for corolocal that can be the storage, but then we probably need to patch eventlet.spawn* to fill it
12:37:25 gibi I will play around
12:37:27 sean-k-mooney certnely not in the notificaiotn fixture which is where we really want to do this
12:37:37 sean-k-mooney ya i might try and play with this too
12:38:40 gibi sean-k-mooney: eventlet patches threading.local to be corolocal.local()
12:38:46 lajoskatona gauzas, gibi: Hi, for rbac discussion do you think Neutron should join to the discussion? (see: https://etherpad.opendev.org/p/policy-popup-yoga-ptg )
12:38:58 lajoskatona bauzas ---^ (sorry)
12:39:45 gibi lajoskatona: for the external event discussion would be good to have somebody from neutron as the client of that api
12:39:51 sean-k-mooney lajoskatona: i think there is work to be done with makeing nova capable of calling neturon where neutron is using scope enforcement
12:40:11 sean-k-mooney and ya the external events is the flip side of that
12:40:46 lajoskatona gibi, sean-k-mooney: thanks, than I add it to next week's shcedule
12:41:15 sean-k-mooney gibi: i still think we need to create some form of oslo.midelware so that we can dicorver a services policy programticaly form teh api
12:41:58 sean-k-mooney right now the operator will need to set the correct scopes ectra in our config file
12:42:05 lajoskatona gibi, bauzas, sean-k-mooney: we have edge session at the same time (1400-1600) try to fix that
12:42:25 bauzas sorry in a meeting
12:42:46 bauzas can you tl;dr ?
12:42:49 bauzas I'm hardly following
12:44:19 sean-k-mooney bauzas: there is a clash between the nova rbac popup session and a neutron? edge session
12:44:51 sean-k-mooney it woudl be good if we could adjust the schduler to accomidate that lajoskatona is that a correct summary
12:45:23 lajoskatona sean-k-mooney: yes
12:45:37 lajoskatona sean-k-mooney, bauzas, gibi: I try to fetch ildikov to see if wee need both hours for edge.....
12:45:39 sean-k-mooney we can also talk about neutorn rback issues in the nova neutron session too if we cant
12:46:08 lajoskatona sean-k-mooney: yeah, worst case
13:01:13 bauzas ah ok
13:01:33 bauzas let's then wait for ildikov but I'm pretty sure we can find other slots
13:30:44 bauzas and then I'm back
16:06:10 melwitt bauzas: sorry to bring this up again but do you think there's a chance you could look at https://review.opendev.org/c/openstack/nova/+/791807 and https://review.opendev.org/c/openstack/nova/+/806629 before the ptg? elodilles is +2 on them and I'm trying to avoid them getting lost
16:08:06 bauzas melwitt: yeah I remember I had to review your patches but I wasn't seeing them on my review priority list, adding them
16:08:30 melwitt thank you bauzas
16:08:46 bauzas melwitt: will be the first patches I look tomorrow
16:29:17 opendevreview Balazs Gibizer proposed openstack/nova master: Prevent leaked eventlets to send notifications https://review.opendev.org/c/openstack/nova/+/814036
16:29:47 gibi sean-k-mooney: ^^ this fixes the local reproduction for me
16:34:11 melwitt gibi: omg, I spent some time looking at these failures yesterday and could not figure out how/why we got "no reply on conductor" and DBNonExistentTable. awesome find 🙌
16:34:46 gibi melwitt: it was hard one, sean-k-mooney and I spent two day figuring it out
16:36:23 gibi the lesson for me is that we probably should not run multiple testcases in a sequence in a same process if those testcases use some kind of parallelism
16:36:36 gibi but I have no viable alternative
16:36:45 melwitt I was completely stumped. I'm so happy y'all figured it out
16:37:56 gibi ... and I still not like eventlets ;)
16:40:51 melwitt haha :)
16:41:46 melwitt speaking of eventlet...
16:42:43 melwitt here's a thing I'd appreciate your eyes on https://review.opendev.org/c/openstack/nova/+/813114 to see if this is the right way to solve it or if I'm missing a better way
16:42:58 melwitt gibi ^
16:43:20 gibi melwitt: added to my queue
16:43:26 melwitt danke
17:04:33 sean-k-mooney most of the work was by gibi but ok ill review that it looks promising
17:04:46 sean-k-mooney it its not that complex at first glance
17:06:00 sean-k-mooney ah you are intercepting spawn
17:06:07 sean-k-mooney and that is where your getting the testcase id
17:06:11 sean-k-mooney and propagating it
17:06:43 sean-k-mooney gibi: so the runtime error will that cause the test to fail?
17:07:15 sean-k-mooney i.e. when https://review.opendev.org/c/openstack/nova/+/814036/1/nova/tests/fixtures/notifications.py#168 is raised
17:07:28 sean-k-mooney which test will fail
17:08:01 sean-k-mooney or will we catch that and just loog it
17:08:40 sean-k-mooney oh the runtim error goes to the eventlet that called notify
17:09:01 sean-k-mooney which is the one that is runnign the backgound and it kill whatever was leaked
17:12:58 sean-k-mooney gibi: by the way would stoping the compute service help in this case.
17:14:29 sean-k-mooney gibi: we start the service here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/tests/functional/integrated_helpers.py#L1125-L1143
17:15:00 sean-k-mooney gibi: if we added a tear_down funciton implmeation that expiclty stop them would that help clean up any running eventlests
17:16:36 sean-k-mooney we have the service refernces so i feel like we shoudl be able to use them to invoke stop https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/service.py#L282-L296
17:16:56 sean-k-mooney that will at least stop the rpc server instnaces
17:18:37 sean-k-mooney hum ok i guess we are at least partly doing that https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/test.py#L438-L441
17:20:25 sean-k-mooney actully no that is not registring it to be involed automaticaly its patching the stop funciton
17:22:27 sean-k-mooney the service fixture calls kill as a cleanup function which in turn calls stop so we are already stoping the services when the serivce fixtuer is disposed of
19:14:21 opendevreview Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649
23:28:05 opendevreview Hang Yang proposed openstack/nova master: Support creating servers with RBAC SGs https://review.opendev.org/c/openstack/nova/+/811521
#openstack-nova - 2021-10-15
07:44:57 bauzas good morning Nova
08:46:35 kashyap gibi[m]: bauzas: Friday shameless plug: I recently summarized a KVM maintainer's talk for LWN on QEMU and software complexity. I think the lessons are interesting for OpenStack too:
08:46:43 kashyap gibi[m]: bauzas: "A QEMU case study in grappling with software complexity" — https://lwn.net/SubscriberLink/872321/221e8d48eb609a38/
08:46:47 bauzas ++
08:47:12 kashyap If you're short on time, read the intro "Sources of complexity", "Ways to fight back", and the short, one-para conclusion.
08:47:34 kashyap (Especially check out the idea of "incomplete transitions")
09:47:32 mdbooth where to look. There's a comment from dansmith that this is a hack: https://github.com/openstack/devstack/blob/82facd6edf7cefac1ab68de4fe9054d7c4cb50db/lib/glance#L291-L294 . Has something invalidated the hack? Does anybody know where 'devstack-system-admin' is supposed to come from?
09:47:32 mdbooth My devstack is failing when it tries to run 'openstack --os-cloud devstack-system-admin registered limit create --service glance --default-limit 10000 --region RegionOne image_size_total' with 'Cloud devstack-system-admin was not found'. This is a fresh install. I don't know what devstack-system-admin is or what creates it, so I'm at a loss for
09:49:19 mdbooth To the best of my knowledge there is no clouds.yaml anywhere on this system. If there is, it was created by devstack and put somewhere I don't know to look for it.
09:49:40 kashyap mdbooth: See this commit in DevStack: 56905820 (Add devstack-system-admin for system scoped actions, 2019-01-08)
09:51:12 mdbooth 👀
09:51:20 kashyap Heh

Earlier   Later