Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-10-14
08:42:00 bauzas anyone hitting it ?
08:42:46 bauzas I'm out of ideas
08:47:21 kashyap bauzas: Why are you installing it on RHEL8.2?
08:47:46 kashyap FWIW, I'd suggest to pick a latest-1 Fedora (or Debian/Ubuntu - if you're comfy w/ it) :)
08:49:15 frickler bauzas: did you check that there is no earlier error already? also 8.2 afaict isn't supported by devstack anymore
09:01:08 opendevreview Rodolfo Alonso proposed openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359
09:13:10 gibi bauzas: hi! did you got the moderator info for the PTG from Ashlee? or should I forward?
09:18:18 bauzas frickler: kashyap: thanks (for some reason, got no pidgin notification when you highlighted me)
09:18:58 bauzas frickler: kashyap: I'll then use RHEL8.4 I guess (I need it for testing the nvidia GPUs, Fedora is not supported for their driver)
09:19:22 bauzas gibi: hmmm, by email ? if yes, nope
09:19:29 bauzas gibi: thanks
09:19:56 gibi email so I forward then
09:20:34 gibi done
09:21:16 bauzas gibi: thanks, will look at it !
09:21:46 gibi also fyi, on Monday I will only be available from 15:00 UTC
09:24:05 gibi rest of the week I will fully available
09:25:15 bauzas ++
09:35:44 kashyap bauzas: Ah, I see.
09:37:42 viks__ hi, I have set `live_migration_completion_timeout=100 & live_migration_timeout_action=force_complete`, now i'm stressing the vm via stress-ng tool with load going up to 300. But my migration is not completing. Why `force_complete` action is not getting kicked in?
10:04:10 opendevreview Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649
12:04:37 gibi sean-k-mooney: you were right, the rpc.NOTIFIER is the global that we facilitate the test case crosstalk https://bugs.launchpad.net/nova/+bug/1946339/comments/7
12:04:52 gibi s/we facilitate/ facilitates/
12:05:30 gibi we reset that global between tests but the nova does dynamically gets the global from the rpc module whathever it is at the moment
12:05:48 gibi so if the global was re-inited it uses the re-inited global for the next notification
12:09:35 gibi I don't see how to fix this from the rpc.NOTIFIER perspective
12:09:50 sean-k-mooney i see i looked at that breifly but did not see how that happened but that is what my gut was telling had to be happening
12:10:02 gibi you have good gut :)
12:11:06 sean-k-mooney so this global https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/rpc.py#L53 is really the issue right
12:11:15 gibi yes,
12:11:27 sean-k-mooney we need that to be mocked in the setup of the test
12:11:29 gibi an whatever code wants to emit a notificiation it uses that global
12:11:50 gibi sean-k-mooney: that won't work as the code grabs the global at the point of time when the notification needs to be emitted
12:12:05 gibi so the first tc will grab it 60 seconds after the tc is finished
12:12:21 gibi and at that time it is already restubbed to the current test case
12:12:39 gibi so it grabbs the new stubbed version that is connected to the current testcase
12:13:06 gibi hence the crosstalk
12:13:37 gibi if nova would grab the global at service startup then yes stubbing would work
12:14:03 sean-k-mooney damnb ok ya that is annoying
12:15:03 gibi it is really due to that the test case executor things that a tc is finished and moves forward but the tc still has greenlets running in the background
12:15:22 gibi + the global :)
12:15:44 gibi I tried killing greenlets at the end of test case but I think I cannot properly kill it
12:16:25 sean-k-mooney we might be able to wait for the notificaiton in that one test but this could affect any set of tests
12:16:45 gibi yes, waiting in each test for each build to finish is a way to solve this
12:16:54 sean-k-mooney so i think we need a more systematic way of mocking this but im not sure how to approch that
12:17:30 gibi yeah probably we need a higher level mock than the stub on rpc.NOTIFIER
12:17:53 gibi I have to think about it
12:18:08 sean-k-mooney we cant just stub out nova.rpc.get_versioned_notifier()
12:18:21 gibi nope
12:18:35 gibi the module level function is also a global
12:19:09 gibi so when the caller say rpc.get_versioned_notifier it gets whatever mocked version the modul has at the moment
12:19:33 gibi and 60 seconds after the first tc, it will be mocked to the current tc not to the first tc
12:20:20 sean-k-mooney it does yes but i was wondering if we coudl have a per test dictionaty of notifieers and do a lookup in that
12:20:39 gibi the caller cannot provide the test case id
12:20:46 gibi afaik
12:21:01 gibi or in other way, what would be the key in the lookuptable?
12:21:04 sean-k-mooney it cant but we can
12:21:33 sean-k-mooney in the fixture we can stash that value
12:22:05 sean-k-mooney so the ideay i had was use a dict with set_default with the test_id as a key and a new fake notifyer as the default
12:22:25 sean-k-mooney then return the result
12:22:44 sean-k-mooney then clear it at the end fo a test run
12:23:25 sean-k-mooney if a long runing eventlet sends a notificaiton after the test we will get a new notifyer
12:23:47 sean-k-mooney instead of the current one
12:23:53 gibi the long running eventlet when calls nova.rpc.get_versioned_notifier it does not provide any tc id, same as if the currntly runnig tc calls nova.rpc.get_versioned_notifier
12:24:25 gibi from the fixture prespective both nova.rpc.get_versioned_notifier call are happening at the current tc time
12:24:31 gibi and providing no id
12:25:15 gibi is there a greenlet specific storage space like threadlocal?
12:25:54 sean-k-mooney i think we shoudl be able to make it sticky to the greenlet yes
12:25:56 gibi somehow we need to mark the long running eventlet with a different id than the current eventlets
12:26:02 sean-k-mooney i feel like i have done this before
12:32:12 gibi an we have to store the tc id automatically in each greenlet nova spawns which is /o\
12:32:48 sean-k-mooney i rememebr trying to use with context managers to create funcitonal test where each isntance of nova compute had a different nova.conf in the past
12:33:42 sean-k-mooney we did not merge it but i was able to make each nova-compute have a differfent view of the global config
12:34:36 sean-k-mooney i have no idea where that his however so i think we can spawn the nova-comptue serivce such that the things we have monkey patched are sticky to that instance but i have no idea if that would out live the test
12:34:53 sean-k-mooney i think those patcher would likely get towrn down when the test funciton ends
12:35:04 sean-k-mooney leading to the same problem
12:35:34 sean-k-mooney gibi: basicaly i was hopign we could use functool.partil or something to carry the extra info
12:36:21 sean-k-mooney gibi: there is https://eventlet.net/doc/modules/corolocal.html
12:36:45 gibi for the partial: for that we need to attach the partial to a thing that is specific to the current test case execution
12:37:08 sean-k-mooney gibi: ya and i dont really know how to do that
12:37:10 gibi for corolocal that can be the storage, but then we probably need to patch eventlet.spawn* to fill it
12:37:25 gibi I will play around
12:37:27 sean-k-mooney certnely not in the notificaiotn fixture which is where we really want to do this
12:37:37 sean-k-mooney ya i might try and play with this too
12:38:40 gibi sean-k-mooney: eventlet patches threading.local to be corolocal.local()
12:38:46 lajoskatona gauzas, gibi: Hi, for rbac discussion do you think Neutron should join to the discussion? (see: https://etherpad.opendev.org/p/policy-popup-yoga-ptg )
12:38:58 lajoskatona bauzas ---^ (sorry)
12:39:45 gibi lajoskatona: for the external event discussion would be good to have somebody from neutron as the client of that api
12:39:51 sean-k-mooney lajoskatona: i think there is work to be done with makeing nova capable of calling neturon where neutron is using scope enforcement
12:40:11 sean-k-mooney and ya the external events is the flip side of that
12:40:46 lajoskatona gibi, sean-k-mooney: thanks, than I add it to next week's shcedule
12:41:15 sean-k-mooney gibi: i still think we need to create some form of oslo.midelware so that we can dicorver a services policy programticaly form teh api
12:41:58 sean-k-mooney right now the operator will need to set the correct scopes ectra in our config file
12:42:05 lajoskatona gibi, bauzas, sean-k-mooney: we have edge session at the same time (1400-1600) try to fix that
12:42:25 bauzas sorry in a meeting
12:42:46 bauzas can you tl;dr ?
12:42:49 bauzas I'm hardly following
12:44:19 sean-k-mooney bauzas: there is a clash between the nova rbac popup session and a neutron? edge session
12:44:51 sean-k-mooney it woudl be good if we could adjust the schduler to accomidate that lajoskatona is that a correct summary
12:45:23 lajoskatona sean-k-mooney: yes
12:45:37 lajoskatona sean-k-mooney, bauzas, gibi: I try to fetch ildikov to see if wee need both hours for edge.....

Earlier   Later