| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-13 | |||
| 17:52:31 | sean-k-mooney | the create_server funciton its calling i think is expected to wait for it to be active | |
| 17:52:52 | sean-k-mooney | oh | |
| 17:53:01 | sean-k-mooney | it not using the one form the integrated helper | |
| 17:53:10 | sean-k-mooney | so ya your right its not waiting | |
| 17:58:46 | gibi | sean-k-mooney: sorry I had to go offline | |
| 17:58:51 | gibi | will read back tomorrow | |
| 21:36:53 | opendevreview | Julia Kreger proposed openstack/nova master: WIP Ironic - Handle instance host on rebalance https://review.opendev.org/c/openstack/nova/+/813897 | |
| 21:53:17 | melwitt | stephenfin: I dunno if you noticed this too but the arm64 non-voting jobs started failing often recently https://zuul.openstack.org/builds?job_name=openstack-tox-py38-arm64&job_name=openstack-tox-py39-arm64+%28non-voting%29&project=openstack%2Fnova and the timing coincided with when a few of the db-related test patches landed. is there any chance it's related? | |
| 21:54:02 | melwitt | this one in particular https://review.opendev.org/c/openstack/nova/+/810291 | |
| 21:55:59 | melwitt | the other two that merged at the same time were https://review.opendev.org/c/openstack/nova/+/810856 and https://review.opendev.org/c/openstack/nova/+/810857 which I thought aren't likely to be related... linking them too just in case | |
| #openstack-nova - 2021-10-14 | |||
| 06:23:37 | opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uWSGI https://review.opendev.org/c/openstack/nova/+/810922 | |
| 07:00:57 | bauzas | good morning Nova | |
| 08:32:42 | opendevreview | alecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | |
| 08:41:56 | bauzas | mmm, I'm stuck trying to install a devstack on RHEL8.2 with a "openstack: command not found" when creating keystone accounts... https://paste.opendev.org/show/809996/ | |
| 08:42:00 | bauzas | anyone hitting it ? | |
| 08:42:46 | bauzas | I'm out of ideas | |
| 08:47:21 | kashyap | bauzas: Why are you installing it on RHEL8.2? | |
| 08:47:46 | kashyap | FWIW, I'd suggest to pick a latest-1 Fedora (or Debian/Ubuntu - if you're comfy w/ it) :) | |
| 08:49:15 | frickler | bauzas: did you check that there is no earlier error already? also 8.2 afaict isn't supported by devstack anymore | |
| 09:01:08 | opendevreview | Rodolfo Alonso proposed openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 09:13:10 | gibi | bauzas: hi! did you got the moderator info for the PTG from Ashlee? or should I forward? | |
| 09:18:18 | bauzas | frickler: kashyap: thanks (for some reason, got no pidgin notification when you highlighted me) | |
| 09:18:58 | bauzas | frickler: kashyap: I'll then use RHEL8.4 I guess (I need it for testing the nvidia GPUs, Fedora is not supported for their driver) | |
| 09:19:22 | bauzas | gibi: hmmm, by email ? if yes, nope | |
| 09:19:29 | bauzas | gibi: thanks | |
| 09:19:56 | gibi | email so I forward then | |
| 09:20:34 | gibi | done | |
| 09:21:16 | bauzas | gibi: thanks, will look at it ! | |
| 09:21:46 | gibi | also fyi, on Monday I will only be available from 15:00 UTC | |
| 09:24:05 | gibi | rest of the week I will fully available | |
| 09:25:15 | bauzas | ++ | |
| 09:35:44 | kashyap | bauzas: Ah, I see. | |
| 09:37:42 | viks__ | hi, I have set `live_migration_completion_timeout=100 & live_migration_timeout_action=force_complete`, now i'm stressing the vm via stress-ng tool with load going up to 300. But my migration is not completing. Why `force_complete` action is not getting kicked in? | |
| 10:04:10 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 12:04:37 | gibi | sean-k-mooney: you were right, the rpc.NOTIFIER is the global that we facilitate the test case crosstalk https://bugs.launchpad.net/nova/+bug/1946339/comments/7 | |
| 12:04:52 | gibi | s/we facilitate/ facilitates/ | |
| 12:05:30 | gibi | we reset that global between tests but the nova does dynamically gets the global from the rpc module whathever it is at the moment | |
| 12:05:48 | gibi | so if the global was re-inited it uses the re-inited global for the next notification | |
| 12:09:35 | gibi | I don't see how to fix this from the rpc.NOTIFIER perspective | |
| 12:09:50 | sean-k-mooney | i see i looked at that breifly but did not see how that happened but that is what my gut was telling had to be happening | |
| 12:10:02 | gibi | you have good gut :) | |
| 12:11:06 | sean-k-mooney | so this global https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/rpc.py#L53 is really the issue right | |
| 12:11:15 | gibi | yes, | |
| 12:11:27 | sean-k-mooney | we need that to be mocked in the setup of the test | |
| 12:11:29 | gibi | an whatever code wants to emit a notificiation it uses that global | |
| 12:11:50 | gibi | sean-k-mooney: that won't work as the code grabs the global at the point of time when the notification needs to be emitted | |
| 12:12:05 | gibi | so the first tc will grab it 60 seconds after the tc is finished | |
| 12:12:21 | gibi | and at that time it is already restubbed to the current test case | |
| 12:12:39 | gibi | so it grabbs the new stubbed version that is connected to the current testcase | |
| 12:13:06 | gibi | hence the crosstalk | |
| 12:13:37 | gibi | if nova would grab the global at service startup then yes stubbing would work | |
| 12:14:03 | sean-k-mooney | damnb ok ya that is annoying | |
| 12:15:03 | gibi | it is really due to that the test case executor things that a tc is finished and moves forward but the tc still has greenlets running in the background | |
| 12:15:22 | gibi | + the global :) | |
| 12:15:44 | gibi | I tried killing greenlets at the end of test case but I think I cannot properly kill it | |
| 12:16:25 | sean-k-mooney | we might be able to wait for the notificaiton in that one test but this could affect any set of tests | |
| 12:16:45 | gibi | yes, waiting in each test for each build to finish is a way to solve this | |
| 12:16:54 | sean-k-mooney | so i think we need a more systematic way of mocking this but im not sure how to approch that | |
| 12:17:30 | gibi | yeah probably we need a higher level mock than the stub on rpc.NOTIFIER | |
| 12:17:53 | gibi | I have to think about it | |
| 12:18:08 | sean-k-mooney | we cant just stub out nova.rpc.get_versioned_notifier() | |
| 12:18:21 | gibi | nope | |
| 12:18:35 | gibi | the module level function is also a global | |
| 12:19:09 | gibi | so when the caller say rpc.get_versioned_notifier it gets whatever mocked version the modul has at the moment | |
| 12:19:33 | gibi | and 60 seconds after the first tc, it will be mocked to the current tc not to the first tc | |
| 12:20:20 | sean-k-mooney | it does yes but i was wondering if we coudl have a per test dictionaty of notifieers and do a lookup in that | |
| 12:20:39 | gibi | the caller cannot provide the test case id | |
| 12:20:46 | gibi | afaik | |
| 12:21:01 | gibi | or in other way, what would be the key in the lookuptable? | |
| 12:21:04 | sean-k-mooney | it cant but we can | |
| 12:21:33 | sean-k-mooney | in the fixture we can stash that value | |
| 12:22:05 | sean-k-mooney | so the ideay i had was use a dict with set_default with the test_id as a key and a new fake notifyer as the default | |
| 12:22:25 | sean-k-mooney | then return the result | |
| 12:22:44 | sean-k-mooney | then clear it at the end fo a test run | |
| 12:23:25 | sean-k-mooney | if a long runing eventlet sends a notificaiton after the test we will get a new notifyer | |
| 12:23:47 | sean-k-mooney | instead of the current one | |
| 12:23:53 | gibi | the long running eventlet when calls nova.rpc.get_versioned_notifier it does not provide any tc id, same as if the currntly runnig tc calls nova.rpc.get_versioned_notifier | |
| 12:24:25 | gibi | from the fixture prespective both nova.rpc.get_versioned_notifier call are happening at the current tc time | |
| 12:24:31 | gibi | and providing no id | |
| 12:25:15 | gibi | is there a greenlet specific storage space like threadlocal? | |
| 12:25:54 | sean-k-mooney | i think we shoudl be able to make it sticky to the greenlet yes | |
| 12:25:56 | gibi | somehow we need to mark the long running eventlet with a different id than the current eventlets | |
| 12:26:02 | sean-k-mooney | i feel like i have done this before | |
| 12:32:12 | gibi | an we have to store the tc id automatically in each greenlet nova spawns which is /o\ | |
| 12:32:48 | sean-k-mooney | i rememebr trying to use with context managers to create funcitonal test where each isntance of nova compute had a different nova.conf in the past | |
| 12:33:42 | sean-k-mooney | we did not merge it but i was able to make each nova-compute have a differfent view of the global config | |
| 12:34:36 | sean-k-mooney | i have no idea where that his however so i think we can spawn the nova-comptue serivce such that the things we have monkey patched are sticky to that instance but i have no idea if that would out live the test | |
| 12:34:53 | sean-k-mooney | i think those patcher would likely get towrn down when the test funciton ends | |
| 12:35:04 | sean-k-mooney | leading to the same problem | |
| 12:35:34 | sean-k-mooney | gibi: basicaly i was hopign we could use functool.partil or something to carry the extra info | |
| 12:36:21 | sean-k-mooney | gibi: there is https://eventlet.net/doc/modules/corolocal.html | |
| 12:36:45 | gibi | for the partial: for that we need to attach the partial to a thing that is specific to the current test case execution | |
| 12:37:08 | sean-k-mooney | gibi: ya and i dont really know how to do that | |
| 12:37:10 | gibi | for corolocal that can be the storage, but then we probably need to patch eventlet.spawn* to fill it | |
| 12:37:25 | gibi | I will play around | |
| 12:37:27 | sean-k-mooney | certnely not in the notificaiotn fixture which is where we really want to do this | |
| 12:37:37 | sean-k-mooney | ya i might try and play with this too | |
| 12:38:40 | gibi | sean-k-mooney: eventlet patches threading.local to be corolocal.local() | |
| 12:38:46 | lajoskatona | gauzas, gibi: Hi, for rbac discussion do you think Neutron should join to the discussion? (see: https://etherpad.opendev.org/p/policy-popup-yoga-ptg ) | |
| 12:38:58 | lajoskatona | bauzas ---^ (sorry) | |