| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-13 | |||
| 16:31:55 | dansmith | stephenfin: you're expected to upgrade all the non-compute services at once, yes, largely because of schema but not only for that | |
| 16:32:25 | stephenfin | Okay, so that being the case, why do we insist that DB columns are removed in a later cycle than the corresponding SQLAlchemy model fields? | |
| 16:32:44 | opendevreview | Merged openstack/nova-specs master: Re-propose Unified Limits in Nova https://review.opendev.org/c/openstack/nova-specs/+/809020 | |
| 16:32:45 | dansmith | so you can apply new schema before you roll any of that new code | |
| 16:32:54 | stephenfin | ah | |
| 16:33:14 | dansmith | schema apply being potentially very expensive, rewriting tables, etc | |
| 16:33:24 | dansmith | it's in the upgrade doc somewhere, I just linked it the other day | |
| 16:33:25 | stephenfin | yup, and having to come first for the additive stuff | |
| 16:33:55 | dansmith | ideally if you've rolled the schema, then "upgrading" is just starting new containers, which can be pretty quick | |
| 16:37:55 | gibi | gmann: ack, I will file a bug to zuul when I have time about the quoting issue | |
| 16:37:59 | sean-k-mooney | provieded we have only made aditive changes in principal for something like an FFU we can fully upgrade the db schema while running n-3 compute nodes and then do all the online migration when we bounce the containers. | |
| 16:38:53 | sean-k-mooney | the queens to train ffu in oo however stop on each release to do the db sync for some reaons | |
| 16:39:42 | sean-k-mooney | so i dont think they have ever actuly done that in practice where tehy did the db sync rom the targent n release while the n-3 contianer where running | |
| 16:40:23 | melwitt | stephenfin: thanks for the detailed explanation about the auto-generation stuff! | |
| 16:40:40 | stephenfin | nw, it's *very* cool, if you ask me | |
| 16:40:42 | stephenfin | zzzeek++ | |
| 16:54:55 | gibi | sean-k-mooney: I manage to make a stable local reproduction for the unshelve func test bug https://bugs.launchpad.net/nova/+bug/1946339/comments/6 | |
| 16:55:25 | gibi | now I just have to debug it to find the leaking global state between the tests | |
| 16:58:04 | sean-k-mooney | that great | |
| 16:58:23 | opendevreview | sean mooney proposed openstack/nova master: [WIP] adress intermitent failure of functional tests https://review.opendev.org/c/openstack/nova/+/813695 | |
| 16:58:31 | sean-k-mooney | i just fixed the pep8 issues with ^ | |
| 16:59:04 | sean-k-mooney | gibi: also thanks for the devstack tip it fixed my placment issue | |
| 17:02:48 | sean-k-mooney | gibi: could it be form this | |
| 17:02:50 | sean-k-mooney | https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_fake.py#L148-L159 | |
| 17:03:05 | sean-k-mooney | gibi: could we be reusing the same exchange between tests | |
| 17:04:05 | sean-k-mooney | if we initalise the fake messaign drver without passing a unique exchange name per test | |
| 17:04:34 | sean-k-mooney | gibi: tox allows use to group test per class correct | |
| 17:04:43 | sean-k-mooney | coudl you try that with your reopducecer and see it that helps | |
| 17:11:58 | sean-k-mooney | gibi: we do try and cleanup the exchanges https://github.com/openstack/nova/blob/7b063e4d0518af3e57872bc0288a94edcd33c19d/nova/tests/fixtures/nova.py#L741-L744 | |
| 17:13:02 | sean-k-mooney | but that does not reset self._default_exchange | |
| 17:13:32 | sean-k-mooney | i guess that is not required | |
| 17:14:19 | sean-k-mooney | self._exchanges.setdefault(name, FakeExchange(name)) shoudl still be a new exchange | |
| 17:52:12 | sean-k-mooney | gibi: just looking at test_description_errors | |
| 17:52:31 | sean-k-mooney | the create_server funciton its calling i think is expected to wait for it to be active | |
| 17:52:52 | sean-k-mooney | oh | |
| 17:53:01 | sean-k-mooney | it not using the one form the integrated helper | |
| 17:53:10 | sean-k-mooney | so ya your right its not waiting | |
| 17:58:46 | gibi | sean-k-mooney: sorry I had to go offline | |
| 17:58:51 | gibi | will read back tomorrow | |
| 21:36:53 | opendevreview | Julia Kreger proposed openstack/nova master: WIP Ironic - Handle instance host on rebalance https://review.opendev.org/c/openstack/nova/+/813897 | |
| 21:53:17 | melwitt | stephenfin: I dunno if you noticed this too but the arm64 non-voting jobs started failing often recently https://zuul.openstack.org/builds?job_name=openstack-tox-py38-arm64&job_name=openstack-tox-py39-arm64+%28non-voting%29&project=openstack%2Fnova and the timing coincided with when a few of the db-related test patches landed. is there any chance it's related? | |
| 21:54:02 | melwitt | this one in particular https://review.opendev.org/c/openstack/nova/+/810291 | |
| 21:55:59 | melwitt | the other two that merged at the same time were https://review.opendev.org/c/openstack/nova/+/810856 and https://review.opendev.org/c/openstack/nova/+/810857 which I thought aren't likely to be related... linking them too just in case | |
| #openstack-nova - 2021-10-14 | |||
| 06:23:37 | opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uWSGI https://review.opendev.org/c/openstack/nova/+/810922 | |
| 07:00:57 | bauzas | good morning Nova | |
| 08:32:42 | opendevreview | alecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | |
| 08:41:56 | bauzas | mmm, I'm stuck trying to install a devstack on RHEL8.2 with a "openstack: command not found" when creating keystone accounts... https://paste.opendev.org/show/809996/ | |
| 08:42:00 | bauzas | anyone hitting it ? | |
| 08:42:46 | bauzas | I'm out of ideas | |
| 08:47:21 | kashyap | bauzas: Why are you installing it on RHEL8.2? | |
| 08:47:46 | kashyap | FWIW, I'd suggest to pick a latest-1 Fedora (or Debian/Ubuntu - if you're comfy w/ it) :) | |
| 08:49:15 | frickler | bauzas: did you check that there is no earlier error already? also 8.2 afaict isn't supported by devstack anymore | |
| 09:01:08 | opendevreview | Rodolfo Alonso proposed openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | |
| 09:13:10 | gibi | bauzas: hi! did you got the moderator info for the PTG from Ashlee? or should I forward? | |
| 09:18:18 | bauzas | frickler: kashyap: thanks (for some reason, got no pidgin notification when you highlighted me) | |
| 09:18:58 | bauzas | frickler: kashyap: I'll then use RHEL8.4 I guess (I need it for testing the nvidia GPUs, Fedora is not supported for their driver) | |
| 09:19:22 | bauzas | gibi: hmmm, by email ? if yes, nope | |
| 09:19:29 | bauzas | gibi: thanks | |
| 09:19:56 | gibi | email so I forward then | |
| 09:20:34 | gibi | done | |
| 09:21:16 | bauzas | gibi: thanks, will look at it ! | |
| 09:21:46 | gibi | also fyi, on Monday I will only be available from 15:00 UTC | |
| 09:24:05 | gibi | rest of the week I will fully available | |
| 09:25:15 | bauzas | ++ | |
| 09:35:44 | kashyap | bauzas: Ah, I see. | |
| 09:37:42 | viks__ | hi, I have set `live_migration_completion_timeout=100 & live_migration_timeout_action=force_complete`, now i'm stressing the vm via stress-ng tool with load going up to 300. But my migration is not completing. Why `force_complete` action is not getting kicked in? | |
| 10:04:10 | opendevreview | Ilya Popov proposed openstack/nova master: Fix to use NUMA cell with more free memory first https://review.opendev.org/c/openstack/nova/+/805649 | |
| 12:04:37 | gibi | sean-k-mooney: you were right, the rpc.NOTIFIER is the global that we facilitate the test case crosstalk https://bugs.launchpad.net/nova/+bug/1946339/comments/7 | |
| 12:04:52 | gibi | s/we facilitate/ facilitates/ | |
| 12:05:30 | gibi | we reset that global between tests but the nova does dynamically gets the global from the rpc module whathever it is at the moment | |
| 12:05:48 | gibi | so if the global was re-inited it uses the re-inited global for the next notification | |
| 12:09:35 | gibi | I don't see how to fix this from the rpc.NOTIFIER perspective | |
| 12:09:50 | sean-k-mooney | i see i looked at that breifly but did not see how that happened but that is what my gut was telling had to be happening | |
| 12:10:02 | gibi | you have good gut :) | |
| 12:11:06 | sean-k-mooney | so this global https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/rpc.py#L53 is really the issue right | |
| 12:11:15 | gibi | yes, | |
| 12:11:27 | sean-k-mooney | we need that to be mocked in the setup of the test | |
| 12:11:29 | gibi | an whatever code wants to emit a notificiation it uses that global | |
| 12:11:50 | gibi | sean-k-mooney: that won't work as the code grabs the global at the point of time when the notification needs to be emitted | |
| 12:12:05 | gibi | so the first tc will grab it 60 seconds after the tc is finished | |
| 12:12:21 | gibi | and at that time it is already restubbed to the current test case | |
| 12:12:39 | gibi | so it grabbs the new stubbed version that is connected to the current testcase | |
| 12:13:06 | gibi | hence the crosstalk | |
| 12:13:37 | gibi | if nova would grab the global at service startup then yes stubbing would work | |
| 12:14:03 | sean-k-mooney | damnb ok ya that is annoying | |
| 12:15:03 | gibi | it is really due to that the test case executor things that a tc is finished and moves forward but the tc still has greenlets running in the background | |
| 12:15:22 | gibi | + the global :) | |
| 12:15:44 | gibi | I tried killing greenlets at the end of test case but I think I cannot properly kill it | |
| 12:16:25 | sean-k-mooney | we might be able to wait for the notificaiton in that one test but this could affect any set of tests | |
| 12:16:45 | gibi | yes, waiting in each test for each build to finish is a way to solve this | |
| 12:16:54 | sean-k-mooney | so i think we need a more systematic way of mocking this but im not sure how to approch that | |
| 12:17:30 | gibi | yeah probably we need a higher level mock than the stub on rpc.NOTIFIER | |
| 12:17:53 | gibi | I have to think about it | |
| 12:18:08 | sean-k-mooney | we cant just stub out nova.rpc.get_versioned_notifier() | |
| 12:18:21 | gibi | nope | |
| 12:18:35 | gibi | the module level function is also a global | |
| 12:19:09 | gibi | so when the caller say rpc.get_versioned_notifier it gets whatever mocked version the modul has at the moment | |
| 12:19:33 | gibi | and 60 seconds after the first tc, it will be mocked to the current tc not to the first tc | |
| 12:20:20 | sean-k-mooney | it does yes but i was wondering if we coudl have a per test dictionaty of notifieers and do a lookup in that | |
| 12:20:39 | gibi | the caller cannot provide the test case id | |