| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-13 | |||
| 13:08:22 | sean-k-mooney | so that shoudl be fixed | |
| 13:08:24 | auniyal | yes, 2.43 | |
| 13:09:37 | sean-k-mooney | so https://paste.opendev.org/show/bE1IV21v1cdSzRKNWKrw/ | |
| 13:09:40 | sean-k-mooney | should be valid | |
| 13:09:48 | sean-k-mooney | just s/tags/tag/ | |
| 13:11:48 | sean-k-mooney | https://github.com/openstack/nova/commit/e80e2511cf825671a479053cc8d41463aab1caaa | |
| 13:11:52 | sean-k-mooney | that is the change that fixed it | |
| 13:12:05 | auniyal | this is full what I have right now - https://paste.opendev.org/show/bymbRPlBXG3nmY1Nm4zy/ | |
| 13:13:03 | sean-k-mooney | this is the api sample test that validates it https://github.com/openstack/nova/blob/master/nova/tests/functional/api_sample_tests/api_samples/servers/v2.42/server-create-req.json.tpl | |
| 13:14:34 | sean-k-mooney | you do not need admin_api=ture by the way | |
| 13:15:05 | auniyal | ack | |
| 13:15:39 | sean-k-mooney | so i think the problem is | |
| 13:15:42 | sean-k-mooney | self.api = self.useFixture( | |
| 13:15:44 | sean-k-mooney | nova_fixtures.OSAPIFixture(api_version='v2.1')).api | |
| 13:16:02 | sean-k-mooney | you are seting api_version but i dont think that will work as you expect | |
| 13:17:47 | bauzas | sorry auniyal can't help, today is the last day before RC1 | |
| 13:18:03 | bauzas | IMHO you need to set the microversion correctly | |
| 13:18:13 | auniyal | so yeah, I tried this nova_fixtures.OSAPIFixture(api_version='v2.42')) and got https://paste.opendev.org/show/bHOTRcnwwpPb1lP8BbH5/ | |
| 13:18:31 | bauzas | that's not how you set the microversion | |
| 13:18:34 | sean-k-mooney | ya so when we set version in teh class like that | |
| 13:18:39 | bauzas | please look at other tests | |
| 13:18:42 | sean-k-mooney | we have explcitly coded the base class to use them | |
| 13:18:47 | bauzas | you have a class value | |
| 13:19:04 | sean-k-mooney | (test.TestCase, integrated_helpers.InstanceHelperMixin) wont make api_version = '2.42' | |
| 13:19:13 | sean-k-mooney | work at the class level | |
| 13:20:06 | sean-k-mooney | this is hte base your inheriting form | |
| 13:20:08 | sean-k-mooney | https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/test.py#L151 | |
| 13:23:08 | sean-k-mooney | the functionaltiy you are tyring to use si part of _IntegratedTestBase | |
| 13:23:10 | sean-k-mooney | https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/tests/functional/integrated_helpers.py#L1239 | |
| 13:24:02 | sean-k-mooney | it does https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/tests/functional/integrated_helpers.py#L1307-L1311 | |
| 13:24:26 | sean-k-mooney | if this is a regression test you shoudl just set the microversion on the fixture | |
| 13:24:49 | auniyal | microversion as latest | |
| 13:24:51 | sean-k-mooney | if this is for geneal testing then you proably shoudl be extendign the exitig test | |
| 13:25:14 | sean-k-mooney | https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/tests/functional/test_servers.py#L61C7-L61C18 | |
| 13:25:43 | sean-k-mooney | auniyal: microveion as lates means nothing of your parten classes dont read that class veraible | |
| 13:25:47 | sean-k-mooney | thats the point im making | |
| 13:26:20 | sean-k-mooney | so first question is why are you writing this test? | |
| 13:27:03 | auniyal | this bug https://bugs.launchpad.net/nova/+bug/1836389 | |
| 13:27:21 | auniyal | I am working on reproducer of it | |
| 13:27:31 | sean-k-mooney | ok so if its a regression test | |
| 13:27:41 | sean-k-mooney | then you should jsut set the api version in the fxiture defintion | |
| 13:28:03 | auniyal | also as I could not find many sonple deve tagging functional test, so thoguth I create for all devices | |
| 13:28:29 | sean-k-mooney | not in the same patch | |
| 13:28:37 | auniyal | yes yes | |
| 13:28:39 | sean-k-mooney | the repoduce should only test the specific bug | |
| 13:29:03 | auniyal | one patch will be a general odule having all tagging test | |
| 13:29:10 | auniyal | and then reproducer | |
| 13:29:43 | auniyal | but yeah, if could write either one first next should be easy | |
| 13:30:09 | auniyal | so update api version in OSAPIFixture ? | |
| 13:30:42 | sean-k-mooney | that or do self.api.microverion=2.43 | |
| 13:30:49 | sean-k-mooney | i generally woudl do that instead | |
| 13:31:00 | auniyal | ack | |
| 13:31:47 | sean-k-mooney | like this https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/tests/functional/regressions/test_bug_1806064.py#L54-L57 | |
| 13:34:47 | auniyal | sean-k-mooney, still same error, "tag was unexpexted" | |
| 13:36:19 | auniyal | so I removed api_versiona and microversion at class level, and add self.api.microverion=2.43 at setup | |
| 13:36:32 | auniyal | after defining self.api | |
| 13:40:59 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 13:53:26 | stephenfin | bauzas, sean-k-mooney: I addressed merge conflicts and removed the job from the gate pipeline ^ | |
| 14:00:11 | bauzas | stephenfin: saw it, you're great | |
| 14:00:26 | bauzas | I definitely want to merge your patch because of the effort you made | |
| 14:10:27 | bauzas | stephenfin: could you make it non-voting at first ? | |
| 14:12:16 | stephenfin | I'd rather not. There's no point in non-voting jobs because no one looks at them. | |
| 14:12:35 | stephenfin | If it breaks our gate and we need to get stuff merged, we can simply disable it until we get around to fixing it. However, outside of a release week, a regression with our SQLAlchemy 2.x support should be prevented from merging. | |
| 14:14:16 | dansmith | then we can just wait until after rc1 to merge? | |
| 14:15:44 | bauzas | we agreed on the Bobcat support envelope | |
| 14:15:46 | bauzas | which is 1.4 | |
| 14:16:18 | bauzas | ideally new-style, hence https://review.opendev.org/c/openstack/nova/+/860829/ being actively reviewed | |
| 14:16:36 | bauzas | so, yeah, I'm ok with awaiting RC1 to be tagged for the zuul job | |
| 14:17:04 | bauzas | yup | |
| 14:26:18 | stephenfin | ack | |
| 14:27:25 | bauzas | stephenfin: I'll keep my +2 but you can ping me once we deliver RC1 | |
| 14:27:41 | bauzas | I'll add this zuul patch into our RC tracking etherpad | |
| 14:27:46 | bauzas | wfy ? | |
| 14:27:51 | stephenfin | sure | |
| 14:28:09 | greatgatsby_ | Hello. Our host aggregates seem to get out of sync with our provider aggregates. There seems to be a `nova-manage placement sync_aggregates` command, but we're confused how they're getting out of sync in the first place. Any suggestions of what could be the cause? This is deployed via kolla-ansible yoga | |
| 14:28:30 | greatgatsby_ | we don't have any idea even where to start looking. Searching logs didn't uncover anything | |
| 14:29:10 | dansmith | greatgatsby_: is the sync command fixing them? | |
| 14:29:47 | greatgatsby_ | it does fix them, but then we'll notice they're suddenly out of sync again. The last time we lost all the provider aggs and couldn't spin up VMs | |
| 14:30:16 | dansmith | suddenly out of sync? in what way? | |
| 14:30:26 | dansmith | they should only change if you add/remove compute nodes from aggregates | |
| 14:30:52 | greatgatsby_ | the host aggs do not match the provider aggs. They do initially, then they keep going out of sync | |
| 14:31:01 | greatgatsby_ | we're not removing nodes | |
| 14:31:20 | dansmith | there is nothing to synchronize if you're not changing the aggregates... | |
| 14:31:52 | dansmith | so I'm not sure what could be going on.. are you doing DB replication and perhaps something is undoing the sync? | |
| 14:32:06 | greatgatsby_ | so the aggs that the sync command sync go out of sync with us (to the best of our knowledge) not doing anything to cause that | |
| 14:33:00 | greatgatsby_ | DB is just setup how kolla-ansible sets it up, we're pretty hands off with the DB right now | |
| 14:33:07 | dansmith | to be clear, the synchronization is 1. make sure all nova aggs exist in placement and 2. make sure host assignments to aggs is mirrored from nova to placement | |
| 14:33:18 | dansmith | so if you're not changing aggs or mappings, there is nothing changing that needs to be sync'd | |
| 14:33:32 | dansmith | presumably nova looks right and placement looks wrong? | |
| 14:34:11 | greatgatsby_ | I agree with that, except we lose the placement aggs. They start of ok, we even have a script that compares the sync, it's all ok for a few days, then some (or all) of the placement aggs get dropped | |
| 14:34:18 | greatgatsby_ | I hope I'm using the correct terminology | |
| 14:34:33 | dansmith | the aggs are disappearing or the mappings? | |
| 14:34:38 | dansmith | (or both)? | |
| 14:34:47 | greatgatsby_ | nova is always right, placement disappear | |
| 14:36:49 | tobias-urdin | interesting observation, stopping a nova-conductor should call ConductorManager's stop() and wait() which isn't implemented or am I blind? the nova.service part calls that. If I stop nova-conductor while nova-compute is mid report_state or doing a periodic task it would fail with MessageTimeout because in nova-conductor we don't finish processing | |
| 14:36:49 | tobias-urdin | before stopping the service? for example we dont stop(), process things until queue is empty (calling wait method)...? perhaps somebody knows before I go digging | |
| 14:36:59 | greatgatsby_ | so when we do `openstack aggregate show <agg-name>` that always shows the correct hosts. When we drill into the resource provider aggregates, those go out of sync | |
| 14:38:54 | dansmith | greatgatsby_: biab, call | |
| 14:39:33 | greatgatsby_ | appreciate the help! I'm just kind of stuck trying to figure out where to even look to figure out what's causing this | |
| 14:48:00 | dansmith | tobias-urdin: yeah nova has no safe graceful shutdown procedure that lets RPC finish what it's doing, unfortunately | |