| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-13 | |||
| 13:28:03 | auniyal | also as I could not find many sonple deve tagging functional test, so thoguth I create for all devices | |
| 13:28:29 | sean-k-mooney | not in the same patch | |
| 13:28:37 | auniyal | yes yes | |
| 13:28:39 | sean-k-mooney | the repoduce should only test the specific bug | |
| 13:29:03 | auniyal | one patch will be a general odule having all tagging test | |
| 13:29:10 | auniyal | and then reproducer | |
| 13:29:43 | auniyal | but yeah, if could write either one first next should be easy | |
| 13:30:09 | auniyal | so update api version in OSAPIFixture ? | |
| 13:30:42 | sean-k-mooney | that or do self.api.microverion=2.43 | |
| 13:30:49 | sean-k-mooney | i generally woudl do that instead | |
| 13:31:00 | auniyal | ack | |
| 13:31:47 | sean-k-mooney | like this https://github.com/openstack/nova/blob/53012f1c55072c42ced267a2b1adef0a669d9f45/nova/tests/functional/regressions/test_bug_1806064.py#L54-L57 | |
| 13:34:47 | auniyal | sean-k-mooney, still same error, "tag was unexpexted" | |
| 13:36:19 | auniyal | so I removed api_versiona and microversion at class level, and add self.api.microverion=2.43 at setup | |
| 13:36:32 | auniyal | after defining self.api | |
| 13:40:59 | opendevreview | Stephen Finucane proposed openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 13:53:26 | stephenfin | bauzas, sean-k-mooney: I addressed merge conflicts and removed the job from the gate pipeline ^ | |
| 14:00:11 | bauzas | stephenfin: saw it, you're great | |
| 14:00:26 | bauzas | I definitely want to merge your patch because of the effort you made | |
| 14:10:27 | bauzas | stephenfin: could you make it non-voting at first ? | |
| 14:12:16 | stephenfin | I'd rather not. There's no point in non-voting jobs because no one looks at them. | |
| 14:12:35 | stephenfin | If it breaks our gate and we need to get stuff merged, we can simply disable it until we get around to fixing it. However, outside of a release week, a regression with our SQLAlchemy 2.x support should be prevented from merging. | |
| 14:14:16 | dansmith | then we can just wait until after rc1 to merge? | |
| 14:15:44 | bauzas | we agreed on the Bobcat support envelope | |
| 14:15:46 | bauzas | which is 1.4 | |
| 14:16:18 | bauzas | ideally new-style, hence https://review.opendev.org/c/openstack/nova/+/860829/ being actively reviewed | |
| 14:16:36 | bauzas | so, yeah, I'm ok with awaiting RC1 to be tagged for the zuul job | |
| 14:17:04 | bauzas | yup | |
| 14:26:18 | stephenfin | ack | |
| 14:27:25 | bauzas | stephenfin: I'll keep my +2 but you can ping me once we deliver RC1 | |
| 14:27:41 | bauzas | I'll add this zuul patch into our RC tracking etherpad | |
| 14:27:46 | bauzas | wfy ? | |
| 14:27:51 | stephenfin | sure | |
| 14:28:09 | greatgatsby_ | Hello. Our host aggregates seem to get out of sync with our provider aggregates. There seems to be a `nova-manage placement sync_aggregates` command, but we're confused how they're getting out of sync in the first place. Any suggestions of what could be the cause? This is deployed via kolla-ansible yoga | |
| 14:28:30 | greatgatsby_ | we don't have any idea even where to start looking. Searching logs didn't uncover anything | |
| 14:29:10 | dansmith | greatgatsby_: is the sync command fixing them? | |
| 14:29:47 | greatgatsby_ | it does fix them, but then we'll notice they're suddenly out of sync again. The last time we lost all the provider aggs and couldn't spin up VMs | |
| 14:30:16 | dansmith | suddenly out of sync? in what way? | |
| 14:30:26 | dansmith | they should only change if you add/remove compute nodes from aggregates | |
| 14:30:52 | greatgatsby_ | the host aggs do not match the provider aggs. They do initially, then they keep going out of sync | |
| 14:31:01 | greatgatsby_ | we're not removing nodes | |
| 14:31:20 | dansmith | there is nothing to synchronize if you're not changing the aggregates... | |
| 14:31:52 | dansmith | so I'm not sure what could be going on.. are you doing DB replication and perhaps something is undoing the sync? | |
| 14:32:06 | greatgatsby_ | so the aggs that the sync command sync go out of sync with us (to the best of our knowledge) not doing anything to cause that | |
| 14:33:00 | greatgatsby_ | DB is just setup how kolla-ansible sets it up, we're pretty hands off with the DB right now | |
| 14:33:07 | dansmith | to be clear, the synchronization is 1. make sure all nova aggs exist in placement and 2. make sure host assignments to aggs is mirrored from nova to placement | |
| 14:33:18 | dansmith | so if you're not changing aggs or mappings, there is nothing changing that needs to be sync'd | |
| 14:33:32 | dansmith | presumably nova looks right and placement looks wrong? | |
| 14:34:11 | greatgatsby_ | I agree with that, except we lose the placement aggs. They start of ok, we even have a script that compares the sync, it's all ok for a few days, then some (or all) of the placement aggs get dropped | |
| 14:34:18 | greatgatsby_ | I hope I'm using the correct terminology | |
| 14:34:33 | dansmith | the aggs are disappearing or the mappings? | |
| 14:34:38 | dansmith | (or both)? | |
| 14:34:47 | greatgatsby_ | nova is always right, placement disappear | |
| 14:36:49 | tobias-urdin | interesting observation, stopping a nova-conductor should call ConductorManager's stop() and wait() which isn't implemented or am I blind? the nova.service part calls that. If I stop nova-conductor while nova-compute is mid report_state or doing a periodic task it would fail with MessageTimeout because in nova-conductor we don't finish processing | |
| 14:36:49 | tobias-urdin | before stopping the service? for example we dont stop(), process things until queue is empty (calling wait method)...? perhaps somebody knows before I go digging | |
| 14:36:59 | greatgatsby_ | so when we do `openstack aggregate show <agg-name>` that always shows the correct hosts. When we drill into the resource provider aggregates, those go out of sync | |
| 14:38:54 | dansmith | greatgatsby_: biab, call | |
| 14:39:33 | greatgatsby_ | appreciate the help! I'm just kind of stuck trying to figure out where to even look to figure out what's causing this | |
| 14:48:00 | dansmith | tobias-urdin: yeah nova has no safe graceful shutdown procedure that lets RPC finish what it's doing, unfortunately | |
| 14:48:42 | dansmith | greatgatsby_: so again, the aggregates themselves are there, it's the host mappings (i.e. which hosts are in which aggregate) that is wrong/ | |
| 14:49:58 | dansmith | greatgatsby_: I think you need to look at the placement access log to see if there is something doing aggregate operations in between it being in sync and being out of sync.. to narrow it down to some sort of DB issue or some external actor un-syncing your aggregates | |
| 14:50:16 | dansmith | greatgatsby_: I'm assuming you're just using normal libvirt and not anything like ironic? | |
| 14:50:41 | greatgatsby_ | correct. We have dev and prod environments, and on prod, 2 hosts were removed from a placement agg but the other 20 or so are still fine. AFAIK nothing unique was done to those computes | |
| 14:51:26 | greatgatsby_ | I grepped through the placement logs for the agg uuid or anything related to aggs and didn't see anything. I'll dig some more there though | |
| 14:52:05 | greatgatsby_ | we're not using ironic | |
| 14:52:06 | tobias-urdin | dansmith: ack good to know, then it's not just me being blind :) out of curiosity does that apply to nova-compute as well, a service stop could interrupt building an instance? (not something one should do but is it possible) | |
| 14:54:00 | dansmith | greatgatsby_: okay, also grep for any operations happening on the hosts that get removed, like if their provider is getting deleted and re-added or something | |
| 14:54:33 | dansmith | greatgatsby_: changing hostnames (which sometimes happens because of bad DNS) or other things could be causing the computes to delete and re-create their providers in placement, which would have the same effect | |
| 14:54:42 | dansmith | tobias-urdin: yep :/ | |
| 14:55:31 | dansmith | tobias-urdin: for instance create, best is to disable the host and let it quiesce, but other things could still be started on that compute like a resize. we were just discussing making this better a bit ago | |
| 14:56:32 | greatgatsby_ | dansmith: thanks, I'll start looking for that | |
| 14:56:47 | greatgatsby_ | greatly appreciate the help! | |
| 14:57:47 | dansmith | greatgatsby_: fwiw, someone a few weeks ago mentioned the same sort of thing, but I never saw a RCA, so I'll be interested to hear the findings | |
| 14:58:31 | dansmith | that was more just out of sync one time and running the sync fixed it IIRC | |
| 14:58:44 | dansmith | certainly possible we've got a bug, but lots of people should be screaming if so | |
| 15:06:36 | tobias-urdin | dansmith: ack, thanks for the info! always nice to get educated on specifics | |
| 15:30:26 | dansmith | sean-k-mooney: were you going to fix this post check? https://review.opendev.org/c/openstack/nova/+/893540 | |
| 15:45:15 | greatgatsby_ | dansmith: I'm going to do an hourly `openstack resource provider list` to a log file. Just to confirm, if something was deleting/re-creating the providers in placement, I would expect to see the uuid change? | |
| 15:46:41 | dansmith | greatgatsby_: not necessarily, although I think in yoga that's probably true | |
| 15:47:06 | dansmith | worth a shot | |
| 15:47:32 | greatgatsby_ | ok - if I find anything I'll be sure to comment back in here | |
| 15:48:39 | opendevreview | Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540 | |
| 15:55:48 | dansmith | greatgatsby_: you might also just query the DB for resource providers for the "created_at" field and see if any of them look too recent | |
| 15:58:49 | opendevreview | Jay Faulkner proposed openstack/nova master: [ironic] Use openstacksdk version with shard support https://review.opendev.org/c/openstack/nova/+/894833 | |
| 15:59:15 | greatgatsby_ | dansmith: excellent, thanks | |
| 16:18:16 | gmann | dansmith: test_evacuate.sh also need to be updated https://review.opendev.org/c/openstack/nova/+/893540/5/roles/run-evacuate-hook/files/test_negative_evacuate.sh#38 | |
| 16:18:33 | dansmith | gmann: I'm not sure it does.. it seems to blow past the failed create just fine | |
| 16:19:30 | gmann | dansmith: but that will be run after negative evacuate test pass https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/tasks/main.yaml#L87 | |
| 16:20:21 | dansmith | ah okay, I thought you meant the setup_evacuate_resources.sh which seems to be fine | |
| 16:20:28 | dansmith | but yeah got it | |
| 16:21:04 | gmann | k | |
| 16:21:36 | bauzas | we are one day before RC1 and still a lot of changes in flight mode :/ | |
| 16:21:51 | bauzas | even if the gate is not flipping that much, that's still limbo dance | |
| 16:22:29 | opendevreview | Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540 | |
| 16:22:48 | dansmith | bauzas: things needing review or just waiting to merge? your etherpad seemed like everything was pretty well on its way when I looked | |
| 16:22:59 | sean-k-mooney | dansmith: oh i forgot about that am not today but i can look at it tomorrow i guess | |
| 16:23:06 | dansmith | sean-k-mooney: already on it | |
| 16:23:18 | sean-k-mooney | oh ok thanks | |
| 16:27:08 | bauzas | dansmith: everything is accepted except the prelude (which is still on my laptop atm) so just a gate update | |
| 16:27:17 | dansmith | bacack | |