Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-09-13
14:29:10 dansmith greatgatsby_: is the sync command fixing them?
14:29:47 greatgatsby_ it does fix them, but then we'll notice they're suddenly out of sync again. The last time we lost all the provider aggs and couldn't spin up VMs
14:30:16 dansmith suddenly out of sync? in what way?
14:30:26 dansmith they should only change if you add/remove compute nodes from aggregates
14:30:52 greatgatsby_ the host aggs do not match the provider aggs. They do initially, then they keep going out of sync
14:31:01 greatgatsby_ we're not removing nodes
14:31:20 dansmith there is nothing to synchronize if you're not changing the aggregates...
14:31:52 dansmith so I'm not sure what could be going on.. are you doing DB replication and perhaps something is undoing the sync?
14:32:06 greatgatsby_ so the aggs that the sync command sync go out of sync with us (to the best of our knowledge) not doing anything to cause that
14:33:00 greatgatsby_ DB is just setup how kolla-ansible sets it up, we're pretty hands off with the DB right now
14:33:07 dansmith to be clear, the synchronization is 1. make sure all nova aggs exist in placement and 2. make sure host assignments to aggs is mirrored from nova to placement
14:33:18 dansmith so if you're not changing aggs or mappings, there is nothing changing that needs to be sync'd
14:33:32 dansmith presumably nova looks right and placement looks wrong?
14:34:11 greatgatsby_ I agree with that, except we lose the placement aggs. They start of ok, we even have a script that compares the sync, it's all ok for a few days, then some (or all) of the placement aggs get dropped
14:34:18 greatgatsby_ I hope I'm using the correct terminology
14:34:33 dansmith the aggs are disappearing or the mappings?
14:34:38 dansmith (or both)?
14:34:47 greatgatsby_ nova is always right, placement disappear
14:36:49 tobias-urdin before stopping the service? for example we dont stop(), process things until queue is empty (calling wait method)...? perhaps somebody knows before I go digging
14:36:49 tobias-urdin interesting observation, stopping a nova-conductor should call ConductorManager's stop() and wait() which isn't implemented or am I blind? the nova.service part calls that. If I stop nova-conductor while nova-compute is mid report_state or doing a periodic task it would fail with MessageTimeout because in nova-conductor we don't finish processing
14:36:59 greatgatsby_ so when we do `openstack aggregate show <agg-name>` that always shows the correct hosts. When we drill into the resource provider aggregates, those go out of sync
14:38:54 dansmith greatgatsby_: biab, call
14:39:33 greatgatsby_ appreciate the help! I'm just kind of stuck trying to figure out where to even look to figure out what's causing this
14:48:00 dansmith tobias-urdin: yeah nova has no safe graceful shutdown procedure that lets RPC finish what it's doing, unfortunately
14:48:42 dansmith greatgatsby_: so again, the aggregates themselves are there, it's the host mappings (i.e. which hosts are in which aggregate) that is wrong/
14:49:58 dansmith greatgatsby_: I think you need to look at the placement access log to see if there is something doing aggregate operations in between it being in sync and being out of sync.. to narrow it down to some sort of DB issue or some external actor un-syncing your aggregates
14:50:16 dansmith greatgatsby_: I'm assuming you're just using normal libvirt and not anything like ironic?
14:50:41 greatgatsby_ correct. We have dev and prod environments, and on prod, 2 hosts were removed from a placement agg but the other 20 or so are still fine. AFAIK nothing unique was done to those computes
14:51:26 greatgatsby_ I grepped through the placement logs for the agg uuid or anything related to aggs and didn't see anything. I'll dig some more there though
14:52:05 greatgatsby_ we're not using ironic
14:52:06 tobias-urdin dansmith: ack good to know, then it's not just me being blind :) out of curiosity does that apply to nova-compute as well, a service stop could interrupt building an instance? (not something one should do but is it possible)
14:54:00 dansmith greatgatsby_: okay, also grep for any operations happening on the hosts that get removed, like if their provider is getting deleted and re-added or something
14:54:33 dansmith greatgatsby_: changing hostnames (which sometimes happens because of bad DNS) or other things could be causing the computes to delete and re-create their providers in placement, which would have the same effect
14:54:42 dansmith tobias-urdin: yep :/
14:55:31 dansmith tobias-urdin: for instance create, best is to disable the host and let it quiesce, but other things could still be started on that compute like a resize. we were just discussing making this better a bit ago
14:56:32 greatgatsby_ dansmith: thanks, I'll start looking for that
14:56:47 greatgatsby_ greatly appreciate the help!
14:57:47 dansmith greatgatsby_: fwiw, someone a few weeks ago mentioned the same sort of thing, but I never saw a RCA, so I'll be interested to hear the findings
14:58:31 dansmith that was more just out of sync one time and running the sync fixed it IIRC
14:58:44 dansmith certainly possible we've got a bug, but lots of people should be screaming if so
15:06:36 tobias-urdin dansmith: ack, thanks for the info! always nice to get educated on specifics
15:30:26 dansmith sean-k-mooney: were you going to fix this post check? https://review.opendev.org/c/openstack/nova/+/893540
15:45:15 greatgatsby_ dansmith: I'm going to do an hourly `openstack resource provider list` to a log file. Just to confirm, if something was deleting/re-creating the providers in placement, I would expect to see the uuid change?
15:46:41 dansmith greatgatsby_: not necessarily, although I think in yoga that's probably true
15:47:06 dansmith worth a shot
15:47:32 greatgatsby_ ok - if I find anything I'll be sure to comment back in here
15:48:39 opendevreview Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540
15:55:48 dansmith greatgatsby_: you might also just query the DB for resource providers for the "created_at" field and see if any of them look too recent
15:58:49 opendevreview Jay Faulkner proposed openstack/nova master: [ironic] Use openstacksdk version with shard support https://review.opendev.org/c/openstack/nova/+/894833
15:59:15 greatgatsby_ dansmith: excellent, thanks
16:18:16 gmann dansmith: test_evacuate.sh also need to be updated https://review.opendev.org/c/openstack/nova/+/893540/5/roles/run-evacuate-hook/files/test_negative_evacuate.sh#38
16:18:33 dansmith gmann: I'm not sure it does.. it seems to blow past the failed create just fine
16:19:30 gmann dansmith: but that will be run after negative evacuate test pass https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/tasks/main.yaml#L87
16:20:21 dansmith ah okay, I thought you meant the setup_evacuate_resources.sh which seems to be fine
16:20:28 dansmith but yeah got it
16:21:04 gmann k
16:21:36 bauzas we are one day before RC1 and still a lot of changes in flight mode :/
16:21:51 bauzas even if the gate is not flipping that much, that's still limbo dance
16:22:29 opendevreview Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540
16:22:48 dansmith bauzas: things needing review or just waiting to merge? your etherpad seemed like everything was pretty well on its way when I looked
16:22:59 sean-k-mooney dansmith: oh i forgot about that am not today but i can look at it tomorrow i guess
16:23:06 dansmith sean-k-mooney: already on it
16:23:18 sean-k-mooney oh ok thanks
16:27:08 bauzas dansmith: everything is accepted except the prelude (which is still on my laptop atm) so just a gate update
16:27:17 dansmith bacack
16:27:20 dansmith or ack even
16:27:35 bauzas bah ack
16:27:37 bauzas :p
16:34:31 opendevreview Sylvain Bauza proposed openstack/nova master: Add a Bobcat prelude section https://review.opendev.org/c/openstack/nova/+/894940
16:36:20 bauzas JayF: I'm lost in translation, does https://review.opendev.org/c/openstack/nova/+/894833/2 mean that ironic shards won't work until we deliver an openstacksdk release ?
16:36:28 bauzas if so, /me is a very sad panda
16:36:38 gmann I will also check etherpad if anything new change needed to merge,
16:39:22 JayF bauzas: johnthetubaguy and I are on a call right now trying to find alternatives
16:39:27 JayF bauzas: I assure you I am the saddest panda
16:40:19 bauzas gmann: prelude just arrived, but maybe we should remove the first bullet point items
16:42:08 bauzas JayF: we still have time to remove the shards from the highlights and remove the conf options, if you want MHO
16:42:23 bauzas or just revert the 3rd patch tbc
16:42:34 JayF that is the worst of all options, I think we have a chance of getting a cleaner fix
16:42:35 bauzas I can hold RC1 just for that
16:42:49 bauzas JayF: a clean fix can't be a client change
16:43:02 JayF we are trying to manipulate it to do what we want w/o the client change
16:43:06 bauzas it's a long-term solution, but this can't happen in the timeframe we hav
16:43:08 JayF that patch is up as the known-working example
16:43:57 bauzas well, the latest patchset is just a noop but yeah
16:44:24 gmann bauzas: noted, will wait for that
16:44:36 bauzas the fact is, I'm surprised we're facing this at the very end, but urgency requires me to do the post-mortem after RC1, not now
16:45:06 bauzas but I'd enjoy any very quick fix
16:45:08 dansmith bauzas: I think it makes sense to communicate the "not super tested" nature of it if we're going to leave it in the prelude, personally
16:45:11 bauzas trust me about it
16:45:25 dansmith given what we normally expect for validation of headline features like that
16:45:41 bauzas dansmith: yeah, and trust me, I can even remove it from the prelude
16:45:59 bauzas actually the highlights worry me much
16:46:12 bauzas since they will be used by the marketing folks for the marketing fest
16:46:32 bauzas fortunately, this isn't merged yet, I'm gonna put a -W until we clarify the state
16:54:10 bauzas JayF: I'll drop now for 2.5 hours but I'll come back
16:54:24 bauzas in case you have options
17:12:41 sean-k-mooney JayF: bauzas i think we use ironic client of rthe shard stuff no?
17:12:50 JayF bauzas: I am convinced a revert is the best path. I'd personally prefer even reverting the peer_list deprecation; but if we leave that deprecation in, can I get some assurance that we won't *remove* peer_list until sharding lands?
17:12:54 sean-k-mooney JayF: stephenfin had a seriese ot move ironic to the sdk
17:12:55 JayF sean-k-mooney: we use a mix of client and sdk

Earlier   Later