Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-09-13
14:51:26 greatgatsby_ I grepped through the placement logs for the agg uuid or anything related to aggs and didn't see anything. I'll dig some more there though
14:52:05 greatgatsby_ we're not using ironic
14:52:06 tobias-urdin dansmith: ack good to know, then it's not just me being blind :) out of curiosity does that apply to nova-compute as well, a service stop could interrupt building an instance? (not something one should do but is it possible)
14:54:00 dansmith greatgatsby_: okay, also grep for any operations happening on the hosts that get removed, like if their provider is getting deleted and re-added or something
14:54:33 dansmith greatgatsby_: changing hostnames (which sometimes happens because of bad DNS) or other things could be causing the computes to delete and re-create their providers in placement, which would have the same effect
14:54:42 dansmith tobias-urdin: yep :/
14:55:31 dansmith tobias-urdin: for instance create, best is to disable the host and let it quiesce, but other things could still be started on that compute like a resize. we were just discussing making this better a bit ago
14:56:32 greatgatsby_ dansmith: thanks, I'll start looking for that
14:56:47 greatgatsby_ greatly appreciate the help!
14:57:47 dansmith greatgatsby_: fwiw, someone a few weeks ago mentioned the same sort of thing, but I never saw a RCA, so I'll be interested to hear the findings
14:58:31 dansmith that was more just out of sync one time and running the sync fixed it IIRC
14:58:44 dansmith certainly possible we've got a bug, but lots of people should be screaming if so
15:06:36 tobias-urdin dansmith: ack, thanks for the info! always nice to get educated on specifics
15:30:26 dansmith sean-k-mooney: were you going to fix this post check? https://review.opendev.org/c/openstack/nova/+/893540
15:45:15 greatgatsby_ dansmith: I'm going to do an hourly `openstack resource provider list` to a log file. Just to confirm, if something was deleting/re-creating the providers in placement, I would expect to see the uuid change?
15:46:41 dansmith greatgatsby_: not necessarily, although I think in yoga that's probably true
15:47:06 dansmith worth a shot
15:47:32 greatgatsby_ ok - if I find anything I'll be sure to comment back in here
15:48:39 opendevreview Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540
15:55:48 dansmith greatgatsby_: you might also just query the DB for resource providers for the "created_at" field and see if any of them look too recent
15:58:49 opendevreview Jay Faulkner proposed openstack/nova master: [ironic] Use openstacksdk version with shard support https://review.opendev.org/c/openstack/nova/+/894833
15:59:15 greatgatsby_ dansmith: excellent, thanks
16:18:16 gmann dansmith: test_evacuate.sh also need to be updated https://review.opendev.org/c/openstack/nova/+/893540/5/roles/run-evacuate-hook/files/test_negative_evacuate.sh#38
16:18:33 dansmith gmann: I'm not sure it does.. it seems to blow past the failed create just fine
16:19:30 gmann dansmith: but that will be run after negative evacuate test pass https://github.com/openstack/nova/blob/master/roles/run-evacuate-hook/tasks/main.yaml#L87
16:20:21 dansmith ah okay, I thought you meant the setup_evacuate_resources.sh which seems to be fine
16:20:28 dansmith but yeah got it
16:21:04 gmann k
16:21:36 bauzas we are one day before RC1 and still a lot of changes in flight mode :/
16:21:51 bauzas even if the gate is not flipping that much, that's still limbo dance
16:22:29 opendevreview Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540
16:22:48 dansmith bauzas: things needing review or just waiting to merge? your etherpad seemed like everything was pretty well on its way when I looked
16:22:59 sean-k-mooney dansmith: oh i forgot about that am not today but i can look at it tomorrow i guess
16:23:06 dansmith sean-k-mooney: already on it
16:23:18 sean-k-mooney oh ok thanks
16:27:08 bauzas dansmith: everything is accepted except the prelude (which is still on my laptop atm) so just a gate update
16:27:17 dansmith bacack
16:27:20 dansmith or ack even
16:27:35 bauzas bah ack
16:27:37 bauzas :p
16:34:31 opendevreview Sylvain Bauza proposed openstack/nova master: Add a Bobcat prelude section https://review.opendev.org/c/openstack/nova/+/894940
16:36:20 bauzas JayF: I'm lost in translation, does https://review.opendev.org/c/openstack/nova/+/894833/2 mean that ironic shards won't work until we deliver an openstacksdk release ?
16:36:28 bauzas if so, /me is a very sad panda
16:36:38 gmann I will also check etherpad if anything new change needed to merge,
16:39:22 JayF bauzas: johnthetubaguy and I are on a call right now trying to find alternatives
16:39:27 JayF bauzas: I assure you I am the saddest panda
16:40:19 bauzas gmann: prelude just arrived, but maybe we should remove the first bullet point items
16:42:08 bauzas JayF: we still have time to remove the shards from the highlights and remove the conf options, if you want MHO
16:42:23 bauzas or just revert the 3rd patch tbc
16:42:34 JayF that is the worst of all options, I think we have a chance of getting a cleaner fix
16:42:35 bauzas I can hold RC1 just for that
16:42:49 bauzas JayF: a clean fix can't be a client change
16:43:02 JayF we are trying to manipulate it to do what we want w/o the client change
16:43:06 bauzas it's a long-term solution, but this can't happen in the timeframe we hav
16:43:08 JayF that patch is up as the known-working example
16:43:57 bauzas well, the latest patchset is just a noop but yeah
16:44:24 gmann bauzas: noted, will wait for that
16:44:36 bauzas the fact is, I'm surprised we're facing this at the very end, but urgency requires me to do the post-mortem after RC1, not now
16:45:06 bauzas but I'd enjoy any very quick fix
16:45:08 dansmith bauzas: I think it makes sense to communicate the "not super tested" nature of it if we're going to leave it in the prelude, personally
16:45:11 bauzas trust me about it
16:45:25 dansmith given what we normally expect for validation of headline features like that
16:45:41 bauzas dansmith: yeah, and trust me, I can even remove it from the prelude
16:45:59 bauzas actually the highlights worry me much
16:46:12 bauzas since they will be used by the marketing folks for the marketing fest
16:46:32 bauzas fortunately, this isn't merged yet, I'm gonna put a -W until we clarify the state
16:54:10 bauzas JayF: I'll drop now for 2.5 hours but I'll come back
16:54:24 bauzas in case you have options
17:12:41 sean-k-mooney JayF: bauzas i think we use ironic client of rthe shard stuff no?
17:12:50 JayF bauzas: I am convinced a revert is the best path. I'd personally prefer even reverting the peer_list deprecation; but if we leave that deprecation in, can I get some assurance that we won't *remove* peer_list until sharding lands?
17:12:54 sean-k-mooney JayF: stephenfin had a seriese ot move ironic to the sdk
17:12:55 JayF sean-k-mooney: we use a mix of client and sdk
17:13:09 sean-k-mooney ok but we didnt land the changes to move to sdk only
17:13:12 JayF sean-k-mooney: which is part of why this was complex to figure out what was going on
17:13:20 JayF https://etherpad.opendev.org/p/nova-sharding-rca I've created this
17:13:24 sean-k-mooney ok
17:13:27 JayF putting notes from our technical research for follow-ups
17:13:34 sean-k-mooney so are we using ironic client for this or not?
17:13:46 JayF all node fetches in Ironic for node listing use sdk
17:13:50 JayF most node updates/writes use client
17:13:52 bauzas JayF: so you basically wanna revert the whole series ?
17:13:53 JayF but that is a very rough line
17:14:11 sean-k-mooney lets not do that just yet
17:14:13 JayF bauzas: I think that's what we have to do. We'd need an SDK change, and there's a sniff of a bug in the Ironic API handling too, which is what pushed me to revert
17:14:23 sean-k-mooney if the specific call we need are functional then i woudl not revert
17:14:36 bauzas sean-k-mooney: we are on the edge of RC1
17:14:47 JayF the entire basis of ironic/nova node sharding is that we have the ability to query nodes limited by shard
17:14:51 bauzas and we highlighted to the marketing about the ironic features
17:14:54 JayF on ironic side we index on shard to make that a fast query
17:15:10 JayF and so if SDK can't add that shard query, doing late filtering or anything like that basically guts the purpose of the change
17:16:05 dansmith if there's any question, we should revert for sure
17:16:07 sean-k-mooney ok its using the sdk
17:16:09 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L616
17:16:23 JayF Yeah I've been on a zoom w/ johnthetubaguy troubleshooting this since like 7am local time, 3 hours
17:16:25 JayF and julia in here too
17:16:29 dansmith we should leave the deprecation in place but we probably need to adjust the text that says to use the other thing
17:16:46 JayF dansmith: please please lets not remove peer_list until shard lands though :/
17:16:49 bauzas dansmith: jay is even proposing to revert the whole stack
17:17:05 JayF bauzas: my care about peer_list is this: we need a single slurp release with peer_list + sharding for migration
17:17:06 sean-k-mooney we are not removing peer_list until D

Earlier   Later