Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-03
16:37:20 sean-k-mooney not form me
16:37:25 bauzas thanks all
16:37:29 bauzas #endmeeting
16:37:29 opendevmeet Meeting ended Tue Oct 3 16:37:29 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:37:29 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.html
16:37:29 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.txt
16:37:29 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.log.html
16:37:59 elodilles thanks o/
16:38:14 dansmith atmark: okay so, there should never be compute services in cell0.. that's why you're seeing the duplicates from the api I think, which we should resolve to start
16:38:41 dansmith atmark: I think you probably also have that node uuid in the compute_nodes table there
16:39:00 dansmith and that if you look, these two compute nodes are pointing at cell0 and not the proper cell, which explains your other issues
16:39:21 sean-k-mooney if it actully got that far it proably create teh placement RP which could cause name collisions if you repoint it ot cell1
16:39:27 dansmith so can you check compute_nodes in cell0, and also check to see if there are any instances in cell0 (other than ones in the build state)
16:40:00 dansmith sean-k-mooney: yes and atmark has already deleted some of that, so we have more work to cleanup there
16:40:09 sean-k-mooney ack
16:43:14 atmark dansmith: i do have those two computes in nova_cell0
16:43:28 dansmith atmark: okay, now things are starting to make sense :)
16:45:06 auniyal sean-k-mooney hopefully you noticed my request in meeting
16:45:09 atmark compute_tables in nova_cell0 https://paste.openstack.org/show/bYtEaxBqcGpSkDhThQY2/
16:45:20 atmark compute_nodes*
16:45:25 auniyal if not please have a look when you have time
16:45:42 dansmith atmark: aha, there's the missing bf7* uuid
16:45:59 sean-k-mooney auniyal: you asked baout https://review.opendev.org/c/openstack/nova/+/885344? but it already has a +2w form dansmith
16:46:05 sean-k-mooney was there something else
16:46:14 auniyal yeah just noticed thanks dansmith
16:46:24 auniyal can you please look for others in topic
16:46:45 sean-k-mooney the others are in other branches
16:47:00 sean-k-mooney so after it has merged i can perhaps take another look
16:47:01 atmark dansmith: bf7* is new resource provider id for compute01. it used to be c017*
16:47:07 auniyal yes same patch backport for other stable branches
16:47:08 sean-k-mooney there is noting to do until the zed patch merges
16:47:11 auniyal till ussuri
16:47:45 dansmith atmark: it's the one I was looking for from the compute logs.. but it needs to be deleted and rebuilt entirely at this point
16:47:53 auniyal if you ad your +2 I'll look for seond later
16:48:17 sean-k-mooney ya so we can look at https://review.opendev.org/c/openstack/nova/+/885345 next but it will fail in the gate until the zed one is merged
16:48:21 dansmith atmark: so before we proceed, we need to check the instances table in there
16:49:08 sean-k-mooney maybe tomorrow can you ping me in the morning
16:49:15 atmark ok, one sec
16:49:16 auniyal ack,
16:49:19 auniyal tanks
16:49:38 sean-k-mooney i dont have time to load context and review before i finish for today but i can look at thing in the morning
16:50:24 atmark dansmith: cell0's instances table https://paste.openstack.org/show/bZuA2J0x09iTDI5JDmKH/
16:50:44 dansmith atmark: okay excellent
16:50:53 dansmith that is both good and expected
16:51:39 dansmith atmark: so first thing to do is figure out what's wrong with those two computes. my guess is that during the upgrade, they got their transport_url pointed at the cell0/superconductor instead of the cell conductor
16:53:33 atmark I see find that in nova.conf right ?
16:53:44 dansmith yep
16:54:16 atmark the transport_url in nova.conf points to rabbitmq
16:55:17 opendevreview Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
16:56:00 dansmith atmark: right it always points at rabbit.. I'm not familiar with kolla but are you running two sets of conductors, one for the top-level and one for the cells?
16:56:12 dansmith easy way to tell is compare your transport_url to a compute that is not borked :)
16:56:13 atmark in the controller i can see transport_url points to nova_cnco1 vhost of rabbitmq
16:56:54 dansmith start with the transport_url on the computes, and compare one of the broken ones to a non-broken one
16:57:44 atmark dansmith: ok, the transport_url on broken computes doesn't have nova_cnco1 vhost
16:58:07 dansmith okay, that's the problem, so fix that and restart nova-compute there
17:01:14 atmark just a sec
17:14:15 atmark dansmith: done
17:15:05 atmark took a while, had to the push change through kolla-ansible
17:15:10 dansmith atmark: okay is it happier on startup now? there might be more work to clean up, but it might also have just worked it out
17:15:18 dansmith by it I mean the compute service in its log
17:16:36 dansmith now that they're not pointing at the wrong transport (you fixed both right?) you can clear the service and node records from cell0's database which should resolve the service list duplication
17:16:52 dansmith if the computes are happy in the log, you probably need to do the placement heal part now
17:18:56 atmark yes, both are fixed. Still seeing `Failed to retrieve allocations for resource provider c017*` on compute01's nova-compute.log https://paste.openstack.org/show/b8j8iC0eV6TOK1QLVBfN/
17:19:00 atmark is that expected?
17:19:59 dansmith atmark: yeah, so what happened is you deleted some stuff, the confused computes re-created their providers with the wrong uuid (the new one in the cell0) and now that it's back, it wants to use/recreate the old one
17:20:23 dansmith atmark: so probably best to nuke the allocations *and* the RP for each, then run the heal and then get them started back up
17:20:49 dansmith it's why I said "you shouldn't have deleted the allocations" earlier, despite the doc saying it's appropriate in some circumstances
17:20:58 dansmith but it's cool, this is confusing and messed up obviously :)
17:24:22 atmark clear the service and node records from cell0's database - I'm safe to remove these records https://paste.openstack.org/show/bxfAMF8OBcUCz2GsHiin/ ?
17:26:05 atmark should I remove entry or just update the deleted field ?
17:26:48 dansmith atmark: you can just set the deleted field to id if you want (that's what service delete does) but really they should go away.. but fine to just mark as deleted to test if you want
17:40:25 opendevreview Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
17:47:20 atmark dansmith: For compute25 - I didn't have to do anything. compute01 - I deleted the RP, restarted nova-compute and then ran heal so issue is now resolved. I learned something new. Thanks a lot.
17:47:48 dansmith atmark: awesome, glad to hear it.. so you cleaned up the service *and* node records from cell0 right?
17:47:49 atmark If I unset alloactions instead of alloactions, will I face the same issue? https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html#solution
17:48:25 atmark dansmith: yes, I decided to remove the records instead of updating deleted field.
17:48:29 dansmith cool
17:48:52 dansmith atmark: not sure what you're asking about. unset .. "allocations instead of allocations"
17:49:31 atmark "allocation unset instead of allocation remove"
17:50:49 dansmith atmark: no I think it would have ended up the same way
17:56:38 atmark is it safe to remove these records in instances table in nova_cell0? https://paste.openstack.org/show/bP2VO4c7Gh6Tb4SIm5Wi/
17:59:19 dansmith yes on the compute nodes.. the instances were all deleted in the last dump, so they belong there and will be cleaned after an archive
17:59:29 dansmith instances *do* belong in cell0, if they failed the schedule
18:10:18 atmark got it. thanks again
18:19:58 opendevreview Merged openstack/nova stable/zed: Fix failed count for anti-affinity check https://review.opendev.org/c/openstack/nova/+/885344
19:30:23 opendevreview Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
#openstack-nova - 2023-10-04
10:03:38 auniyal Hi sean-k-mooney, reminder ping for https://review.opendev.org/q/topic:bug%252F1996732
10:05:24 auniyal can you also please quick lookon https://review.opendev.org/q/topic:bug%252F2008102 only 2 patches are pending to merge
10:16:55 auniyal thanks sean-k-mooney
11:58:17 opendevreview Merged openstack/nova stable/xena: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/882921
15:24:20 opendevreview Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
15:24:34 opendevreview Dan Smith proposed openstack/nova master: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
17:09:49 frickler anyone willing tp push a small status check URL fix over the edge? https://review.opendev.org/c/openstack/nova/+/895770
17:09:55 frickler *to
17:11:14 dansmith frickler: the new one gives me a 404
17:11:18 dansmith https://docs.openstack.org/nova/latest/admin/hw_machine_type.html
17:11:40 dansmith second new one seems right tho
17:15:58 frickler humm, I admit I only checked the second one because that's what I stumbled about and then just duplicated the change for the other URL
17:16:07 frickler let me try to find a proper URL
17:17:11 frickler ah, dashes instead of underscores, will amend, thx for noticing this

Earlier   Later