Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-10-03
16:31:09 bauzas as I said, I need to do bedtime readings
16:31:20 bauzas nothing urges
16:31:22 sean-k-mooney but we can prepare to EOL Ussuri and victoria and asses Wallaby
16:31:41 elodilles sean-k-mooney: i remember, X,W,V, hence, i'm not sure about Y :)
16:31:46 bauzas but yeah, EOLing up to Victoria is honestly a goal for me :)
16:32:08 sean-k-mooney eoling Victoria allows us to drop 18.04
16:32:12 sean-k-mooney form all our ci
16:32:45 elodilles yeah, ussuri is the last 18.04 based series
16:33:00 sean-k-mooney so that i think has a lot of value
16:33:12 bauzas yeah
16:33:18 dansmith why?
16:33:25 bauzas in terms of CI
16:33:45 dansmith I mean I'm fine with EOLing V but is 18.04 adding a lot of pain I'm unaware of?
16:33:56 bauzas if nova stops supporting ussuri, that would pull the other projects to do the same
16:34:04 sean-k-mooney dansmith: 18.04 is EOL form a canonical point of view
16:34:10 sean-k-mooney so its not entirly unsecurity patched
16:34:38 sean-k-mooney it went out of supprot in march this year
16:35:03 bauzas worth adding it into a PTG topic then
16:35:08 dansmith okay just for that reason, fine, I just am not aware of any actual problems
16:35:10 bauzas I take the action
16:35:43 sean-k-mooney we dont get any similar benifts again until zed
16:36:14 dansmith bauzas: you said this meeting would be quick
16:36:18 bauzas ok, moving on
16:36:20 bauzas and yeah
16:36:26 sean-k-mooney zed droped centos 8 and python 3.6
16:36:30 bauzas my bad, I opened a can
16:36:37 elodilles :D
16:36:45 bauzas #topic Open discussion
16:36:55 bauzas (nothing)
16:37:09 bauzas so, anything else or can I return you 23 mins of your time ?
16:37:20 sean-k-mooney not form me
16:37:25 bauzas thanks all
16:37:29 opendevmeet Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.log.html
16:37:29 opendevmeet Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.txt
16:37:29 opendevmeet Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.html
16:37:29 opendevmeet Meeting ended Tue Oct 3 16:37:29 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
16:37:29 bauzas #endmeeting
16:37:59 elodilles thanks o/
16:38:14 dansmith atmark: okay so, there should never be compute services in cell0.. that's why you're seeing the duplicates from the api I think, which we should resolve to start
16:38:41 dansmith atmark: I think you probably also have that node uuid in the compute_nodes table there
16:39:00 dansmith and that if you look, these two compute nodes are pointing at cell0 and not the proper cell, which explains your other issues
16:39:21 sean-k-mooney if it actully got that far it proably create teh placement RP which could cause name collisions if you repoint it ot cell1
16:39:27 dansmith so can you check compute_nodes in cell0, and also check to see if there are any instances in cell0 (other than ones in the build state)
16:40:00 dansmith sean-k-mooney: yes and atmark has already deleted some of that, so we have more work to cleanup there
16:40:09 sean-k-mooney ack
16:43:14 atmark dansmith: i do have those two computes in nova_cell0
16:43:28 dansmith atmark: okay, now things are starting to make sense :)
16:45:06 auniyal sean-k-mooney hopefully you noticed my request in meeting
16:45:09 atmark compute_tables in nova_cell0 https://paste.openstack.org/show/bYtEaxBqcGpSkDhThQY2/
16:45:20 atmark compute_nodes*
16:45:25 auniyal if not please have a look when you have time
16:45:42 dansmith atmark: aha, there's the missing bf7* uuid
16:45:59 sean-k-mooney auniyal: you asked baout https://review.opendev.org/c/openstack/nova/+/885344? but it already has a +2w form dansmith
16:46:05 sean-k-mooney was there something else
16:46:14 auniyal yeah just noticed thanks dansmith
16:46:24 auniyal can you please look for others in topic
16:46:45 sean-k-mooney the others are in other branches
16:47:00 sean-k-mooney so after it has merged i can perhaps take another look
16:47:01 atmark dansmith: bf7* is new resource provider id for compute01. it used to be c017*
16:47:07 auniyal yes same patch backport for other stable branches
16:47:08 sean-k-mooney there is noting to do until the zed patch merges
16:47:11 auniyal till ussuri
16:47:45 dansmith atmark: it's the one I was looking for from the compute logs.. but it needs to be deleted and rebuilt entirely at this point
16:47:53 auniyal if you ad your +2 I'll look for seond later
16:48:17 sean-k-mooney ya so we can look at https://review.opendev.org/c/openstack/nova/+/885345 next but it will fail in the gate until the zed one is merged
16:48:21 dansmith atmark: so before we proceed, we need to check the instances table in there
16:49:08 sean-k-mooney maybe tomorrow can you ping me in the morning
16:49:15 atmark ok, one sec
16:49:16 auniyal ack,
16:49:19 auniyal tanks
16:49:38 sean-k-mooney i dont have time to load context and review before i finish for today but i can look at thing in the morning
16:50:24 atmark dansmith: cell0's instances table https://paste.openstack.org/show/bZuA2J0x09iTDI5JDmKH/
16:50:44 dansmith atmark: okay excellent
16:50:53 dansmith that is both good and expected
16:51:39 dansmith atmark: so first thing to do is figure out what's wrong with those two computes. my guess is that during the upgrade, they got their transport_url pointed at the cell0/superconductor instead of the cell conductor
16:53:33 atmark I see find that in nova.conf right ?
16:53:44 dansmith yep
16:54:16 atmark the transport_url in nova.conf points to rabbitmq
16:55:17 opendevreview Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246
16:56:00 dansmith atmark: right it always points at rabbit.. I'm not familiar with kolla but are you running two sets of conductors, one for the top-level and one for the cells?
16:56:12 dansmith easy way to tell is compare your transport_url to a compute that is not borked :)
16:56:13 atmark in the controller i can see transport_url points to nova_cnco1 vhost of rabbitmq
16:56:54 dansmith start with the transport_url on the computes, and compare one of the broken ones to a non-broken one
16:57:44 atmark dansmith: ok, the transport_url on broken computes doesn't have nova_cnco1 vhost
16:58:07 dansmith okay, that's the problem, so fix that and restart nova-compute there
17:01:14 atmark just a sec
17:14:15 atmark dansmith: done
17:15:05 atmark took a while, had to the push change through kolla-ansible
17:15:10 dansmith atmark: okay is it happier on startup now? there might be more work to clean up, but it might also have just worked it out
17:15:18 dansmith by it I mean the compute service in its log
17:16:36 dansmith now that they're not pointing at the wrong transport (you fixed both right?) you can clear the service and node records from cell0's database which should resolve the service list duplication
17:16:52 dansmith if the computes are happy in the log, you probably need to do the placement heal part now
17:18:56 atmark yes, both are fixed. Still seeing `Failed to retrieve allocations for resource provider c017*` on compute01's nova-compute.log https://paste.openstack.org/show/b8j8iC0eV6TOK1QLVBfN/
17:19:00 atmark is that expected?
17:19:59 dansmith atmark: yeah, so what happened is you deleted some stuff, the confused computes re-created their providers with the wrong uuid (the new one in the cell0) and now that it's back, it wants to use/recreate the old one
17:20:23 dansmith atmark: so probably best to nuke the allocations *and* the RP for each, then run the heal and then get them started back up
17:20:49 dansmith it's why I said "you shouldn't have deleted the allocations" earlier, despite the doc saying it's appropriate in some circumstances
17:20:58 dansmith but it's cool, this is confusing and messed up obviously :)
17:24:22 atmark clear the service and node records from cell0's database - I'm safe to remove these records https://paste.openstack.org/show/bxfAMF8OBcUCz2GsHiin/ ?
17:26:05 atmark should I remove entry or just update the deleted field ?

Earlier   Later