| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-03 | |||
| 16:27:37 | bauzas | because we EOLd Train with pain | |
| 16:28:01 | bauzas | and I would want us to be a little more aggresive in terms of cleanup :) | |
| 16:28:22 | sean-k-mooney | i would have to check the resolution but i think everyting pre Wallaby should be removed under the propsal | |
| 16:28:31 | sean-k-mooney | but i dont recall off the top of my head | |
| 16:29:03 | sean-k-mooney | https://github.com/openstack/governance/blob/master/resolutions/20230724-unmaintained-branches.rst#transition | |
| 16:29:13 | sean-k-mooney | he last 3 active Extended Maintenance branches are automatically transitioned to Unmaintained branches. | |
| 16:29:49 | elodilles | ++ | |
| 16:29:56 | sean-k-mooney | so thats W,X,Y | |
| 16:30:08 | sean-k-mooney | unless im off by one which i could be | |
| 16:30:14 | elodilles | (this is the merged resolution: https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branches.html ) | |
| 16:30:49 | sean-k-mooney | so we dont need to fully figure this out now | |
| 16:31:09 | bauzas | as I said, I need to do bedtime readings | |
| 16:31:20 | bauzas | nothing urges | |
| 16:31:22 | sean-k-mooney | but we can prepare to EOL Ussuri and victoria and asses Wallaby | |
| 16:31:41 | elodilles | sean-k-mooney: i remember, X,W,V, hence, i'm not sure about Y :) | |
| 16:31:46 | bauzas | but yeah, EOLing up to Victoria is honestly a goal for me :) | |
| 16:32:08 | sean-k-mooney | eoling Victoria allows us to drop 18.04 | |
| 16:32:12 | sean-k-mooney | form all our ci | |
| 16:32:45 | elodilles | yeah, ussuri is the last 18.04 based series | |
| 16:33:00 | sean-k-mooney | so that i think has a lot of value | |
| 16:33:12 | bauzas | yeah | |
| 16:33:18 | dansmith | why? | |
| 16:33:25 | bauzas | in terms of CI | |
| 16:33:45 | dansmith | I mean I'm fine with EOLing V but is 18.04 adding a lot of pain I'm unaware of? | |
| 16:33:56 | bauzas | if nova stops supporting ussuri, that would pull the other projects to do the same | |
| 16:34:04 | sean-k-mooney | dansmith: 18.04 is EOL form a canonical point of view | |
| 16:34:10 | sean-k-mooney | so its not entirly unsecurity patched | |
| 16:34:38 | sean-k-mooney | it went out of supprot in march this year | |
| 16:35:03 | bauzas | worth adding it into a PTG topic then | |
| 16:35:08 | dansmith | okay just for that reason, fine, I just am not aware of any actual problems | |
| 16:35:10 | bauzas | I take the action | |
| 16:35:43 | sean-k-mooney | we dont get any similar benifts again until zed | |
| 16:36:14 | dansmith | bauzas: you said this meeting would be quick | |
| 16:36:18 | bauzas | ok, moving on | |
| 16:36:20 | bauzas | and yeah | |
| 16:36:26 | sean-k-mooney | zed droped centos 8 and python 3.6 | |
| 16:36:30 | bauzas | my bad, I opened a can | |
| 16:36:37 | elodilles | :D | |
| 16:36:45 | bauzas | #topic Open discussion | |
| 16:36:55 | bauzas | (nothing) | |
| 16:37:09 | bauzas | so, anything else or can I return you 23 mins of your time ? | |
| 16:37:20 | sean-k-mooney | not form me | |
| 16:37:25 | bauzas | thanks all | |
| 16:37:29 | bauzas | #endmeeting | |
| 16:37:29 | opendevmeet | Meeting ended Tue Oct 3 16:37:29 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:37:29 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.html | |
| 16:37:29 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.txt | |
| 16:37:29 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-10-03-16.00.log.html | |
| 16:37:59 | elodilles | thanks o/ | |
| 16:38:14 | dansmith | atmark: okay so, there should never be compute services in cell0.. that's why you're seeing the duplicates from the api I think, which we should resolve to start | |
| 16:38:41 | dansmith | atmark: I think you probably also have that node uuid in the compute_nodes table there | |
| 16:39:00 | dansmith | and that if you look, these two compute nodes are pointing at cell0 and not the proper cell, which explains your other issues | |
| 16:39:21 | sean-k-mooney | if it actully got that far it proably create teh placement RP which could cause name collisions if you repoint it ot cell1 | |
| 16:39:27 | dansmith | so can you check compute_nodes in cell0, and also check to see if there are any instances in cell0 (other than ones in the build state) | |
| 16:40:00 | dansmith | sean-k-mooney: yes and atmark has already deleted some of that, so we have more work to cleanup there | |
| 16:40:09 | sean-k-mooney | ack | |
| 16:43:14 | atmark | dansmith: i do have those two computes in nova_cell0 | |
| 16:43:28 | dansmith | atmark: okay, now things are starting to make sense :) | |
| 16:45:06 | auniyal | sean-k-mooney hopefully you noticed my request in meeting | |
| 16:45:09 | atmark | compute_tables in nova_cell0 https://paste.openstack.org/show/bYtEaxBqcGpSkDhThQY2/ | |
| 16:45:20 | atmark | compute_nodes* | |
| 16:45:25 | auniyal | if not please have a look when you have time | |
| 16:45:42 | dansmith | atmark: aha, there's the missing bf7* uuid | |
| 16:45:59 | sean-k-mooney | auniyal: you asked baout https://review.opendev.org/c/openstack/nova/+/885344? but it already has a +2w form dansmith | |
| 16:46:05 | sean-k-mooney | was there something else | |
| 16:46:14 | auniyal | yeah just noticed thanks dansmith | |
| 16:46:24 | auniyal | can you please look for others in topic | |
| 16:46:45 | sean-k-mooney | the others are in other branches | |
| 16:47:00 | sean-k-mooney | so after it has merged i can perhaps take another look | |
| 16:47:01 | atmark | dansmith: bf7* is new resource provider id for compute01. it used to be c017* | |
| 16:47:07 | auniyal | yes same patch backport for other stable branches | |
| 16:47:08 | sean-k-mooney | there is noting to do until the zed patch merges | |
| 16:47:11 | auniyal | till ussuri | |
| 16:47:45 | dansmith | atmark: it's the one I was looking for from the compute logs.. but it needs to be deleted and rebuilt entirely at this point | |
| 16:47:53 | auniyal | if you ad your +2 I'll look for seond later | |
| 16:48:17 | sean-k-mooney | ya so we can look at https://review.opendev.org/c/openstack/nova/+/885345 next but it will fail in the gate until the zed one is merged | |
| 16:48:21 | dansmith | atmark: so before we proceed, we need to check the instances table in there | |
| 16:49:08 | sean-k-mooney | maybe tomorrow can you ping me in the morning | |
| 16:49:15 | atmark | ok, one sec | |
| 16:49:16 | auniyal | ack, | |
| 16:49:19 | auniyal | tanks | |
| 16:49:38 | sean-k-mooney | i dont have time to load context and review before i finish for today but i can look at thing in the morning | |
| 16:50:24 | atmark | dansmith: cell0's instances table https://paste.openstack.org/show/bZuA2J0x09iTDI5JDmKH/ | |
| 16:50:44 | dansmith | atmark: okay excellent | |
| 16:50:53 | dansmith | that is both good and expected | |
| 16:51:39 | dansmith | atmark: so first thing to do is figure out what's wrong with those two computes. my guess is that during the upgrade, they got their transport_url pointed at the cell0/superconductor instead of the cell conductor | |
| 16:53:33 | atmark | I see find that in nova.conf right ? | |
| 16:53:44 | dansmith | yep | |
| 16:54:16 | atmark | the transport_url in nova.conf points to rabbitmq | |
| 16:55:17 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 16:56:00 | dansmith | atmark: right it always points at rabbit.. I'm not familiar with kolla but are you running two sets of conductors, one for the top-level and one for the cells? | |
| 16:56:12 | dansmith | easy way to tell is compare your transport_url to a compute that is not borked :) | |
| 16:56:13 | atmark | in the controller i can see transport_url points to nova_cnco1 vhost of rabbitmq | |
| 16:56:54 | dansmith | start with the transport_url on the computes, and compare one of the broken ones to a non-broken one | |
| 16:57:44 | atmark | dansmith: ok, the transport_url on broken computes doesn't have nova_cnco1 vhost | |
| 16:58:07 | dansmith | okay, that's the problem, so fix that and restart nova-compute there | |
| 17:01:14 | atmark | just a sec | |
| 17:14:15 | atmark | dansmith: done | |
| 17:15:05 | atmark | took a while, had to the push change through kolla-ansible | |
| 17:15:10 | dansmith | atmark: okay is it happier on startup now? there might be more work to clean up, but it might also have just worked it out | |