| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-03 | |||
| 14:59:45 | atmark | these are duplicate entries https://paste.openstack.org/show/bBM1U8uQkSgpsNh6gLFU/ | |
| 15:00:03 | bauzas | reminder : nova meeting in 1 hour by now here | |
| 15:01:54 | dansmith | atmark: okay you'll want to find the instances that are assigned to the orphaned computes | |
| 15:02:12 | atmark | compute01's nova-compute.log https://paste.openstack.org/show/bGYG9FcesMGyFRgYH08x/ | |
| 15:02:12 | dansmith | I'm surprised to see that they have identical names | |
| 15:04:39 | opendevreview | Merged openstack/nova-specs master: Amend spec to add more details around encryption secrets https://review.opendev.org/c/openstack/nova-specs/+/887905 | |
| 15:06:02 | dansmith | huh okay I guess we can have duplicate hostnames in service records, even for the same binary | |
| 15:06:07 | atmark | dansmith: so I found the and tried to heal allocations but throws not descriptive `Error:` https://paste.openstack.org/show/bFSAcoAlJvZdpCrxzRyh/ | |
| 15:06:23 | atmark | found the instances* | |
| 15:06:33 | dansmith | atmark: yeah that's not going to help because they have different nodes in the DB as well | |
| 15:08:00 | dansmith | atmark: so the best thing you can do, if able, is to migrate the affected instances away from the duplicate services and then delete the dupes after they're empty | |
| 15:11:51 | atmark | Cold migrate? Getting `No server with a name or ID of '' exists. when doing live migrate | |
| 15:12:17 | dansmith | any migration | |
| 15:12:39 | dansmith | is that the whole error message? | |
| 15:13:29 | atmark | yes, that's from the openstack client | |
| 15:15:27 | opendevreview | Merged openstack/nova-specs master: Re-propose using extend volume completion action for 2024.1 https://review.opendev.org/c/openstack/nova-specs/+/895648 | |
| 15:15:58 | dansmith | like it's complaining about the instance id being unknown? | |
| 15:16:11 | dansmith | are you able to do a server show on that instance? | |
| 15:24:06 | atmark | dansmith: disregard the previous error, my for loop syntax to live migrate was wrong | |
| 15:24:20 | atmark | getting `Compute service of compute25.cnco1 is unavailable at this time.` | |
| 15:24:23 | dansmith | heh okay | |
| 15:24:46 | dansmith | atmark: is that just live or did you try cold as well? | |
| 15:24:53 | atmark | live | |
| 15:24:56 | atmark | let me try cold | |
| 15:25:43 | dansmith | so I think this might be because you've got these duplicated service records, so it's probably picking the wrong one.. usually what happens is people get duplicated *node* records, but the service record shows as up, so this works | |
| 15:26:25 | dansmith | picking the wrong one meaning it thinks the compute node is down even though it's not (obviously).. the migrating is what we need to correct the node mismatch, which you *also* have | |
| 15:26:34 | atmark | `Service is unavailable at this time.` for cold migrate | |
| 15:29:00 | dansmith | can you select out the host and node fields for one of the affected instances direct from the database? | |
| 15:35:22 | atmark | dansmith: https://paste.openstack.org/show/b0oiRoqWDHHgataIK7jO/ | |
| 15:37:40 | dansmith | atmark: and can you query out the relevant records from services and compute_nodes? | |
| 15:38:22 | dansmith | it's weird that the compute is having trouble finding the right compute node record (placement errors in the compute logs) and that the instance is being associated with the downed duplicate service record | |
| 15:38:29 | atmark | these are instances affected https://paste.openstack.org/show/bM6wA9HPHbuNzGmNEqPB/ | |
| 15:39:28 | dansmith | yeah so just a little background, we associate instances, nodes and services very loosely with these hostnames, which is terrible and fragile, as you're noticing.. work has been done recently to try to make this better (if that's any consolation) | |
| 15:43:13 | atmark | services and compute_nodes table https://paste.openstack.org/show/bujkfzse6gko5nV197ue/ | |
| 15:44:24 | dansmith | hmm, okay, so there is only one non-deleted service of each name right? | |
| 15:44:55 | dansmith | a record is deleted when there is a non-zero value in the deleted column | |
| 15:47:08 | dansmith | I'm not sure why you're seeing them in the services list through the openstack client in that case | |
| 15:47:31 | atmark | for compute01, I ran `openstack resource provider allocation delete $consumerid`, deleted the resource provider for the compute, restarted the nova compute to let it recreate resource provider entry a | |
| 15:47:58 | atmark | upon healing allocation, it wont just throws an error | |
| 15:48:13 | atmark | it just throws an non descriptive error* | |
| 15:48:34 | dansmith | yeah, well (a) you shouldn't have done that but (b) I think it's failing to create a new provider with the same name | |
| 15:49:12 | dansmith | also that has nothing to do with these records, FWIW | |
| 15:49:22 | dansmith | atmark: your compute nodes list doesn't show the deleted field, can you update with that field? | |
| 15:50:26 | atmark | https://paste.openstack.org/show/b6Ng2S3XwYQoL8G0mJvr/ with deleted field | |
| 15:52:57 | dansmith | so the uuid in our compute01 logs shows a node uuid of bf7a7e8e-c486-42b2-8dc4-00b2d5f70f86 | |
| 15:53:08 | dansmith | but I don't see that in your nodes list | |
| 15:53:15 | dansmith | can you see if any compute node has that uuid in the DB? | |
| 15:53:24 | dansmith | atmark: I guess I should also ask - do you have multiple compute cells? | |
| 15:53:49 | atmark | (a) - I followed what's in RH KB https://access.redhat.com/solutions/5308941. After the upgrade, nova-compute on those was throwing `Conflicting resource provider name` | |
| 15:54:33 | dansmith | are you running OSP? | |
| 15:55:03 | atmark | atmark: Yes, two cells: cnco1 and cnco2 but I only use cnco1 cell | |
| 15:55:35 | atmark | dansmith: No. I have kolla-ansible deployment | |
| 15:55:39 | dansmith | meaning there are no compute nodes in the other cell? but it's mapped? | |
| 15:56:08 | atmark | show databases https://paste.openstack.org/show/b26YFJxzw33hNLa9pVLw/ | |
| 15:56:22 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/837321 | |
| 15:56:39 | dansmith | re: cells I'm just trying to suss out if there are any duplicates happening because of that.. can you check the services table in the other cell to see if any of them have the same hostname of either of these affected ones just to be sure? | |
| 15:56:42 | atmark | nova_cnco1 is where all computes are mapped | |
| 15:57:29 | dansmith | atmark: we're going to have to pause here for the compute meeting in a few minutes but if you're still around after we can resume | |
| 15:57:53 | atmark | got it | |
| 15:57:57 | atmark | appreaciate the help | |
| 15:58:33 | atmark | re: can you check the services table in the other cell - nova_cell0? | |
| 15:59:57 | dansmith | no you said you have a cnco2 cell | |
| 16:00:01 | atmark | nova_cell0 https://paste.openstack.org/show/by9WvQgKARdZRppniRoY/ | |
| 16:00:06 | dansmith | cell0 is a different thing | |
| 16:00:09 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:09 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:09 | opendevmeet | Meeting started Tue Oct 3 16:00:09 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:09 | bauzas | #startmeeting nova | |
| 16:00:19 | bauzas | sorry guys, meeting by now | |
| 16:00:31 | elodilles | o/ | |
| 16:00:35 | dansmith | o/ | |
| 16:00:35 | bauzas | atmark: I can also help you once we're done with the meeting, shall be quick | |
| 16:00:50 | bauzas | heya folks | |
| 16:00:59 | opendevreview | Merged openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 16:01:08 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:38 | gibi | o/ | |
| 16:01:42 | bauzas | ok, I guess we can start | |
| 16:01:50 | bauzas | #topic Bugs (stuck/critical) | |
| 16:01:58 | Uggla | o/ | |
| 16:02:04 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:02:04 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 44 new untriaged bugs (-1 since the last meeting) | |
| 16:02:04 | bauzas | #info No Critical bug | |
| 16:02:10 | bauzas | gibi: happy with triaging some bugs ? | |
| 16:02:29 | bauzas | or do you want to skip it given your other prios ? | |
| 16:03:27 | gibi | bauzas: I can try | |
| 16:03:47 | bauzas | gibi: no worries, as I always say, it's a stretch goal, best-effort | |
| 16:04:12 | bauzas | and fwiw, in general when it's me, the number of bugs is raising :blushes: :facepalm: | |
| 16:04:31 | bauzas | but thanks, appreciated given the known situation | |
| 16:04:39 | bauzas | #info bug baton is gibi | |
| 16:04:43 | bauzas | moving on | |
| 16:04:47 | bauzas | #topic Gate status | |
| 16:04:52 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:04:56 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:05:02 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:05:04 | bauzas | all greens | |
| 16:05:08 | bauzas | nothing I spotted | |
| 16:05:26 | bauzas | that's a silent week and I appreciate it :) | |
| 16:05:41 | bauzas | any CI failures you guys spotted N? | |
| 16:06:03 | bauzas | (doh about the N?, thanks french keyboard layout) | |
| 16:06:23 | bauzas | looks nothing ditto | |