| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-03 | |||
| 15:29:00 | dansmith | can you select out the host and node fields for one of the affected instances direct from the database? | |
| 15:35:22 | atmark | dansmith: https://paste.openstack.org/show/b0oiRoqWDHHgataIK7jO/ | |
| 15:37:40 | dansmith | atmark: and can you query out the relevant records from services and compute_nodes? | |
| 15:38:22 | dansmith | it's weird that the compute is having trouble finding the right compute node record (placement errors in the compute logs) and that the instance is being associated with the downed duplicate service record | |
| 15:38:29 | atmark | these are instances affected https://paste.openstack.org/show/bM6wA9HPHbuNzGmNEqPB/ | |
| 15:39:28 | dansmith | yeah so just a little background, we associate instances, nodes and services very loosely with these hostnames, which is terrible and fragile, as you're noticing.. work has been done recently to try to make this better (if that's any consolation) | |
| 15:43:13 | atmark | services and compute_nodes table https://paste.openstack.org/show/bujkfzse6gko5nV197ue/ | |
| 15:44:24 | dansmith | hmm, okay, so there is only one non-deleted service of each name right? | |
| 15:44:55 | dansmith | a record is deleted when there is a non-zero value in the deleted column | |
| 15:47:08 | dansmith | I'm not sure why you're seeing them in the services list through the openstack client in that case | |
| 15:47:31 | atmark | for compute01, I ran `openstack resource provider allocation delete $consumerid`, deleted the resource provider for the compute, restarted the nova compute to let it recreate resource provider entry a | |
| 15:47:58 | atmark | upon healing allocation, it wont just throws an error | |
| 15:48:13 | atmark | it just throws an non descriptive error* | |
| 15:48:34 | dansmith | yeah, well (a) you shouldn't have done that but (b) I think it's failing to create a new provider with the same name | |
| 15:49:12 | dansmith | also that has nothing to do with these records, FWIW | |
| 15:49:22 | dansmith | atmark: your compute nodes list doesn't show the deleted field, can you update with that field? | |
| 15:50:26 | atmark | https://paste.openstack.org/show/b6Ng2S3XwYQoL8G0mJvr/ with deleted field | |
| 15:52:57 | dansmith | so the uuid in our compute01 logs shows a node uuid of bf7a7e8e-c486-42b2-8dc4-00b2d5f70f86 | |
| 15:53:08 | dansmith | but I don't see that in your nodes list | |
| 15:53:15 | dansmith | can you see if any compute node has that uuid in the DB? | |
| 15:53:24 | dansmith | atmark: I guess I should also ask - do you have multiple compute cells? | |
| 15:53:49 | atmark | (a) - I followed what's in RH KB https://access.redhat.com/solutions/5308941. After the upgrade, nova-compute on those was throwing `Conflicting resource provider name` | |
| 15:54:33 | dansmith | are you running OSP? | |
| 15:55:03 | atmark | atmark: Yes, two cells: cnco1 and cnco2 but I only use cnco1 cell | |
| 15:55:35 | atmark | dansmith: No. I have kolla-ansible deployment | |
| 15:55:39 | dansmith | meaning there are no compute nodes in the other cell? but it's mapped? | |
| 15:56:08 | atmark | show databases https://paste.openstack.org/show/b26YFJxzw33hNLa9pVLw/ | |
| 15:56:22 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/837321 | |
| 15:56:39 | dansmith | re: cells I'm just trying to suss out if there are any duplicates happening because of that.. can you check the services table in the other cell to see if any of them have the same hostname of either of these affected ones just to be sure? | |
| 15:56:42 | atmark | nova_cnco1 is where all computes are mapped | |
| 15:57:29 | dansmith | atmark: we're going to have to pause here for the compute meeting in a few minutes but if you're still around after we can resume | |
| 15:57:53 | atmark | got it | |
| 15:57:57 | atmark | appreaciate the help | |
| 15:58:33 | atmark | re: can you check the services table in the other cell - nova_cell0? | |
| 15:59:57 | dansmith | no you said you have a cnco2 cell | |
| 16:00:01 | atmark | nova_cell0 https://paste.openstack.org/show/by9WvQgKARdZRppniRoY/ | |
| 16:00:06 | dansmith | cell0 is a different thing | |
| 16:00:09 | bauzas | #startmeeting nova | |
| 16:00:09 | opendevmeet | Meeting started Tue Oct 3 16:00:09 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:09 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:09 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:19 | bauzas | sorry guys, meeting by now | |
| 16:00:31 | elodilles | o/ | |
| 16:00:35 | bauzas | atmark: I can also help you once we're done with the meeting, shall be quick | |
| 16:00:35 | dansmith | o/ | |
| 16:00:50 | bauzas | heya folks | |
| 16:00:59 | opendevreview | Merged openstack/nova master: Add job to test with SQLAlchemy master (2.x) https://review.opendev.org/c/openstack/nova/+/886230 | |
| 16:01:08 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:38 | gibi | o/ | |
| 16:01:42 | bauzas | ok, I guess we can start | |
| 16:01:50 | bauzas | #topic Bugs (stuck/critical) | |
| 16:01:58 | Uggla | o/ | |
| 16:02:04 | bauzas | #info No Critical bug | |
| 16:02:04 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 44 new untriaged bugs (-1 since the last meeting) | |
| 16:02:04 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:02:10 | bauzas | gibi: happy with triaging some bugs ? | |
| 16:02:29 | bauzas | or do you want to skip it given your other prios ? | |
| 16:03:27 | gibi | bauzas: I can try | |
| 16:03:47 | bauzas | gibi: no worries, as I always say, it's a stretch goal, best-effort | |
| 16:04:12 | bauzas | and fwiw, in general when it's me, the number of bugs is raising :blushes: :facepalm: | |
| 16:04:31 | bauzas | but thanks, appreciated given the known situation | |
| 16:04:39 | bauzas | #info bug baton is gibi | |
| 16:04:43 | bauzas | moving on | |
| 16:04:47 | bauzas | #topic Gate status | |
| 16:04:52 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:04:56 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:05:02 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |
| 16:05:04 | bauzas | all greens | |
| 16:05:08 | bauzas | nothing I spotted | |
| 16:05:26 | bauzas | that's a silent week and I appreciate it :) | |
| 16:05:41 | bauzas | any CI failures you guys spotted N? | |
| 16:06:03 | bauzas | (doh about the N?, thanks french keyboard layout) | |
| 16:06:23 | bauzas | looks nothing ditto | |
| 16:06:28 | bauzas | that's cool \o/ | |
| 16:06:38 | bauzas | #topic Release Planning | |
| 16:06:45 | bauzas | #link https://releases.openstack.org/caracal/schedule.html | |
| 16:07:03 | bauzas | #info Nova deadlines aren't set yet until we agree on them at the PTG | |
| 16:07:11 | bauzas | #info Bobcat GA planned tomorrow | |
| 16:07:18 | bauzas | mostly a FYI | |
| 16:07:40 | bauzas | #info Specs can be reproposed for 2024.1 Caracal timeframe | |
| 16:07:55 | bauzas | I've been a round of reviews but I still need to do my homework on some | |
| 16:08:05 | bauzas | I've been doing* | |
| 16:08:13 | bauzas | (with the verb, it's better) | |
| 16:08:35 | bauzas | #info Caracal-1 milestone in 6 weeks | |
| 16:08:42 | bauzas | I don't expect any deadline around caracal-1 so hold my beer | |
| 16:09:24 | sean-k-mooney | well | |
| 16:09:47 | sean-k-mooney | the only "deadlien" we really had around m1 ins the past was a gentil | |
| 16:09:56 | sean-k-mooney | encurrament to please have your specs ready for review | |
| 16:10:10 | bauzas | yeah, for sure, good point | |
| 16:10:16 | sean-k-mooney | because many are on PTO for a lot of the time between m2 and m1 | |
| 16:10:28 | sean-k-mooney | so prehaps not a deadline | |
| 16:10:34 | sean-k-mooney | but i do want to do a spec review day | |
| 16:10:35 | bauzas | once we agree on the deadlines, be sure I'll probably tell this | |
| 16:10:44 | bauzas | yeah, sure too | |
| 16:10:46 | sean-k-mooney | somewhere between m1 and decemeber 13th ish | |
| 16:10:50 | bauzas | fwiw, I started looking at some specs | |
| 16:11:03 | bauzas | but we can align ourselves on a review day, this is helpful | |
| 16:11:12 | bauzas | and yeah about the calendar | |
| 16:11:22 | sean-k-mooney | when is m1 this release | |
| 16:11:27 | bauzas | for some unexpected reason, people leave around Christmas period, doh. | |