| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-03 | |||
| 13:15:30 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 13:15:35 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 13:15:35 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 13:15:36 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 13:15:37 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 13:15:38 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 13:15:39 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 13:15:40 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 13:15:40 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 13:15:41 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part) https://review.opendev.org/c/openstack/nova/+/860287 | |
| 13:15:41 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 13:15:42 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860288 | |
| 13:15:43 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 13:15:43 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 13:15:45 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (only API exception) https://review.opendev.org/c/openstack/nova/+/885752 | |
| 13:15:45 | opendevreview | ribaudr proposed openstack/nova master: Add share lock/unlock and restrict visibility https://review.opendev.org/c/openstack/nova/+/890340 | |
| 13:15:47 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (API) https://review.opendev.org/c/openstack/nova/+/850499 | |
| 13:15:47 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 13:15:49 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 13:15:49 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 13:15:51 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 13:15:51 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 13:15:53 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 13:15:53 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 13:15:55 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 13:15:55 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 13:15:57 | opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | |
| 13:31:36 | bauzas | sean-k-mooney: please try to avoid telling our contributors that we would have new linting enhancements until we have a consensus, please | |
| 13:32:20 | sean-k-mooney | bauzas: i didnt say we will i said i wanted to get them ready before the ptg | |
| 13:32:34 | bauzas | okay | |
| 13:33:06 | sean-k-mooney | i would very strongly like to do this however | |
| 13:34:44 | bauzas | tbc, I'm okay with a new project, my only concerns are around to Nova | |
| 13:35:10 | sean-k-mooney | im mainly concerned about nova and our deliverables | |
| 14:56:00 | atmark | Hello! Yesterday, I upgraded Victoria to Wallaby using kolla-ansible. Out of 48 computes, I found 2 computes duplicated entries in `openstack compute service list`. It won't let me delete rouge entry | |
| 14:57:51 | atmark | I tried deleting the resource allocations and tried to do heal allocations but it throws an error | |
| 14:58:56 | dansmith | atmark: won't let you delete service records because why? it thinks they have instances running on them? | |
| 14:59:03 | atmark | dansmith: yes | |
| 14:59:45 | atmark | these are duplicate entries https://paste.openstack.org/show/bBM1U8uQkSgpsNh6gLFU/ | |
| 15:00:03 | bauzas | reminder : nova meeting in 1 hour by now here | |
| 15:01:54 | dansmith | atmark: okay you'll want to find the instances that are assigned to the orphaned computes | |
| 15:02:12 | atmark | compute01's nova-compute.log https://paste.openstack.org/show/bGYG9FcesMGyFRgYH08x/ | |
| 15:02:12 | dansmith | I'm surprised to see that they have identical names | |
| 15:04:39 | opendevreview | Merged openstack/nova-specs master: Amend spec to add more details around encryption secrets https://review.opendev.org/c/openstack/nova-specs/+/887905 | |
| 15:06:02 | dansmith | huh okay I guess we can have duplicate hostnames in service records, even for the same binary | |
| 15:06:07 | atmark | dansmith: so I found the and tried to heal allocations but throws not descriptive `Error:` https://paste.openstack.org/show/bFSAcoAlJvZdpCrxzRyh/ | |
| 15:06:23 | atmark | found the instances* | |
| 15:06:33 | dansmith | atmark: yeah that's not going to help because they have different nodes in the DB as well | |
| 15:08:00 | dansmith | atmark: so the best thing you can do, if able, is to migrate the affected instances away from the duplicate services and then delete the dupes after they're empty | |
| 15:11:51 | atmark | Cold migrate? Getting `No server with a name or ID of '' exists. when doing live migrate | |
| 15:12:17 | dansmith | any migration | |
| 15:12:39 | dansmith | is that the whole error message? | |
| 15:13:29 | atmark | yes, that's from the openstack client | |
| 15:15:27 | opendevreview | Merged openstack/nova-specs master: Re-propose using extend volume completion action for 2024.1 https://review.opendev.org/c/openstack/nova-specs/+/895648 | |
| 15:15:58 | dansmith | like it's complaining about the instance id being unknown? | |
| 15:16:11 | dansmith | are you able to do a server show on that instance? | |
| 15:24:06 | atmark | dansmith: disregard the previous error, my for loop syntax to live migrate was wrong | |
| 15:24:20 | atmark | getting `Compute service of compute25.cnco1 is unavailable at this time.` | |
| 15:24:23 | dansmith | heh okay | |
| 15:24:46 | dansmith | atmark: is that just live or did you try cold as well? | |
| 15:24:53 | atmark | live | |
| 15:24:56 | atmark | let me try cold | |
| 15:25:43 | dansmith | so I think this might be because you've got these duplicated service records, so it's probably picking the wrong one.. usually what happens is people get duplicated *node* records, but the service record shows as up, so this works | |
| 15:26:25 | dansmith | picking the wrong one meaning it thinks the compute node is down even though it's not (obviously).. the migrating is what we need to correct the node mismatch, which you *also* have | |
| 15:26:34 | atmark | `Service is unavailable at this time.` for cold migrate | |
| 15:29:00 | dansmith | can you select out the host and node fields for one of the affected instances direct from the database? | |
| 15:35:22 | atmark | dansmith: https://paste.openstack.org/show/b0oiRoqWDHHgataIK7jO/ | |
| 15:37:40 | dansmith | atmark: and can you query out the relevant records from services and compute_nodes? | |
| 15:38:22 | dansmith | it's weird that the compute is having trouble finding the right compute node record (placement errors in the compute logs) and that the instance is being associated with the downed duplicate service record | |
| 15:38:29 | atmark | these are instances affected https://paste.openstack.org/show/bM6wA9HPHbuNzGmNEqPB/ | |
| 15:39:28 | dansmith | yeah so just a little background, we associate instances, nodes and services very loosely with these hostnames, which is terrible and fragile, as you're noticing.. work has been done recently to try to make this better (if that's any consolation) | |
| 15:43:13 | atmark | services and compute_nodes table https://paste.openstack.org/show/bujkfzse6gko5nV197ue/ | |
| 15:44:24 | dansmith | hmm, okay, so there is only one non-deleted service of each name right? | |
| 15:44:55 | dansmith | a record is deleted when there is a non-zero value in the deleted column | |
| 15:47:08 | dansmith | I'm not sure why you're seeing them in the services list through the openstack client in that case | |
| 15:47:31 | atmark | for compute01, I ran `openstack resource provider allocation delete $consumerid`, deleted the resource provider for the compute, restarted the nova compute to let it recreate resource provider entry a | |
| 15:47:58 | atmark | upon healing allocation, it wont just throws an error | |
| 15:48:13 | atmark | it just throws an non descriptive error* | |
| 15:48:34 | dansmith | yeah, well (a) you shouldn't have done that but (b) I think it's failing to create a new provider with the same name | |
| 15:49:12 | dansmith | also that has nothing to do with these records, FWIW | |
| 15:49:22 | dansmith | atmark: your compute nodes list doesn't show the deleted field, can you update with that field? | |
| 15:50:26 | atmark | https://paste.openstack.org/show/b6Ng2S3XwYQoL8G0mJvr/ with deleted field | |
| 15:52:57 | dansmith | so the uuid in our compute01 logs shows a node uuid of bf7a7e8e-c486-42b2-8dc4-00b2d5f70f86 | |
| 15:53:08 | dansmith | but I don't see that in your nodes list | |
| 15:53:15 | dansmith | can you see if any compute node has that uuid in the DB? | |
| 15:53:24 | dansmith | atmark: I guess I should also ask - do you have multiple compute cells? | |
| 15:53:49 | atmark | (a) - I followed what's in RH KB https://access.redhat.com/solutions/5308941. After the upgrade, nova-compute on those was throwing `Conflicting resource provider name` | |
| 15:54:33 | dansmith | are you running OSP? | |
| 15:55:03 | atmark | atmark: Yes, two cells: cnco1 and cnco2 but I only use cnco1 cell | |
| 15:55:35 | atmark | dansmith: No. I have kolla-ansible deployment | |
| 15:55:39 | dansmith | meaning there are no compute nodes in the other cell? but it's mapped? | |
| 15:56:08 | atmark | show databases https://paste.openstack.org/show/b26YFJxzw33hNLa9pVLw/ | |
| 15:56:22 | opendevreview | Elod Illes proposed openstack/nova stable/victoria: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/837321 | |
| 15:56:39 | dansmith | re: cells I'm just trying to suss out if there are any duplicates happening because of that.. can you check the services table in the other cell to see if any of them have the same hostname of either of these affected ones just to be sure? | |
| 15:56:42 | atmark | nova_cnco1 is where all computes are mapped | |
| 15:57:29 | dansmith | atmark: we're going to have to pause here for the compute meeting in a few minutes but if you're still around after we can resume | |
| 15:57:53 | atmark | got it | |
| 15:57:57 | atmark | appreaciate the help | |
| 15:58:33 | atmark | re: can you check the services table in the other cell - nova_cell0? | |
| 15:59:57 | dansmith | no you said you have a cnco2 cell | |
| 16:00:01 | atmark | nova_cell0 https://paste.openstack.org/show/by9WvQgKARdZRppniRoY/ | |