| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-01-13 | |||
| 10:25:28 | auniyal | https://paste.openstack.org/show/bL5nGTkXQjNgVMltv6fL/ | |
| 10:26:26 | auniyal | same msg isstill coming on compute restart | |
| 10:28:07 | gibi | yepp | |
| 10:29:23 | gibi | so there are two separate issues at least 1) we made that exception a hard fail since PCI in Placement as we too eagerly translate the exception 2) for some unknown reason in you env nova-compute cannot update the disk inventory at startup | |
| 10:34:56 | gibi | auniyal: could you put your nova-cpu.conf in paste.openstack.org? | |
| 10:37:00 | auniyal | here https://paste.openstack.org/show/bqpnfND7kpB29US0UA6P/ | |
| 10:37:28 | gibi | thanks | |
| 10:39:33 | gibi | OK so this is a compute with rbd configured | |
| 10:40:14 | auniyal | can't create new VM - https://paste.openstack.org/show/b2TQPF1lLwbFj3QZsaeC/ | |
| 10:40:39 | gibi | for the 2) I assume, but no proof yet, that for some reason the compute tries to modify the total DISK_GB total inventory to a value that is smaller than the current usage in placement causing placement to reject the change | |
| 10:42:19 | gibi | in the last log you pasted, it seems that the compute try to report an inventory without any DISK_GB meaning that any existing DISK_GB inventory is deleted, and placement rejects that as there is still usage from the RC | |
| 10:43:40 | gibi | is this a devstack that built with rbd in the first place or was it built with local disk first and then reconfigured with rbd on after? | |
| 10:44:12 | auniyal | no reconfiguration, only ceph/rbd | |
| 10:48:51 | gibi | https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L9456-L9473 this is the code that causes that DISK_GB is reported. | |
| 10:49:31 | gibi | so for some reason disk_gb is 0 there | |
| 10:49:52 | gibi | but you could try to prove that by adding some extra logs around that code | |
| 10:50:39 | gibi | that disk_gb is filled in https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L8306-L8307 in case of rbd | |
| 10:50:51 | gibi | based on the info from the rbd pool | |
| 10:53:09 | gibi | so you should check what the rbd reports for the vms pool | |
| 10:58:22 | auniyal | https://paste.openstack.org/show/b6WyabNygyB5hiTBUGdN/ | |
| 10:58:40 | auniyal | in between delimeters -- ## | |
| 11:04:59 | gibi | yepp that proofs that compute tries to remove the DISK_GB inventory in placement due to getting 0 total available disk from rbd | |
| 11:05:41 | gibi | so you should check your the vms rbd ceph pool in you deployment | |
| 11:05:50 | gibi | why it is reporting 0 disk | |
| 11:14:34 | auniyal | sudo rbd ls volumes hangs | |
| 11:16:55 | auniyal | gibi - https://paste.openstack.org/show/bzYaK4id1g94RyidGpcU/ | |
| 11:17:18 | auniyal | no keyring found | |
| 11:22:31 | auniyal | https://paste.openstack.org/show/bfgQbk0ZCHeDPzszszSj/ | |
| 12:13:56 | gibi | OK. so you have some ceph issues. | |
| 12:14:13 | gibi | I suggest to ping storage folks to get help debugging it | |
| 12:26:49 | gibi | I filed a separate bug for the missleading error message conflating DISK_GB with PCI in Placement https://bugs.launchpad.net/nova/+bug/2093879 | |
| 12:30:51 | gibi | bauzas: could you check my ping from this morning regarding the spec freeze exception? | |
| 12:33:29 | opendevreview | Balazs Gibizer proposed openstack/placement stable/2023.2: Add round-robin candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938947 | |
| 12:41:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: [CI][nova-next]test with placement ac breadth-first https://review.opendev.org/c/openstack/nova/+/937275 | |
| 12:43:20 | bauzas | sorry I missed your ping | |
| 12:45:09 | bauzas | gibi: I just +Wd your spec | |
| 12:45:15 | gibi | bauzas: thanks | |
| 12:57:41 | opendevreview | Merged openstack/nova-specs master: Image properties in server show https://review.opendev.org/c/openstack/nova-specs/+/938910 | |
| 13:15:40 | opendevreview | Merged openstack/nova-specs master: vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/936775 | |
| 13:23:53 | auniyal | gibi ack, thanks | |
| 13:30:49 | PrzemekK | How to connect encrypted volume (LUKS) to different VM ? Is it just detach/atach ? | |
| 13:34:20 | sean-k-mooney | PrzemekK: that depend if the vm is in the same project and the user that issues the attach hass acesses to the encyption secret then i think that shoudl work | |
| 13:34:55 | sean-k-mooney | detach for encycped vs non encypted volumes is the same more or less | |
| 13:35:50 | opendevreview | Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova-specs/+/939091 | |
| 13:42:09 | PrzemekK | ok. Same project / user admin. I want backup via Commvault and strange error that unable create snapshot. It creates snapshot and attach it to backup machine | |
| 14:15:53 | sean-k-mooney | so an admin cannot retrive the secret from barbican, but if the admin user that is doign the backup is added as a member of the project. a generic user with the admin cannot access secrets stored in barbican | |
| 14:16:35 | sean-k-mooney | * is added as a member of the project it shoudl be able to retrive it. | |
| 14:16:56 | sean-k-mooney | *with the admin role cannot ... | |
| 14:17:55 | sean-k-mooney | PrzemekK: by the way commvaults backup soltuion is a purly out of tree impelmeation and may or may not be supproted by your openstack vendor or work with encypted volumes | |
| 14:18:33 | sean-k-mooney | if it relies on attaching the volume to a backup vm for exampel that wont work with Boot form volume guests | |
| 14:36:31 | dansmith | sean-k-mooney: did we have a bug for the iso+gpt thing or was it just something we stumbled upon? I can't find it in the bug tracker if so | |
| 14:37:22 | sean-k-mooney | we have a bug for generic iso support and after we got that working a new oslo.utils releasel with the gpt issue was cut | |
| 14:37:36 | sean-k-mooney | i can see if i can find the other bug if you want to at least track it as related | |
| 14:37:53 | dansmith | I found the generic iso problem, but it's way old | |
| 14:38:09 | dansmith | I mean, predates and is unrelated to the gpt thing | |
| 14:38:28 | sean-k-mooney | the multi format issue came out of https://review.opendev.org/c/openstack/nova/+/909611 | |
| 14:38:45 | dansmith | right | |
| 14:38:55 | sean-k-mooney | so i have comemnted on https://bugs.launchpad.net/nova/+bug/2054446 | |
| 14:39:20 | sean-k-mooney | that iso supprot in general was only partly broken i.e. isos that supprot block device booting stilll worked | |
| 14:39:34 | sean-k-mooney | so its not really the same thing but they are releated | |
| 14:39:44 | sean-k-mooney | i dont know if you want to file a spereate bug or not | |
| 14:39:54 | dansmith | yeah related by virtue of us causing that fix another issue | |
| 14:40:06 | dansmith | just wondering what to use for a related-bug on my fix patch.. I can file one with basically the details from the comments on the other patch, | |
| 14:40:14 | dansmith | but it will be pretty much just paperwork at that point | |
| 14:40:14 | sean-k-mooney | dansmith: the intoduction fo the gpt inspctor came out of the ironic cve is that correct | |
| 14:40:22 | dansmith | no | |
| 14:40:42 | sean-k-mooney | oh ok i tought it was motivated by that orginally. | |
| 14:40:56 | dansmith | it came out of the original thing, to get us to stop using raw for two things, it just wasn't critical for the qcow fix of course | |
| 14:41:11 | sean-k-mooney | dansmith: paperwork wise i would just tack it however you find simplest unless bauzas has a specific request of for a dedicate bug | |
| 14:41:22 | dansmith | yeah, so I was just going to say: | |
| 14:41:50 | dansmith | bauzas: this fix needs to get in: https://review.opendev.org/c/openstack/nova/+/931833 we noticed it because it caused a problem for the fix of another bug, but we never filed a bug for this specifically, | |
| 14:42:01 | dansmith | since it was just on the heels of landing the initial gpt detection support | |
| 14:42:17 | bauzas | dansmith: context ? | |
| 14:42:20 | dansmith | so if we need paperwork, please say, but if not, could you review? | |
| 14:42:35 | bauzas | want me to review ? | |
| 14:42:37 | bauzas | ack, ok | |
| 14:42:47 | bauzas | I already added myself for reviewing it indeed | |
| 14:42:57 | dansmith | thanks | |
| 14:43:22 | dansmith | I've got a tangled web of reviews for disk inspection across several projects and I need to start reducing that before I lose it :) | |
| 14:53:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: [CI][nova-next]test with placement ac breadth-first https://review.opendev.org/c/openstack/nova/+/937275 | |
| 15:02:29 | sean-k-mooney | gibi: ^ fyi see comment inline, im pretty sure peers is not the correct group | |
| 15:04:05 | sean-k-mooney | im gong to see if i can quickly update https://review.opendev.org/c/openstack/nova/+/933365 to adress the review feedback. if i cant get that to a mergable state today however im goign to have to ask someone else to take it over to unblock the requirements patch | |
| 15:04:45 | gibi | sean-k-mooney: thanksh | |
| 15:04:48 | gibi | sean-k-mooney: thank | |
| 15:04:51 | gibi | s | |
| 15:06:47 | opendevreview | Balazs Gibizer proposed openstack/nova master: [CI][nova-next]test with placement ac breadth-first https://review.opendev.org/c/openstack/nova/+/937275 | |
| 15:34:37 | opendevreview | sean mooney proposed openstack/nova master: [eventlet] update nova tests for eventlet 0.37.0 https://review.opendev.org/c/openstack/nova/+/933365 | |
| 15:35:57 | opendevreview | sean mooney proposed openstack/nova master: [eventlet] update nova tests for eventlet 0.38.2 https://review.opendev.org/c/openstack/nova/+/933365 | |
| 15:36:50 | sean-k-mooney | hberaud ^ i think that might be enough. the issue we had with the funcitoal test seam to have been resolved with the oslo.log release as well | |
| 15:38:37 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: DNM: Test eventlet bump https://review.opendev.org/c/openstack/nova/+/938879 | |
| 15:47:09 | sean-k-mooney | frickler: i rebased ^ to confirm if my other change is enogh but it works locally so i suspect it is | |
| 15:48:13 | sean-k-mooney | locally im emulating the test env by doing " .tox/functional/bin/python3 -m pip install -U -c https://opendev.org/openstack/requirements/raw/commit/27ded6c22e5f1ea9a16c8d3dabbd7fad72775f00/upper-constraints.txt -r requirements.txt -r test-requirements.txt eventlet" | |
| 15:48:25 | sean-k-mooney | that the url to the uc bump patch | |
| 16:43:50 | MengyangZhang[m] | sean-k-mooney: Hey sean, are you planning to talk about this https://review.opendev.org/c/openstack/nova-specs/+/932653 on Thur? I see it listed on the agenda of Thu's cinder-nova meeting. Just want to give a quick update about. I already brought it up to cinder last week and they gave the green light since there was no cinder changes needed. | |
| 16:43:51 | MengyangZhang[m] | I was wondering what's next step regarding submitting the proposed code changes. Since the spec freeze has passed I heard, would it be possible to follow the specless workflow? Happy to discuss it in the next nova meeting. | |
| 17:35:06 | sean-k-mooney | MengyangZhang[m]: so technially we are past the spec approval deadline for this release. it was last thrusday | |
| 17:36:21 | sean-k-mooney | so procedurely this would need a spec freeze excption. it might be better however to plan this for next cycle but start on the impementation in parallel | |
| 18:08:49 | sean-k-mooney | hi folks https://review.opendev.org/c/openstack/nova/+/933365 is green and so is https://review.opendev.org/c/openstack/nova/+/938879/2 so i think we are ok to proceed with merging that | |
| 18:09:20 | sean-k-mooney | gibi: melwitt: would ye be able to review https://review.opendev.org/c/openstack/nova/+/933365/3 | |
| 18:11:56 | melwitt | sean-k-mooney: did you ever find some reason that the new eventlet cause the waitall() calls to change? | |