| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-01-13 | |||
| 07:18:02 | opendevreview | Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044 | |
| 07:48:39 | opendevreview | Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044 | |
| 08:40:27 | gibi | bauzas: when you are up. I would like to formally ask for a spec freeze exception for https://review.opendev.org/c/openstack/nova-specs/+/938910 | |
| 08:40:35 | gibi | This is a spin off of the vTPM live migration spec | |
| 08:41:09 | gibi | but the impact is moved out of it as a) it has API impact b) it is independently useful feature | |
| 08:41:30 | gibi | we already have plenty of core approval on it so it does not seems to controversial | |
| 08:43:28 | gibi | the assignee of the impl is a questionmark today but just because we want to give the opportunity to auniyal or ratailor to pick it up. But if it does not fit to their timeline for E then I can be the fallback to push the implementation. Fortunately it is a small impact one | |
| 08:44:09 | gibi | more context is in the comment thread https://review.opendev.org/c/openstack/nova-specs/+/938843/1#message-c65ec478107012ab4c141bd79812528a74f7761d | |
| 09:10:10 | auniyal | hey gibi, can you please have a look at https://bugs.launchpad.net/nova/+bug/2093869/, nova-place placement bug | |
| 09:10:51 | ratailor | gibi, I could try to do that. | |
| 09:11:57 | gibi | ratailor: cool. Thanks for stepping up | |
| 09:12:01 | gibi | auniyal: looking | |
| 09:38:37 | opendevreview | Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933 | |
| 09:46:04 | gibi | auniyal: so this is a system with PCI in Placement enabled and after a while, even if the devstack sits idle the compute starts failing with the conflict exception | |
| 09:53:28 | gibi | auniyal: you wrote that "If we recreate resource provider, then new once does not have DISK_GB resource-class | |
| 09:53:31 | gibi | " | |
| 09:53:58 | gibi | does it mean you stopped the compute, deleted the RP tree, and then started the compute? | |
| 09:54:16 | gibi | did you have VMs running on this compute? | |
| 09:56:34 | gibi | do you just enabled PCI in Placement or also have some devices configured via [pci]device_spec config option? | |
| 09:56:56 | auniyal | that time, I teventually endup deleting R-P and creating new, | |
| 09:56:56 | auniyal | that time, I had VM, which was failing to start usin nova cmd, - on checking compute logs I saw this error. | |
| 09:56:56 | auniyal | not in right now- but I encountered the same issue, earlier (do not this VM-setup anymore ) | |
| 09:57:17 | auniyal | no | |
| 09:57:17 | auniyal | > enabled PCI in Placement or also have some devices configured via [pci]device_spec config option? | |
| 09:57:30 | auniyal | both time, there is no PCI changes | |
| 09:58:16 | auniyal | or any update in any service confs, the system is as same as how devstack deploy it | |
| 09:59:07 | auniyal | only thing is, I have ceph enabled for storage | |
| 09:59:39 | opendevreview | Dmitriy Chubinidze proposed openstack/nova master: Adding link for RabbitMQ installation during nova deployment on controller node. https://review.opendev.org/c/openstack/nova/+/938702 | |
| 10:03:06 | auniyal | meant, I do not have that setup anymore, and this is a new setup having similar issue | |
| 10:03:06 | auniyal | >>> not in right now- but I encountered the same issue, earlier (do not this VM-setup anymore ) | |
| 10:17:22 | gibi | auniyal: so the error is coming from https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/compute/resource_tracker.py#L1360-L1370 Interstingly the comment in the code contradicts with the actual InventoryInUse exception in the logs. So I think the assumption that all the InventoryInUse exception from update_from_provider_tree is realted to PCI in Placement is wrong | |
| 10:18:13 | gibi | so that exception translation needs to be fixed to only react on real exceptions from the PCI in Placement codepath and do not react on other InventoryInUse exceptions. | |
| 10:18:35 | gibi | Still I'm wondering why the exception is raised in you env | |
| 10:19:01 | gibi | could you replace the raise exception.PlacementPciException(error=str(e)) | |
| 10:19:09 | auniyal | yes, I was surprised seeing PlacementPciException ! | |
| 10:19:10 | gibi | line with a raise e | |
| 10:19:25 | gibi | and try to start up the compute | |
| 10:20:04 | auniyal | ack, please give a min | |
| 10:23:56 | auniyal | gibi, https://paste.openstack.org/show/bqQZYiVU8NA5PdRxn7Rm/ | |
| 10:24:54 | auniyal | compute went uo though, in compute service list | |
| 10:25:28 | auniyal | https://paste.openstack.org/show/bL5nGTkXQjNgVMltv6fL/ | |
| 10:26:26 | auniyal | same msg isstill coming on compute restart | |
| 10:28:07 | gibi | yepp | |
| 10:29:23 | gibi | so there are two separate issues at least 1) we made that exception a hard fail since PCI in Placement as we too eagerly translate the exception 2) for some unknown reason in you env nova-compute cannot update the disk inventory at startup | |
| 10:34:56 | gibi | auniyal: could you put your nova-cpu.conf in paste.openstack.org? | |
| 10:37:00 | auniyal | here https://paste.openstack.org/show/bqpnfND7kpB29US0UA6P/ | |
| 10:37:28 | gibi | thanks | |
| 10:39:33 | gibi | OK so this is a compute with rbd configured | |
| 10:40:14 | auniyal | can't create new VM - https://paste.openstack.org/show/b2TQPF1lLwbFj3QZsaeC/ | |
| 10:40:39 | gibi | for the 2) I assume, but no proof yet, that for some reason the compute tries to modify the total DISK_GB total inventory to a value that is smaller than the current usage in placement causing placement to reject the change | |
| 10:42:19 | gibi | in the last log you pasted, it seems that the compute try to report an inventory without any DISK_GB meaning that any existing DISK_GB inventory is deleted, and placement rejects that as there is still usage from the RC | |
| 10:43:40 | gibi | is this a devstack that built with rbd in the first place or was it built with local disk first and then reconfigured with rbd on after? | |
| 10:44:12 | auniyal | no reconfiguration, only ceph/rbd | |
| 10:48:51 | gibi | https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L9456-L9473 this is the code that causes that DISK_GB is reported. | |
| 10:49:31 | gibi | so for some reason disk_gb is 0 there | |
| 10:49:52 | gibi | but you could try to prove that by adding some extra logs around that code | |
| 10:50:39 | gibi | that disk_gb is filled in https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L8306-L8307 in case of rbd | |
| 10:50:51 | gibi | based on the info from the rbd pool | |
| 10:53:09 | gibi | so you should check what the rbd reports for the vms pool | |
| 10:58:22 | auniyal | https://paste.openstack.org/show/b6WyabNygyB5hiTBUGdN/ | |
| 10:58:40 | auniyal | in between delimeters -- ## | |
| 11:04:59 | gibi | yepp that proofs that compute tries to remove the DISK_GB inventory in placement due to getting 0 total available disk from rbd | |
| 11:05:41 | gibi | so you should check your the vms rbd ceph pool in you deployment | |
| 11:05:50 | gibi | why it is reporting 0 disk | |
| 11:14:34 | auniyal | sudo rbd ls volumes hangs | |
| 11:16:55 | auniyal | gibi - https://paste.openstack.org/show/bzYaK4id1g94RyidGpcU/ | |
| 11:17:18 | auniyal | no keyring found | |
| 11:22:31 | auniyal | https://paste.openstack.org/show/bfgQbk0ZCHeDPzszszSj/ | |
| 12:13:56 | gibi | OK. so you have some ceph issues. | |
| 12:14:13 | gibi | I suggest to ping storage folks to get help debugging it | |
| 12:26:49 | gibi | I filed a separate bug for the missleading error message conflating DISK_GB with PCI in Placement https://bugs.launchpad.net/nova/+bug/2093879 | |
| 12:30:51 | gibi | bauzas: could you check my ping from this morning regarding the spec freeze exception? | |
| 12:33:29 | opendevreview | Balazs Gibizer proposed openstack/placement stable/2023.2: Add round-robin candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938947 | |
| 12:41:38 | opendevreview | Balazs Gibizer proposed openstack/nova master: [CI][nova-next]test with placement ac breadth-first https://review.opendev.org/c/openstack/nova/+/937275 | |
| 12:43:20 | bauzas | sorry I missed your ping | |
| 12:45:09 | bauzas | gibi: I just +Wd your spec | |
| 12:45:15 | gibi | bauzas: thanks | |
| 12:57:41 | opendevreview | Merged openstack/nova-specs master: Image properties in server show https://review.opendev.org/c/openstack/nova-specs/+/938910 | |
| 13:15:40 | opendevreview | Merged openstack/nova-specs master: vTPM live migration https://review.opendev.org/c/openstack/nova-specs/+/936775 | |
| 13:23:53 | auniyal | gibi ack, thanks | |
| 13:30:49 | PrzemekK | How to connect encrypted volume (LUKS) to different VM ? Is it just detach/atach ? | |
| 13:34:20 | sean-k-mooney | PrzemekK: that depend if the vm is in the same project and the user that issues the attach hass acesses to the encyption secret then i think that shoudl work | |
| 13:34:55 | sean-k-mooney | detach for encycped vs non encypted volumes is the same more or less | |
| 13:35:50 | opendevreview | Takashi Natsume proposed openstack/nova-specs master: Create specs directory for 2025.2 Flamingo https://review.opendev.org/c/openstack/nova-specs/+/939091 | |
| 13:42:09 | PrzemekK | ok. Same project / user admin. I want backup via Commvault and strange error that unable create snapshot. It creates snapshot and attach it to backup machine | |
| 14:15:53 | sean-k-mooney | so an admin cannot retrive the secret from barbican, but if the admin user that is doign the backup is added as a member of the project. a generic user with the admin cannot access secrets stored in barbican | |
| 14:16:35 | sean-k-mooney | * is added as a member of the project it shoudl be able to retrive it. | |
| 14:16:56 | sean-k-mooney | *with the admin role cannot ... | |
| 14:17:55 | sean-k-mooney | PrzemekK: by the way commvaults backup soltuion is a purly out of tree impelmeation and may or may not be supproted by your openstack vendor or work with encypted volumes | |
| 14:18:33 | sean-k-mooney | if it relies on attaching the volume to a backup vm for exampel that wont work with Boot form volume guests | |
| 14:36:31 | dansmith | sean-k-mooney: did we have a bug for the iso+gpt thing or was it just something we stumbled upon? I can't find it in the bug tracker if so | |
| 14:37:22 | sean-k-mooney | we have a bug for generic iso support and after we got that working a new oslo.utils releasel with the gpt issue was cut | |
| 14:37:36 | sean-k-mooney | i can see if i can find the other bug if you want to at least track it as related | |
| 14:37:53 | dansmith | I found the generic iso problem, but it's way old | |
| 14:38:09 | dansmith | I mean, predates and is unrelated to the gpt thing | |
| 14:38:28 | sean-k-mooney | the multi format issue came out of https://review.opendev.org/c/openstack/nova/+/909611 | |
| 14:38:45 | dansmith | right | |
| 14:38:55 | sean-k-mooney | so i have comemnted on https://bugs.launchpad.net/nova/+bug/2054446 | |
| 14:39:20 | sean-k-mooney | that iso supprot in general was only partly broken i.e. isos that supprot block device booting stilll worked | |
| 14:39:34 | sean-k-mooney | so its not really the same thing but they are releated | |