Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-01-10
16:51:49 dansmith I hate that we have to make more work, but I also think this is long overdue, so I'm glad for the excuse :)
16:57:23 gibi yeah this feels like a low effort high value change regardless of the vTPM work
16:57:42 opendevreview Balazs Gibizer proposed openstack/nova-specs master: Image properties in server show https://review.opendev.org/c/openstack/nova-specs/+/938910
16:57:55 gibi I've fixed the comments ^^
17:00:23 dansmith I've +2d :)
17:01:34 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.2: Add a global limit on the number of allocation candidates https://review.opendev.org/c/openstack/placement/+/938939
17:01:35 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.2: Factor out allocation candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938940
17:01:35 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.2: Add round-robin candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938941
17:01:38 gibi dansmith: thansk
17:01:43 gibi thanks even ()
17:01:51 gibi :)
17:03:04 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.1: Add a global limit on the number of allocation candidates https://review.opendev.org/c/openstack/placement/+/938942
17:03:04 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.1: Factor out allocation candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938943
17:03:05 opendevreview Balazs Gibizer proposed openstack/placement stable/2024.1: Add round-robin candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938944
17:04:15 opendevreview Balazs Gibizer proposed openstack/placement stable/2023.2: Add a global limit on the number of allocation candidates https://review.opendev.org/c/openstack/placement/+/938945
17:04:15 opendevreview Balazs Gibizer proposed openstack/placement stable/2023.2: Factor out allocation candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938946
17:04:16 opendevreview Balazs Gibizer proposed openstack/placement stable/2023.2: Add round-robin candidate generation strategy https://review.opendev.org/c/openstack/placement/+/938947
21:51:15 opendevreview Balazs Gibizer proposed openstack/nova-specs master: Image properties in server show https://review.opendev.org/c/openstack/nova-specs/+/938910
21:54:58 melwitt gibi: I had +W ^ but then saw in the backscroll this is supposed to wait for a spec freeze exception, so I pulled it from the gate
22:05:22 sean-k-mooney im also +2 on that too but ya we can chat about it in the team meeting on tuesday
22:26:01 opendevreview Ghanshyam proposed openstack/nova-specs master: Propose API policy service and manager role spec https://review.opendev.org/c/openstack/nova-specs/+/937650
#openstack-nova - 2025-01-11
07:57:25 gibi melwitt: sean-k-mooney: thanks, yeah I will formally ask for an exception on Monday from bauzas and we can discuss it on the meeting Tuesday
15:43:55 opendevreview Dmitriy Chubinidze proposed openstack/nova master: doc: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680
16:15:44 opendevreview Dmitriy Chubinidze proposed openstack/nova master: doc: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680
16:49:47 opendevreview Dmitriy Chubinidze proposed openstack/nova master: doc: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680
21:45:37 opendevreview Dmitriy Chubinidze proposed openstack/nova master: doc: nova service_user wrong auth_url https://review.opendev.org/c/openstack/nova/+/938680
#openstack-nova - 2025-01-13
04:55:50 opendevreview Takashi Kajinami proposed openstack/os-resource-classes master: Fix outdated envlist https://review.opendev.org/c/openstack/os-resource-classes/+/939018
05:04:15 opendevreview Takashi Kajinami proposed openstack/python-novaclient master: Remove environment for Python 3.8 https://review.opendev.org/c/openstack/python-novaclient/+/939022
05:11:14 opendevreview Takashi Kajinami proposed openstack/nova master: Drop environment for Python 3.8 https://review.opendev.org/c/openstack/nova/+/939027
05:37:55 opendevreview Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044
06:08:17 opendevreview Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044
07:18:02 opendevreview Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044
07:48:39 opendevreview Hemanth N proposed openstack/nova master: Add memory stats to compute monitor https://review.opendev.org/c/openstack/nova/+/939044
08:40:27 gibi bauzas: when you are up. I would like to formally ask for a spec freeze exception for https://review.opendev.org/c/openstack/nova-specs/+/938910
08:40:35 gibi This is a spin off of the vTPM live migration spec
08:41:09 gibi but the impact is moved out of it as a) it has API impact b) it is independently useful feature
08:41:30 gibi we already have plenty of core approval on it so it does not seems to controversial
08:43:28 gibi the assignee of the impl is a questionmark today but just because we want to give the opportunity to auniyal or ratailor to pick it up. But if it does not fit to their timeline for E then I can be the fallback to push the implementation. Fortunately it is a small impact one
08:44:09 gibi more context is in the comment thread https://review.opendev.org/c/openstack/nova-specs/+/938843/1#message-c65ec478107012ab4c141bd79812528a74f7761d
09:10:10 auniyal hey gibi, can you please have a look at https://bugs.launchpad.net/nova/+bug/2093869/, nova-place placement bug
09:10:51 ratailor gibi, I could try to do that.
09:11:57 gibi ratailor: cool. Thanks for stepping up
09:12:01 gibi auniyal: looking
09:38:37 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933
09:46:04 gibi auniyal: so this is a system with PCI in Placement enabled and after a while, even if the devstack sits idle the compute starts failing with the conflict exception
09:53:28 gibi auniyal: you wrote that "If we recreate resource provider, then new once does not have DISK_GB resource-class
09:53:31 gibi "
09:53:58 gibi does it mean you stopped the compute, deleted the RP tree, and then started the compute?
09:54:16 gibi did you have VMs running on this compute?
09:56:34 gibi do you just enabled PCI in Placement or also have some devices configured via [pci]device_spec config option?
09:56:56 auniyal not in right now- but I encountered the same issue, earlier (do not this VM-setup anymore )
09:56:56 auniyal that time, I had VM, which was failing to start usin nova cmd, - on checking compute logs I saw this error.
09:56:56 auniyal that time, I teventually endup deleting R-P and creating new,
09:57:17 auniyal > enabled PCI in Placement or also have some devices configured via [pci]device_spec config option?
09:57:17 auniyal no
09:57:30 auniyal both time, there is no PCI changes
09:58:16 auniyal or any update in any service confs, the system is as same as how devstack deploy it
09:59:07 auniyal only thing is, I have ceph enabled for storage
09:59:39 opendevreview Dmitriy Chubinidze proposed openstack/nova master: Adding link for RabbitMQ installation during nova deployment on controller node. https://review.opendev.org/c/openstack/nova/+/938702
10:03:06 auniyal >>> not in right now- but I encountered the same issue, earlier (do not this VM-setup anymore )
10:03:06 auniyal meant, I do not have that setup anymore, and this is a new setup having similar issue
10:17:22 gibi auniyal: so the error is coming from https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/compute/resource_tracker.py#L1360-L1370 Interstingly the comment in the code contradicts with the actual InventoryInUse exception in the logs. So I think the assumption that all the InventoryInUse exception from update_from_provider_tree is realted to PCI in Placement is wrong
10:18:13 gibi so that exception translation needs to be fixed to only react on real exceptions from the PCI in Placement codepath and do not react on other InventoryInUse exceptions.
10:18:35 gibi Still I'm wondering why the exception is raised in you env
10:19:01 gibi could you replace the raise exception.PlacementPciException(error=str(e))
10:19:09 auniyal yes, I was surprised seeing PlacementPciException !
10:19:10 gibi line with a raise e
10:19:25 gibi and try to start up the compute
10:20:04 auniyal ack, please give a min
10:23:56 auniyal gibi, https://paste.openstack.org/show/bqQZYiVU8NA5PdRxn7Rm/
10:24:54 auniyal compute went uo though, in compute service list
10:25:28 auniyal https://paste.openstack.org/show/bL5nGTkXQjNgVMltv6fL/
10:26:26 auniyal same msg isstill coming on compute restart
10:28:07 gibi yepp
10:29:23 gibi so there are two separate issues at least 1) we made that exception a hard fail since PCI in Placement as we too eagerly translate the exception 2) for some unknown reason in you env nova-compute cannot update the disk inventory at startup
10:34:56 gibi auniyal: could you put your nova-cpu.conf in paste.openstack.org?
10:37:00 auniyal here https://paste.openstack.org/show/bqpnfND7kpB29US0UA6P/
10:37:28 gibi thanks
10:39:33 gibi OK so this is a compute with rbd configured
10:40:14 auniyal can't create new VM - https://paste.openstack.org/show/b2TQPF1lLwbFj3QZsaeC/
10:40:39 gibi for the 2) I assume, but no proof yet, that for some reason the compute tries to modify the total DISK_GB total inventory to a value that is smaller than the current usage in placement causing placement to reject the change
10:42:19 gibi in the last log you pasted, it seems that the compute try to report an inventory without any DISK_GB meaning that any existing DISK_GB inventory is deleted, and placement rejects that as there is still usage from the RC
10:43:40 gibi is this a devstack that built with rbd in the first place or was it built with local disk first and then reconfigured with rbd on after?
10:44:12 auniyal no reconfiguration, only ceph/rbd
10:48:51 gibi https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L9456-L9473 this is the code that causes that DISK_GB is reported.
10:49:31 gibi so for some reason disk_gb is 0 there
10:49:52 gibi but you could try to prove that by adding some extra logs around that code
10:50:39 gibi that disk_gb is filled in https://github.com/openstack/nova/blob/a459467899d2b406aa8cf530ae481255eaf3c957/nova/virt/libvirt/driver.py#L8306-L8307 in case of rbd
10:50:51 gibi based on the info from the rbd pool
10:53:09 gibi so you should check what the rbd reports for the vms pool
10:58:22 auniyal https://paste.openstack.org/show/b6WyabNygyB5hiTBUGdN/
10:58:40 auniyal in between delimeters -- ##
11:04:59 gibi yepp that proofs that compute tries to remove the DISK_GB inventory in placement due to getting 0 total available disk from rbd
11:05:41 gibi so you should check your the vms rbd ceph pool in you deployment
11:05:50 gibi why it is reporting 0 disk
11:14:34 auniyal sudo rbd ls volumes hangs
11:16:55 auniyal gibi - https://paste.openstack.org/show/bzYaK4id1g94RyidGpcU/
11:17:18 auniyal no keyring found
11:22:31 auniyal https://paste.openstack.org/show/bfgQbk0ZCHeDPzszszSj/
12:13:56 gibi OK. so you have some ceph issues.

Earlier   Later