Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-11
21:24:32 jizaymes_ hey all. Having lots of issues getting nova-compute to start after upgrading from 2025.1 to 2025.2 -- get errors about 'Resource provider data migration failed fatally during startup'
21:26:20 jizaymes_ Resource provider data migration failed fatally during startup for node sc01.: nova.exception.ReshapeFailed: Resource provider inventory and allocation data migration failed: {"errors": [{"status": 403, "title": "Forbidden", "detail": "Access was denied to this resource.\n\n Policy does not allow placement:reshaper:reshape to be performed. ", "code": "placement.undefined_code", "request_id": "..."}]}
21:26:20 jizaymes_ AMD SEV support detected \n kernel doesn't support AMD SEV-ES \n Requesting provider tree reshape in order to move memory encryption context inventory from the root compute node provider sc01 to a child provider.
21:27:35 jizaymes_ if the kernel says its not enabled i'm not sure why it would try to move stuff under it, but in any case, adding that policy to placement yields a "this is already in the default policy" message so lost as to how where to look next.
#openstack-nova - 2025-11-12
01:48:07 opendevreview Steve Baker proposed openstack/nova master: Add VNC console support for the Ironic driver https://review.opendev.org/c/openstack/nova/+/942528
07:39:55 opendevreview Nicolai Ruckel proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/959682
07:41:55 opendevreview Rajesh Tailor proposed openstack/nova-specs master: Show finish_time field in instance action show https://review.opendev.org/c/openstack/nova-specs/+/929780
08:44:04 nicolairuckel sean-k-mooney, sorry for the delay. I didn't realize that my responses were just drafts until today. But I think I'm finally getting the hang of Gerrit.
08:44:29 sean-k-mooney nicolairuckel: :)
08:45:06 sean-k-mooney ya the fact they are nto sent until you hit reply is slightly difefnt then gitlab and github which send them by defualt unless you explcitly tell it your doign a review
08:45:57 nicolairuckel It's funny because I used to complain about their default because it creates so many notifications. :D
08:46:16 sean-k-mooney nicolairuckel: sound like at least localy you not whave a mostly working version.
08:46:30 sean-k-mooney nicolairuckel: ya i dnot use notifcation on github basically ever
08:46:54 sean-k-mooney its an entirly broken feature given the signal to noise ratio
08:48:17 sean-k-mooney nicolairuckel: so what im thinking currently is we fix nvram preservation for rebooo. we do the saem for vtpm
08:48:30 nicolairuckel Same for GitLab where people just ignore every email and you have to notify them separately if there are relevant updates to a merge request.
08:48:39 nicolairuckel sean-k-mooney, sounds good to me
08:48:49 sean-k-mooney then we proably will need a spec to fix cold migration and maybe shelve as i think that will need net new desigin work
08:49:01 nicolairuckel and then we can look into the cold migration separately as that seems a lot more complicated
08:49:04 nicolairuckel right
08:49:05 sean-k-mooney the cold migration part is actully not hard and might be a blueprint or bug
08:49:25 sean-k-mooney shelve is harder because we need to decised where we woudl store the tpm and nvram data while the vm is not on a host
08:49:58 nicolairuckel ah, I didn't look into that yet but that makes sense to me
08:50:10 sean-k-mooney for cold migration we just need to copy the file to the correct location and deal with deleting it when you confirm ro revert
08:51:23 nicolairuckel similar to the old version of my patch?
08:51:36 sean-k-mooney kind of but diferent
08:52:02 sean-k-mooney we will need to use the removeet filestyem supprot to copy the file between hosts rahter then within the same host
08:52:40 nicolairuckel I see
08:54:18 sean-k-mooney this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/remotefs.py#L172-L316
08:54:46 sean-k-mooney we just need to use create_dir and copy_file
08:55:05 sean-k-mooney and then that will use ssh or rsync to move it
08:55:12 sean-k-mooney depend9ign on which you ahve enabeld
08:55:44 sean-k-mooney we do that via https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L314-L351
08:56:19 sean-k-mooney for cold migraiton you will jsut pass the host paramater
08:57:21 nicolairuckel thanks, I will look into that
08:58:10 nicolairuckel then I'll update the commit message of my patch accordingly
08:58:55 sean-k-mooney ill see if i can test this again once you have the new verison with the release note up although proably not today
09:00:01 nicolairuckel ah, how do I do the release note? I wasn't able to find documentation on that yet.
09:00:26 sean-k-mooney given what you knwo know for this it woudl be good to sync with your coworker on updating https://review.opendev.org/c/openstack/nova/+/955657
09:01:08 sean-k-mooney ah its pretty simple to create one do tox -e venv reno new <some-string-with-dashes>
09:01:23 sean-k-mooney tox -e venv -- preserve-nvram
09:01:47 sean-k-mooney that will tempelate out a new file in the release notes folder
09:02:06 sean-k-mooney you can delete 99% of the yaml content but keep the --- and the fixes section
09:02:26 sean-k-mooney then just wirte a short operator focused descrtipion of what was fixed
09:02:46 sean-k-mooney you can then test it locally with tox -e releasenotes
09:03:44 sean-k-mooney https://docs.openstack.org/nova/latest/contributor/releasenotes.html
09:10:46 nicolairuckel we agreed on that this is a bugfix, right/
09:15:39 opendevreview Nicolai Ruckel proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/959682
09:15:54 nicolairuckel okay, then it should be ready for you to test now
09:41:23 sean-k-mooney nicolairuckel:just realiased what version of libvirt you were checkign agaisnt sorry
09:41:50 sean-k-mooney we only suppport libvirt 8.0.0 and above so we do not need to check if preserving the nvram is supproted
09:42:31 sean-k-mooney nicolairuckel: for the vtpm change this was needed because it need 8.9.0
09:42:42 sean-k-mooney but nvram only needs 2.3.0
09:44:26 nicolairuckel ah, then I can remove that
09:45:35 sean-k-mooney yes im also wonderign about these changes https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/functional/libvirt/test_uefi.py
09:45:59 sean-k-mooney is this a side effect of when you remvoed os.append(nvram)
09:46:37 sean-k-mooney when your testing the generation fo the xml https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/unit/virt/libvirt/test_config.py
09:46:47 sean-k-mooney you are assert ign the nvram element is present which is correct
09:47:50 sean-k-mooney it still feels like you have mroe changes tehn are required but i have not gone line by line to verify
09:48:26 nicolairuckel you're right about the first one for sure.
09:49:04 nicolairuckel Let me change that, the version check and run the tests locally again
09:49:54 sean-k-mooney ack. this is pretty close to correct at thsi point so we jus tneed to get the last few bit fixed up and we can proceed with ti and start backproting once we get someone else to review as well
09:51:55 nicolairuckel I guess I can remove everything to _may_keep_nvram then, right?
09:52:22 sean-k-mooney ya
09:52:53 sean-k-mooney what i woudl do is create a clean clone and apply just the https://review.opendev.org/c/openstack/nova/+/959682/14/nova/virt/libvirt/guest.py and https://review.opendev.org/c/openstack/nova/+/959682/14/nova/virt/libvirt/driver.py changes
09:53:14 sean-k-mooney - the _may_keep_nvram bits then run the test and see what breaks
09:53:35 sean-k-mooney that will tell you which parts of the rest of the chagne are actully needed
09:54:39 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/unit/virt/libvirt/test_guest.py will be and https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/unit/virt/libvirt/test_driver.py
09:55:09 sean-k-mooney i dont think you need https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/unit/virt/test_virt_drivers.py
09:55:13 sean-k-mooney but maybe
09:56:09 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/unit/virt/libvirt/test_config.py is good for extra test coverage but im not sure https://review.opendev.org/c/openstack/nova/+/959682/14/nova/tests/functional/libvirt/test_uefi.py is
09:56:19 sean-k-mooney so i fyou can confirm that it woudl be good
09:56:51 nicolairuckel I'll do that.
09:57:47 nicolairuckel I'm not sure if I'll be able to do that today though.
09:58:24 sean-k-mooney thats ok just let us know when it ready
10:32:38 opendevreview sean mooney proposed openstack/nova master: add functional repoducer for bug 2048837 https://review.opendev.org/c/openstack/nova/+/919979
10:32:39 opendevreview sean mooney proposed openstack/nova master: ensure correct cleanup of multi-attach volumes https://review.opendev.org/c/openstack/nova/+/916322
10:33:28 sean-k-mooney gibi: bauzas i realsised that ^ is still open. woudl ye mind looking at those again
10:33:40 bauzas sean-k-mooney: ack I can try
11:34:06 gibi sean-k-mooney: added inline comments in the fix
11:37:34 sean-k-mooney gibi: so many of those comemnt i dismissed as wont do in the previous revision
11:37:46 sean-k-mooney partly because if dont wnat to reopen the formating holy wars
11:38:58 sean-k-mooney well these comments https://review.opendev.org/c/openstack/nova/+/916322/8/nova/compute/manager.py im still readying the rest
11:42:16 sean-k-mooney gibi: good question on the thread case.
11:42:34 sean-k-mooney what i can do is aquire the named lock before i add it to the list
11:43:16 sean-k-mooney that way i think only one thread will be able to do that
11:43:41 sean-k-mooney so ill reorder https://review.opendev.org/c/openstack/nova/+/916322/8/nova/utils.py#1196 and 1197
11:43:46 sean-k-mooney that shoudl fix that.
12:21:58 gibi sean-k-mooney: Im not sure that lock is enough
12:23:05 sean-k-mooney well the sharign between thread you are descibign is not inteded to be supprote dbut ill comment more on the review
12:23:08 gibi there can be guard([a,b,c]) at the staget of a released but b is not released yet by T1 in exit. Then T2 can enter and take lock a and manipulate the list
12:24:02 gibi the doc of FairLockGuard does not really explain what is supported and what is not
12:24:16 sean-k-mooney ya i can improve that
12:28:17 sean-k-mooney to me its a logic bug to share a single instance of a context manager between treads
12:34:06 gibi OK so T1 and T2 has its own context manager but both using the same list of names to aquire the same set of locks
12:34:28 gibi that is the way we have shared locks but not shared context managers
12:34:44 gibi that is not intuitive so needs to be documented
12:35:11 gibi normally you do `with self.lock` in both T1 and T2 so the context manager (=lock) is shared
12:38:47 sahid o/
12:38:58 sahid I have noticed this function during reboot: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4286

Earlier   Later