| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-16 | |||
| 12:09:54 | ioni | hi guys | |
| 12:10:47 | ioni | i have an issue with deleting an instance after upgrading to Zed(I think). The error I'm having is: Requested operation is not valid: cannot undefine domain with nvram | |
| 12:10:50 | ioni | https://paste.xinu.at/qvp4dm/ | |
| 12:11:14 | ioni | the only thing that I changed is to add hw_machine_type=q35 to that specific image | |
| 12:11:23 | ioni | and is kinda random | |
| 12:11:51 | ioni | to delete the instance successfully i have to run virsh undefine domain --nvram first | |
| 12:12:25 | ioni | the problems occurs randomly on instances that sometimes are newly built with that image and deleted after | |
| 12:22:48 | ioni | i think i need to add to the image hw_firmware_type uefi for nova to understand to undefine nvram first before undefining the domain | |
| 13:17:56 | opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898319 | |
| 13:20:12 | opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898319 | |
| 13:24:55 | opendevreview | ribaudr proposed openstack/nova-specs master: Re-propose "Allow local scaphandre directory to be mapped to an instance using virtiofs" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898320 | |
| 14:43:31 | dansmith | we've got something going on with the live migration jobs and maybe others | |
| 14:43:35 | dansmith | lots of post failures | |
| 14:46:44 | dansmith | "msg": "The `lxml` module is not importable. Check the requirements." | |
| 14:52:25 | dansmith | sean-k-mooney: do you know how we get ansible modules installed on the hosts? like the virt module, which has requirements for libvirt-python, etc | |
| 14:56:33 | opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | |
| 14:59:09 | opendevreview | Tobias Urdin proposed openstack/nova master: [WIP] Handle scaling of cputune.shares https://review.opendev.org/c/openstack/nova/+/898326 | |
| 14:59:46 | tobias-urdin | sean-k-mooney[m]: feedback on ^ – doesn't handle all cases yet but i dont want to spend time on it if it's just garbage then we'll just carry something similar downstream | |
| 14:59:46 | tobias-urdin | sean-k-mooney[m]: feedback on ^ – doesn't handle all cases yet but i dont want to spend time on it if it's just garbage then we'll just carry something similar downstream | |
| 15:08:14 | dansmith | sean-k-mooney: https://6df0d90b08f38024b3b0-ddacc1cdacca3582cad1fbbc53a5babe.ssl.cf2.rackcdn.com/897674/1/check/nova-live-migration/beaa1e0/job-output.txt | |
| 15:10:10 | sean-k-mooney | dansmith: so my guess is we need to set the interperter path for the remote node | |
| 15:11:24 | dansmith | sean-k-mooney: to what? something inside our new venv? | |
| 15:11:36 | sean-k-mooney | to the python3 binary in the venv | |
| 15:11:59 | dansmith | still, something changed over the weekend and I don't know what that is.. maybe the ansible module got changed or libvirt-python changed to a soft req on lxml or something? | |
| 15:12:38 | sean-k-mooney | ya did any changes happen in devstack we can look at this after the meeting | |
| 15:13:24 | dansmith | not really seeing anything | |
| 15:33:38 | opendevreview | Dan Smith proposed openstack/nova master: Install lxml before we need it in post-run https://review.opendev.org/c/openstack/nova/+/898435 | |
| 16:34:41 | dansmith | sean-k-mooney: bauzas this ^ seems to be working, it already passed the live-migration-ceph job | |
| 16:35:01 | sean-k-mooney | sweet i have it open but had not refreshed | |
| 16:35:08 | sean-k-mooney | ah i assume your watching the live output | |
| 16:35:55 | dansmith | yup | |
| 16:36:09 | dansmith | just passed the main job as well | |
| 16:39:02 | sean-k-mooney | added link to the passing job where it ran and left a +2 | |
| 16:39:48 | sean-k-mooney | if we start seeing this type of dep issue often i woudl prefer to use bindep or similar to store the packages in a file or move the list of packagges to a role default | |
| 16:39:54 | sean-k-mooney | but for now this is fine | |
| 16:41:42 | dansmith | ack, well, I'd like to get the gate unblocked at least | |
| 16:42:26 | dansmith | I know bindep is what we use for that stuff normally, but I figured it was more aimed at setup for running the services and not just for ansible stuff to work | |
| 16:42:34 | dansmith | I suppose it's also usable | |
| 16:42:45 | sean-k-mooney | bindep has tags | |
| 16:42:51 | sean-k-mooney | so we can use [ansible] | |
| 16:43:02 | sean-k-mooney | for example if we wanted to store ansible only deps or whatever | |
| 16:43:22 | dansmith | ack | |
| 16:43:44 | sean-k-mooney | we do use the "test" tag for functional/unit tests deps | |
| 16:43:47 | dansmith | so maybe I should just slap this into bindep.txt instead of doing this in the ansible? | |
| 16:43:59 | dansmith | or maybe just follow this up with that so we can unblock the gate | |
| 16:44:20 | sean-k-mooney | well bindep is likely not aviable on the host outside of the venv where devstack installed it | |
| 16:44:33 | sean-k-mooney | so if we want to go that way lets do it in a followup | |
| 16:44:47 | sean-k-mooney | as first we need to decied how to make bindep avaialable | |
| 16:45:30 | dansmith | oh, we install bindep in the venv to manage deps on the host? I figured that was installed and managed before we got to devstack at all | |
| 16:45:46 | dansmith | yeah, let me queue up something behind this if for no other reason than to serve as a reminder | |
| 16:46:05 | sean-k-mooney | for the devstack job i belive its in the venv for the unit test jobs its installed in the job | |
| 16:46:09 | sean-k-mooney | before we invoke tox | |
| 16:46:45 | sean-k-mooney | https://github.com/openstack/nova/blob/9dceb735f6926c71a8c2e2c6ef8b5e0a1363041d/bindep.txt#L24 | |
| 16:46:58 | sean-k-mooney | lxml is a dep in bindep already for what its worth | |
| 16:47:04 | sean-k-mooney | but its marked as a test dep | |
| 16:47:21 | sean-k-mooney | so its not installed by default by devstack | |
| 16:47:45 | sean-k-mooney | actully no | |
| 16:47:50 | sean-k-mooney | thats not the python bindings | |
| 16:48:01 | sean-k-mooney | that the clib that lxml uses | |
| 16:48:09 | dansmith | yeah, neither is libvirt-python, but that's clearly installed which is weird that it works | |
| 16:48:17 | dansmith | er, python3-libvirt | |
| 16:48:52 | sean-k-mooney | i belive that will be install by devstack seperatly | |
| 16:49:15 | sean-k-mooney | https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova_plugins/functions-libvirt#L96 | |
| 16:49:42 | sean-k-mooney | and here for ubuntu https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova_plugins/functions-libvirt#L72C87-L72C102 | |
| 16:49:49 | opendevreview | Dan Smith proposed openstack/nova master: Move lxml dep to bindep.txt https://review.opendev.org/c/openstack/nova/+/898439 | |
| 16:50:14 | dansmith | sean-k-mooney: right but I would expect python3-libvirt to have a hard dep on python3-lxml, but perhaps not | |
| 16:50:35 | sean-k-mooney | i dont think it does | |
| 16:50:41 | sean-k-mooney | we need lxml to parse the xml | |
| 16:50:49 | sean-k-mooney | but the xml is generated by libvirt | |
| 16:50:54 | sean-k-mooney | which would use the c lib | |
| 16:50:59 | sean-k-mooney | not the python bindings | |
| 16:51:49 | dansmith | I guess I expected some of the python bindings to manipulate xml chunks themselves, but perhaps not | |
| 16:52:46 | sean-k-mooney | so just confirmign that his would work devstack invokes bindep if teh bindep file exits for a project here in setup_package https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/inc/python#L465 | |
| 16:55:49 | sean-k-mooney | that is called form _setup_package_with_constraints_edit which is called form setup_deploy here https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/inc/python#L359 | |
| 16:56:08 | sean-k-mooney | and we call that for nova here https://github.com/openstack/devstack/blob/ca4d5132e63752878620c4e4f374d98d433b3f52/lib/nova#L971 | |
| 16:56:25 | sean-k-mooney | so based on that call graph | |
| 16:56:31 | sean-k-mooney | we are not specifying any profile | |
| 16:56:54 | sean-k-mooney | i.e. it sthe same as just runing bindep in the nova root effectivly | |
| 16:57:26 | sean-k-mooney | so we just need to confirm that https://review.opendev.org/c/openstack/nova/+/898439 make it included in that properly | |
| 16:57:43 | sean-k-mooney | ah you added no tag so yes it will be | |
| 17:36:26 | dansmith | bauzas: can you +W this to unblock the gate please? https://review.opendev.org/c/openstack/nova/+/898435 | |
| 17:36:45 | bauzas | the quickiest I was able to | |
| 17:36:55 | dansmith | thanks | |
| 17:38:47 | dansmith | sean-k-mooney: the bindep approach doesn't leave us with lxml installed | |
| 17:41:30 | sean-k-mooney | ack i wonder if we are removing it in devstack to aovid pip issues | |
| 17:42:41 | sean-k-mooney | id dont see anything that would remove it but lets stick with teh simple solution for now | |
| 17:42:56 | dansmith | idk, but glad we approved the first solution in the short term :) | |
| 17:51:04 | sean-k-mooney | dansmith: for what its worth i dont see any packages being installed | |
| 17:51:17 | dansmith | sean-k-mooney: bindep packages you mean? | |
| 17:51:30 | dansmith | I didn't see it try lxml that's for sure | |
| 17:51:48 | sean-k-mooney | ya bindep is installed bu ti dont see ay execution of setup_package that actully installs anything | |
| 17:52:16 | sean-k-mooney | so that might have silently been broken at some point | |
| 17:52:38 | sean-k-mooney | im sing things like inc/python:_setup_package_with_constraints_edit:387 : [[ /opt/stack/neutron == -bindep* ]] | |
| 17:53:06 | dansmith | I'd have expected more breakage in that case | |
| 17:54:01 | sean-k-mooney | ah so reading the code | |
| 17:54:08 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/inc/python#L385 | |
| 17:54:20 | sean-k-mooney | the fintion is checkign if it was invoked with -bindep | |
| 17:54:23 | sean-k-mooney | as a flag | |
| 17:54:33 | sean-k-mooney | and poping it | |
| 17:54:39 | sean-k-mooney | and we are not doign that | |