| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-29 | |||
| 10:36:13 | lyarwood | yeah that might be my fault sorry | |
| 10:36:32 | lyarwood | actually yeah it is | |
| 10:36:33 | lyarwood | actually yeah it is | |
| 10:36:51 | lyarwood | sean-k-mooney: yeah I did bring it up upstream as well as it's also a bugfix IMHO | |
| 10:36:52 | lyarwood | sean-k-mooney: yeah I did bring it up upstream as well as it's also a bugfix IMHO | |
| 10:37:52 | gibi | it is nobody's fault, we needed those fixes | |
| 10:37:52 | gibi | it is nobody's fault, we needed those fixes | |
| 11:08:30 | openstackgerrit | Vlad Gusev proposed openstack/nova master: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/764435 | |
| 11:08:30 | openstackgerrit | Vlad Gusev proposed openstack/nova master: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/764435 | |
| 11:10:56 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 12:13:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/788720 | |
| 12:13:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Move the guest.get_disk test to test_guest https://review.opendev.org/c/openstack/nova/+/788721 | |
| 12:13:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Enable mypy on libvirt/guest.py https://review.opendev.org/c/openstack/nova/+/788722 | |
| 12:13:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Follow up type hints for a634103 https://review.opendev.org/c/openstack/nova/+/788723 | |
| 12:13:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724 | |
| 12:13:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725 | |
| 12:14:24 | gibi | lyarwood, sean-k-mooney: ^^ here is the backport series. I'm not 100% confident about it as there was hairy conflicts in the tests | |
| 12:14:25 | gibi | lyarwood, sean-k-mooney: ^^ here is the backport series. I'm not 100% confident about it as there was hairy conflicts in the tests | |
| 12:14:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726 | |
| 12:26:38 | gibi | lyarwood: fyi, there is a rescue + volume + RBD bug in our untriaged list https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:26:40 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:26:40 | gibi | lyarwood: fyi, there is a rescue + volume + RBD bug in our untriaged list https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:29:04 | sean-k-mooney | this is hte rbd images backend | |
| 12:29:04 | sean-k-mooney | this is hte rbd images backend | |
| 12:29:31 | sean-k-mooney | and a bfv guest | |
| 12:29:32 | sean-k-mooney | and a bfv guest | |
| 12:29:55 | lyarwood | gibi: ack yeah just on a call but I'll triage it today | |
| 12:29:56 | lyarwood | gibi: ack yeah just on a call but I'll triage it today | |
| 12:30:41 | gibi | thanks | |
| 12:31:34 | sean-k-mooney | it sound like we are just not creating the rescue disk in ceph but trying to use it | |
| 12:31:34 | sean-k-mooney | it sound like we are just not creating the rescue disk in ceph but trying to use it | |
| 12:32:16 | sean-k-mooney | i wonder is that becaue of the fact the vm is bfv. | |
| 12:32:17 | sean-k-mooney | i wonder is that becaue of the fact the vm is bfv. | |
| 12:34:20 | sean-k-mooney | they are on victoria so they should have https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/virt-bfv-instance-rescue.html | |
| 12:34:21 | sean-k-mooney | they are on victoria so they should have https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/virt-bfv-instance-rescue.html | |
| 12:35:52 | sean-k-mooney | my guess is that we are creating the rescue disk as a cinder volumn but then trying to use it form teh vms pool since the rbd images backend is enabeld | |
| 12:35:52 | sean-k-mooney | my guess is that we are creating the rescue disk as a cinder volumn but then trying to use it form teh vms pool since the rbd images backend is enabeld | |
| 12:43:58 | sean-k-mooney | so this is going to use the image_backed specified in the nova.conf so the disk it returns would be basedon the rbd image backend which will use the vms pool | |
| 12:43:58 | sean-k-mooney | so this is going to use the image_backed specified in the nova.conf so the disk it returns would be basedon the rbd image backend which will use the vms pool | |
| 12:44:00 | sean-k-mooney | https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L5170 | |
| 12:44:00 | sean-k-mooney | https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L5170 | |
| 12:47:16 | sean-k-mooney | _create_image is going too check if its a bfv guest here https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4500 | |
| 12:47:16 | sean-k-mooney | _create_image is going too check if its a bfv guest here https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4500 | |
| 12:48:36 | sean-k-mooney | which will be passed to _create_and_inject_local_root https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4578 | |
| 12:48:37 | sean-k-mooney | which will be passed to _create_and_inject_local_root https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4578 | |
| 12:49:25 | sean-k-mooney | which will not create teh image becuase bfv willl be true https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4664 | |
| 12:49:25 | sean-k-mooney | which will not create teh image becuase bfv willl be true https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4664 | |
| 12:49:46 | sean-k-mooney | so we will just take the else branch | |
| 12:49:46 | sean-k-mooney | so we will just take the else branch | |
| 12:50:09 | sean-k-mooney | lyarwood: gibi ^ pretty sure that is the root cause of https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:50:10 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:50:10 | sean-k-mooney | lyarwood: gibi ^ pretty sure that is the root cause of https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:50:10 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:50:33 | gibi | sean-k-mooney: nice analysis | |
| 12:50:33 | gibi | sean-k-mooney: nice analysis | |
| 12:53:42 | lyarwood | hmm I was sure we had tempest tests for this | |
| 12:53:42 | lyarwood | hmm I was sure we had tempest tests for this | |
| 12:53:53 | lyarwood | and I also verified it during the bfv rescue work a few cycles ago | |
| 12:53:54 | lyarwood | and I also verified it during the bfv rescue work a few cycles ago | |
| 12:53:55 | lyarwood | * lyarwood looks | |
| 13:26:46 | lyarwood | sean-k-mooney: thanks for the pointers, the real issue is that the request was even allowed as we don't support bfv rescue outside of also requesting a stable device rescue. | |
| 13:26:46 | lyarwood | sean-k-mooney: thanks for the pointers, the real issue is that the request was even allowed as we don't support bfv rescue outside of also requesting a stable device rescue. | |
| 13:27:33 | lyarwood | appears I never encoded that in the compute API so it's accepted and passed down to the virt driver that was never changed to support this with legacy rescue attempts (rescue device first etc) | |
| 13:27:33 | lyarwood | appears I never encoded that in the compute API so it's accepted and passed down to the virt driver that was never changed to support this with legacy rescue attempts (rescue device first etc) | |
| 13:27:57 | lyarwood | stable device rescues of bfv instances works just fine and is tested in tempest | |
| 13:27:57 | lyarwood | stable device rescues of bfv instances works just fine and is tested in tempest | |
| 13:28:09 | lyarwood | I'll write up a regression test and add some logic in the compute API to avoid this | |
| 13:28:09 | lyarwood | I'll write up a regression test and add some logic in the compute API to avoid this | |
| 13:29:52 | lyarwood | https://docs.openstack.org/nova/latest/user/rescue.html#instance-rescue I did at least call it out in the docs | |
| 13:29:52 | lyarwood | https://docs.openstack.org/nova/latest/user/rescue.html#instance-rescue I did at least call it out in the docs | |
| 13:35:35 | sean-k-mooney | lyarwood: ok so for the bug i guess we coudl do two things. one explain how to use stabel rescue | |
| 13:35:35 | sean-k-mooney | lyarwood: ok so for the bug i guess we coudl do two things. one explain how to use stabel rescue | |
| 13:35:49 | sean-k-mooney | and second update it to track blocking it in the api with a 400 | |
| 13:35:49 | sean-k-mooney | and second update it to track blocking it in the api with a 400 | |
| 13:36:45 | lyarwood | yup indeed, I'll sort both out shortly | |
| 13:36:45 | lyarwood | yup indeed, I'll sort both out shortly | |
| 14:07:48 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1926375 seen this? | |
| 14:07:49 | openstack | Launchpad bug 1926375 in OpenStack Compute (nova) "nova-compute service failed to start up" [Undecided,New] | |
| 14:07:49 | lyarwood | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1926375 seen this? | |
| 14:07:50 | openstack | Launchpad bug 1926375 in OpenStack Compute (nova) "nova-compute service failed to start up" [Undecided,New] | |
| 14:21:49 | sean-k-mooney | lyarwood: i can see why that would happen | |
| 14:21:50 | sean-k-mooney | lyarwood: i can see why that would happen | |
| 14:22:20 | sean-k-mooney | if the binding profile in the neutorn port was currupted and the pci_slot key was removed that would happen | |
| 14:22:20 | sean-k-mooney | if the binding profile in the neutorn port was currupted and the pci_slot key was removed that would happen | |
| 14:24:33 | gibi | sean-k-mooney: yeah, but we cannot really prove anyithing as the logs does not have an instance uuid or port uuid to map it back to the neutron state | |
| 14:24:33 | gibi | sean-k-mooney: yeah, but we cannot really prove anyithing as the logs does not have an instance uuid or port uuid to map it back to the neutron state | |
| 14:25:01 | sean-k-mooney | well we can see plugin='ovs',port_profile=VIFPortProfileOpenVSwitch | |
| 14:25:01 | sean-k-mooney | well we can see plugin='ovs',port_profile=VIFPortProfileOpenVSwitch | |
| 14:25:10 | sean-k-mooney | and .plug_hw_veb | |
| 14:25:10 | sean-k-mooney | and .plug_hw_veb | |
| 14:25:24 | sean-k-mooney | so this is hardware offloaded ovs which does not support trusted VF | |
| 14:25:24 | sean-k-mooney | so this is hardware offloaded ovs which does not support trusted VF | |
| 14:28:06 | gibi | ohh, so we only support trusted VF with the sriov agent? | |
| 14:28:06 | gibi | ohh, so we only support trusted VF with the sriov agent? | |
| 14:28:12 | gibi | I did not know that | |
| 14:28:12 | gibi | I did not know that | |
| 14:28:46 | sean-k-mooney | if it works with anything else its by acident wew never extended support to anything else | |
| 14:28:46 | sean-k-mooney | if it works with anything else its by acident wew never extended support to anything else | |
| 14:30:41 | gibi | it seems they are using it :) | |
| 14:30:41 | gibi | it seems they are using it :) | |