| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-04-29 | |||
| 10:18:37 | gibi | fingers crossed | |
| 10:18:37 | gibi | fingers crossed | |
| 10:19:18 | gibi | lyarwood: it is a grenade run, so I guess before the upgrade it have to run the old code from Wallaby | |
| 10:19:18 | gibi | lyarwood: it is a grenade run, so I guess before the upgrade it have to run the old code from Wallaby | |
| 10:19:34 | lyarwood | aaaaaaaah | |
| 10:19:34 | lyarwood | aaaaaaaah | |
| 10:19:37 | lyarwood | right | |
| 10:19:37 | lyarwood | right | |
| 10:19:42 | lyarwood | I didn't even think | |
| 10:19:42 | lyarwood | I didn't even think | |
| 10:19:55 | lyarwood | so is this still v to w? | |
| 10:19:55 | lyarwood | so is this still v to w? | |
| 10:20:05 | gibi | I think it is w to master | |
| 10:20:05 | gibi | I think it is w to master | |
| 10:20:13 | gibi | but we only merged the fix to master | |
| 10:20:13 | gibi | but we only merged the fix to master | |
| 10:20:19 | sean-k-mooney | oh ya | |
| 10:20:19 | sean-k-mooney | oh ya | |
| 10:20:58 | lyarwood | we did talk about backporting your change to wallaby | |
| 10:20:59 | lyarwood | we did talk about backporting your change to wallaby | |
| 10:21:08 | sean-k-mooney | well grenade shoudl be wallaby to xena | |
| 10:21:08 | sean-k-mooney | well grenade shoudl be wallaby to xena | |
| 10:21:15 | lyarwood | * lyarwood needs to jump on a call brb | |
| 10:21:36 | lyarwood | sean-k-mooney: yeah I think this is the live migration test between the two | |
| 10:21:36 | lyarwood | sean-k-mooney: yeah I think this is the live migration test between the two | |
| 10:21:40 | lyarwood | sean-k-mooney: with the compute still on w | |
| 10:21:40 | lyarwood | sean-k-mooney: with the compute still on w | |
| 10:22:59 | sean-k-mooney | back porting the change for libvirt events to wallaby would be nice form a downstrem perspecitve too | |
| 10:23:00 | sean-k-mooney | back porting the change for libvirt events to wallaby would be nice form a downstrem perspecitve too | |
| 10:23:53 | sean-k-mooney | we talked about trying that downstream but while i like the idea of doing that i was not personally sure we wanted to diverge form upstream in this case | |
| 10:23:53 | sean-k-mooney | we talked about trying that downstream but while i like the idea of doing that i was not personally sure we wanted to diverge form upstream in this case | |
| 10:24:56 | gibi | lyarwood: the delta between w and current master should be small, so I can fire up the cherry-picks | |
| 10:24:56 | gibi | lyarwood: the delta between w and current master should be small, so I can fire up the cherry-picks | |
| 10:29:46 | gibi | ahh there are already merge conflicts :/ | |
| 10:29:47 | gibi | ahh there are already merge conflicts :/ | |
| 10:36:13 | lyarwood | yeah that might be my fault sorry | |
| 10:36:13 | lyarwood | yeah that might be my fault sorry | |
| 10:36:32 | lyarwood | actually yeah it is | |
| 10:36:33 | lyarwood | actually yeah it is | |
| 10:36:51 | lyarwood | sean-k-mooney: yeah I did bring it up upstream as well as it's also a bugfix IMHO | |
| 10:36:52 | lyarwood | sean-k-mooney: yeah I did bring it up upstream as well as it's also a bugfix IMHO | |
| 10:37:52 | gibi | it is nobody's fault, we needed those fixes | |
| 10:37:52 | gibi | it is nobody's fault, we needed those fixes | |
| 11:08:30 | openstackgerrit | Vlad Gusev proposed openstack/nova master: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/764435 | |
| 11:08:30 | openstackgerrit | Vlad Gusev proposed openstack/nova master: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/764435 | |
| 11:10:56 | openstackgerrit | Daniel Bengtsson proposed openstack/nova master: Use the new type HostDomainOpt. https://review.opendev.org/c/openstack/nova/+/788240 | |
| 12:13:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/788720 | |
| 12:13:06 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Move the guest.get_disk test to test_guest https://review.opendev.org/c/openstack/nova/+/788721 | |
| 12:13:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Enable mypy on libvirt/guest.py https://review.opendev.org/c/openstack/nova/+/788722 | |
| 12:13:08 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Follow up type hints for a634103 https://review.opendev.org/c/openstack/nova/+/788723 | |
| 12:13:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/788724 | |
| 12:13:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Move instance power state check to _detach_with_retry https://review.opendev.org/c/openstack/nova/+/788725 | |
| 12:14:24 | gibi | lyarwood, sean-k-mooney: ^^ here is the backport series. I'm not 100% confident about it as there was hairy conflicts in the tests | |
| 12:14:25 | gibi | lyarwood, sean-k-mooney: ^^ here is the backport series. I'm not 100% confident about it as there was hairy conflicts in the tests | |
| 12:14:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/wallaby: Consolidate device detach error handling https://review.opendev.org/c/openstack/nova/+/788726 | |
| 12:26:38 | gibi | lyarwood: fyi, there is a rescue + volume + RBD bug in our untriaged list https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:26:40 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:26:40 | gibi | lyarwood: fyi, there is a rescue + volume + RBD bug in our untriaged list https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:29:04 | sean-k-mooney | this is hte rbd images backend | |
| 12:29:04 | sean-k-mooney | this is hte rbd images backend | |
| 12:29:31 | sean-k-mooney | and a bfv guest | |
| 12:29:32 | sean-k-mooney | and a bfv guest | |
| 12:29:55 | lyarwood | gibi: ack yeah just on a call but I'll triage it today | |
| 12:29:56 | lyarwood | gibi: ack yeah just on a call but I'll triage it today | |
| 12:30:41 | gibi | thanks | |
| 12:31:34 | sean-k-mooney | it sound like we are just not creating the rescue disk in ceph but trying to use it | |
| 12:31:34 | sean-k-mooney | it sound like we are just not creating the rescue disk in ceph but trying to use it | |
| 12:32:16 | sean-k-mooney | i wonder is that becaue of the fact the vm is bfv. | |
| 12:32:17 | sean-k-mooney | i wonder is that becaue of the fact the vm is bfv. | |
| 12:34:20 | sean-k-mooney | they are on victoria so they should have https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/virt-bfv-instance-rescue.html | |
| 12:34:21 | sean-k-mooney | they are on victoria so they should have https://specs.openstack.org/openstack/nova-specs/specs/ussuri/implemented/virt-bfv-instance-rescue.html | |
| 12:35:52 | sean-k-mooney | my guess is that we are creating the rescue disk as a cinder volumn but then trying to use it form teh vms pool since the rbd images backend is enabeld | |
| 12:35:52 | sean-k-mooney | my guess is that we are creating the rescue disk as a cinder volumn but then trying to use it form teh vms pool since the rbd images backend is enabeld | |
| 12:43:58 | sean-k-mooney | so this is going to use the image_backed specified in the nova.conf so the disk it returns would be basedon the rbd image backend which will use the vms pool | |
| 12:43:58 | sean-k-mooney | so this is going to use the image_backed specified in the nova.conf so the disk it returns would be basedon the rbd image backend which will use the vms pool | |
| 12:44:00 | sean-k-mooney | https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L5170 | |
| 12:44:00 | sean-k-mooney | https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L5170 | |
| 12:47:16 | sean-k-mooney | _create_image is going too check if its a bfv guest here https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4500 | |
| 12:47:16 | sean-k-mooney | _create_image is going too check if its a bfv guest here https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4500 | |
| 12:48:36 | sean-k-mooney | which will be passed to _create_and_inject_local_root https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4578 | |
| 12:48:37 | sean-k-mooney | which will be passed to _create_and_inject_local_root https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4578 | |
| 12:49:25 | sean-k-mooney | which will not create teh image becuase bfv willl be true https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4664 | |
| 12:49:25 | sean-k-mooney | which will not create teh image becuase bfv willl be true https://github.com/openstack/nova/blob/eba9d596daa91d8f702b719afb88cb89f2d5bb32/nova/virt/libvirt/driver.py#L4664 | |
| 12:49:46 | sean-k-mooney | so we will just take the else branch | |
| 12:49:46 | sean-k-mooney | so we will just take the else branch | |
| 12:50:09 | sean-k-mooney | lyarwood: gibi ^ pretty sure that is the root cause of https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:50:10 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:50:10 | sean-k-mooney | lyarwood: gibi ^ pretty sure that is the root cause of https://bugs.launchpad.net/nova/+bug/1926601 | |
| 12:50:10 | openstack | Launchpad bug 1926601 in OpenStack Compute (nova) "Rescuing RBD volume-backed instance does not work" [Undecided,New] | |
| 12:50:33 | gibi | sean-k-mooney: nice analysis | |
| 12:50:33 | gibi | sean-k-mooney: nice analysis | |
| 12:53:42 | lyarwood | hmm I was sure we had tempest tests for this | |
| 12:53:42 | lyarwood | hmm I was sure we had tempest tests for this | |
| 12:53:53 | lyarwood | and I also verified it during the bfv rescue work a few cycles ago | |
| 12:53:54 | lyarwood | and I also verified it during the bfv rescue work a few cycles ago | |
| 12:53:55 | lyarwood | * lyarwood looks | |
| 13:26:46 | lyarwood | sean-k-mooney: thanks for the pointers, the real issue is that the request was even allowed as we don't support bfv rescue outside of also requesting a stable device rescue. | |
| 13:26:46 | lyarwood | sean-k-mooney: thanks for the pointers, the real issue is that the request was even allowed as we don't support bfv rescue outside of also requesting a stable device rescue. | |
| 13:27:33 | lyarwood | appears I never encoded that in the compute API so it's accepted and passed down to the virt driver that was never changed to support this with legacy rescue attempts (rescue device first etc) | |
| 13:27:33 | lyarwood | appears I never encoded that in the compute API so it's accepted and passed down to the virt driver that was never changed to support this with legacy rescue attempts (rescue device first etc) | |