| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 12:04:17 | sean-k-mooney | chateaulav: yes. and you can always author new commits at the end of the chain and move them with an interactive rebase if need | |
| 12:05:14 | chateaulav | thanks for that last confirmation! your mentorship is much appreciated. | |
| 12:05:30 | sean-k-mooney | but basically gerrit is intended to work with freature branches and it tack each chage to a review with the change-id in the commit message | |
| 12:05:41 | sean-k-mooney | chateaulav: no worries glad to help | |
| 12:06:46 | sean-k-mooney | so rebases or change to a commit will not create a new review if the change id does not change and it will just update teh exsit review with a new revsion | |
| 12:15:14 | gibi | kashyap: so what do you think about "Device virtio-disk1 is already in the process of unplug" error? Should we increase the amount of time we wait before we retry the detach? | |
| 12:17:07 | gibi | it is configurable with CONF.libvirt.device_detach_timeout | |
| 12:17:40 | gibi | it is 20 sec by default | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/821840 | |
| 12:23:36 | bauzas | kashyap: sorry, was at lunch (isolated but in the kitchen tho) | |
| 12:24:35 | gibi | kashyap: I've pushed https://review.opendev.org/c/openstack/devstack/+/828705 to see if longer timeout helps or not | |
| 12:29:18 | sean-k-mooney | gibi: i tought you implmented an event based retry | |
| 12:29:33 | gibi | sean-k-mooney: it is event based but with a timeout | |
| 12:29:46 | gibi | so if the event came then we stop waiting | |
| 12:29:48 | sean-k-mooney | and when it times out we give up | |
| 12:29:52 | gibi | but if the event never cames we give up | |
| 12:30:00 | sean-k-mooney | ya ok | |
| 12:30:02 | gibi | more preciesly we retry | |
| 12:30:07 | sean-k-mooney | well no | |
| 12:30:13 | sean-k-mooney | retrying would be wrong | |
| 12:30:29 | sean-k-mooney | since we know that is an error in qemu and will abort the detach | |
| 12:30:48 | sean-k-mooney | at least in current verions in old version it was undefiend behavior | |
| 12:30:55 | gibi | I remember that even with libvirt 6.0.0 retry was needed in some cases | |
| 12:31:09 | kashyap | bauzas: Don't worry | |
| 12:31:12 | gibi | but maybe that was just the case of not waiting enough | |
| 12:31:14 | sean-k-mooney | its qemu rather then libvirt that i think is important here | |
| 12:31:19 | kashyap | gibi: Reading back; went for some air | |
| 12:31:51 | gibi | sean-k-mooney: ack, then qemu 4.2.0 vs 6.2.0 | |
| 12:31:56 | sean-k-mooney | gibi: the orginal behavior change was qemu consider a second detach request to be an error and aborting | |
| 12:32:00 | kashyap | gibi: What I'm wondering is why is the unplug still in the process - what is holding up the unplug. Let me chat w/ the libvirt block dev | |
| 12:32:00 | sean-k-mooney | yes | |
| 12:32:09 | gibi | kashyap: thanks | |
| 12:32:21 | gibi | kashyap: could be the guest keeping the dev busy? | |
| 12:32:37 | sean-k-mooney | kashyap: unplug requries the guest kernel to cooperate | |
| 12:32:40 | kashyap | gibi: Right, this is a negative test of server rescue, right? I'm trying to look at the exact test | |
| 12:32:41 | gibi | sean-k-mooney: do you have a qemu version number from which we should never retry? | |
| 12:32:51 | sean-k-mooney | so if the guest is not fully booted or busy that can delay it | |
| 12:32:58 | kashyap | sean-k-mooney: But note: there's no QEMU guest agent installed here | |
| 12:33:10 | sean-k-mooney | kashyap: it is not realted to the guest agent | |
| 12:33:39 | sean-k-mooney | its related to hardware interupts that are sent by qemu that guest must process | |
| 12:34:22 | sean-k-mooney | either via achi or the pci native hotplug mechium depend on you machinetype and qemu version | |
| 12:34:25 | gibi | kashyap: we have a positive test test_stable_device_rescue_disk_virtio_with_volume_attached and a negative test_stable_device_rescue_disk_virtio_with_volume_attached both failing | |
| 12:34:29 | gibi | ahh | |
| 12:34:34 | gibi | this is the negative test_rescued_vm_detach_volume | |
| 12:35:05 | kashyap | Yeah, this is negative that's failing | |
| 12:35:16 | kashyap | What exactly is the negative test doing? /me looks... | |
| 12:35:18 | sean-k-mooney | gibi: in terms of the exact vesion i think its in the release notes but ill see if i can find it | |
| 12:35:43 | kashyap | sean-k-mooney: Are you confident it is "related to hardware interupts that are sent by QEMU?" What evidence there is for it? | |
| 12:36:12 | gibi | sean-k-mooney: thanks. if we know the version number then I can craft a patch that conditionally set the detach attempts to 1 if the qemu is new enough | |
| 12:37:02 | sean-k-mooney | kashyap: we dont know that for certin and in fact i think https://bugzilla.redhat.com/show_bug.cgi?id=2007129 is a large part of the problem | |
| 12:37:55 | sean-k-mooney | unless you use virtio-scsi which we dont by default each volume attach and deatch is a pci hotplug form teh guest perspective as we add a seperte pci device for each virtio-blk device | |
| 12:38:28 | frickler | kashyap: for downloading logs, I think wget may have issues because the source is swift and not a "normal" webserver. you may want to look at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/download-logs.sh and maybe filter if you need only some subdirs | |
| 12:38:44 | kashyap | Okay, that bug is about PCI hotplug emulation | |
| 12:38:46 | sean-k-mooney | kashyap: that bug might be unrealted but artom suggested it might be in a different context | |
| 12:38:52 | kashyap | sean-k-mooney: Do you know what the negative test is exaclty doing? | |
| 12:39:09 | sean-k-mooney | i have not looked expictly | |
| 12:39:34 | sean-k-mooney | i guess botting into rescue mode and detaching a cinder volume | |
| 12:40:21 | gibi | yepp it does that, I try to correlate that with the logs | |
| 12:40:21 | sean-k-mooney | https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/servers/test_server_rescue_negative.py#L136 | |
| 12:40:26 | kashyap | So it is trying to rescue a paused instance, and a non-existing instance | |
| 12:40:44 | sean-k-mooney | no | |
| 12:41:17 | sean-k-mooney | its booting a vm, attaching a volume, then puting it in rescue mode which reboot with a new root disk | |
| 12:41:26 | sean-k-mooney | waiting for ti to get to rescue meaning its running | |
| 12:41:57 | sean-k-mooney | then asserting that detach raises a 409 conflict | |
| 12:42:25 | sean-k-mooney | there is not paused instance | |
| 12:42:47 | kashyap | sean-k-mooney: Well. What do you think this is doing, then? - test_rescue_non_existent_server()? | |
| 12:43:01 | kashyap | And test_rescue_paused_instance() | |
| 12:43:22 | sean-k-mooney | its just asserting that if the server does not exist the rescue call returns a 404 | |
| 12:43:40 | kashyap | And there are also: test_rescued_vm_attach_volume() and test_rescued_vm_detach_volume() | |
| 12:43:42 | sean-k-mooney | paused is asserting that you cant call resuce when its paused | |
| 12:43:56 | sean-k-mooney | yep | |
| 12:43:59 | sean-k-mooney | how is this relevent | |
| 12:44:38 | sean-k-mooney | the test all look valid and are asserting what i woudl expect | |
| 12:44:47 | kashyap | It is relevant in the sense that these are the different tests being run here | |
| 12:45:09 | sean-k-mooney | yes they are differnt test but they are not using the same vm | |
| 12:45:18 | kashyap | Yep, noted | |
| 12:45:22 | sean-k-mooney | so they should have no impact on each other | |
| 12:45:58 | sean-k-mooney | although i will say if we get very very unlucky the random uuid could colide with a real on ein the non_existent_instance test | |
| 12:46:24 | sean-k-mooney | statistically however we are not going to get uuid collisions | |
| 12:48:46 | gibi | wait a bit I think we are looking at the wrong test case | |
| 12:49:29 | kashyap | gibi: So you don't think this is the failing test? test_rescued_vm_detach_volume()? | |
| 12:49:40 | gibi | Im confused | |
| 12:49:51 | gibi | Im looking at this run https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/controller/logs/index.html | |
| 12:50:29 | kashyap | Same here. | |
| 12:53:10 | kashyap | gibi: If we can narrow down the exact test action that is causing this, then we can debug it further from libvirt/QEMU angle. | |
| 12:53:26 | gibi | yeah I'm trying that | |
| 12:53:42 | gibi | I mean trying to collect the steps the test took | |
| 12:53:43 | kashyap | Sure, no rush. (I just want to arrive at a reproducer w/ just libvirt - APIs or shell) | |
| 12:55:53 | gibi | https://paste.opendev.org/show/bEtFDBoLqDfotDMPDOq8/ | |
| 12:56:16 | gibi | OK so that test case is failing similarly than the other so we can look at any of the too | |
| 12:56:26 | gibi | that paste has the steps from test_rescued_vm_detach_volume | |
| 12:57:06 | gibi | so yeah the log correlates with the test case | |
| 12:57:39 | kashyap | gibi: So, it is indeed test_rescued_vm_detach_volume(), then | |
| 12:57:48 | gibi | we boot an instance then rescue it (basically destory the domina and start it again with different disk config) then we attempt to detach a volume while the rescue domin is running | |
| 12:57:52 | gibi | kashyap: yes | |
| 12:58:02 | kashyap | gibi: Please post this info in the bug as a record. | |
| 12:58:11 | gibi | kashyap: but the other test_stable_device_rescue_disk_virtio_with_volume_attached also failing similarly | |
| 12:58:23 | gibi | kashyap: hence my confusion | |
| 12:58:37 | kashyap | Hm, so both positive and negative are failing similarly | |