| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 11:05:45 | bauzas | kashyap: sean-k-mooney: context is some TripleO CI blocked https://bugs.launchpad.net/tripleo/+bug/1960310 | |
| 11:05:55 | kashyap | sean-k-mooney: I wouldn't be so confident about regressions in libvirt/QEMU without evidence. | |
| 11:06:05 | bauzas | which looks to be due b/c of https://bugs.launchpad.net/nova/+bug/1960346 | |
| 11:06:07 | kashyap | When it comes to bugs, I follow "seeing is believing" | |
| 11:06:25 | bauzas | kashyap: if you see the last bug, you'll see gibi saying we use a new libvirt/qemu version | |
| 11:06:41 | bauzas | but it seems to be a regression | |
| 11:06:57 | kashyap | bauzas: Regression where? Me looks | |
| 11:07:24 | frickler | oh, it's not only devstack, but also tripleo, then devstack is even more out of the boat ;) | |
| 11:08:55 | bauzas | kashyap: see https://zuul.openstack.org/build/3e24d977991d4536b6279afd7f3b5d56/log/controller/logs/screen-n-cpu.txt?severity=4#49433 | |
| 11:09:12 | kashyap | bauzas: Yep, already noticed it | |
| 11:09:27 | kashyap | bauzas: Looking for libvirtd logs w/ QEMU filters | |
| 11:09:46 | kashyap | bauzas: Have you got the affected instance ID from the logs | |
| 11:09:58 | bauzas | lemme try to find one | |
| 11:11:47 | kashyap | 1074c6fa-12fe-40a8-b1d5-a47a49018d9f | |
| 11:11:48 | kashyap | ? | |
| 11:12:02 | bauzas | for this job, yes | |
| 11:15:18 | chateaulav | sean-k-mooney: alright then im on the right track, got that all setup and inline. | |
| 11:29:08 | chateaulav | sean-k-mooney: and then with any further changes, i would use interactive rebase to edit and add any files to each specific commit and the submit the topic for review. | |
| 11:29:17 | kashyap | bauzas: Do you know why I'm unable to fetch all the instance log files with a simple `wget`? | |
| 11:29:23 | kashyap | I'm tryin this: | |
| 11:29:40 | kashyap | $> wget -r -nH -nd -np -R "index.html*" https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/controller/logs/libvirt/libvirt/qemu/ | |
| 11:42:59 | kashyap | Anyway, it's not required; ignore the above | |
| 11:56:33 | kashyap | Duh, I wish Launchpad didn't break the complete formatting in text messages by wrapping text | |
| 12:04:17 | sean-k-mooney | chateaulav: yes. and you can always author new commits at the end of the chain and move them with an interactive rebase if need | |
| 12:05:14 | chateaulav | thanks for that last confirmation! your mentorship is much appreciated. | |
| 12:05:30 | sean-k-mooney | but basically gerrit is intended to work with freature branches and it tack each chage to a review with the change-id in the commit message | |
| 12:05:41 | sean-k-mooney | chateaulav: no worries glad to help | |
| 12:06:46 | sean-k-mooney | so rebases or change to a commit will not create a new review if the change id does not change and it will just update teh exsit review with a new revsion | |
| 12:15:14 | gibi | kashyap: so what do you think about "Device virtio-disk1 is already in the process of unplug" error? Should we increase the amount of time we wait before we retry the detach? | |
| 12:17:07 | gibi | it is configurable with CONF.libvirt.device_detach_timeout | |
| 12:17:40 | gibi | it is 20 sec by default | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/821840 | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 12:23:36 | bauzas | kashyap: sorry, was at lunch (isolated but in the kitchen tho) | |
| 12:24:35 | gibi | kashyap: I've pushed https://review.opendev.org/c/openstack/devstack/+/828705 to see if longer timeout helps or not | |
| 12:29:18 | sean-k-mooney | gibi: i tought you implmented an event based retry | |
| 12:29:33 | gibi | sean-k-mooney: it is event based but with a timeout | |
| 12:29:46 | gibi | so if the event came then we stop waiting | |
| 12:29:48 | sean-k-mooney | and when it times out we give up | |
| 12:29:52 | gibi | but if the event never cames we give up | |
| 12:30:00 | sean-k-mooney | ya ok | |
| 12:30:02 | gibi | more preciesly we retry | |
| 12:30:07 | sean-k-mooney | well no | |
| 12:30:13 | sean-k-mooney | retrying would be wrong | |
| 12:30:29 | sean-k-mooney | since we know that is an error in qemu and will abort the detach | |
| 12:30:48 | sean-k-mooney | at least in current verions in old version it was undefiend behavior | |
| 12:30:55 | gibi | I remember that even with libvirt 6.0.0 retry was needed in some cases | |
| 12:31:09 | kashyap | bauzas: Don't worry | |
| 12:31:12 | gibi | but maybe that was just the case of not waiting enough | |
| 12:31:14 | sean-k-mooney | its qemu rather then libvirt that i think is important here | |
| 12:31:19 | kashyap | gibi: Reading back; went for some air | |
| 12:31:51 | gibi | sean-k-mooney: ack, then qemu 4.2.0 vs 6.2.0 | |
| 12:31:56 | sean-k-mooney | gibi: the orginal behavior change was qemu consider a second detach request to be an error and aborting | |
| 12:32:00 | sean-k-mooney | yes | |
| 12:32:00 | kashyap | gibi: What I'm wondering is why is the unplug still in the process - what is holding up the unplug. Let me chat w/ the libvirt block dev | |
| 12:32:09 | gibi | kashyap: thanks | |
| 12:32:21 | gibi | kashyap: could be the guest keeping the dev busy? | |
| 12:32:37 | sean-k-mooney | kashyap: unplug requries the guest kernel to cooperate | |
| 12:32:40 | kashyap | gibi: Right, this is a negative test of server rescue, right? I'm trying to look at the exact test | |
| 12:32:41 | gibi | sean-k-mooney: do you have a qemu version number from which we should never retry? | |
| 12:32:51 | sean-k-mooney | so if the guest is not fully booted or busy that can delay it | |
| 12:32:58 | kashyap | sean-k-mooney: But note: there's no QEMU guest agent installed here | |
| 12:33:10 | sean-k-mooney | kashyap: it is not realted to the guest agent | |
| 12:33:39 | sean-k-mooney | its related to hardware interupts that are sent by qemu that guest must process | |
| 12:34:22 | sean-k-mooney | either via achi or the pci native hotplug mechium depend on you machinetype and qemu version | |
| 12:34:25 | gibi | kashyap: we have a positive test test_stable_device_rescue_disk_virtio_with_volume_attached and a negative test_stable_device_rescue_disk_virtio_with_volume_attached both failing | |
| 12:34:29 | gibi | ahh | |
| 12:34:34 | gibi | this is the negative test_rescued_vm_detach_volume | |
| 12:35:05 | kashyap | Yeah, this is negative that's failing | |
| 12:35:16 | kashyap | What exactly is the negative test doing? /me looks... | |
| 12:35:18 | sean-k-mooney | gibi: in terms of the exact vesion i think its in the release notes but ill see if i can find it | |
| 12:35:43 | kashyap | sean-k-mooney: Are you confident it is "related to hardware interupts that are sent by QEMU?" What evidence there is for it? | |
| 12:36:12 | gibi | sean-k-mooney: thanks. if we know the version number then I can craft a patch that conditionally set the detach attempts to 1 if the qemu is new enough | |
| 12:37:02 | sean-k-mooney | kashyap: we dont know that for certin and in fact i think https://bugzilla.redhat.com/show_bug.cgi?id=2007129 is a large part of the problem | |
| 12:37:55 | sean-k-mooney | unless you use virtio-scsi which we dont by default each volume attach and deatch is a pci hotplug form teh guest perspective as we add a seperte pci device for each virtio-blk device | |
| 12:38:28 | frickler | kashyap: for downloading logs, I think wget may have issues because the source is swift and not a "normal" webserver. you may want to look at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/download-logs.sh and maybe filter if you need only some subdirs | |
| 12:38:44 | kashyap | Okay, that bug is about PCI hotplug emulation | |
| 12:38:46 | sean-k-mooney | kashyap: that bug might be unrealted but artom suggested it might be in a different context | |
| 12:38:52 | kashyap | sean-k-mooney: Do you know what the negative test is exaclty doing? | |
| 12:39:09 | sean-k-mooney | i have not looked expictly | |
| 12:39:34 | sean-k-mooney | i guess botting into rescue mode and detaching a cinder volume | |
| 12:40:21 | sean-k-mooney | https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/servers/test_server_rescue_negative.py#L136 | |
| 12:40:21 | gibi | yepp it does that, I try to correlate that with the logs | |
| 12:40:26 | kashyap | So it is trying to rescue a paused instance, and a non-existing instance | |
| 12:40:44 | sean-k-mooney | no | |
| 12:41:17 | sean-k-mooney | its booting a vm, attaching a volume, then puting it in rescue mode which reboot with a new root disk | |
| 12:41:26 | sean-k-mooney | waiting for ti to get to rescue meaning its running | |
| 12:41:57 | sean-k-mooney | then asserting that detach raises a 409 conflict | |
| 12:42:25 | sean-k-mooney | there is not paused instance | |
| 12:42:47 | kashyap | sean-k-mooney: Well. What do you think this is doing, then? - test_rescue_non_existent_server()? | |
| 12:43:01 | kashyap | And test_rescue_paused_instance() | |
| 12:43:22 | sean-k-mooney | its just asserting that if the server does not exist the rescue call returns a 404 | |
| 12:43:40 | kashyap | And there are also: test_rescued_vm_attach_volume() and test_rescued_vm_detach_volume() | |
| 12:43:42 | sean-k-mooney | paused is asserting that you cant call resuce when its paused | |
| 12:43:56 | sean-k-mooney | yep | |
| 12:43:59 | sean-k-mooney | how is this relevent | |
| 12:44:38 | sean-k-mooney | the test all look valid and are asserting what i woudl expect | |
| 12:44:47 | kashyap | It is relevant in the sense that these are the different tests being run here | |
| 12:45:09 | sean-k-mooney | yes they are differnt test but they are not using the same vm | |
| 12:45:18 | kashyap | Yep, noted | |