Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-10
10:58:02 sean-k-mooney chateaulav: you have 1 local branch for all commits in a feature
10:58:28 sean-k-mooney chateaulav: you should be able to check out that branch locally and run all the code form the top of the branch
10:58:52 sean-k-mooney e.g. run the unit/func tests and or deploy nova and execute the code
11:02:03 sean-k-mooney frickler: im pretty sure any change in the detach behavior is a libvirt/qemu regression likely related to the rework they are doing in this arrea for pci passthtough
11:02:24 sean-k-mooney regarding https://bugs.launchpad.net/nova/+bug/1960346
11:05:00 bauzas sean-k-mooney: yup we'd appreciate some qemu expert on this one
11:05:07 bauzas kashyap: maybe you can help ?
11:05:22 kashyap bauzas: Hey, reading back
11:05:45 bauzas kashyap: sean-k-mooney: context is some TripleO CI blocked https://bugs.launchpad.net/tripleo/+bug/1960310
11:05:55 kashyap sean-k-mooney: I wouldn't be so confident about regressions in libvirt/QEMU without evidence.
11:06:05 bauzas which looks to be due b/c of https://bugs.launchpad.net/nova/+bug/1960346
11:06:07 kashyap When it comes to bugs, I follow "seeing is believing"
11:06:25 bauzas kashyap: if you see the last bug, you'll see gibi saying we use a new libvirt/qemu version
11:06:41 bauzas but it seems to be a regression
11:06:57 kashyap bauzas: Regression where? Me looks
11:07:24 frickler oh, it's not only devstack, but also tripleo, then devstack is even more out of the boat ;)
11:08:55 bauzas kashyap: see https://zuul.openstack.org/build/3e24d977991d4536b6279afd7f3b5d56/log/controller/logs/screen-n-cpu.txt?severity=4#49433
11:09:12 kashyap bauzas: Yep, already noticed it
11:09:27 kashyap bauzas: Looking for libvirtd logs w/ QEMU filters
11:09:46 kashyap bauzas: Have you got the affected instance ID from the logs
11:09:58 bauzas lemme try to find one
11:11:47 kashyap 1074c6fa-12fe-40a8-b1d5-a47a49018d9f
11:11:48 kashyap ?
11:12:02 bauzas for this job, yes
11:15:18 chateaulav sean-k-mooney: alright then im on the right track, got that all setup and inline.
11:29:08 chateaulav sean-k-mooney: and then with any further changes, i would use interactive rebase to edit and add any files to each specific commit and the submit the topic for review.
11:29:17 kashyap bauzas: Do you know why I'm unable to fetch all the instance log files with a simple `wget`?
11:29:23 kashyap I'm tryin this:
11:29:40 kashyap $> wget -r -nH -nd -np -R "index.html*" https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/controller/logs/libvirt/libvirt/qemu/
11:42:59 kashyap Anyway, it's not required; ignore the above
11:56:33 kashyap Duh, I wish Launchpad didn't break the complete formatting in text messages by wrapping text
12:04:17 sean-k-mooney chateaulav: yes. and you can always author new commits at the end of the chain and move them with an interactive rebase if need
12:05:14 chateaulav thanks for that last confirmation! your mentorship is much appreciated.
12:05:30 sean-k-mooney but basically gerrit is intended to work with freature branches and it tack each chage to a review with the change-id in the commit message
12:05:41 sean-k-mooney chateaulav: no worries glad to help
12:06:46 sean-k-mooney so rebases or change to a commit will not create a new review if the change id does not change and it will just update teh exsit review with a new revsion
12:15:14 gibi kashyap: so what do you think about "Device virtio-disk1 is already in the process of unplug" error? Should we increase the amount of time we wait before we retry the detach?
12:17:07 gibi it is configurable with CONF.libvirt.device_detach_timeout
12:17:40 gibi it is 20 sec by default
12:22:48 opendevreview Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324
12:22:48 opendevreview Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/821840
12:23:36 bauzas kashyap: sorry, was at lunch (isolated but in the kitchen tho)
12:24:35 gibi kashyap: I've pushed https://review.opendev.org/c/openstack/devstack/+/828705 to see if longer timeout helps or not
12:29:18 sean-k-mooney gibi: i tought you implmented an event based retry
12:29:33 gibi sean-k-mooney: it is event based but with a timeout
12:29:46 gibi so if the event came then we stop waiting
12:29:48 sean-k-mooney and when it times out we give up
12:29:52 gibi but if the event never cames we give up
12:30:00 sean-k-mooney ya ok
12:30:02 gibi more preciesly we retry
12:30:07 sean-k-mooney well no
12:30:13 sean-k-mooney retrying would be wrong
12:30:29 sean-k-mooney since we know that is an error in qemu and will abort the detach
12:30:48 sean-k-mooney at least in current verions in old version it was undefiend behavior
12:30:55 gibi I remember that even with libvirt 6.0.0 retry was needed in some cases
12:31:09 kashyap bauzas: Don't worry
12:31:12 gibi but maybe that was just the case of not waiting enough
12:31:14 sean-k-mooney its qemu rather then libvirt that i think is important here
12:31:19 kashyap gibi: Reading back; went for some air
12:31:51 gibi sean-k-mooney: ack, then qemu 4.2.0 vs 6.2.0
12:31:56 sean-k-mooney gibi: the orginal behavior change was qemu consider a second detach request to be an error and aborting
12:32:00 kashyap gibi: What I'm wondering is why is the unplug still in the process - what is holding up the unplug. Let me chat w/ the libvirt block dev
12:32:00 sean-k-mooney yes
12:32:09 gibi kashyap: thanks
12:32:21 gibi kashyap: could be the guest keeping the dev busy?
12:32:37 sean-k-mooney kashyap: unplug requries the guest kernel to cooperate
12:32:40 kashyap gibi: Right, this is a negative test of server rescue, right? I'm trying to look at the exact test
12:32:41 gibi sean-k-mooney: do you have a qemu version number from which we should never retry?
12:32:51 sean-k-mooney so if the guest is not fully booted or busy that can delay it
12:32:58 kashyap sean-k-mooney: But note: there's no QEMU guest agent installed here
12:33:10 sean-k-mooney kashyap: it is not realted to the guest agent
12:33:39 sean-k-mooney its related to hardware interupts that are sent by qemu that guest must process
12:34:22 sean-k-mooney either via achi or the pci native hotplug mechium depend on you machinetype and qemu version
12:34:25 gibi kashyap: we have a positive test test_stable_device_rescue_disk_virtio_with_volume_attached and a negative test_stable_device_rescue_disk_virtio_with_volume_attached both failing
12:34:29 gibi ahh
12:34:34 gibi this is the negative test_rescued_vm_detach_volume
12:35:05 kashyap Yeah, this is negative that's failing
12:35:16 kashyap What exactly is the negative test doing? /me looks...
12:35:18 sean-k-mooney gibi: in terms of the exact vesion i think its in the release notes but ill see if i can find it
12:35:43 kashyap sean-k-mooney: Are you confident it is "related to hardware interupts that are sent by QEMU?" What evidence there is for it?
12:36:12 gibi sean-k-mooney: thanks. if we know the version number then I can craft a patch that conditionally set the detach attempts to 1 if the qemu is new enough
12:37:02 sean-k-mooney kashyap: we dont know that for certin and in fact i think https://bugzilla.redhat.com/show_bug.cgi?id=2007129 is a large part of the problem
12:37:55 sean-k-mooney unless you use virtio-scsi which we dont by default each volume attach and deatch is a pci hotplug form teh guest perspective as we add a seperte pci device for each virtio-blk device
12:38:28 frickler kashyap: for downloading logs, I think wget may have issues because the source is swift and not a "normal" webserver. you may want to look at https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/download-logs.sh and maybe filter if you need only some subdirs
12:38:44 kashyap Okay, that bug is about PCI hotplug emulation
12:38:46 sean-k-mooney kashyap: that bug might be unrealted but artom suggested it might be in a different context
12:38:52 kashyap sean-k-mooney: Do you know what the negative test is exaclty doing?
12:39:09 sean-k-mooney i have not looked expictly
12:39:34 sean-k-mooney i guess botting into rescue mode and detaching a cinder volume
12:40:21 gibi yepp it does that, I try to correlate that with the logs
12:40:21 sean-k-mooney https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/servers/test_server_rescue_negative.py#L136
12:40:26 kashyap So it is trying to rescue a paused instance, and a non-existing instance
12:40:44 sean-k-mooney no
12:41:17 sean-k-mooney its booting a vm, attaching a volume, then puting it in rescue mode which reboot with a new root disk
12:41:26 sean-k-mooney waiting for ti to get to rescue meaning its running
12:41:57 sean-k-mooney then asserting that detach raises a 409 conflict
12:42:25 sean-k-mooney there is not paused instance
12:42:47 kashyap sean-k-mooney: Well. What do you think this is doing, then? - test_rescue_non_existent_server()?
12:43:01 kashyap And test_rescue_paused_instance()
12:43:22 sean-k-mooney its just asserting that if the server does not exist the rescue call returns a 404

Earlier   Later