| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-10 | |||
| 08:23:16 | bauzas | oh then the other way around | |
| 08:23:46 | bauzas | __ministry: so you're asking about resources:VGPU=X where X>1 | |
| 08:24:16 | bauzas | I'm unfortunate to say there is some limitation from libvirt due to some nasty bug | |
| 08:24:48 | __ministry | yep. it like we can attach multiple volume to nova instance. | |
| 08:24:50 | __ministry | ok | |
| 08:27:48 | bauzas | __ministry: https://bugs.launchpad.net/nova/+bug/1758086 | |
| 08:33:23 | __ministry | bauzas: thank you. ^.^ | |
| 08:35:16 | bauzas | __ministry: that being said, this is a nvidia driver limitation, you're welcome to test again another newer GRID release version | |
| 08:52:07 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add properties to set advanced QXL video RAM settings https://review.opendev.org/c/openstack/nova/+/828674 | |
| 08:59:17 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 09:09:56 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add property to set number of screens per video adapter https://review.opendev.org/c/openstack/nova/+/828676 | |
| 10:09:30 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491 | |
| 10:09:57 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | |
| 10:09:58 | opendevreview | Balazs Gibizer proposed openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | |
| 10:10:08 | opendevreview | Balazs Gibizer proposed openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | |
| 10:11:38 | gibi | gmann: thanks for the comment in the any-traits series. I restored the legacy behavior in https://review.opendev.org/c/openstack/placement/+/826491/8 | |
| 10:11:47 | gibi | melwitt: I had to respin the any-traits series due to ^^ | |
| 10:27:45 | opendevreview | Manuel Bentele proposed openstack/nova master: libvirt: Add configuration options to set SPICE compression settings https://review.opendev.org/c/openstack/nova/+/828675 | |
| 10:58:02 | sean-k-mooney | chateaulav: you have 1 local branch for all commits in a feature | |
| 10:58:28 | sean-k-mooney | chateaulav: you should be able to check out that branch locally and run all the code form the top of the branch | |
| 10:58:52 | sean-k-mooney | e.g. run the unit/func tests and or deploy nova and execute the code | |
| 11:02:03 | sean-k-mooney | frickler: im pretty sure any change in the detach behavior is a libvirt/qemu regression likely related to the rework they are doing in this arrea for pci passthtough | |
| 11:02:24 | sean-k-mooney | regarding https://bugs.launchpad.net/nova/+bug/1960346 | |
| 11:05:00 | bauzas | sean-k-mooney: yup we'd appreciate some qemu expert on this one | |
| 11:05:07 | bauzas | kashyap: maybe you can help ? | |
| 11:05:22 | kashyap | bauzas: Hey, reading back | |
| 11:05:45 | bauzas | kashyap: sean-k-mooney: context is some TripleO CI blocked https://bugs.launchpad.net/tripleo/+bug/1960310 | |
| 11:05:55 | kashyap | sean-k-mooney: I wouldn't be so confident about regressions in libvirt/QEMU without evidence. | |
| 11:06:05 | bauzas | which looks to be due b/c of https://bugs.launchpad.net/nova/+bug/1960346 | |
| 11:06:07 | kashyap | When it comes to bugs, I follow "seeing is believing" | |
| 11:06:25 | bauzas | kashyap: if you see the last bug, you'll see gibi saying we use a new libvirt/qemu version | |
| 11:06:41 | bauzas | but it seems to be a regression | |
| 11:06:57 | kashyap | bauzas: Regression where? Me looks | |
| 11:07:24 | frickler | oh, it's not only devstack, but also tripleo, then devstack is even more out of the boat ;) | |
| 11:08:55 | bauzas | kashyap: see https://zuul.openstack.org/build/3e24d977991d4536b6279afd7f3b5d56/log/controller/logs/screen-n-cpu.txt?severity=4#49433 | |
| 11:09:12 | kashyap | bauzas: Yep, already noticed it | |
| 11:09:27 | kashyap | bauzas: Looking for libvirtd logs w/ QEMU filters | |
| 11:09:46 | kashyap | bauzas: Have you got the affected instance ID from the logs | |
| 11:09:58 | bauzas | lemme try to find one | |
| 11:11:47 | kashyap | 1074c6fa-12fe-40a8-b1d5-a47a49018d9f | |
| 11:11:48 | kashyap | ? | |
| 11:12:02 | bauzas | for this job, yes | |
| 11:15:18 | chateaulav | sean-k-mooney: alright then im on the right track, got that all setup and inline. | |
| 11:29:08 | chateaulav | sean-k-mooney: and then with any further changes, i would use interactive rebase to edit and add any files to each specific commit and the submit the topic for review. | |
| 11:29:17 | kashyap | bauzas: Do you know why I'm unable to fetch all the instance log files with a simple `wget`? | |
| 11:29:23 | kashyap | I'm tryin this: | |
| 11:29:40 | kashyap | $> wget -r -nH -nd -np -R "index.html*" https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e2/828280/1/check/devstack-platform-centos-9-stream/3e24d97/controller/logs/libvirt/libvirt/qemu/ | |
| 11:42:59 | kashyap | Anyway, it's not required; ignore the above | |
| 11:56:33 | kashyap | Duh, I wish Launchpad didn't break the complete formatting in text messages by wrapping text | |
| 12:04:17 | sean-k-mooney | chateaulav: yes. and you can always author new commits at the end of the chain and move them with an interactive rebase if need | |
| 12:05:14 | chateaulav | thanks for that last confirmation! your mentorship is much appreciated. | |
| 12:05:30 | sean-k-mooney | but basically gerrit is intended to work with freature branches and it tack each chage to a review with the change-id in the commit message | |
| 12:05:41 | sean-k-mooney | chateaulav: no worries glad to help | |
| 12:06:46 | sean-k-mooney | so rebases or change to a commit will not create a new review if the change id does not change and it will just update teh exsit review with a new revsion | |
| 12:15:14 | gibi | kashyap: so what do you think about "Device virtio-disk1 is already in the process of unplug" error? Should we increase the amount of time we wait before we retry the detach? | |
| 12:17:07 | gibi | it is configurable with CONF.libvirt.device_detach_timeout | |
| 12:17:40 | gibi | it is 20 sec by default | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 12:22:48 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/821840 | |
| 12:23:36 | bauzas | kashyap: sorry, was at lunch (isolated but in the kitchen tho) | |
| 12:24:35 | gibi | kashyap: I've pushed https://review.opendev.org/c/openstack/devstack/+/828705 to see if longer timeout helps or not | |
| 12:29:18 | sean-k-mooney | gibi: i tought you implmented an event based retry | |
| 12:29:33 | gibi | sean-k-mooney: it is event based but with a timeout | |
| 12:29:46 | gibi | so if the event came then we stop waiting | |
| 12:29:48 | sean-k-mooney | and when it times out we give up | |
| 12:29:52 | gibi | but if the event never cames we give up | |
| 12:30:00 | sean-k-mooney | ya ok | |
| 12:30:02 | gibi | more preciesly we retry | |
| 12:30:07 | sean-k-mooney | well no | |
| 12:30:13 | sean-k-mooney | retrying would be wrong | |
| 12:30:29 | sean-k-mooney | since we know that is an error in qemu and will abort the detach | |
| 12:30:48 | sean-k-mooney | at least in current verions in old version it was undefiend behavior | |
| 12:30:55 | gibi | I remember that even with libvirt 6.0.0 retry was needed in some cases | |
| 12:31:09 | kashyap | bauzas: Don't worry | |
| 12:31:12 | gibi | but maybe that was just the case of not waiting enough | |
| 12:31:14 | sean-k-mooney | its qemu rather then libvirt that i think is important here | |
| 12:31:19 | kashyap | gibi: Reading back; went for some air | |
| 12:31:51 | gibi | sean-k-mooney: ack, then qemu 4.2.0 vs 6.2.0 | |
| 12:31:56 | sean-k-mooney | gibi: the orginal behavior change was qemu consider a second detach request to be an error and aborting | |
| 12:32:00 | kashyap | gibi: What I'm wondering is why is the unplug still in the process - what is holding up the unplug. Let me chat w/ the libvirt block dev | |
| 12:32:00 | sean-k-mooney | yes | |
| 12:32:09 | gibi | kashyap: thanks | |
| 12:32:21 | gibi | kashyap: could be the guest keeping the dev busy? | |
| 12:32:37 | sean-k-mooney | kashyap: unplug requries the guest kernel to cooperate | |
| 12:32:40 | kashyap | gibi: Right, this is a negative test of server rescue, right? I'm trying to look at the exact test | |
| 12:32:41 | gibi | sean-k-mooney: do you have a qemu version number from which we should never retry? | |
| 12:32:51 | sean-k-mooney | so if the guest is not fully booted or busy that can delay it | |
| 12:32:58 | kashyap | sean-k-mooney: But note: there's no QEMU guest agent installed here | |
| 12:33:10 | sean-k-mooney | kashyap: it is not realted to the guest agent | |
| 12:33:39 | sean-k-mooney | its related to hardware interupts that are sent by qemu that guest must process | |
| 12:34:22 | sean-k-mooney | either via achi or the pci native hotplug mechium depend on you machinetype and qemu version | |
| 12:34:25 | gibi | kashyap: we have a positive test test_stable_device_rescue_disk_virtio_with_volume_attached and a negative test_stable_device_rescue_disk_virtio_with_volume_attached both failing | |
| 12:34:29 | gibi | ahh | |
| 12:34:34 | gibi | this is the negative test_rescued_vm_detach_volume | |
| 12:35:05 | kashyap | Yeah, this is negative that's failing | |
| 12:35:16 | kashyap | What exactly is the negative test doing? /me looks... | |
| 12:35:18 | sean-k-mooney | gibi: in terms of the exact vesion i think its in the release notes but ill see if i can find it | |
| 12:35:43 | kashyap | sean-k-mooney: Are you confident it is "related to hardware interupts that are sent by QEMU?" What evidence there is for it? | |
| 12:36:12 | gibi | sean-k-mooney: thanks. if we know the version number then I can craft a patch that conditionally set the detach attempts to 1 if the qemu is new enough | |
| 12:37:02 | sean-k-mooney | kashyap: we dont know that for certin and in fact i think https://bugzilla.redhat.com/show_bug.cgi?id=2007129 is a large part of the problem | |