| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-23 | |||
| 14:50:42 | sean-k-mooney | gibi: https://bugs.launchpad.net/nova/+bug/1761798 | |
| 14:50:57 | sean-k-mooney | danpb found ^ | |
| 14:52:02 | gibi | nice catch, so we hit this before | |
| 14:52:27 | gibi | I guess it is infrequent bug if we hit it like 3 times in 2 years | |
| 14:53:40 | gibi | there is no qemu version differences in that grenade run | |
| 14:53:51 | sean-k-mooney | ya so it might be a regression | |
| 14:53:57 | sean-k-mooney | in qemu 4.2 | |
| 14:54:12 | gibi | could be | |
| 14:54:58 | sean-k-mooney | we are not seeing the error in the instance log unless im looking at the wrong one | |
| 14:58:35 | gibi | yeah I did not find it either there | |
| 14:58:56 | sean-k-mooney | oh its instance-0000001a | |
| 14:58:59 | sean-k-mooney | not 1b | |
| 14:59:47 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt | |
| 15:00:22 | gibi | not too chatty 'virtio: bogus descriptor or out of resources' | |
| 15:00:42 | sean-k-mooney | ya that is before the migration i think | |
| 15:01:20 | sean-k-mooney | this is the otehr node https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvirt/qemu/instance-0000001a.txt | |
| 15:02:10 | gibi | ahh you found it | |
| 15:02:11 | sean-k-mooney | that has the error in the nova.conf | |
| 15:02:13 | sean-k-mooney | yep | |
| 15:02:38 | sean-k-mooney | so it look like we have the inial virtio error before the migration is started | |
| 15:02:44 | openstackgerrit | Merged openstack/nova-specs master: Remove tenant_id https://review.opendev.org/c/openstack/nova-specs/+/737241 | |
| 15:03:37 | gibi | that initial error could be a hint for the qemu devs what went wrong | |
| 15:04:03 | sean-k-mooney | ya so this does still look like a qemu issue | |
| 15:04:25 | sean-k-mooney | well or libvirt but our live config for the disk is basically the bare minium | |
| 15:06:21 | gibi | sean-k-mooney: do we just need to re-target the 1761798 to qemu in launchpad or qemu upstream bugs tracked elsewhere? | |
| 15:06:56 | sean-k-mooney | its just | |
| 15:07:03 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <devices> | |
| 15:07:05 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <disk type="file" device="disk"> | |
| 15:07:07 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <driver name="qemu" type="qcow2" cache="none"/> | |
| 15:07:09 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <source file="/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk"/> | |
| 15:07:11 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <target dev="vda" bus="virtio"/> | |
| 15:07:13 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: </disk> | |
| 15:07:34 | sean-k-mooney | gibi: i think they use bugzilla but they have a launchpad too i think | |
| 15:07:53 | sean-k-mooney | im not sure fi we need to retarget or just add them | |
| 15:08:34 | sean-k-mooney | this could be a libvirt bug with regards to the options that it generates when invoking qemu | |
| 15:09:17 | sean-k-mooney | its generating | |
| 15:09:20 | sean-k-mooney | -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ | |
| 15:09:22 | sean-k-mooney | -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ | |
| 15:09:24 | sean-k-mooney | -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on | |
| 15:09:42 | sean-k-mooney | but i dont really see a clear smoking gun there | |
| 15:10:11 | sean-k-mooney | its adding a bunch of stuff we did not ask for but none of ti looks invalid | |
| 15:18:31 | gibi | added qemu project as affected (keep nova as you suggested) | |
| 15:20:52 | sean-k-mooney | gibi: kashyap has ping one of the qemu migration experts for input | |
| 15:21:06 | kashyap | Yeah, Dave Gilbert from upstream QEMU | |
| 15:21:17 | openstack | bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang | |
| 15:21:17 | sean-k-mooney | there is an old https://bugzilla.redhat.com/show_bug.cgi?id=1450524 downstream bug | |
| 15:21:29 | sean-k-mooney | but it was closed as they could not reliably repoduce it | |
| 15:21:34 | kashyap | He's a fixed a similar-sounding issue as above --^ It was also triggerred by OpenStack CI | |
| 15:21:42 | kashyap | s/a fixed/fixed/ | |
| 15:22:07 | sean-k-mooney | gibi: the main delta there being virtio-ballon vs virtio-blk | |
| 15:38:42 | openstack | Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed] | |
| 15:38:42 | kashyap | sean-k-mooney: Also see this one: https://bugs.launchpad.net/nova/+bug/1737625 | |
| 15:38:58 | kashyap | Especially my comment#2, FWIW | |
| 15:45:15 | openstackgerrit | Merged openstack/nova master: Fix the instance_uuid for test https://review.opendev.org/c/openstack/nova/+/763171 | |
| 15:45:20 | openstack | bugzilla.redhat.com bug 1895702 in openstack-neutron "Nova – VM is created in spite of the fact that used port is missing fixed IP" [High,New] - Assigned to rhos-maint | |
| 15:45:20 | sean-k-mooney | gibi: by the way this might be of interest to you https://bugzilla.redhat.com/show_bug.cgi?id=1895702 | |
| 15:46:05 | openstackgerrit | Merged openstack/nova master: doc: Update the PTL guide https://review.opendev.org/c/openstack/nova/+/762727 | |
| 15:46:38 | sean-k-mooney | kashyap we might want to combine these or at least cross link them | |
| 15:47:01 | gibi | sean-k-mooney: interesing edge case where nova allows booting without IP | |
| 15:47:24 | kashyap | sean-k-mooney: So ... the ideal person to help debug this would be Stefan (he's on PTO), who is the 'virtio-blk' expert | |
| 15:47:26 | sean-k-mooney | gibi: ya i think its actully a neutron bug but you were asking about could we enable that behavior | |
| 15:47:46 | kashyap | That said, bugs like these, deep in the bowels of QEMU, require a debug QEMU with probes installed in the environment ... | |
| 15:47:52 | gibi | sean-k-mooney: yes, I got asked about it internally | |
| 15:48:03 | kashyap | ... and pray to the JuJu at the bottom of the sea that the bug reproduces | |
| 15:48:06 | sean-k-mooney | kashyap: if it starts blocking the ci then we will need to take more imediate action but until then i think it can wait till they are back form pto | |
| 15:48:37 | kashyap | sean-k-mooney: I highly doubt it can "block"; I just read Dave's explanation on the rare occurrence it can hit | |
| 15:48:45 | kashyap | As the last known occurrence in the CI seems to be 3 years ago | |
| 15:49:02 | kashyap | (But I'll comment further in the bug) | |
| 15:49:30 | gibi | yeah, i have no problem ignoring this qemu issue if it happens once in two years | |
| 15:50:01 | sean-k-mooney | yep same | |
| 15:50:15 | sean-k-mooney | we just know that it can fail this way | |
| 15:50:18 | kashyap | sean-k-mooney: Also, what's happening in the guest? | |
| 15:50:32 | kashyap | Is it just a garden-variety CirrOS instance just idling? | |
| 15:50:35 | sean-k-mooney | its just a migtrion of a cirror guest | |
| 15:50:41 | sean-k-mooney | yep | |
| 15:51:20 | sean-k-mooney | so beyound the ocational dhcp request or other idel work it should be doing nothing most of the time | |
| 15:54:55 | kashyap | sean-k-mooney: Dave says if it's a migrate-during-a-reboot, it could be a potential problem | |
| 15:54:58 | kashyap | That said ... | |
| 15:55:38 | kashyap | sean-k-mooney: gibi: I learn from Dave that we get that "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state. | |
| 15:55:44 | kashyap | And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted. | |
| 15:55:52 | kashyap | (I'll add a comment in the bug from 2018 with notes) | |
| 15:58:22 | gibi | kashyap: thanks | |
| 16:10:36 | stephenfin | bauzas: Think you could take a look at this this week? https://review.opendev.org/c/openstack/nova/+/756549 | |
| 16:10:49 | bauzas | stephenfin: sure, will look | |
| 16:10:53 | stephenfin | ta | |
| 16:14:13 | openstackgerrit | Merged openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication https://review.opendev.org/c/openstack/nova-specs/+/759828 | |
| 16:28:53 | openstackgerrit | Merged openstack/nova master: Ignore InstanceNotFound exception when host rejoin cluster https://review.opendev.org/c/openstack/nova/+/757053 | |
| 16:39:14 | stephenfin | dansmith: I know it's your final day before the new year, but if you've a chance to provide guidance at [1] I'd appreciate it. Would like to close that out this year :) https://review.opendev.org/c/openstack/nova/+/759114/3/nova/db/sqlalchemy/models.py#1212 | |
| 16:40:19 | dansmith | stephenfin: ack | |
| 16:50:11 | openstack | Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed] | |
| 16:50:11 | kashyap | gibi: sean-k-mooney: I've outlined some notes here after my chat with QEMU migration maintainer, FWIW: https://bugs.launchpad.net/nova/+bug/1737625/comments/4 | |
| 16:51:03 | sean-k-mooney | kashyap: we might want to use the other bug | |
| 16:51:13 | sean-k-mooney | and close that as a duplicate | |
| 16:51:35 | kashyap | sean-k-mooney: Isn't this more accurate? It is talking about 'virtio-blk' and the Gate issue also is virtio-blk | |
| 16:51:44 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 16:51:44 | sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1761798 | |
| 16:51:56 | sean-k-mooney | kashyap: no i dont think it is i think its the same underlying issue | |
| 16:52:56 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 16:52:56 | kashyap | In any case, I'm copying my comment into this one, too: https://bugs.launchpad.net/nova/+bug/1761798 | |
| 18:14:26 | openstackgerrit | Merged openstack/nova master: virt: Remove 'change_instance_metadata' API https://review.opendev.org/c/openstack/nova/+/749316 | |