| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-11-23 | |||
| 12:19:41 | sean-k-mooney | infact it looks identical | |
| 12:19:49 | sean-k-mooney | that was your intent? | |
| 12:20:16 | stephenfin | look at the bottom three lines | |
| 12:20:17 | sean-k-mooney | i set my line wrap in emails to 150 so i dont think my line wrapping really kicks in often | |
| 12:20:27 | sean-k-mooney | oh | |
| 12:20:28 | sean-k-mooney | ya | |
| 12:20:32 | stephenfin | ah, I want my own replies to be wrapped at 80 | |
| 12:20:39 | stephenfin | but I don't want to wrap others | |
| 12:20:45 | sean-k-mooney | ya i dont it drives me nuts | |
| 12:20:49 | stephenfin | because it doesn't reflow them properly | |
| 12:21:03 | stephenfin | this definitely affects you less so | |
| 12:21:37 | sean-k-mooney | part of the reason i set it to 150 instead of 120 is to not reflow text | |
| 12:21:43 | sean-k-mooney | *code | |
| 12:22:50 | stephenfin | I just select those rare code blocks and mark them as preformatted | |
| 12:30:41 | sean-k-mooney | oh vdpa live migration patches | |
| 12:40:42 | openstackgerrit | Merged openstack/nova stable/ussuri: compute: Skip cinder_encryption_key_id check when booting from volume https://review.opendev.org/c/openstack/nova/+/752486 | |
| 12:58:24 | openstack | bug 1897528 in OpenStack Compute (nova) "32bit pci domain number is not supported" [High,In progress] https://launchpad.net/bugs/1897528 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 12:58:24 | openstackgerrit | Merged openstack/nova master: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/756696 | |
| 12:59:06 | openstackgerrit | Merged openstack/nova master: Add os-volume_attachments reference docs https://review.opendev.org/c/openstack/nova/+/760971 | |
| 13:17:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support sys.argv in wsgi app https://review.opendev.org/c/openstack/nova/+/763750 | |
| 14:21:16 | gibi | is is a known grenade issue or is it something new? https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/screen-n-cpu.txt?severity=0#13482 | |
| 14:21:26 | gibi | "qemu-system-x86_64: VQ 0 size 0x80 Guest index 0xb8 inconsistent with Host index 0xe0: delta 0xffd8" | |
| 14:22:07 | sean-k-mooney | looking | |
| 14:29:38 | gibi | I only see one hit in logstash and that is the same as I linked above http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22qemu-system-x86_64%3A%20VQ%5C%22%20AND%20message%3A%5C%22inconsistent%20with%20Host%5C%22 | |
| 14:29:44 | gibi | so I guess it is new | |
| 14:31:34 | sean-k-mooney | ya it looks like a qemu bug | |
| 14:32:05 | sean-k-mooney | so this is a migratoin | |
| 14:32:11 | gibi | it is a live migration | |
| 14:32:13 | sean-k-mooney | a live migration to be precise | |
| 14:32:15 | sean-k-mooney | yep | |
| 14:32:16 | gibi | yepp | |
| 14:33:07 | sean-k-mooney | and the error is statign that the virtqueue size in the guest is different form the that of the destiation host | |
| 14:34:14 | sean-k-mooney | specfically the dest host size quemes are larger | |
| 14:34:53 | sean-k-mooney | this is for the disk too | |
| 14:34:58 | sean-k-mooney | not the nic | |
| 14:35:12 | sean-k-mooney | we have config options which i dislike to set the nic queue size | |
| 14:35:32 | sean-k-mooney | but we dont have any way to set the queues for the virtio block device | |
| 14:35:51 | gibi | so the two qemu version has different default queue sizes I assume | |
| 14:36:11 | sean-k-mooney | maybe | |
| 14:36:14 | sean-k-mooney | this is grenade? | |
| 14:36:27 | sean-k-mooney | so we are not changing os right? | |
| 14:36:42 | sean-k-mooney | im not sure if we should be changign the qemu version here | |
| 14:36:45 | sean-k-mooney | we may be | |
| 14:37:58 | sean-k-mooney | this is something i would expect libvirt to ensure compatiabily for by the way | |
| 14:39:17 | gibi | it is a grenade run. I'm not sure we have different qemus on the nodes in this run | |
| 14:39:26 | gibi | bauzas: could you hit https://review.opendev.org/c/openstack/nova-specs/+/759828 when you have time? | |
| 14:39:39 | bauzas | gibi: will look fersur | |
| 14:39:45 | gibi | bauzas: thanks | |
| 14:40:49 | stephenfin | I was just about to ping bauzas about the same spec :) | |
| 14:41:23 | stephenfin | The new UI says if a patch has +2, but not who left it | |
| 14:41:31 | stephenfin | that used to be a toggle option in Gerrit 2.x | |
| 14:42:57 | gibi | sean-k-mooney: the qemu versions are the same on the two compute in that grenade run qemu version: 4.2.1Debian 1:4.2-3ubuntu6.7~cloud0 | |
| 14:44:49 | dansmith | gibi: thanks :) | |
| 14:44:58 | gibi | dansmith: I'm an active user | |
| 14:45:15 | dansmith | gibi: sweet, might just be you and me at this point :) | |
| 14:45:53 | gibi | I will keep it working until it seriously breaks | |
| 14:46:05 | gibi | as it is part of my workflow | |
| 14:48:45 | lyarwood | FWIW I also use it | |
| 14:49:15 | sean-k-mooney | gibi: just asking danpb about it downstream he expecs this to be handeled by the qemu machine type | |
| 14:50:01 | sean-k-mooney | we is -machine pc-i440fx-4.2,accel=tcg,usb=off,dump-guest-core=off | |
| 14:50:42 | openstack | Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed] | |
| 14:50:42 | sean-k-mooney | gibi: https://bugs.launchpad.net/nova/+bug/1761798 | |
| 14:50:57 | sean-k-mooney | danpb found ^ | |
| 14:52:02 | gibi | nice catch, so we hit this before | |
| 14:52:27 | gibi | I guess it is infrequent bug if we hit it like 3 times in 2 years | |
| 14:53:40 | gibi | there is no qemu version differences in that grenade run | |
| 14:53:51 | sean-k-mooney | ya so it might be a regression | |
| 14:53:57 | sean-k-mooney | in qemu 4.2 | |
| 14:54:12 | gibi | could be | |
| 14:54:58 | sean-k-mooney | we are not seeing the error in the instance log unless im looking at the wrong one | |
| 14:58:35 | gibi | yeah I did not find it either there | |
| 14:58:56 | sean-k-mooney | oh its instance-0000001a | |
| 14:58:59 | sean-k-mooney | not 1b | |
| 14:59:47 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/subnode-2/libvirt/qemu/instance-0000001a.txt | |
| 15:00:22 | gibi | not too chatty 'virtio: bogus descriptor or out of resources' | |
| 15:00:42 | sean-k-mooney | ya that is before the migration i think | |
| 15:01:20 | sean-k-mooney | this is the otehr node https://zuul.opendev.org/t/openstack/build/d50877ae15db4022b82f4bb1d1d52cea/log/logs/libvirt/qemu/instance-0000001a.txt | |
| 15:02:10 | gibi | ahh you found it | |
| 15:02:11 | sean-k-mooney | that has the error in the nova.conf | |
| 15:02:13 | sean-k-mooney | yep | |
| 15:02:38 | sean-k-mooney | so it look like we have the inial virtio error before the migration is started | |
| 15:02:44 | openstackgerrit | Merged openstack/nova-specs master: Remove tenant_id https://review.opendev.org/c/openstack/nova-specs/+/737241 | |
| 15:03:37 | gibi | that initial error could be a hint for the qemu devs what went wrong | |
| 15:04:03 | sean-k-mooney | ya so this does still look like a qemu issue | |
| 15:04:25 | sean-k-mooney | well or libvirt but our live config for the disk is basically the bare minium | |
| 15:06:21 | gibi | sean-k-mooney: do we just need to re-target the 1761798 to qemu in launchpad or qemu upstream bugs tracked elsewhere? | |
| 15:06:56 | sean-k-mooney | its just | |
| 15:07:03 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <devices> | |
| 15:07:05 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <disk type="file" device="disk"> | |
| 15:07:07 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <driver name="qemu" type="qcow2" cache="none"/> | |
| 15:07:09 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <source file="/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk"/> | |
| 15:07:11 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <target dev="vda" bus="virtio"/> | |
| 15:07:13 | sean-k-mooney | Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: </disk> | |
| 15:07:34 | sean-k-mooney | gibi: i think they use bugzilla but they have a launchpad too i think | |
| 15:07:53 | sean-k-mooney | im not sure fi we need to retarget or just add them | |
| 15:08:34 | sean-k-mooney | this could be a libvirt bug with regards to the options that it generates when invoking qemu | |
| 15:09:17 | sean-k-mooney | its generating | |
| 15:09:20 | sean-k-mooney | -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ | |
| 15:09:22 | sean-k-mooney | -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \ | |
| 15:09:24 | sean-k-mooney | -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on | |