Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-11-23
15:02:10 gibi ahh you found it
15:02:11 sean-k-mooney that has the error in the nova.conf
15:02:13 sean-k-mooney yep
15:02:38 sean-k-mooney so it look like we have the inial virtio error before the migration is started
15:02:44 openstackgerrit Merged openstack/nova-specs master: Remove tenant_id https://review.opendev.org/c/openstack/nova-specs/+/737241
15:03:37 gibi that initial error could be a hint for the qemu devs what went wrong
15:04:03 sean-k-mooney ya so this does still look like a qemu issue
15:04:25 sean-k-mooney well or libvirt but our live config for the disk is basically the bare minium
15:06:21 gibi sean-k-mooney: do we just need to re-target the 1761798 to qemu in launchpad or qemu upstream bugs tracked elsewhere?
15:06:56 sean-k-mooney its just
15:07:03 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <devices>
15:07:05 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <disk type="file" device="disk">
15:07:07 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <driver name="qemu" type="qcow2" cache="none"/>
15:07:09 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <source file="/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk"/>
15:07:11 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: <target dev="vda" bus="virtio"/>
15:07:13 sean-k-mooney Nov 20 14:25:11.510321 ubuntu-bionic-ovh-bhs1-0021872194 nova-compute[9647]: </disk>
15:07:34 sean-k-mooney gibi: i think they use bugzilla but they have a launchpad too i think
15:07:53 sean-k-mooney im not sure fi we need to retarget or just add them
15:08:34 sean-k-mooney this could be a libvirt bug with regards to the options that it generates when invoking qemu
15:09:17 sean-k-mooney its generating
15:09:20 sean-k-mooney -blockdev '{"driver":"file","filename":"/opt/stack/data/nova/instances/2c468d92-4b19-426a-8c25-16b4624c21a4/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
15:09:22 sean-k-mooney -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \
15:09:24 sean-k-mooney -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on
15:09:42 sean-k-mooney but i dont really see a clear smoking gun there
15:10:11 sean-k-mooney its adding a bunch of stuff we did not ask for but none of ti looks invalid
15:18:31 gibi added qemu project as affected (keep nova as you suggested)
15:20:52 sean-k-mooney gibi: kashyap has ping one of the qemu migration experts for input
15:21:06 kashyap Yeah, Dave Gilbert from upstream QEMU
15:21:17 sean-k-mooney there is an old https://bugzilla.redhat.com/show_bug.cgi?id=1450524 downstream bug
15:21:17 openstack bugzilla.redhat.com bug 1450524 in qemu-kvm-rhev "qemu-kvm: VQ 0 size 0x100 Guest index 0x2010 inconsistent with Host index 0x2171: delta 0xfe9f" [High,Closed: worksforme] - Assigned to jasowang
15:21:29 sean-k-mooney but it was closed as they could not reliably repoduce it
15:21:34 kashyap He's a fixed a similar-sounding issue as above --^ It was also triggerred by OpenStack CI
15:21:42 kashyap s/a fixed/fixed/
15:22:07 sean-k-mooney gibi: the main delta there being virtio-ballon vs virtio-blk
15:38:42 kashyap sean-k-mooney: Also see this one: https://bugs.launchpad.net/nova/+bug/1737625
15:38:42 openstack Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed]
15:38:58 kashyap Especially my comment#2, FWIW
15:45:15 openstackgerrit Merged openstack/nova master: Fix the instance_uuid for test https://review.opendev.org/c/openstack/nova/+/763171
15:45:20 sean-k-mooney gibi: by the way this might be of interest to you https://bugzilla.redhat.com/show_bug.cgi?id=1895702
15:45:20 openstack bugzilla.redhat.com bug 1895702 in openstack-neutron "Nova – VM is created in spite of the fact that used port is missing fixed IP" [High,New] - Assigned to rhos-maint
15:46:05 openstackgerrit Merged openstack/nova master: doc: Update the PTL guide https://review.opendev.org/c/openstack/nova/+/762727
15:46:38 sean-k-mooney kashyap we might want to combine these or at least cross link them
15:47:01 gibi sean-k-mooney: interesing edge case where nova allows booting without IP
15:47:24 kashyap sean-k-mooney: So ... the ideal person to help debug this would be Stefan (he's on PTO), who is the 'virtio-blk' expert
15:47:26 sean-k-mooney gibi: ya i think its actully a neutron bug but you were asking about could we enable that behavior
15:47:46 kashyap That said, bugs like these, deep in the bowels of QEMU, require a debug QEMU with probes installed in the environment ...
15:47:52 gibi sean-k-mooney: yes, I got asked about it internally
15:48:03 kashyap ... and pray to the JuJu at the bottom of the sea that the bug reproduces
15:48:06 sean-k-mooney kashyap: if it starts blocking the ci then we will need to take more imediate action but until then i think it can wait till they are back form pto
15:48:37 kashyap sean-k-mooney: I highly doubt it can "block"; I just read Dave's explanation on the rare occurrence it can hit
15:48:45 kashyap As the last known occurrence in the CI seems to be 3 years ago
15:49:02 kashyap (But I'll comment further in the bug)
15:49:30 gibi yeah, i have no problem ignoring this qemu issue if it happens once in two years
15:50:01 sean-k-mooney yep same
15:50:15 sean-k-mooney we just know that it can fail this way
15:50:18 kashyap sean-k-mooney: Also, what's happening in the guest?
15:50:32 kashyap Is it just a garden-variety CirrOS instance just idling?
15:50:35 sean-k-mooney its just a migtrion of a cirror guest
15:50:41 sean-k-mooney yep
15:51:20 sean-k-mooney so beyound the ocational dhcp request or other idel work it should be doing nothing most of the time
15:54:55 kashyap sean-k-mooney: Dave says if it's a migrate-during-a-reboot, it could be a potential problem
15:54:58 kashyap That said ...
15:55:38 kashyap sean-k-mooney: gibi: I learn from Dave that we get that "guest index inconsistent" error when the migrated RAM is inconsistent with the migrated 'virtio' device state.
15:55:44 kashyap And a common case is where a 'virtio' device does an operation after the vCPU is stopped and after RAM has been transmitted.
15:55:52 kashyap (I'll add a comment in the bug from 2018 with notes)
15:58:22 gibi kashyap: thanks
16:10:36 stephenfin bauzas: Think you could take a look at this this week? https://review.opendev.org/c/openstack/nova/+/756549
16:10:49 bauzas stephenfin: sure, will look
16:10:53 stephenfin ta
16:14:13 openstackgerrit Merged openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication https://review.opendev.org/c/openstack/nova-specs/+/759828
16:28:53 openstackgerrit Merged openstack/nova master: Ignore InstanceNotFound exception when host rejoin cluster https://review.opendev.org/c/openstack/nova/+/757053
16:39:14 stephenfin dansmith: I know it's your final day before the new year, but if you've a chance to provide guidance at [1] I'd appreciate it. Would like to close that out this year :) https://review.opendev.org/c/openstack/nova/+/759114/3/nova/db/sqlalchemy/models.py#1212
16:40:19 dansmith stephenfin: ack
16:50:11 kashyap gibi: sean-k-mooney: I've outlined some notes here after my chat with QEMU migration maintainer, FWIW: https://bugs.launchpad.net/nova/+bug/1737625/comments/4
16:50:11 openstack Launchpad bug 1737625 in OpenStack Compute (nova) "live migration failed with "Failed to load virtio-blk:virtio" (libvirt/qemu)" [Undecided,Confirmed]
16:51:03 sean-k-mooney kashyap: we might want to use the other bug
16:51:13 sean-k-mooney and close that as a duplicate
16:51:35 kashyap sean-k-mooney: Isn't this more accurate? It is talking about 'virtio-blk' and the Gate issue also is virtio-blk
16:51:44 sean-k-mooney https://bugs.launchpad.net/nova/+bug/1761798
16:51:44 openstack Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]
16:51:56 sean-k-mooney kashyap: no i dont think it is i think its the same underlying issue
16:52:56 kashyap In any case, I'm copying my comment into this one, too: https://bugs.launchpad.net/nova/+bug/1761798
16:52:56 openstack Launchpad bug 1761798 in OpenStack Compute (nova) "live migration intermittently fails in CI with "VQ 0 size 0x80 Guest index 0x12c inconsistent with Host index 0x134: delta 0xfff8"" [Medium,Confirmed]
18:14:26 openstackgerrit Merged openstack/nova master: virt: Remove 'change_instance_metadata' API https://review.opendev.org/c/openstack/nova/+/749316
18:42:34 openstackgerrit Stephen Finucane proposed openstack/nova master: virt: Remove 'reset_network' API https://review.opendev.org/c/openstack/nova/+/749315
18:42:34 openstackgerrit Stephen Finucane proposed openstack/nova master: virt: Remove 'get_all_bw_counters' API https://review.opendev.org/c/openstack/nova/+/749312
18:42:35 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Remove 'BandwidthUsage', 'BandwidthUsageList' https://review.opendev.org/c/openstack/nova/+/759114
18:42:35 openstackgerrit Stephen Finucane proposed openstack/nova master: db: Drop models for removed services, features https://review.opendev.org/c/openstack/nova/+/698706
19:12:20 sean-k-mooney stephenfin: ah those are related to xenapi removal
19:16:50 sean-k-mooney i tought it was related to the compute_monitors config intially
19:16:57 sean-k-mooney https://github.com/openstack/nova/blob/120d866df7ab81b666c0e43b97705a90483533bc/nova/conf/compute.py#L107-L126
19:24:02 sean-k-mooney stephenfin: by the way we talked about the metrics weigher and weighing on things that were not compute weights in the past
19:24:06 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/compute/monitors/__init__.py#L40-L48
19:24:17 sean-k-mooney the monitors are a stevaldor entry point
19:24:30 sean-k-mooney and the numa memory bandwtih ones were actully plugins not in tree
19:24:57 sean-k-mooney https://github.com/openstack/nova/blob/c250aca00ba89b6fad86bfaebb3083e9f5c55f3b/nova/compute/monitors/__init__.py#L44 the doc used to reference them years ago
19:28:43 sean-k-mooney well it was propsoed upstream https://review.opendev.org/c/openstack/nova/+/270344
19:28:53 sean-k-mooney but it was not merged
19:31:20 sean-k-mooney https://review.opendev.org/c/openstack/nova-specs/+/180983 was the spec
21:21:27 sean-k-mooney gibi: stephenfin i just realised that https://review.opendev.org/c/openstack/nova/+/653506 is still not merged but its been so long that i have lost most of the context. its still proably should be fixed but im not sure when ill get to it. hopefully soon but i tought this was merged years ago...

Earlier   Later