| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-07 | |||
| 12:17:28 | kashyap | lyarwood: So I just posted #9 | |
| 12:17:40 | kashyap | To copy/paste my point-1 from there: | |
| 12:17:45 | kashyap | "- DEVICE_DELETED is the event that QEMU sends to libvirt, *once* the device was removed by the guest, so that libvirt can clean-up. So if we see DEVICE_DELETED that means the device was successfully detached from QEMU's point of view (therefore, from the guest's PoV, too)" | |
| 12:18:32 | lyarwood | kashyap: right sorry I'm just working out how to instrument things in CI at the moment | |
| 12:18:35 | kashyap | lyarwood: Are you using a new kernel rebuilt with it? | |
| 12:18:43 | lyarwood | kashyap: detach works correctly in the env at the moment | |
| 12:19:02 | lyarwood | kashyap: I'm just trying to figure out what I need to capture during a run to show things are delayed in the guestos | |
| 12:19:28 | lyarwood | kashyap: and yeah 5.3.0-26-generic is the kernel | |
| 12:20:27 | kashyap | lyarwood: So, Igor (KVM/QEMU dev) says: "You'd could watch for udev events as indirect result of unplug events for specific device subsystem" | |
| 12:21:26 | lyarwood | kashyap: I don't think cirros is using udev tbh | |
| 12:22:17 | kashyap | lyarwood: Nod; I've actually snipped out his first part where he admits he isn't familiar w/ 'acpid' | |
| 12:23:48 | lyarwood | https://git.busybox.net/busybox/tree/util-linux/acpid.c it's not even the old version I was used to tbh | |
| 12:25:17 | kashyap | lyarwood: I'm curious if your test with slightly "better resources" for the guest fixes it | |
| 12:25:33 | kashyap | lyarwood: Also can you tell what's the buggy guest configuration? If you don't mind posting the guest XML... | |
| 12:25:50 | lyarwood | kashyap: I still saw a few failures | |
| 12:26:14 | kashyap | So it's not the resources allocated to the guest | |
| 12:26:28 | lyarwood | kashyap: that was in reference to the host guest running openstack FWIW | |
| 12:26:37 | lyarwood | kashyap: correct | |
| 12:26:50 | lyarwood | kashyap: CI nodes run with 1 vCPU and 8GB of RAM at the moment | |
| 12:27:06 | lyarwood | kashyap: the instances have 1 vCPU and 128MB of RAM | |
| 12:27:29 | kashyap | lyarwood: BTW, haven't we "proved" that it is the guest OS that is buggy when you can't reproduce it w/ other guest OSes? :) | |
| 12:27:40 | kashyap | (Thx for the guest config) | |
| 12:27:50 | lyarwood | kashyap: I'd just like to capture the actual events to prove it | |
| 12:28:20 | kashyap | Nod. Seems notoriously difficult so far from my interactions | |
| 12:28:52 | kashyap | lyarwood: I guess your approach w/ this rebuilt kernel w/ ACPI debug is to reproduce the prob and watch for output in 'dmesg'? | |
| 12:30:44 | lyarwood | kashyap: yeah, I shouldn't need to rebuild the kernel | |
| 12:30:56 | lyarwood | kashyap: I just need to work out a way of providing command line args to the instances | |
| 12:31:04 | lyarwood | kashyap: and then capture their console logs on failure | |
| 12:39:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [Trivial] Replace ref of policy.json to policy.yaml https://review.opendev.org/749821 | |
| 12:39:52 | gmann | dansmith: sean-k-mooney gibi policy file default change is ready now- https://review.opendev.org/#/c/748059/9 | |
| 12:43:01 | gibi | gmann: ack | |
| 12:45:21 | openstackgerrit | sean mooney proposed openstack/nova master: add functional regression test for bug #1888395 https://review.opendev.org/747454 | |
| 12:45:21 | openstack | bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to sean mooney (sean-k-mooney) | |
| 12:45:21 | openstackgerrit | sean mooney proposed openstack/nova master: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/742180 | |
| 13:10:04 | sean-k-mooney | stephenfin: another one for your review queue https://review.opendev.org/#/q/topic:bug/1860555+(status:open+OR+status:merged) althouhg that is still WIP so lower priority but that might be the cause of our downstream issue | |
| 13:21:08 | sean-k-mooney | gmann: im just poping out to grab lunch but ill try and take a look at the polcy change when i get back. its not really my area but ill take a look in anycase | |
| 13:21:45 | gmann | sure, thanks | |
| 13:22:04 | kashyap | lyarwood: BTW, can you please link to the latest error logs from upstream? I can't find them here - https://zuul.opendev.org/t/openstack/build/9290c83e18a741a5bdab4e28de5eedb7/log/ | |
| 13:22:23 | kashyap | lyarwood: I'm looking for the offending guest QEMU command-line and its guest kernel version | |
| 13:23:08 | gibi | gmann: only have a request in the reno https://review.opendev.org/#/c/748059 but overall looks good to me | |
| 13:23:55 | gmann | gibi: thanks. updating. | |
| 13:23:56 | kashyap | lyarwood: The reason for the above details is because one of the QEMU devs say "lack of CPU time doesn't make sense [as a potential cause], as hot[un]plug events should be porcessed sooner or later" | |
| 13:25:46 | kashyap | lyarwood: I think I should find the logs here (for the latest failing -focal logs): https://review.opendev.org/#/c/734029/ | |
| 13:26:29 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301 is a recent example | |
| 13:26:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:08 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301/log/controller/logs/libvirt/qemu/instance-0000007a_log.txt is the QEMU log for an instance that hit this | |
| 13:27:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:21 | lyarwood | kashyap: 1dec20ff-922e-4bed-a97f-1699f114e74b | |
| 13:27:45 | kashyap | lyarwood: Thank you; do you have the guest kernel version? (Or the CirrOS version - then I can figure out the kernel version) | |
| 13:27:59 | lyarwood | kashyap: pretty sure it's the same as the version I listed earlier | |
| 13:28:06 | lyarwood | 5.3.0-26-generic | |
| 13:28:08 | kashyap | Ah, okay; was about to guess as much. Thank you | |
| 13:28:22 | gmann | gibi: updated, added bug in cmt msg also | |
| 13:28:42 | lyarwood | kashyap: just modified the cirros image in my test env to use debug ACPI btw | |
| 13:28:54 | lyarwood | kashyap: just hacking tempest to dump the console log / dmesg on failure | |
| 13:29:08 | kashyap | Ah, cool | |
| 13:35:34 | gibi | gmann: thanks, +2 | |
| 13:35:57 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [Trivial] Replace ref of policy.json to policy.yaml https://review.opendev.org/749821 | |
| 13:36:22 | gmann | gibi: thanks. ^^ this is trivial one to replace the ref of policy.json in doc and test | |
| 13:37:23 | gibi | looking | |
| 13:45:34 | gibi | sean-k-mooney: I have a question at https://review.opendev.org/#/c/742180/11 | |
| 13:49:57 | bauzas | sean-k-mooney: sooooo, we build the VIFs once we are in the compute service, right? | |
| 13:51:05 | bauzas | well, answering myself | |
| 13:51:21 | bauzas | right, only when we add the fixed IP to an instance | |
| 13:52:26 | bauzas | which is called either after creating the instance in the compute, or when adding the fixed IP directly to an instance by the API... | |
| 14:00:32 | kashyap | lyarwood: So, I just combed through the libvirtd log surrounding the QMP 'device_del' (which does the detach), and here's the little fragment: https://kashyapc.fedorapeople.org/CirrOS_device_detach_issues/libvirtd-log-surrounding-device_del.txt | |
| 14:00:34 | bauzas | gibi: sean-k-mooney: question, should we look at the segments if someone asks the API to put a port to an existance ? | |
| 14:00:59 | bauzas | if so... | |
| 14:01:16 | kashyap | lyarwood: It all looks "clean" until here to me: | |
| 14:01:16 | kashyap | 2020-09-03 20:01:53.019+0000: 65328: debug : qemuMonitorJSONIOProcessEvent:205 : handle DEVICE_DELETED handler=0x7f0230572840 data=0x55d556edf3c0 | |
| 14:01:19 | kashyap | 2020-09-03 20:01:53.019+0000: 65328: debug : qemuMonitorJSONHandleDeviceDeleted:1287 : missing device in device deleted event | |
| 14:01:20 | gibi | port will be bound and I guess neutron will fail the binding if there is no segment on the given host | |
| 14:01:47 | gibi | as far as I remember interface_attach is a call so the error will propagate back the user | |
| 14:01:57 | gibi | bauzas: ^^ | |
| 14:02:18 | bauzas | gibi: ok, so Neutron will check it ? | |
| 14:02:23 | bauzas | if so, fine | |
| 14:02:27 | gibi | I assume, yes | |
| 14:02:30 | bauzas | cool | |
| 14:02:36 | gibi | as neutron would need to assign an ip | |
| 14:02:40 | gibi | during the binding | |
| 14:02:44 | bauzas | anyway, we could provide a caveat documentation if no | |
| 14:03:08 | bauzas | anyway, today is the last day I'm trying to work on this | |
| 14:03:27 | bauzas | gibi: sean-k-mooneyif you have changes you want to me to review, lemme know | |
| 14:03:38 | bauzas | and then I'll review them tomorrow | |
| 14:03:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events https://review.opendev.org/749929 | |
| 14:04:31 | gibi | bauzas: sriov attach is ready, sean-k-mooney is alread +1 on it and the bottom has +2s from stephenfin. series starts here https://review.opendev.org/#/c/741436 | |
| 14:04:41 | bauzas | gibi: ack, will look | |
| 14:04:48 | gibi | thanks! | |
| 14:05:28 | bauzas | gibi: now that I work on some network features, I know better the related files ;) | |
| 14:05:39 | gibi | :) | |
| 14:07:27 | lyarwood | kashyap: yeah that's long after tempest has stopped waiting for the volume to be detached | |
| 14:07:35 | lyarwood | kashyap: let me grab some logs in pastebin | |
| 14:07:55 | kashyap | lyarwood: I've got some contextual stuff here: https://kashyapc.fedorapeople.org/CirrOS_device_detach_issues/ | |
| 14:16:29 | lyarwood | kashyap: http://paste.openstack.org/show/797545/ - AFAICT n-cpu stops trying to detach the volume much earlier than the libvirtd logs you've posted | |
| 14:18:12 | kashyap | lyarwood: Okay, I perhaps need to look further up; let me see if I can see this "Unable to detach" thing in the log | |
| 14:19:14 | kashyap | lyarwood: I'm stumped - I don't see why that "Unable to detach ..." isn't captured here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3a/734029/2/check/devstack-platform-focal/c3ab542/controller/logs/libvirt/libvirtd_log.txt | |
| 14:19:49 | kashyap | (Beaware the above log size: gzip-compressed - 8.2MB; uncompressed - 118MB) | |
| 14:20:00 | lyarwood | 721862 2020-09-03 19:58:35.443+0000: 65331: debug : qemuDomainDeleteDevice:128 : Detaching of device virtio-disk1 failed and no event arrived | |
| 14:20:14 | lyarwood | ^ kashyap I think that's what we are after | |
| 14:20:28 | kashyap | lyarwood: "Huzzah", that's right | |