| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-07 | |||
| 11:45:43 | kashyap | On my Fedora laptop I see: | |
| 11:46:00 | kashyap | $> systemctl | grep -i acpi sys-devices-platform-thinkpad_acpi-leds-tpacpi::kbd_backlight.device loaded active plugged /sys/devices/platform/thinkpad_acpi/leds/tpacpi::kbd_backlight | |
| 11:46:04 | kashyap | systemd-backlight@leds:tpacpi::kbd_backlight.service loaded active exited Load/Save Screen Backlight Brightness of leds:tpacpi::kbd_backlight | |
| 11:46:15 | sean-k-mooney | lyarwood: so i have been suggesting we should look into useing alpine instead of cirros going forward in the gate. its still does not use systemd but its one of the lightest weight distros i know of and unlike cirros its still maintained regurally | |
| 11:46:19 | kashyap | (So some Thinkpad-related ACPI events are handled) | |
| 11:46:41 | sean-k-mooney | kashyap: that sound like a lenovo extention | |
| 11:46:51 | sean-k-mooney | rather then generic support | |
| 11:47:02 | kashyap | lyarwood: Back to your original question - yeah, we need to find the "event" (IIRC, DEVICE_DELETED - need to double-check) that libvirtsends to the guest OS | |
| 11:47:22 | lyarwood | kashyap: do you know what that actually maps to in terms of what the guestOS sees? | |
| 11:47:41 | lyarwood | kashyap: an ACPI event right but any idea what type etc? | |
| 11:47:52 | kashyap | lyarwood: Not top off my head, perhaps Michal from libvirt might know; he worked on the 'udev' integration | |
| 11:48:25 | lyarwood | kashyap: could you ask and I'll work out a while of capturing that within the guestOS itself | |
| 11:48:43 | kashyap | lyarwood: Yeah, just asked; he's AFK. I'm checking w/ a couple of others | |
| 11:49:02 | lyarwood | sean-k-mooney: tbh it's a little silly that we are using it in CI and running nodes with such little resource as well tbh | |
| 11:49:36 | sean-k-mooney | lyarwood: well we dont have enogh disk/ram to use something much hevier | |
| 11:49:50 | sean-k-mooney | not without reducing concurancy at least | |
| 11:50:41 | sean-k-mooney | cirros made sense when it was activly maintained and updated | |
| 11:59:55 | kashyap | lyarwood: Do you have access to the guest? If so - is this present in it: /sys/module/pci_hotplug? | |
| 12:01:53 | sean-k-mooney | kashyap: cirrus uses a striped down ubuntu 18.04 kernel so it may not be | |
| 12:01:56 | lyarwood | kashyap: yeah that's there, I assume I can enable that | |
| 12:02:04 | lyarwood | kashyap: debug that is | |
| 12:03:09 | lyarwood | and yeah was just reading https://blog.chrishowie.com/2019/09/19/hot-swapping-virtio-disks-on-qemu/ so it's a PCI hot remove with virtio-blk that makes sense | |
| 12:03:27 | sean-k-mooney | yep it is | |
| 12:03:47 | sean-k-mooney | that why i was asserting that virtio-scsi or q35 might help | |
| 12:04:25 | kashyap | lyarwood: So I learn that's the part (the /sys/module/pci_hotplug) which is responsible for hotplug/hotunplug events | |
| 12:04:34 | sean-k-mooney | virtio-scsi woudl be the simplest thing to enable | |
| 12:04:50 | lyarwood | sean-k-mooney: well if it the guestOS can't process the request to detach I don't think changing the underlying bus is going to help tbh | |
| 12:05:05 | kashyap | lyarwood: So I just chatted w/ a couple of QEMU devs; and it seems notoriously difficult to detect this. Way too low-level ... | |
| 12:05:10 | sean-k-mooney | lyarwood: well it wont be a pci hotplug anymore | |
| 12:05:31 | sean-k-mooney | lyarwood: it will be a scsi detach | |
| 12:05:32 | lyarwood | sean-k-mooney: true but the guest would still need to handle the SCSI command (?) to detach | |
| 12:05:52 | gibi | stephenfin: fyi, I have a question in https://review.opendev.org/#/c/746945/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py@a370 | |
| 12:05:57 | sean-k-mooney | yes proably but i think that would be more relyable | |
| 12:06:03 | kashyap | lyarwood: A snippet: | |
| 12:06:04 | kashyap | <kashyap> Hiya, a ranodm question: on monitor command 'device_del' (for device detach), would you happen to know how exactly does it manifest in the guest? | |
| 12:06:07 | kashyap | Answer (from Igor): guest gets SCI interrupt, next thing it reads status from GPE block and calls appropriate AML handler (it's all done within guest kernel) | |
| 12:06:40 | kashyap | Answer 2 (from DanPB): "you'll get <insert hand waving> an ACPI unplug event something in the guest needs to respond to this event for it to complete" | |
| 12:11:02 | jangutter_ | kashyap: on physical hw I've hotplugged and unplugged SATA/SCSI/USB devices for ages, but I've NEVER done so with a PCIe device. | |
| 12:11:38 | sean-k-mooney | gibi: stephenfin can i get your eyes on this https://review.opendev.org/#/c/738432/ | |
| 12:12:08 | sean-k-mooney | i want to get that bug fix merged before m3 if we can so we can backport it to train | |
| 12:12:33 | kashyap | jangutter: Yeap, noted | |
| 12:12:54 | kashyap | lyarwood: So Jiri from libvirt also suggests to get the communication w/ QEMU monitor | |
| 12:13:12 | sean-k-mooney | gibi: stephenfin im also hoping to get https://review.opendev.org/#/q/topic:bug/1888395+(status:open+OR+status:merged) merged soon bug im going to adress artoms nits now | |
| 12:16:43 | lyarwood | kashyap: yeah tracking that, I see the DEVICE_DELETED events | |
| 12:17:07 | lyarwood | kashyap: I've used https://www.kernel.org/doc/html/latest/firmware-guide/acpi/debug.html to enable ACPI debug for the ACPI_PCI_COMPONENT | |
| 12:17:12 | lyarwood | kashyap: within the guestos | |
| 12:17:17 | lyarwood | kashyap: lets see if that helps | |
| 12:17:28 | kashyap | lyarwood: So I just posted #9 | |
| 12:17:40 | kashyap | To copy/paste my point-1 from there: | |
| 12:17:45 | kashyap | "- DEVICE_DELETED is the event that QEMU sends to libvirt, *once* the device was removed by the guest, so that libvirt can clean-up. So if we see DEVICE_DELETED that means the device was successfully detached from QEMU's point of view (therefore, from the guest's PoV, too)" | |
| 12:18:32 | lyarwood | kashyap: right sorry I'm just working out how to instrument things in CI at the moment | |
| 12:18:35 | kashyap | lyarwood: Are you using a new kernel rebuilt with it? | |
| 12:18:43 | lyarwood | kashyap: detach works correctly in the env at the moment | |
| 12:19:02 | lyarwood | kashyap: I'm just trying to figure out what I need to capture during a run to show things are delayed in the guestos | |
| 12:19:28 | lyarwood | kashyap: and yeah 5.3.0-26-generic is the kernel | |
| 12:20:27 | kashyap | lyarwood: So, Igor (KVM/QEMU dev) says: "You'd could watch for udev events as indirect result of unplug events for specific device subsystem" | |
| 12:21:26 | lyarwood | kashyap: I don't think cirros is using udev tbh | |
| 12:22:17 | kashyap | lyarwood: Nod; I've actually snipped out his first part where he admits he isn't familiar w/ 'acpid' | |
| 12:23:48 | lyarwood | https://git.busybox.net/busybox/tree/util-linux/acpid.c it's not even the old version I was used to tbh | |
| 12:25:17 | kashyap | lyarwood: I'm curious if your test with slightly "better resources" for the guest fixes it | |
| 12:25:33 | kashyap | lyarwood: Also can you tell what's the buggy guest configuration? If you don't mind posting the guest XML... | |
| 12:25:50 | lyarwood | kashyap: I still saw a few failures | |
| 12:26:14 | kashyap | So it's not the resources allocated to the guest | |
| 12:26:28 | lyarwood | kashyap: that was in reference to the host guest running openstack FWIW | |
| 12:26:37 | lyarwood | kashyap: correct | |
| 12:26:50 | lyarwood | kashyap: CI nodes run with 1 vCPU and 8GB of RAM at the moment | |
| 12:27:06 | lyarwood | kashyap: the instances have 1 vCPU and 128MB of RAM | |
| 12:27:29 | kashyap | lyarwood: BTW, haven't we "proved" that it is the guest OS that is buggy when you can't reproduce it w/ other guest OSes? :) | |
| 12:27:40 | kashyap | (Thx for the guest config) | |
| 12:27:50 | lyarwood | kashyap: I'd just like to capture the actual events to prove it | |
| 12:28:20 | kashyap | Nod. Seems notoriously difficult so far from my interactions | |
| 12:28:52 | kashyap | lyarwood: I guess your approach w/ this rebuilt kernel w/ ACPI debug is to reproduce the prob and watch for output in 'dmesg'? | |
| 12:30:44 | lyarwood | kashyap: yeah, I shouldn't need to rebuild the kernel | |
| 12:30:56 | lyarwood | kashyap: I just need to work out a way of providing command line args to the instances | |
| 12:31:04 | lyarwood | kashyap: and then capture their console logs on failure | |
| 12:39:06 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: [Trivial] Replace ref of policy.json to policy.yaml https://review.opendev.org/749821 | |
| 12:39:52 | gmann | dansmith: sean-k-mooney gibi policy file default change is ready now- https://review.opendev.org/#/c/748059/9 | |
| 12:43:01 | gibi | gmann: ack | |
| 12:45:21 | openstackgerrit | sean mooney proposed openstack/nova master: add functional regression test for bug #1888395 https://review.opendev.org/747454 | |
| 12:45:21 | openstack | bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to sean mooney (sean-k-mooney) | |
| 12:45:21 | openstackgerrit | sean mooney proposed openstack/nova master: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/742180 | |
| 13:10:04 | sean-k-mooney | stephenfin: another one for your review queue https://review.opendev.org/#/q/topic:bug/1860555+(status:open+OR+status:merged) althouhg that is still WIP so lower priority but that might be the cause of our downstream issue | |
| 13:21:08 | sean-k-mooney | gmann: im just poping out to grab lunch but ill try and take a look at the polcy change when i get back. its not really my area but ill take a look in anycase | |
| 13:21:45 | gmann | sure, thanks | |
| 13:22:04 | kashyap | lyarwood: BTW, can you please link to the latest error logs from upstream? I can't find them here - https://zuul.opendev.org/t/openstack/build/9290c83e18a741a5bdab4e28de5eedb7/log/ | |
| 13:22:23 | kashyap | lyarwood: I'm looking for the offending guest QEMU command-line and its guest kernel version | |
| 13:23:08 | gibi | gmann: only have a request in the reno https://review.opendev.org/#/c/748059 but overall looks good to me | |
| 13:23:55 | gmann | gibi: thanks. updating. | |
| 13:23:56 | kashyap | lyarwood: The reason for the above details is because one of the QEMU devs say "lack of CPU time doesn't make sense [as a potential cause], as hot[un]plug events should be porcessed sooner or later" | |
| 13:25:46 | kashyap | lyarwood: I think I should find the logs here (for the latest failing -focal logs): https://review.opendev.org/#/c/734029/ | |
| 13:26:29 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301 is a recent example | |
| 13:26:36 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:08 | lyarwood | kashyap: https://zuul.opendev.org/t/openstack/build/eee0dc94780c4555b376f17c4f50c301/log/controller/logs/libvirt/qemu/instance-0000007a_log.txt is the QEMU log for an instance that hit this | |
| 13:27:11 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | |
| 13:27:21 | lyarwood | kashyap: 1dec20ff-922e-4bed-a97f-1699f114e74b | |
| 13:27:45 | kashyap | lyarwood: Thank you; do you have the guest kernel version? (Or the CirrOS version - then I can figure out the kernel version) | |
| 13:27:59 | lyarwood | kashyap: pretty sure it's the same as the version I listed earlier | |
| 13:28:06 | lyarwood | 5.3.0-26-generic | |
| 13:28:08 | kashyap | Ah, okay; was about to guess as much. Thank you | |
| 13:28:22 | gmann | gibi: updated, added bug in cmt msg also | |