| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-07 | |||
| 09:18:41 | elod | gibi: thx! \o/ | |
| 09:24:46 | luyao | stephenfin: Hi, not sure whether you saw my message last week, just a kind remind for vpmem-enhencement https://review.opendev.org/#/q/topic:bp/vpmem-enhancement+(status:open+OR+status:merged) . I move the patch 'improve orphans tracking' to the last of the patch sequence (we don't have any big concern on the other 3 patches I think), I redefined those orphans in updated patch since previous version involved the | |
| 09:24:47 | openstack | bug 1879878 in OpenStack Compute (nova) "VM become Error after confirming resize with Error info CPUUnpinningInvalid on source node " [Medium,In progress] https://launchpad.net/bugs/1879878 - Assigned to Stephen Finucane (stephenfinucane) | |
| 09:24:47 | luyao | bug #1879878, and I didn't notice you have fixed it. | |
| 09:25:25 | stephenfin | luyao: ack, will take a look this afternoon | |
| 09:25:43 | luyao | stephenfin: Thank you in advance. :) | |
| 09:34:52 | bauzas | gosh, morning hell here | |
| 09:47:00 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Support vGPU in nova and cyborg interaction https://review.opendev.org/750116 | |
| 09:59:46 | nightmare_unreal | live-migration force node is not working with v2.67 , In v.268 force live migration was completely removed but it should work with v2.67 | |
| 10:10:53 | bauzas | why don't we have good api reference for https://docs.openstack.org/api-ref/network/v2/index.html#subnets ? | |
| 10:11:24 | noonedeadpunk | hi everyone. any chance you know why nova-compute may fail on centos7 for master and ussuri that way? https://zuul.opendev.org/t/openstack/build/6add842202c34390959d5fd0bd6fc83b/log/logs/host/nova-compute.service.journal-15-37-20.log.txt#2729-2768 | |
| 10:13:19 | noonedeadpunk | centos8, debian,ubuntu feel ok with the same setup path and configs | |
| 10:13:45 | bauzas | gibi: do you know how I can know the API reference for getting the subnets information ? | |
| 10:13:54 | bauzas | gibi: looking at https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html#example | |
| 10:14:05 | bauzas | gibi: it looks to me you can get the related segment of a subnet | |
| 10:14:15 | bauzas | openstack subnet show my_subnet --c segment_id | |
| 10:14:56 | bauzas | but i guess it's a neutron extension | |
| 10:17:37 | bauzas | ahah, that's the reference which is confusing | |
| 10:17:56 | bauzas | https://docs.openstack.org/api-ref/network/v2/index.html?expanded=show-subnet-details-detail#show-subnet-details | |
| 10:33:24 | gibi | bauzas: sorry, I was afk | |
| 11:27:12 | lee1 | kashyap: morning, random question, have you had to debug device detach issues before through libvirt and the guestOS and if so do you have any tips? | |
| 11:27:29 | openstack | Launchpad bug 1882521 in OpenStack Compute (nova) "Failing device detachments on Focal" [Critical,Confirmed] - Assigned to Lee Yarwood (lyarwood) | |
| 11:27:29 | lyarwood | kashyap: context is https://bugs.launchpad.net/nova/+bug/1882521 | |
| 11:27:51 | kashyap | Ack; back here in a min :) | |
| 11:28:20 | lyarwood | kashyap: I can reproduce while running the full suite of tests and I'm pretty sure it's just an issue of the guestOS (cirros) not being able to process the request but I just want to prove it somehow | |
| 11:28:23 | lyarwood | ack np | |
| 11:29:34 | kashyap | lyarwood: lee1 was your nick too, I suppose? | |
| 11:32:05 | kashyap | lyarwood: I vaguely recall some triaging some device detach issues; but I forget the details. Gimme a few | |
| 11:32:22 | kashyap | lyarwood: Are you also implying that this is not reproducible with non-CirrOS guests? | |
| 11:34:03 | kashyap | Okay, you say as much in #5 | |
| 11:34:05 | kashyap | "Each time this has been hit however it appears that the Guest OS (cirros) isn't able to react to the ACPI request to detach the disk device. " | |
| 11:34:45 | lyarwood | kashyap: yeah that's my feeling at the moment, I'm looking to prove it now | |
| 11:35:07 | lyarwood | kashyap: just trying to work out how to capture the moment libvirt / QEMU signal the guestOS to detach the device | |
| 11:35:24 | kashyap | lyarwood: Right, configure the debug log filters, it should definitely give us some clues | |
| 11:35:25 | lyarwood | kashyap: and then work out how to capture that in the guest, AFAICT dmesg doesn't list it | |
| 11:35:40 | kashyap | `journalctl`? | |
| 11:37:01 | lyarwood | kashyap: cirros doesn't have systemd | |
| 11:37:08 | kashyap | Darn, I keep forgetting | |
| 11:37:44 | lyarwood | ah it's using acpid | |
| 11:40:43 | sean-k-mooney1 | i need to try and find time to test alpine as a cirros alternitive | |
| 11:40:47 | kashyap | lyarwood: Fedora doesn't do it? | |
| 11:40:56 | kashyap | (As in, 'acpid' daemon) | |
| 11:41:39 | sean-k-mooney1 | fedora uses systemd for udev im not cure if that will handel acpid too | |
| 11:41:50 | sean-k-mooney1 | *sure | |
| 11:45:35 | kashyap | sean-k-mooney: 'systemd' can handle some ACPI events; not all - https://wiki.archlinux.org/index.php/Power_management#ACPI_events | |
| 11:45:43 | kashyap | On my Fedora laptop I see: | |
| 11:46:00 | kashyap | $> systemctl | grep -i acpi sys-devices-platform-thinkpad_acpi-leds-tpacpi::kbd_backlight.device loaded active plugged /sys/devices/platform/thinkpad_acpi/leds/tpacpi::kbd_backlight | |
| 11:46:04 | kashyap | systemd-backlight@leds:tpacpi::kbd_backlight.service loaded active exited Load/Save Screen Backlight Brightness of leds:tpacpi::kbd_backlight | |
| 11:46:15 | sean-k-mooney | lyarwood: so i have been suggesting we should look into useing alpine instead of cirros going forward in the gate. its still does not use systemd but its one of the lightest weight distros i know of and unlike cirros its still maintained regurally | |
| 11:46:19 | kashyap | (So some Thinkpad-related ACPI events are handled) | |
| 11:46:41 | sean-k-mooney | kashyap: that sound like a lenovo extention | |
| 11:46:51 | sean-k-mooney | rather then generic support | |
| 11:47:02 | kashyap | lyarwood: Back to your original question - yeah, we need to find the "event" (IIRC, DEVICE_DELETED - need to double-check) that libvirtsends to the guest OS | |
| 11:47:22 | lyarwood | kashyap: do you know what that actually maps to in terms of what the guestOS sees? | |
| 11:47:41 | lyarwood | kashyap: an ACPI event right but any idea what type etc? | |
| 11:47:52 | kashyap | lyarwood: Not top off my head, perhaps Michal from libvirt might know; he worked on the 'udev' integration | |
| 11:48:25 | lyarwood | kashyap: could you ask and I'll work out a while of capturing that within the guestOS itself | |
| 11:48:43 | kashyap | lyarwood: Yeah, just asked; he's AFK. I'm checking w/ a couple of others | |
| 11:49:02 | lyarwood | sean-k-mooney: tbh it's a little silly that we are using it in CI and running nodes with such little resource as well tbh | |
| 11:49:36 | sean-k-mooney | lyarwood: well we dont have enogh disk/ram to use something much hevier | |
| 11:49:50 | sean-k-mooney | not without reducing concurancy at least | |
| 11:50:41 | sean-k-mooney | cirros made sense when it was activly maintained and updated | |
| 11:59:55 | kashyap | lyarwood: Do you have access to the guest? If so - is this present in it: /sys/module/pci_hotplug? | |
| 12:01:53 | sean-k-mooney | kashyap: cirrus uses a striped down ubuntu 18.04 kernel so it may not be | |
| 12:01:56 | lyarwood | kashyap: yeah that's there, I assume I can enable that | |
| 12:02:04 | lyarwood | kashyap: debug that is | |
| 12:03:09 | lyarwood | and yeah was just reading https://blog.chrishowie.com/2019/09/19/hot-swapping-virtio-disks-on-qemu/ so it's a PCI hot remove with virtio-blk that makes sense | |
| 12:03:27 | sean-k-mooney | yep it is | |
| 12:03:47 | sean-k-mooney | that why i was asserting that virtio-scsi or q35 might help | |
| 12:04:25 | kashyap | lyarwood: So I learn that's the part (the /sys/module/pci_hotplug) which is responsible for hotplug/hotunplug events | |
| 12:04:34 | sean-k-mooney | virtio-scsi woudl be the simplest thing to enable | |
| 12:04:50 | lyarwood | sean-k-mooney: well if it the guestOS can't process the request to detach I don't think changing the underlying bus is going to help tbh | |
| 12:05:05 | kashyap | lyarwood: So I just chatted w/ a couple of QEMU devs; and it seems notoriously difficult to detect this. Way too low-level ... | |
| 12:05:10 | sean-k-mooney | lyarwood: well it wont be a pci hotplug anymore | |
| 12:05:31 | sean-k-mooney | lyarwood: it will be a scsi detach | |
| 12:05:32 | lyarwood | sean-k-mooney: true but the guest would still need to handle the SCSI command (?) to detach | |
| 12:05:52 | gibi | stephenfin: fyi, I have a question in https://review.opendev.org/#/c/746945/6/nova/tests/functional/libvirt/test_pci_sriov_servers.py@a370 | |
| 12:05:57 | sean-k-mooney | yes proably but i think that would be more relyable | |
| 12:06:03 | kashyap | lyarwood: A snippet: | |
| 12:06:04 | kashyap | <kashyap> Hiya, a ranodm question: on monitor command 'device_del' (for device detach), would you happen to know how exactly does it manifest in the guest? | |
| 12:06:07 | kashyap | Answer (from Igor): guest gets SCI interrupt, next thing it reads status from GPE block and calls appropriate AML handler (it's all done within guest kernel) | |
| 12:06:40 | kashyap | Answer 2 (from DanPB): "you'll get <insert hand waving> an ACPI unplug event something in the guest needs to respond to this event for it to complete" | |
| 12:11:02 | jangutter_ | kashyap: on physical hw I've hotplugged and unplugged SATA/SCSI/USB devices for ages, but I've NEVER done so with a PCIe device. | |
| 12:11:38 | sean-k-mooney | gibi: stephenfin can i get your eyes on this https://review.opendev.org/#/c/738432/ | |
| 12:12:08 | sean-k-mooney | i want to get that bug fix merged before m3 if we can so we can backport it to train | |
| 12:12:33 | kashyap | jangutter: Yeap, noted | |
| 12:12:54 | kashyap | lyarwood: So Jiri from libvirt also suggests to get the communication w/ QEMU monitor | |
| 12:13:12 | sean-k-mooney | gibi: stephenfin im also hoping to get https://review.opendev.org/#/q/topic:bug/1888395+(status:open+OR+status:merged) merged soon bug im going to adress artoms nits now | |
| 12:16:43 | lyarwood | kashyap: yeah tracking that, I see the DEVICE_DELETED events | |
| 12:17:07 | lyarwood | kashyap: I've used https://www.kernel.org/doc/html/latest/firmware-guide/acpi/debug.html to enable ACPI debug for the ACPI_PCI_COMPONENT | |
| 12:17:12 | lyarwood | kashyap: within the guestos | |
| 12:17:17 | lyarwood | kashyap: lets see if that helps | |
| 12:17:28 | kashyap | lyarwood: So I just posted #9 | |
| 12:17:40 | kashyap | To copy/paste my point-1 from there: | |
| 12:17:45 | kashyap | "- DEVICE_DELETED is the event that QEMU sends to libvirt, *once* the device was removed by the guest, so that libvirt can clean-up. So if we see DEVICE_DELETED that means the device was successfully detached from QEMU's point of view (therefore, from the guest's PoV, too)" | |
| 12:18:32 | lyarwood | kashyap: right sorry I'm just working out how to instrument things in CI at the moment | |
| 12:18:35 | kashyap | lyarwood: Are you using a new kernel rebuilt with it? | |
| 12:18:43 | lyarwood | kashyap: detach works correctly in the env at the moment | |
| 12:19:02 | lyarwood | kashyap: I'm just trying to figure out what I need to capture during a run to show things are delayed in the guestos | |
| 12:19:28 | lyarwood | kashyap: and yeah 5.3.0-26-generic is the kernel | |