Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-18
10:36:38 lyarwood okay done ^ gibi I've put the MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING removal patch at the base of that series so we don't hold up your device detach rewrite
11:11:21 gibi lyarwood: thanks. I rebased my effort top of that
11:11:52 gibi lyarwood: btw, the event handling patch seems to work well on the gate, I still have to add some unit tests and I will consider splitting the patch as it is getting huge
11:13:46 lyarwood gibi: ack cool, it's on my list to review later today either way
11:14:04 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: trivial: Resolve (most) flake8 3.x issues https://review.opendev.org/c/openstack/nova/+/770943
11:14:05 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Switch to hacking 2.x https://review.opendev.org/c/openstack/nova/+/770944
11:14:15 gibi lyarwood: have you heard back from libvirt folks about the timeout length in libvirt?
11:14:21 lyarwood elod: ^ not sure how you feel about this btw
11:15:01 lyarwood gibi: oh I didn't reach out to them after our discussion last week as it looks like the failure case is sync and doesn't have a timeout
11:15:15 lyarwood gibi: I can confirm with them if you'd like
11:15:46 songwenping__ sean-k-mooney, gibi, bauzas: hi guys, i review your chat record, do you means when the host is power off, we should set arqs to UNKNOWN state through cyborg-api; when the host is power on and during the cyborg-agent init, we should set arqs to PROVISIONING and create mdevs, and then set them to BOUND when the mdevs created.
11:15:51 gibi lyarwood: I trust your judgement. I misunderstood you previously about the timeout then, I though we have sync and async case as well
11:16:14 lyarwood kashyap: ^ not sure if you'd know, basically we are trying to work out if libvirt has an internal timeout before it emits the DeviceRemovalFailedEvent event
11:16:24 openstackgerrit YumengBao proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116
11:16:30 kashyap lyarwood: Hi, incidentally, just reading here. :-)
11:16:47 gibi but then I we got back to the point where we don't know how long to wait for the event. if we timeout too soon then the retry will cause the same issue in libvirt as before
11:16:49 lyarwood kashyap: AFAICT it doesn't and would just emit that event synchronously on a failure
11:17:36 lyarwood gibi: but I think it's safer to wait for longer now
11:17:39 kashyap lyarwood: Hmm, what's the exact event name? This? VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED
11:17:49 gibi kashyap: yes
11:18:01 lyarwood kashyap: but the failure one
11:18:23 kashyap Right; just trying to find the precise name
11:18:40 lyarwood VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED
11:18:53 gibi lyarwood: we can wait longer yes, I can bump the current 60sec to longer and see what happens on the gate. current I see some timeout in tempest with 60sec (then retry succeeds) so we can at least see if the timeout case go away if I bump the timeout value in the driver
11:19:30 kashyap lyarwood: Okay; Peter Krempa / Michal from upstream libvirt might know; let me check on OFTC
11:19:52 lyarwood gibi: oh it doesn't need to be 60 seconds between attempts
11:20:08 lyarwood gibi: our previous issues were caused with a 5 second retry or lower iirc
11:20:14 gibi lyarwood: it is 60sec now and still see one or two timeout in a tempest run
11:20:24 lyarwood gibi: not between requests
11:20:36 lyarwood gibi: it builds up, the first is 5 seconds
11:21:17 openstackgerrit sean mooney proposed openstack/nova-specs master: add port scoped numa affinity spec https://review.opendev.org/c/openstack/nova-specs/+/765901
11:21:30 kashyap lyarwood: gibi: So, from the libvirt ocmmit that introduced the said event, it says the following:
11:21:33 kashyap [quote]
11:21:35 kashyap Since we didn't opt to use one single event for device lifecycle for a
11:21:38 kashyap VM we are missing one last event if the device removal failed. This
11:21:40 kashyap event will be emitted once we asked to eject the device but for some
11:21:43 kashyap reason it is not possible.
11:21:45 kashyap [/quote]
11:22:38 lyarwood right, I've also looked at the code and it looks like it's just a direct synchronous failure with that call, IOW it doesn't come after a timeout.
11:22:46 kashyap Looking at the code, I don't see particularly any timeout: https://www.redhat.com/archives/libvir-list/2016-April/msg00196.html
11:22:50 kashyap Jinx
11:22:57 kashyap But let me confirm w/ Peter
11:23:26 gibi lyarwood: but then who the event base code can help compared to the current retry code. that retry also wait syncronosly for libvirt to return
11:23:48 gibi if the event also sync then waiting for the event is equivalent to wait for the detach call to return from libvirt
11:23:59 gibi or do I miss something?
11:26:26 gibi I can verify something from the logs. I can check if there is a case when the detach returns before and then the detach event arrives significantly later
11:26:38 gibi if there is such case then waiting for the event make sense
11:26:40 kashyap gibi: Can you give a link to your patch, again, please?
11:26:47 gibi https://review.opendev.org/c/openstack/nova/+/770246
11:26:52 kashyap Thx
11:33:46 kashyap gibi: lyarwood: So ... it indeed looks like a sync failure; danpb on #virt also says this: that event is issued when ACPI refuses the unplug explicitly.
11:34:00 kashyap gibi: lyarwood: But if the guest OS is just slow or not responding, we don't get that event.
11:34:22 gibi kashyap: so there is a case when we eventually need to time out an retry?
11:34:47 gibi the case when the guest OS did not respond
11:35:13 kashyap gibi: Yeah, looks liks so -- how else do we take into acount the the guest OS's slowness or unresponsiveness
11:35:42 gibi but we should not retry too soon to avoid interrupting an ongoing detach in libvirt as that caused our issue in detach in the first place
11:36:15 kashyap Yeah, exactly that too...
11:36:29 kashyap gibi: Also, wait:
11:36:56 kashyap gibi: DanPB reminds: "if you get that event I don't htink you want to retry - that's an unconditional failure IIUC"
11:38:11 gibi kashyap: so if we get the failure event then we fail, if we get timeout we retry. if we get a sync failure about device missing then we retry with persistent=False
11:39:47 Yumeng hi sean-k-mooney, gibi, bauzas. Thank you for the discussion on Jan 15 about vGPU spec. I agree sean's suggestion about host reboot case and  updated a new patch . Pls help to review at your convenience.Thanks! https://review.opendev.org/c/openstack/nova-specs/+/750116
11:40:04 lyarwood gibi: yeah sorry had to go afk
11:40:27 hemanth_n lyarwood: could you please review https://review.opendev.org/c/openstack/nova/+/761725 when you get some time
11:40:40 lyarwood hemanth_n: ack will do
11:40:47 kashyap gibi: How does the sync failure of the missing device manifest?
11:41:15 gibi kashyap: https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#45
11:41:24 lyarwood kashyap: if the device has already been removed from the persistent config but remains in the live config
11:41:27 gibi we get a libvirtError with VIR_ERR_DEVICE_MISSING
11:41:50 gibi as far as I see there is no failed event in this case
11:42:18 lyarwood gibi: yeah I don't think libvirt gets as far as calling QEMU
11:42:29 lyarwood gibi: for QEMU to then emit the event back to libvirt etc
11:42:39 kashyap Thx, both
11:44:06 kashyap Ah, looks like VIR_ERR_DEVICE_MISSING was added in libvirt for Nova :)
11:44:13 gibi ooh :)
11:44:43 kashyap Otherwise, we were `grep`ing for error message; which is unreliable
11:45:04 kashyap Proper error code was due anyway; Nova just uncovered the bug
11:46:13 gibi ahh I know that. I thought there was some deeper reason. Yeas lyarwood just removed the grepping from nova as we have new enough libvirt to rely on the error code instead
11:50:46 gibi Logs look good. I see cases where the detach sequence is the following:
11:50:46 gibi * first attempt times out after 60sec waiting for the event from libvirt
11:50:46 gibi * second attempt fails synchronously with device not found
11:50:47 gibi * third attempt with persistent=False succeeds
11:51:11 gibi so I assume the first attempt only removed the device from the persistent config
11:51:15 gibi hence the second attempt fail
11:51:29 gibi then the third succeed as it does not try to remove it from the persistent config
11:53:08 lyarwood yeah the first should always remove it from the persistent config
11:53:19 lyarwood so you can call the second with persistent=False
11:53:23 lyarwood as we did previously
11:54:42 kashyap gibi: FWIW, I summarized the IRC chat here: https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#2330
11:55:02 gibi lyarwood: still that first timeout seems wrong, why not we get an event
11:55:34 lyarwood gibi: I think it's a single event once detached from both the persistent and live config
11:55:53 gibi lyarwood: and we need to send two detach to have detach it from both config?
11:56:06 gibi if yes then it make sense
12:08:56 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info' https://review.opendev.org/c/openstack/nova/+/744199
12:13:19 lyarwood gibi: one request can detach the device from both
12:13:40 lyarwood gibi: it just depends on the guestOS for the device to be detached from the live config
12:15:20 gibi lyarwood: so the first detach attempt triggers the detach from both the persistent and the live config. The detach call returns, but we timeout waiting for the event. So I have to assume that either a) we didn't wait long enough to get the event b) we _have to_ send two detach to get the device detached from the live config.
12:16:09 gibi or c) libvirt does not send the event in some case
12:16:22 gibi but that seems wrong
12:17:33 lyarwood right so for a and b we retry the detach
12:17:48 lyarwood we can even check if the device is in either config before doing that

Earlier   Later