Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-18
02:06:41 openstackgerrit Keigo Noha proposed openstack/nova stable/rocky: Change default num_retries for glance to 3 https://review.opendev.org/c/openstack/nova/+/771120
07:04:55 openstackgerrit Adit Sarfaty proposed openstack/nova master: Retry on vmware create_vm when it fails https://review.opendev.org/c/openstack/nova/+/764586
09:20:12 openstackgerrit Lee Yarwood proposed openstack/nova master: api: Log os-resetState as an instance action https://review.opendev.org/c/openstack/nova/+/770926
09:20:41 lyarwood ^ stephenfin should be good now, there was however a weird but unrelated grenade failure in the last run
09:20:53 lyarwood https://bugs.launchpad.net/tempest/+bug/1912178
09:20:55 openstack Launchpad bug 1912178 in tempest "Links are not allowed as constraints (pip 20.2.4)" [Undecided,New]
09:20:55 stephenfin ack
10:08:17 lyarwood did we depreacte xen support in the libvirt driver?
10:10:08 lyarwood https://review.opendev.org/c/openstack/nova/+/749055 yup
10:10:40 lyarwood stephenfin: you didn't have a series to remove xen already did you?
10:10:52 stephenfin from libvirt?
10:10:57 stephenfin I did, yeah
10:11:15 lyarwood ah where? I only see the xenapi stuff
10:11:26 stephenfin https://review.opendev.org/q/topic:%22bp%252Fremove-deprecated-libvirt-virt-types%22+(status:open%20OR%20status:merged)
10:11:42 lyarwood *facepalm* sorry the same topic, of course!
10:12:07 lyarwood I'll rebase my version bump on that series and give it a review
10:36:33 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702
10:36:34 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Drop support for UML https://review.opendev.org/c/openstack/nova/+/743230
10:36:35 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen https://review.opendev.org/c/openstack/nova/+/743231
10:36:35 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754700
10:36:38 lyarwood okay done ^ gibi I've put the MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING removal patch at the base of that series so we don't hold up your device detach rewrite
11:11:21 gibi lyarwood: thanks. I rebased my effort top of that
11:11:52 gibi lyarwood: btw, the event handling patch seems to work well on the gate, I still have to add some unit tests and I will consider splitting the patch as it is getting huge
11:13:46 lyarwood gibi: ack cool, it's on my list to review later today either way
11:14:04 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: trivial: Resolve (most) flake8 3.x issues https://review.opendev.org/c/openstack/nova/+/770943
11:14:05 openstackgerrit Lee Yarwood proposed openstack/nova stable/train: Switch to hacking 2.x https://review.opendev.org/c/openstack/nova/+/770944
11:14:15 gibi lyarwood: have you heard back from libvirt folks about the timeout length in libvirt?
11:14:21 lyarwood elod: ^ not sure how you feel about this btw
11:15:01 lyarwood gibi: oh I didn't reach out to them after our discussion last week as it looks like the failure case is sync and doesn't have a timeout
11:15:15 lyarwood gibi: I can confirm with them if you'd like
11:15:46 songwenping__ sean-k-mooney, gibi, bauzas: hi guys, i review your chat record, do you means when the host is power off, we should set arqs to UNKNOWN state through cyborg-api; when the host is power on and during the cyborg-agent init, we should set arqs to PROVISIONING and create mdevs, and then set them to BOUND when the mdevs created.
11:15:51 gibi lyarwood: I trust your judgement. I misunderstood you previously about the timeout then, I though we have sync and async case as well
11:16:14 lyarwood kashyap: ^ not sure if you'd know, basically we are trying to work out if libvirt has an internal timeout before it emits the DeviceRemovalFailedEvent event
11:16:24 openstackgerrit YumengBao proposed openstack/nova-specs master: Libvirt supports composing cyborg owned vGPU accelerator into domain XML https://review.opendev.org/c/openstack/nova-specs/+/750116
11:16:30 kashyap lyarwood: Hi, incidentally, just reading here. :-)
11:16:47 gibi but then I we got back to the point where we don't know how long to wait for the event. if we timeout too soon then the retry will cause the same issue in libvirt as before
11:16:49 lyarwood kashyap: AFAICT it doesn't and would just emit that event synchronously on a failure
11:17:36 lyarwood gibi: but I think it's safer to wait for longer now
11:17:39 kashyap lyarwood: Hmm, what's the exact event name? This? VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED
11:17:49 gibi kashyap: yes
11:18:01 lyarwood kashyap: but the failure one
11:18:23 kashyap Right; just trying to find the precise name
11:18:40 lyarwood VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED
11:18:53 gibi lyarwood: we can wait longer yes, I can bump the current 60sec to longer and see what happens on the gate. current I see some timeout in tempest with 60sec (then retry succeeds) so we can at least see if the timeout case go away if I bump the timeout value in the driver
11:19:30 kashyap lyarwood: Okay; Peter Krempa / Michal from upstream libvirt might know; let me check on OFTC
11:19:52 lyarwood gibi: oh it doesn't need to be 60 seconds between attempts
11:20:08 lyarwood gibi: our previous issues were caused with a 5 second retry or lower iirc
11:20:14 gibi lyarwood: it is 60sec now and still see one or two timeout in a tempest run
11:20:24 lyarwood gibi: not between requests
11:20:36 lyarwood gibi: it builds up, the first is 5 seconds
11:21:17 openstackgerrit sean mooney proposed openstack/nova-specs master: add port scoped numa affinity spec https://review.opendev.org/c/openstack/nova-specs/+/765901
11:21:30 kashyap lyarwood: gibi: So, from the libvirt ocmmit that introduced the said event, it says the following:
11:21:33 kashyap [quote]
11:21:35 kashyap Since we didn't opt to use one single event for device lifecycle for a
11:21:38 kashyap VM we are missing one last event if the device removal failed. This
11:21:40 kashyap event will be emitted once we asked to eject the device but for some
11:21:43 kashyap reason it is not possible.
11:21:45 kashyap [/quote]
11:22:38 lyarwood right, I've also looked at the code and it looks like it's just a direct synchronous failure with that call, IOW it doesn't come after a timeout.
11:22:46 kashyap Looking at the code, I don't see particularly any timeout: https://www.redhat.com/archives/libvir-list/2016-April/msg00196.html
11:22:50 kashyap Jinx
11:22:57 kashyap But let me confirm w/ Peter
11:23:26 gibi lyarwood: but then who the event base code can help compared to the current retry code. that retry also wait syncronosly for libvirt to return
11:23:48 gibi if the event also sync then waiting for the event is equivalent to wait for the detach call to return from libvirt
11:23:59 gibi or do I miss something?
11:26:26 gibi I can verify something from the logs. I can check if there is a case when the detach returns before and then the detach event arrives significantly later
11:26:38 gibi if there is such case then waiting for the event make sense
11:26:40 kashyap gibi: Can you give a link to your patch, again, please?
11:26:47 gibi https://review.opendev.org/c/openstack/nova/+/770246
11:26:52 kashyap Thx
11:33:46 kashyap gibi: lyarwood: So ... it indeed looks like a sync failure; danpb on #virt also says this: that event is issued when ACPI refuses the unplug explicitly.
11:34:00 kashyap gibi: lyarwood: But if the guest OS is just slow or not responding, we don't get that event.
11:34:22 gibi kashyap: so there is a case when we eventually need to time out an retry?
11:34:47 gibi the case when the guest OS did not respond
11:35:13 kashyap gibi: Yeah, looks liks so -- how else do we take into acount the the guest OS's slowness or unresponsiveness
11:35:42 gibi but we should not retry too soon to avoid interrupting an ongoing detach in libvirt as that caused our issue in detach in the first place
11:36:15 kashyap Yeah, exactly that too...
11:36:29 kashyap gibi: Also, wait:
11:36:56 kashyap gibi: DanPB reminds: "if you get that event I don't htink you want to retry - that's an unconditional failure IIUC"
11:38:11 gibi kashyap: so if we get the failure event then we fail, if we get timeout we retry. if we get a sync failure about device missing then we retry with persistent=False
11:39:47 Yumeng hi sean-k-mooney, gibi, bauzas. Thank you for the discussion on Jan 15 about vGPU spec. I agree sean's suggestion about host reboot case and  updated a new patch . Pls help to review at your convenience.Thanks! https://review.opendev.org/c/openstack/nova-specs/+/750116
11:40:04 lyarwood gibi: yeah sorry had to go afk
11:40:27 hemanth_n lyarwood: could you please review https://review.opendev.org/c/openstack/nova/+/761725 when you get some time
11:40:40 lyarwood hemanth_n: ack will do
11:40:47 kashyap gibi: How does the sync failure of the missing device manifest?
11:41:15 gibi kashyap: https://review.opendev.org/c/openstack/nova/+/770246/7/nova/virt/libvirt/driver.py#45
11:41:24 lyarwood kashyap: if the device has already been removed from the persistent config but remains in the live config
11:41:27 gibi we get a libvirtError with VIR_ERR_DEVICE_MISSING
11:41:50 gibi as far as I see there is no failed event in this case
11:42:18 lyarwood gibi: yeah I don't think libvirt gets as far as calling QEMU
11:42:29 lyarwood gibi: for QEMU to then emit the event back to libvirt etc
11:42:39 kashyap Thx, both
11:44:06 kashyap Ah, looks like VIR_ERR_DEVICE_MISSING was added in libvirt for Nova :)
11:44:13 gibi ooh :)
11:44:43 kashyap Otherwise, we were `grep`ing for error message; which is unreliable
11:45:04 kashyap Proper error code was due anyway; Nova just uncovered the bug
11:46:13 gibi ahh I know that. I thought there was some deeper reason. Yeas lyarwood just removed the grepping from nova as we have new enough libvirt to rely on the error code instead
11:50:46 gibi Logs look good. I see cases where the detach sequence is the following:
11:50:46 gibi * first attempt times out after 60sec waiting for the event from libvirt
11:50:46 gibi * second attempt fails synchronously with device not found

Earlier   Later