Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-18
15:54:38 gibi lyarwood: I think devstack edits the constraints list based on what repos/libs you asked in the local.conf to be cloned
15:55:09 gibi lyarwood: as far as I understand the issue appears if somehow the pip version gets updated in your devstack vm
15:55:28 gibi lyarwood: for me, in a clean ubuntu 20.04 the first devstack stach.sh works
15:55:45 gibi but subsequnt unstack.sh && stack.sh fails with the above rror
15:56:47 lyarwood gibi: yeah I was about to say I've not touched the version of pip in this env
15:57:10 gibi lyarwood: I suspect that the stach.sh runs some update if it is not in a clean vM
15:57:14 gibi but I have no proof
16:00:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: WIP/DNM libvirt: Start emitting DeviceRemovedEvent and DeviceRemovalFailedEvent events https://review.opendev.org/c/openstack/nova/+/749929
16:00:50 openstackgerrit Balazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach https://review.opendev.org/c/openstack/nova/+/770246
16:02:00 lyarwood gibi: fun, so I've updated pip and it seems to be working now
16:02:15 gibi hm, interesting
16:14:28 stephenfin bauzas: If you've a chance this week, care to look at https://review.opendev.org/q/topic:bp/compact-db-migrations-wallaby ? dansmith and gibi have already looked through it, so I need a final pair of eyes
16:14:43 bauzas sure, will do
16:14:56 stephenfin you can ignore the apidb series at the bottom for now - I suspect I need to rework some of that
16:15:01 stephenfin cool, thanks
16:25:13 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info' https://review.opendev.org/c/openstack/nova/+/744199
16:29:54 openstackgerrit Stephen Finucane proposed openstack/nova master: Add generate schemas tool https://review.opendev.org/c/openstack/nova/+/769796
16:35:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen https://review.opendev.org/c/openstack/nova/+/743231
16:35:52 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754700
16:51:53 openstackgerrit Merged openstack/python-novaclient master: Deprecate agent commands and APIs https://review.opendev.org/c/openstack/python-novaclient/+/769068
16:52:46 gibi lyarwood, kashyap`: back from some testing. If a domain has both a persistent and a live config, then detaching from only the persistent domain first works but does not generate an event. detaching from the live first then detaching from the persistent second does not work none of them generate event (but if the 3rd attempt is live again then that works).
16:53:10 kashyap` gibi: On a call; but reading (as I'm not speaking) ...
16:53:37 gibi so it seems we always need to detach from the persistent first, don't wait for any event, then detach from the live, and wait for event
16:54:19 kashyap` Wait, you're contrasting "persistent" vs. "live" -- my brain was expecting either persistent vs. transient; or live vs offline :-)
16:54:41 gibi the code call them persistent and live
16:54:57 gibi this is where I learned :)
16:55:39 gibi detach from persistent is detach with VIR_DOMAIN_AFFECT_CONFIG
16:55:49 gibi detach from live is detach with VIR_DOMAIN_AFFECT_LIVE flag
16:56:21 kashyap` (Sometimes libvirt terminology gives a damn headache)
16:56:48 kashyap` (For ther record, I was referring to: https://wiki.libvirt.org/page/VM_lifecycle#Transient_guest_domains_vs_Persistent_guest_domains)
16:57:05 kashyap` gibi: Hmm, but I _think_ I see what you mean
16:57:49 kashyap` "live config" often implies it's not persistent (i.e. it is "transient")
16:57:58 kashyap` Either way; back to the original issue
17:00:09 gibi so
17:00:21 gibi I can make it work it just wierdly order dependent
17:00:34 kashyap` gibi: Hmm, so that behaviour of "no event when detaching from live config first" sounds strange and inconsistent.
17:00:35 gibi and also the persistent side does not generate event
17:00:44 gibi kashyap`: yeah
17:01:04 kashyap` gibi: Let me ask Peter; if this is a "bug or a feature" ;-)
17:03:35 gibi kashyap`: could be some kind of complication in my test as well. I try to re-test it with a simpler tempest test as now there is rescue involved as well
17:05:58 kashyap` gibi: I want to phrase the (complex) problem in my own words, so tell me if this is correct:
17:06:18 kashyap` (Because I'm not clear in _which_ combination the event is generated and in which not)
17:06:48 kashyap` - Assume a domain has both a persistent (i.e. inactive) and a live config.
17:06:51 kashyap` - Detach a device from the inactive config first --> detach successful; but does not generate an event
17:06:54 kashyap` - Detach a device from the live config first, then detach it from the persistent config --> the second detach fails; and no event generated in both cases
17:07:37 kashyap` gibi: Is that correct? (With the final point being: if the third attempt in the last step is a live config --> then the detach works)
17:08:05 gibi kashyap`: Detach a device from the live config first, then detach it from the
17:08:09 gibi persistent config
17:08:40 gibi in this case both detach returns without generating event, and the device remains attached (at least in the live config)
17:09:22 kashyap` Aah --> both of them fail (i.e. detach not succesful) and nor do they generate an event?
17:10:30 gibi nova only checks the live config so I can only say that the detach failed from the live config as teh device remains there.
17:10:40 gibi but meanwhile I simplifed the test
17:10:48 gibi and now
17:11:15 gibi I can start with live detach first and it succeeds and generates event as well
17:11:40 gibi so there is some complication my first test scenario that makes the behavior different
17:11:59 kashyap` gibi: The event is _REMOVAL_FAILED, yeah?
17:12:14 kashyap` gibi: If you like, I can start an upstream thread w/ the libvirt folks on libvir-list, and Cc you
17:13:24 gibi kashyap`: when I get event it is VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED
17:13:36 gibi kashyap`: I think I've never seen VIR_DOMAIN_EVENT_ID_DEVICE_REMOVAL_FAILED in my tests ever
17:13:36 kashyap` Err, sorry; it's a succesful removal
17:13:58 gibi kashyap`: let's not rush with that mail, I think I tested too much first
17:14:16 kashyap` gibi: No rush at all :-) Just outlining as an option
17:14:20 gibi and I have to figure out what extra the nova rescue does
17:14:28 gibi that makes the behavior different
17:14:49 gibi because with a simple ACTIVE instance the live first scenario works
17:14:51 gibi as expected
17:14:58 gibi removes the device and generates event
17:15:16 kashyap` Right; gibi - while it's fresh in your mind, do you want to quickly add a note on the scenarios you've tested and the ones that are "weird"?
17:15:35 kashyap` Doesn't have to be now; I know it's EoD in CET. Tomm is fine too
17:15:42 gibi sure I will comment on the patch
17:16:38 gibi lyarwood: you mentioned that I could check the device both in the persistent config and in the live config. do you have a pointer how to check in the persistent config?
17:16:43 kashyap` Excellent; this hot unplug thing is always muddy; so many variables involved.
17:22:42 kashyap` gibi: I wonder if lyarwood meant 'nodedev-list'
17:22:59 kashyap` (No, wait - that's for listing devices on the host)
17:27:09 kashyap` Normally, there should be a '--persitent' flag for most of the listing ('domiflist', 'domblklist', et al) APIs.
17:27:31 gibi kashyap`: yeah, I'm doing this from the nova code
17:27:55 kashyap` gibi: So what do you want to check? Query the presence of a device from peristent config?
17:28:21 gibi yes
17:28:33 gibi nova currently query the live config only
17:31:05 kashyap` gibi: Are you referring to this?
17:31:05 kashyap` device_name_for_logging=self.vif_driver.get_vif_devname(vif),
17:31:08 kashyap` live=live,
17:33:00 gibi kashyap`: https://github.com/openstack/nova/blob/5b65812ef2c6ac46bea30e2e603284eac232339e/nova/virt/libvirt/guest.py#L315
17:33:28 gibi this is how nova checks if a device is in the config and I think it is checking the live config
17:34:06 gibi kashyap`: for vifs the code is here https://github.com/openstack/nova/blob/5b65812ef2c6ac46bea30e2e603284eac232339e/nova/virt/libvirt/guest.py#L230
17:35:10 gibi I guess the key is self._domain.XMLDesc(0) in both case but I failed to find the proper doc what 0 means and what else I can plug there
17:35:31 gibi anyhow I have to drop off
17:35:33 gibi for today
17:35:33 kashyap` gibi: Ah, it's all custom parsing of the XML
17:35:42 gibi yep
17:35:47 kashyap` Yeah, I need to head out shortly too; we can work it out tomm
17:35:55 gibi kashyap`: thanks for the help, let's continue this tomorrow
17:35:59 gibi o/
18:42:02 gmann gibi: stephenfin can either of you check this- https://review.opendev.org/c/openstack/placement/+/764394
18:56:30 elod lyarwood: sorry, I was off today, but will look into the patches tomorrow
19:01:35 openstackgerrit sean mooney proposed openstack/nova-specs master: libvirt: support vdpa kernel networking https://review.opendev.org/c/openstack/nova-specs/+/764999
19:04:02 sean-k-mooney artom: im proably going to call it there for today let me know if there is anything else you want me to adress in ^
19:04:20 sean-k-mooney it will be tomorrow before i get to it proably
19:04:35 artom sean-k-mooney, cool, thanks!
19:04:56 artom Guyze.

Earlier   Later