| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 09:30:24 | gibi | being it a qemu bug I don't really think we as openstack can find a solution | |
| 09:30:43 | bauzas | I do agree | |
| 09:30:47 | gibi | but yes, be clear that we don't know a viable workaround on Focal | |
| 09:31:10 | bauzas | the workaround could be "don't use volumes ?" | |
| 09:31:13 | bauzas | shit, I can't emoji | |
| 09:31:18 | gibi | that is not a WA :D | |
| 09:31:33 | kashyap | gibi: So for the Focal QEMU problem, the moment we can figure out a libvirt-level reproducer that can really speeden up the root-cause-analysis | |
| 09:31:49 | kashyap | gibi: The tricky part here is, it is reproducible only in a Tempest-based setup :-( | |
| 09:32:21 | gibi | kashyap: I see | |
| 09:33:24 | gibi | bauzas: to answer your earlier question about talking to the TC, I haven't doen that (yet). I will raise the issue to them | |
| 09:33:25 | kashyap | gibi: One approach I want to try (once I get these other pressing thing off my chest) is to setup a CPU-intensive workload, and do an attach+detach on a guest launched w/ plain libvirt | |
| 09:33:51 | kashyap | ... and then pray to the Juju under the sea that the issue will reprocuce | |
| 09:34:00 | bauzas | gibi: thanks | |
| 09:39:01 | gibi | bauzas: will you add the limitation to our relesenotes in a separate patch from the https://review.opendev.org/#/c/751045/3/releasenotes/notes/victoria-prelude-9b4c16ff8c6e7f3e.yaml ? | |
| 09:39:09 | gibi | can I approve ^^? | |
| 09:39:39 | bauzas | gibi: yes, I'll add another prelude item in a separate change once lyarwood documents the issue in another reno change | |
| 09:40:11 | gibi | ack | |
| 09:40:14 | gibi | thanks | |
| 09:40:19 | bauzas | gibi: lyarwood: tbc, this is what I see : a documented 'issues' note that explains the problem and the absence of known solution | |
| 09:40:42 | bauzas | and a prelude item that would only refer to this problem | |
| 09:40:56 | gibi | sure, that works for me | |
| 09:46:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | |
| 09:46:26 | openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | |
| 09:46:26 | lyarwood | bauzas: ^ | |
| 09:46:26 | bauzas | hah | |
| 09:46:31 | bauzas | looking now | |
| 09:46:38 | gibi | lyarwood: looking too | |
| 09:47:15 | bauzas | lyarwood: absolutely well written | |
| 09:47:21 | bauzas | lyarwood: I would +3 if I could | |
| 09:47:22 | gibi | looks good to me too | |
| 09:57:42 | lyarwood | stephenfin: https://review.opendev.org/#/c/746984/ thoughts on this btw | |
| 10:02:29 | openstackgerrit | Merged openstack/nova master: fakelibvirt: Use versionutils to set min versions found in the driver https://review.opendev.org/749707 | |
| 10:03:42 | openstackgerrit | Merged openstack/nova master: Victoria 22.0.0 prelude section https://review.opendev.org/751045 | |
| 10:11:24 | gibi | bauzas, lyarwood, gmann: pinged the TC about the our Focal limitation http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017390.html | |
| 10:16:34 | stephenfin | gibi: Do you still have your SR-IOV environment available? If so, could you run a test for me? | |
| 10:16:42 | gibi | stephenfin: sure | |
| 10:16:43 | lyarwood | gibi: ack thanks | |
| 10:16:59 | lyarwood | melwitt: https://review.opendev.org/#/c/747361/ - would you mind hitting this stable backport series today? | |
| 10:17:39 | stephenfin | gibi: Could you try (a) pausing and (b) stopping a guest and then attaching an SR-IOV-based VIF to it? | |
| 10:18:41 | gibi | so try to attach it while it is pause or stopped | |
| 10:18:49 | stephenfin | yup | |
| 10:18:59 | stephenfin | and ideally detach also, but that's less critical | |
| 10:19:05 | gibi | OK, I will check it soon | |
| 10:19:16 | stephenfin | I want to see if libvirt complains and if the guest handles it when it's resumed | |
| 10:19:18 | stephenfin | ack, thanks :) | |
| 10:20:42 | vinay_m | hi , regarding this bug : https://bugs.launchpad.net/nova/+bug/1893238 | |
| 10:20:42 | openstack | Launchpad bug 1893238 in OpenStack Compute (nova) "nova document search result links not working" [Medium,Triaged] | |
| 10:20:58 | stephenfin | (reason being that we call attach_device without the live or persistent flags, so I suspect libvirt might crash and burn without them https://github.com/openstack/nova/blob/f47bde539c8/nova/virt/libvirt/driver.py#L4425) | |
| 10:21:48 | vinay_m | is search implementation is diff/seperate from nova service ? | |
| 10:22:09 | vinay_m | means like is it not involved with nova code part | |
| 10:24:36 | sean-k-mooney | stephenfin: i kind of figured out why the func test failed. its not the fakelibvirt changes | |
| 10:24:52 | sean-k-mooney | at least not directly | |
| 10:25:00 | stephenfin | vinay_m: the search functionality is a Sphinx feature so no, it's not a nova issue | |
| 10:25:29 | sean-k-mooney | the source_dev elemnt is different in one of the comarisons so it does not find the device to detach | |
| 10:27:30 | gibi | stephenfin: boot, pause, attach, unpause. No error, the running guest sees the new device | |
| 10:28:41 | stephenfin | gibi: Okay, cool. I wonder what the point of those flags is so :-\ | |
| 10:28:52 | stephenfin | gibi: Could you run one final check? If you attach the device then stop and start the guest, does it stick around? | |
| 10:29:12 | sean-k-mooney | stephenfin: im really thinking we might want to remove tehm and pass true in all cases | |
| 10:29:52 | gibi | detaching while paused leads to a failure and a retry loop | |
| 10:30:01 | sean-k-mooney | gibi: yep it should | |
| 10:30:29 | vinay_m | stephenfin: thx | |
| 10:30:39 | sean-k-mooney | libvirt will wait for the guest to respond to the acpi hot unplug interupt | |
| 10:30:44 | sean-k-mooney | and since its paused it wont | |
| 10:32:05 | stephenfin | sean-k-mooney: Yeah, I'm not even sure what the point of them is. By passing 'live=True', we'll be passing the 'VIR_DOMAIN_AFFECT_LIVE' flag to the 'attachDeviceFlags' call | |
| 10:32:42 | stephenfin | libvirt says that "VIR_DOMAIN_AFFECT_LIVE specifies that the device shall be allocated to the active domain instance only and is not added to the persisted domain configuration." | |
| 10:33:04 | stephenfin | but if the instance is shutoff, then by definition there isn't an active domain so that should fail | |
| 10:33:24 | stephenfin | "Note that the target hypervisor must return an error if unable to satisfy flags." | |
| 10:33:36 | sean-k-mooney | ah ok | |
| 10:33:46 | sean-k-mooney | i didnt know about the error requiremetn | |
| 10:34:21 | sean-k-mooney | well i did but not with that wording | |
| 10:34:46 | stephenfin | my thinking was simply to set the 'VIR_DOMAIN_AFFECT_LIVE' and 'VIR_DOMAIN_AFFECT_CONFIG' flags based on checking 'domain.isActive()' and 'domain.isPersistent()' respectively | |
| 10:35:00 | stephenfin | but if it's that simple, why do you even need the flags | |
| 10:35:02 | sean-k-mooney | ya | |
| 10:35:21 | sean-k-mooney | i think that should work | |
| 10:35:32 | sean-k-mooney | can you add a patch to try it | |
| 10:35:47 | gibi | stephenfin: boot, attach, stop, start: guest keeps the attached device | |
| 10:35:59 | sean-k-mooney | as i said i know we have had some issues where the live and persistent domains get out of sync | |
| 10:36:15 | stephenfin | weeeeeird | |
| 10:36:23 | stephenfin | might have to ask danpb about this one | |
| 10:36:30 | sean-k-mooney | gibi: start recreated the xml form start so its going to work in the nova cases | |
| 10:36:39 | sean-k-mooney | even if we dont update the persistent domain | |
| 10:37:07 | sean-k-mooney | i kindof wish we did not have a persistnet domain in nova | |
| 10:37:13 | sean-k-mooney | just use teh traisiant one | |
| 10:37:38 | sean-k-mooney | it would remove a subset of bugs | |
| 10:37:43 | stephenfin | sean-k-mooney: ah, that's a fair point | |
| 10:37:58 | lyarwood | sean-k-mooney: I thought that was the plan in W? | |
| 10:38:07 | lyarwood | sean-k-mooney: to look into removing the persistent domain that is | |
| 10:38:13 | stephenfin | I was thinking of reboot in the libvirt sense, where you have to have a persistent domain | |
| 10:38:14 | sean-k-mooney | lyarwood: add it to the ptg/spec list | |
| 10:38:26 | sean-k-mooney | lyarwood: that is what we talked about after that CVE | |
| 10:38:33 | lyarwood | yup doing it now | |
| 10:38:34 | sean-k-mooney | did you bring it up upstream yet | |
| 10:39:20 | sean-k-mooney | lyarwood: stephenfin is looking at the device attach/detach flag we use and tryign to figure out why we pass live/persited true/false in places | |
| 10:39:26 | lyarwood | I thought we had talked about it here but obviously not sorry | |
| 10:39:46 | lyarwood | ah | |
| 10:39:48 | lyarwood | so | |
| 10:39:53 | lyarwood | that's because the initial attempt to detach | |
| 10:40:04 | lyarwood | might only detach from the inactive config | |
| 10:40:24 | lyarwood | after that we only want to try the live config | |
| 10:40:33 | stephenfin | lyarwood: not for PCI devices | |