| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 09:15:39 | kashyap | lyarwood: Is this the Focal topic? | |
| 09:16:25 | kashyap | stephenfin: gibi: lyarwood: In general: in the past I've gone to Ubuntu's -stable IRC channel to point out missing virt-related backports | |
| 09:16:30 | kashyap | But that's not scalable... | |
| 09:17:20 | bauzas | I'm sure that raising the problem to the TC and eventually the Foundation would send a signal that Canonical would be foolish to not capture | |
| 09:18:07 | bauzas | because if we eventually say "sorry, but you can race on volume attach/detach with Focal and you have no other options but to consider other distros for this release if you BFV", that would certainly put some prios | |
| 09:18:30 | bauzas | that's my expectation at least | |
| 09:20:12 | bauzas | lyarwood: do you feel brave enough to write the problem and the absence of solution in a reno note ? | |
| 09:25:53 | lyarwood | bauzas: I can try | |
| 09:26:01 | bauzas | thanks | |
| 09:27:04 | gibi | so on Focal the only solution is to downgrade libvirt? | |
| 09:27:20 | gibi | so on Focal the only solution is to downgrade qemu | |
| 09:27:25 | bauzas | gibi: there is no solution for Focal, IIUC | |
| 09:27:36 | bauzas | that's the problem | |
| 09:27:57 | bauzas | or I'd rather say "there is no solution found yet for Focal" | |
| 09:28:11 | bauzas | this leaves people investigating a potential workaround | |
| 09:28:14 | gibi | there is no way to use an older qemu there? | |
| 09:28:29 | bauzas | gibi: lyarwood said this is though to downgrade | |
| 09:28:38 | bauzas | due to the canonical support model | |
| 09:28:54 | bauzas | we would then ask to install an old qemu version that's not supporter | |
| 09:28:56 | bauzas | supported | |
| 09:29:00 | lyarwood | gibi / bauzas ; Yeah I can't speak for Canonical but we (RH) wouldn't support that with our distros | |
| 09:29:33 | bauzas | which makes sense tbh | |
| 09:29:37 | gibi | OK, then simply state what we know in the reno. | |
| 09:29:45 | gibi | don't have to be sugarcoated | |
| 09:29:49 | lyarwood | kk | |
| 09:29:50 | gibi | there is an open qemu bug to link to | |
| 09:29:52 | bauzas | gibi: we also need to state we haven't identified yet a solution. | |
| 09:30:23 | bauzas | if I was an operator, I'd surely reconsider upgrading if I feel I could be impacted | |
| 09:30:24 | gibi | being it a qemu bug I don't really think we as openstack can find a solution | |
| 09:30:43 | bauzas | I do agree | |
| 09:30:47 | gibi | but yes, be clear that we don't know a viable workaround on Focal | |
| 09:31:10 | bauzas | the workaround could be "don't use volumes ?" | |
| 09:31:13 | bauzas | shit, I can't emoji | |
| 09:31:18 | gibi | that is not a WA :D | |
| 09:31:33 | kashyap | gibi: So for the Focal QEMU problem, the moment we can figure out a libvirt-level reproducer that can really speeden up the root-cause-analysis | |
| 09:31:49 | kashyap | gibi: The tricky part here is, it is reproducible only in a Tempest-based setup :-( | |
| 09:32:21 | gibi | kashyap: I see | |
| 09:33:24 | gibi | bauzas: to answer your earlier question about talking to the TC, I haven't doen that (yet). I will raise the issue to them | |
| 09:33:25 | kashyap | gibi: One approach I want to try (once I get these other pressing thing off my chest) is to setup a CPU-intensive workload, and do an attach+detach on a guest launched w/ plain libvirt | |
| 09:33:51 | kashyap | ... and then pray to the Juju under the sea that the issue will reprocuce | |
| 09:34:00 | bauzas | gibi: thanks | |
| 09:39:01 | gibi | bauzas: will you add the limitation to our relesenotes in a separate patch from the https://review.opendev.org/#/c/751045/3/releasenotes/notes/victoria-prelude-9b4c16ff8c6e7f3e.yaml ? | |
| 09:39:09 | gibi | can I approve ^^? | |
| 09:39:39 | bauzas | gibi: yes, I'll add another prelude item in a separate change once lyarwood documents the issue in another reno change | |
| 09:40:11 | gibi | ack | |
| 09:40:14 | gibi | thanks | |
| 09:40:19 | bauzas | gibi: lyarwood: tbc, this is what I see : a documented 'issues' note that explains the problem and the absence of known solution | |
| 09:40:42 | bauzas | and a prelude item that would only refer to this problem | |
| 09:40:56 | gibi | sure, that works for me | |
| 09:46:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: releasenote: Add known issue for bug #1894804 https://review.opendev.org/752654 | |
| 09:46:26 | openstack | bug 1894804 in qemu (Ubuntu) "Second DEVICE_DELETED event missing during virtio-blk disk device detach" [Undecided,New] https://launchpad.net/bugs/1894804 | |
| 09:46:26 | lyarwood | bauzas: ^ | |
| 09:46:26 | bauzas | hah | |
| 09:46:31 | bauzas | looking now | |
| 09:46:38 | gibi | lyarwood: looking too | |
| 09:47:15 | bauzas | lyarwood: absolutely well written | |
| 09:47:21 | bauzas | lyarwood: I would +3 if I could | |
| 09:47:22 | gibi | looks good to me too | |
| 09:57:42 | lyarwood | stephenfin: https://review.opendev.org/#/c/746984/ thoughts on this btw | |
| 10:02:29 | openstackgerrit | Merged openstack/nova master: fakelibvirt: Use versionutils to set min versions found in the driver https://review.opendev.org/749707 | |
| 10:03:42 | openstackgerrit | Merged openstack/nova master: Victoria 22.0.0 prelude section https://review.opendev.org/751045 | |
| 10:11:24 | gibi | bauzas, lyarwood, gmann: pinged the TC about the our Focal limitation http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017390.html | |
| 10:16:34 | stephenfin | gibi: Do you still have your SR-IOV environment available? If so, could you run a test for me? | |
| 10:16:42 | gibi | stephenfin: sure | |
| 10:16:43 | lyarwood | gibi: ack thanks | |
| 10:16:59 | lyarwood | melwitt: https://review.opendev.org/#/c/747361/ - would you mind hitting this stable backport series today? | |
| 10:17:39 | stephenfin | gibi: Could you try (a) pausing and (b) stopping a guest and then attaching an SR-IOV-based VIF to it? | |
| 10:18:41 | gibi | so try to attach it while it is pause or stopped | |
| 10:18:49 | stephenfin | yup | |
| 10:18:59 | stephenfin | and ideally detach also, but that's less critical | |
| 10:19:05 | gibi | OK, I will check it soon | |
| 10:19:16 | stephenfin | I want to see if libvirt complains and if the guest handles it when it's resumed | |
| 10:19:18 | stephenfin | ack, thanks :) | |
| 10:20:42 | vinay_m | hi , regarding this bug : https://bugs.launchpad.net/nova/+bug/1893238 | |
| 10:20:42 | openstack | Launchpad bug 1893238 in OpenStack Compute (nova) "nova document search result links not working" [Medium,Triaged] | |
| 10:20:58 | stephenfin | (reason being that we call attach_device without the live or persistent flags, so I suspect libvirt might crash and burn without them https://github.com/openstack/nova/blob/f47bde539c8/nova/virt/libvirt/driver.py#L4425) | |
| 10:21:48 | vinay_m | is search implementation is diff/seperate from nova service ? | |
| 10:22:09 | vinay_m | means like is it not involved with nova code part | |
| 10:24:36 | sean-k-mooney | stephenfin: i kind of figured out why the func test failed. its not the fakelibvirt changes | |
| 10:24:52 | sean-k-mooney | at least not directly | |
| 10:25:00 | stephenfin | vinay_m: the search functionality is a Sphinx feature so no, it's not a nova issue | |
| 10:25:29 | sean-k-mooney | the source_dev elemnt is different in one of the comarisons so it does not find the device to detach | |
| 10:27:30 | gibi | stephenfin: boot, pause, attach, unpause. No error, the running guest sees the new device | |
| 10:28:41 | stephenfin | gibi: Okay, cool. I wonder what the point of those flags is so :-\ | |
| 10:28:52 | stephenfin | gibi: Could you run one final check? If you attach the device then stop and start the guest, does it stick around? | |
| 10:29:12 | sean-k-mooney | stephenfin: im really thinking we might want to remove tehm and pass true in all cases | |
| 10:29:52 | gibi | detaching while paused leads to a failure and a retry loop | |
| 10:30:01 | sean-k-mooney | gibi: yep it should | |
| 10:30:29 | vinay_m | stephenfin: thx | |
| 10:30:39 | sean-k-mooney | libvirt will wait for the guest to respond to the acpi hot unplug interupt | |
| 10:30:44 | sean-k-mooney | and since its paused it wont | |
| 10:32:05 | stephenfin | sean-k-mooney: Yeah, I'm not even sure what the point of them is. By passing 'live=True', we'll be passing the 'VIR_DOMAIN_AFFECT_LIVE' flag to the 'attachDeviceFlags' call | |
| 10:32:42 | stephenfin | libvirt says that "VIR_DOMAIN_AFFECT_LIVE specifies that the device shall be allocated to the active domain instance only and is not added to the persisted domain configuration." | |
| 10:33:04 | stephenfin | but if the instance is shutoff, then by definition there isn't an active domain so that should fail | |
| 10:33:24 | stephenfin | "Note that the target hypervisor must return an error if unable to satisfy flags." | |
| 10:33:36 | sean-k-mooney | ah ok | |
| 10:33:46 | sean-k-mooney | i didnt know about the error requiremetn | |
| 10:34:21 | sean-k-mooney | well i did but not with that wording | |
| 10:34:46 | stephenfin | my thinking was simply to set the 'VIR_DOMAIN_AFFECT_LIVE' and 'VIR_DOMAIN_AFFECT_CONFIG' flags based on checking 'domain.isActive()' and 'domain.isPersistent()' respectively | |
| 10:35:00 | stephenfin | but if it's that simple, why do you even need the flags | |