| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-18 | |||
| 09:01:28 | lyarwood | ubuntu-- https://zuul.opendev.org/t/openstack/build/fc4fd2d6fcd14618899060c4e84edca0/log/job-output.txt#6579 one small thing to fix here for the evacuation tests also | |
| 09:01:44 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.binary_type/integer_types/string_types https://review.opendev.org/728094 | |
| 09:01:52 | lyarwood | bauzas: and you mean outside of https://review.opendev.org/#/c/746981/6/releasenotes/notes/victoria-libvirt-version-bump-e1a09b3a72ee56a4.yaml | |
| 09:01:58 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (1/2) https://review.opendev.org/728109 | |
| 09:02:24 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117 | |
| 09:02:59 | openstackgerrit | Takashi Natsume proposed openstack/nova master: doc: correct the link to user/flavor.rst https://review.opendev.org/740176 | |
| 09:03:09 | bauzas | lyarwood: no, it's a matter of communicating the caveat we found with Focal | |
| 09:03:16 | openstackgerrit | Takashi Natsume proposed openstack/nova master: doc: Fix rendering in the accelerator support doc https://review.opendev.org/751574 | |
| 09:03:29 | bauzas | because TC and the Foundation would communicate on the fact that service projects do support Focal | |
| 09:03:36 | bauzas | but we regress on a thing | |
| 09:03:46 | bauzas | so, ideally, I would comment this in a prelude section | |
| 09:04:08 | lyarwood | bauzas: ah sorry right can we include a known issues section? | |
| 09:04:13 | bauzas | lyarwood: I guess that RHEL-based distros are also impacted by the volume attach issue since it's a qemu problem ? | |
| 09:04:23 | bauzas | lyarwood: that'd be the right position I guess | |
| 09:05:47 | bauzas | lyarwood: reno supports an 'issues' section https://docs.openstack.org/reno/latest/user/usage.html#editing-a-release-note | |
| 09:06:16 | lyarwood | bauzas: I've not been able to reproduce it on el distros yet | |
| 09:06:28 | bauzas | ah shit | |
| 09:06:38 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_FILE_BACKED_DISCARD_VERSION https://review.opendev.org/746982 | |
| 09:06:38 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/746981 | |
| 09:06:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_BETTER_SIGKILL_HANDLING https://review.opendev.org/746984 | |
| 09:06:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_NATIVE_TLS_VERSION https://review.opendev.org/746983 | |
| 09:06:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_{LIBVIRT,QEMU}_PMEM_SUPPORT https://review.opendev.org/746986 | |
| 09:06:40 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIDEO_MODEL_VERSIONS https://review.opendev.org/746985 | |
| 09:06:41 | openstackgerrit | Lee Yarwood proposed openstack/nova master: test_evacuate.sh: Support libvirt-bin and libvirtd systemd services https://review.opendev.org/752650 | |
| 09:06:42 | bauzas | so it's not purely a qemu thing | |
| 09:07:38 | lyarwood | we believe it is but it also appears to be caused by overall load on the host etc so it's super awkward to reproduce | |
| 09:08:35 | bauzas | gibi: removed the -2 and +W the base patch | |
| 09:09:03 | bauzas | lyarwood: what would be a recommendation for operators using Focal ? | |
| 09:09:17 | bauzas | if we document a known issue | |
| 09:09:39 | bauzas | I mean, is there any found workaround with Focal ? | |
| 09:09:55 | lyarwood | no, there's not. | |
| 09:09:59 | lyarwood | they could hit this issue | |
| 09:10:03 | bauzas | or should we document as a known issue 'hold your OS upgrade if you consider running Victoria" | |
| 09:10:30 | lyarwood | well that's the thing we can't if we drop support for bionic | |
| 09:10:39 | bauzas | lyarwood: can't they try to override the base qemu versions by downgrading them using a Train UCA ? | |
| 09:10:40 | lyarwood | if we support both then we can say that | |
| 09:10:57 | lyarwood | bauzas: I doubt ubuntu would support that for a few release tbh | |
| 09:11:05 | bauzas | lyarwood: I'm in favor of continuing to support Bionic for that exact reason | |
| 09:11:30 | bauzas | this whole thing sucks. | |
| 09:11:40 | lyarwood | bauzas: if we can it would make our lives alot easier | |
| 09:12:36 | bauzas | honestly, we would then have to write "if you wanna use Victoria and you feel you're impacted by the bug, please consider running another distro that's not impacted" ? | |
| 09:12:56 | bauzas | this is absolutely serious. | |
| 09:13:28 | bauzas | can't we try to find Canonical engineers to help us ? | |
| 09:13:44 | bauzas | sahid isn't here AFAICS | |
| 09:13:59 | bauzas | jamespage neither. | |
| 09:14:03 | stephenfin | sahid left Canonical and is now...somewhere else | |
| 09:14:04 | lyarwood | kashyap has been trying to engage their QEMU engineer | |
| 09:14:15 | lyarwood | yes I did mean engineer | |
| 09:14:18 | lyarwood | 8| | |
| 09:14:52 | bauzas | gibi: have you engaged with the TC about this very critical issue ? | |
| 09:15:04 | lyarwood | brb I need more coffee | |
| 09:15:14 | bauzas | hopefully gmann did the liaison | |
| 09:15:39 | kashyap | lyarwood: Is this the Focal topic? | |
| 09:16:25 | kashyap | stephenfin: gibi: lyarwood: In general: in the past I've gone to Ubuntu's -stable IRC channel to point out missing virt-related backports | |
| 09:16:30 | kashyap | But that's not scalable... | |
| 09:17:20 | bauzas | I'm sure that raising the problem to the TC and eventually the Foundation would send a signal that Canonical would be foolish to not capture | |
| 09:18:07 | bauzas | because if we eventually say "sorry, but you can race on volume attach/detach with Focal and you have no other options but to consider other distros for this release if you BFV", that would certainly put some prios | |
| 09:18:30 | bauzas | that's my expectation at least | |
| 09:20:12 | bauzas | lyarwood: do you feel brave enough to write the problem and the absence of solution in a reno note ? | |
| 09:25:53 | lyarwood | bauzas: I can try | |
| 09:26:01 | bauzas | thanks | |
| 09:27:04 | gibi | so on Focal the only solution is to downgrade libvirt? | |
| 09:27:20 | gibi | so on Focal the only solution is to downgrade qemu | |
| 09:27:25 | bauzas | gibi: there is no solution for Focal, IIUC | |
| 09:27:36 | bauzas | that's the problem | |
| 09:27:57 | bauzas | or I'd rather say "there is no solution found yet for Focal" | |
| 09:28:11 | bauzas | this leaves people investigating a potential workaround | |
| 09:28:14 | gibi | there is no way to use an older qemu there? | |
| 09:28:29 | bauzas | gibi: lyarwood said this is though to downgrade | |
| 09:28:38 | bauzas | due to the canonical support model | |
| 09:28:54 | bauzas | we would then ask to install an old qemu version that's not supporter | |
| 09:28:56 | bauzas | supported | |
| 09:29:00 | lyarwood | gibi / bauzas ; Yeah I can't speak for Canonical but we (RH) wouldn't support that with our distros | |
| 09:29:33 | bauzas | which makes sense tbh | |
| 09:29:37 | gibi | OK, then simply state what we know in the reno. | |
| 09:29:45 | gibi | don't have to be sugarcoated | |
| 09:29:49 | lyarwood | kk | |
| 09:29:50 | gibi | there is an open qemu bug to link to | |
| 09:29:52 | bauzas | gibi: we also need to state we haven't identified yet a solution. | |
| 09:30:23 | bauzas | if I was an operator, I'd surely reconsider upgrading if I feel I could be impacted | |
| 09:30:24 | gibi | being it a qemu bug I don't really think we as openstack can find a solution | |
| 09:30:43 | bauzas | I do agree | |
| 09:30:47 | gibi | but yes, be clear that we don't know a viable workaround on Focal | |
| 09:31:10 | bauzas | the workaround could be "don't use volumes ?" | |
| 09:31:13 | bauzas | shit, I can't emoji | |
| 09:31:18 | gibi | that is not a WA :D | |
| 09:31:33 | kashyap | gibi: So for the Focal QEMU problem, the moment we can figure out a libvirt-level reproducer that can really speeden up the root-cause-analysis | |
| 09:31:49 | kashyap | gibi: The tricky part here is, it is reproducible only in a Tempest-based setup :-( | |
| 09:32:21 | gibi | kashyap: I see | |
| 09:33:24 | gibi | bauzas: to answer your earlier question about talking to the TC, I haven't doen that (yet). I will raise the issue to them | |
| 09:33:25 | kashyap | gibi: One approach I want to try (once I get these other pressing thing off my chest) is to setup a CPU-intensive workload, and do an attach+detach on a guest launched w/ plain libvirt | |
| 09:33:51 | kashyap | ... and then pray to the Juju under the sea that the issue will reprocuce | |
| 09:34:00 | bauzas | gibi: thanks | |
| 09:39:01 | gibi | bauzas: will you add the limitation to our relesenotes in a separate patch from the https://review.opendev.org/#/c/751045/3/releasenotes/notes/victoria-prelude-9b4c16ff8c6e7f3e.yaml ? | |
| 09:39:09 | gibi | can I approve ^^? | |
| 09:39:39 | bauzas | gibi: yes, I'll add another prelude item in a separate change once lyarwood documents the issue in another reno change | |
| 09:40:11 | gibi | ack | |
| 09:40:14 | gibi | thanks | |
| 09:40:19 | bauzas | gibi: lyarwood: tbc, this is what I see : a documented 'issues' note that explains the problem and the absence of known solution | |
| 09:40:42 | bauzas | and a prelude item that would only refer to this problem | |