| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-15 | |||
| 15:05:29 | sean-k-mooney | oh your fixing that issue | |
| 15:05:36 | sean-k-mooney | that is on my todo list | |
| 15:05:46 | sean-k-mooney | i personally would add tests to oslo utils | |
| 15:07:02 | sean-k-mooney | thats breaking fedora with the virt preview repo enabeld currently | |
| 15:07:20 | sean-k-mooney | alshtough im not sure it affct any distros default packages yet? | |
| 15:07:46 | MarkMielke | right | |
| 15:08:25 | MarkMielke | qemu 4.1 provides important nVMX patches | |
| 15:08:51 | MarkMielke | so not necessarily in distro | |
| 15:09:00 | MarkMielke | can use qemu 4.1 on RHEL 7 if need be | |
| 15:09:40 | MarkMielke | all of the tests for outputs seem to be in nova | |
| 15:09:50 | MarkMielke | seems like unit test confusion :-) | |
| 15:09:52 | sean-k-mooney | non of them shoudl be in nova | |
| 15:10:02 | sean-k-mooney | they should all be in oslo_untils | |
| 15:10:18 | sean-k-mooney | but i think that code was original ported from nova | |
| 15:10:25 | sean-k-mooney | so i guess they did not mvoe the tests | |
| 15:11:15 | MarkMielke | I want to add a test, but I'm not familiar with setting it up from scratch, and I don't want to take on doing it all :-) | |
| 15:11:16 | sean-k-mooney | nova really should not be asserting the parseing behavior of oslo code so i sustpect the nova tests are just legacy test we did not remove | |
| 15:11:46 | MarkMielke | two character fix :-( hehe | |
| 15:11:49 | sean-k-mooney | you could add more test in nova but personally i think those are techdebt | |
| 15:12:11 | MarkMielke | i agree from observing | |
| 15:12:23 | sean-k-mooney | well you could file a bug for the lack of test in oslo_utils | |
| 15:12:44 | MarkMielke | good idea | |
| 15:12:47 | sean-k-mooney | and submit the ptach without tests. you would have to use a depend on to test the feature in nova | |
| 15:13:15 | sean-k-mooney | but we are pass the non client lib freeze so i dont exepct there to be another oslo release in Train | |
| 15:13:31 | MarkMielke | hmm - i wondered about that | |
| 15:13:45 | MarkMielke | how important patches get in to dependency libraries | |
| 15:14:02 | sean-k-mooney | form a rhel perspective we can always backport this downstream | |
| 15:14:16 | MarkMielke | patches to the packages, you mean? | |
| 15:14:35 | sean-k-mooney | but i dont really consider this an important patch unless your saying the advance virt strema now ships this by default on rhel 7 | |
| 15:15:03 | MarkMielke | well i wondered in general - importance depends upon perspective, so i agree others might not consider it important | |
| 15:15:12 | MarkMielke | qemu 4.1 is important for nested virtualization use case | |
| 15:15:23 | MarkMielke | it's the first release that officially supports nested virtualization live migration | |
| 15:15:24 | sean-k-mooney | nested virt work without it | |
| 15:15:36 | MarkMielke | nah, they've eliminated it in 3.1 and 4.0 | |
| 15:15:40 | sean-k-mooney | nested virt live migration also work without it | |
| 15:15:41 | MarkMielke | because it was fundamentally broken | |
| 15:15:52 | MarkMielke | they have a live migration blocker implemented | |
| 15:15:56 | MarkMielke | that many distros disable | |
| 15:16:00 | sean-k-mooney | you mean live migat of th l1 guest | |
| 15:16:01 | MarkMielke | it's basically unsafe | |
| 15:16:03 | sean-k-mooney | not the l2 guest | |
| 15:16:30 | MarkMielke | there is special KVM state in the L0 hypervisor that isn't transferred prior to qemu 4.1 | |
| 15:16:38 | MarkMielke | basically results in undefined memory at the target | |
| 15:17:12 | MarkMielke | undefined behaviour is the result ... better not actually be doing nested virtualization previously :-) | |
| 15:17:13 | sean-k-mooney | right but are you saying that qemu 4.1 fix live migration of l1 vms that contin l2 vm or live migration of l2 vms | |
| 15:17:18 | MarkMielke | yes | |
| 15:17:26 | sean-k-mooney | ok that is different | |
| 15:17:38 | MarkMielke | the patches were added - and it passes for Oracle and others | |
| 15:17:40 | sean-k-mooney | you could previosly safely migrate the l2 guests | |
| 15:17:45 | MarkMielke | unfortunately i have a use case that it doesn't work for | |
| 15:17:55 | MarkMielke | but they've tried to solve all the edge cases | |
| 15:18:29 | MarkMielke | it uses several new kernel features only in Linux 4.19 and later | |
| 15:18:37 | MarkMielke | so RHEL 8 would be a candidate for some of it | |
| 15:18:50 | sean-k-mooney | rhel 8 is based on 4.18 | |
| 15:18:59 | sean-k-mooney | but ya it could be backported | |
| 15:19:11 | MarkMielke | yuck i thought it used 4.19 | |
| 15:19:22 | sean-k-mooney | nope which is annoying | |
| 15:19:34 | sean-k-mooney | but rhel kernel version dont really mena anything | |
| 15:19:37 | MarkMielke | Red Hat will just backport most of the patches anyways :-) | |
| 15:19:51 | sean-k-mooney | rhel 7 was 3.x and has patch from 4.15 | |
| 15:21:02 | sean-k-mooney | i work for redhat by the way which is why i said we could backport the oslo utils change if it was need becaue we stated shiping qemu 4.1 by default on rhel 7 with the advance vert stream | |
| 15:21:12 | MarkMielke | https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a | |
| 15:21:15 | sean-k-mooney | but i dont think we have or we would have got a bug report | |
| 15:21:21 | MarkMielke | there are a few patches in the set... but that's one of the important ones | |
| 15:24:26 | MarkMielke | I've been tracking these changes and the need for them since 2018, so it being almost usable is great. :-) | |
| 15:25:14 | MarkMielke | lots of nVMX patches in Linux 5.0, 5.1, and 5.2 though... so 4.18 will just be annoying to patch to for you | |
| 15:26:06 | sean-k-mooney | out of interest why to you need to migrate the l1 guests? are the l1 guest your tenant guest and they are using nested virt on your cloud? | |
| 15:26:40 | MarkMielke | in my particular use case - the users have "simulations" of network equipment running in guests | |
| 15:26:44 | MarkMielke | previously they were on physical workstations | |
| 15:26:53 | MarkMielke | now they are in virtual machines | |
| 15:27:10 | MarkMielke | could be an embedded Linux using KVM, or could be Vxworks | |
| 15:27:24 | MarkMielke | so those machines are basically not safe to live migration until now | |
| 15:27:37 | MarkMielke | i've found the only safe recipe is one by one: | |
| 15:27:52 | sean-k-mooney | yes but live migration is really only for mantaince | |
| 15:27:56 | MarkMielke | 1) confirm no L2 guests running in the L1 hypervisor | |
| 15:27:57 | sean-k-mooney | is that your usecase | |
| 15:28:00 | MarkMielke | 2) rmmod kvm_intel | |
| 15:28:04 | MarkMielke | 3) live migrate | |
| 15:28:07 | MarkMielke | 4) modprobe kvm_intel | |
| 15:28:13 | MarkMielke | it's terribly painful for 1700+ machines | |
| 15:28:32 | MarkMielke | yes, maintenance - basically to patch the hypervisors | |
| 15:28:44 | sean-k-mooney | if you do step 1 then why do you need 2 and 4 | |
| 15:28:49 | MarkMielke | or to balance load if necessary (although because of the limitations we basically ensure it is never necessary) | |
| 15:28:57 | sean-k-mooney | if there are no l2 guest it should be safe to migrate the l1 | |
| 15:28:58 | MarkMielke | because they might have used KVM in past | |
| 15:29:10 | MarkMielke | if they used it, and then not using it - it still transfers bad state | |
| 15:29:18 | MarkMielke | undefined memory where the KVM state should be | |
| 15:29:32 | MarkMielke | the qemu people say it's dangerous always | |
| 15:29:37 | MarkMielke | but i haven't hit one problem doing the above | |
| 15:30:05 | MarkMielke | i believe it triggers re-initialization of the memory | |
| 15:30:07 | sean-k-mooney | ok i was not aware it permently tainted the guest | |
| 15:30:17 | MarkMielke | yeah that's why they made live migration blocker | |
| 15:30:24 | MarkMielke | they don't want to support it at all after they realized how bad it was | |
| 15:30:44 | sean-k-mooney | they have not comunicated that well | |
| 15:30:48 | MarkMielke | nope :-) | |
| 15:30:59 | MarkMielke | they just went from it working sort of | |
| 15:31:10 | sean-k-mooney | if they did we might have considerd blocking it in nova | |
| 15:31:11 | MarkMielke | to live migration fails if VMX is enabled on the guest :-) | |
| 15:31:25 | MarkMielke | but Ubuntu, Fedora, etc. disable the live migration blocker | |
| 15:31:30 | MarkMielke | which I also did in my custom builds | |