| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-15 | |||
| 15:11:49 | sean-k-mooney | you could add more test in nova but personally i think those are techdebt | |
| 15:12:11 | MarkMielke | i agree from observing | |
| 15:12:23 | sean-k-mooney | well you could file a bug for the lack of test in oslo_utils | |
| 15:12:44 | MarkMielke | good idea | |
| 15:12:47 | sean-k-mooney | and submit the ptach without tests. you would have to use a depend on to test the feature in nova | |
| 15:13:15 | sean-k-mooney | but we are pass the non client lib freeze so i dont exepct there to be another oslo release in Train | |
| 15:13:31 | MarkMielke | hmm - i wondered about that | |
| 15:13:45 | MarkMielke | how important patches get in to dependency libraries | |
| 15:14:02 | sean-k-mooney | form a rhel perspective we can always backport this downstream | |
| 15:14:16 | MarkMielke | patches to the packages, you mean? | |
| 15:14:35 | sean-k-mooney | but i dont really consider this an important patch unless your saying the advance virt strema now ships this by default on rhel 7 | |
| 15:15:03 | MarkMielke | well i wondered in general - importance depends upon perspective, so i agree others might not consider it important | |
| 15:15:12 | MarkMielke | qemu 4.1 is important for nested virtualization use case | |
| 15:15:23 | MarkMielke | it's the first release that officially supports nested virtualization live migration | |
| 15:15:24 | sean-k-mooney | nested virt work without it | |
| 15:15:36 | MarkMielke | nah, they've eliminated it in 3.1 and 4.0 | |
| 15:15:40 | sean-k-mooney | nested virt live migration also work without it | |
| 15:15:41 | MarkMielke | because it was fundamentally broken | |
| 15:15:52 | MarkMielke | they have a live migration blocker implemented | |
| 15:15:56 | MarkMielke | that many distros disable | |
| 15:16:00 | sean-k-mooney | you mean live migat of th l1 guest | |
| 15:16:01 | MarkMielke | it's basically unsafe | |
| 15:16:03 | sean-k-mooney | not the l2 guest | |
| 15:16:30 | MarkMielke | there is special KVM state in the L0 hypervisor that isn't transferred prior to qemu 4.1 | |
| 15:16:38 | MarkMielke | basically results in undefined memory at the target | |
| 15:17:12 | MarkMielke | undefined behaviour is the result ... better not actually be doing nested virtualization previously :-) | |
| 15:17:13 | sean-k-mooney | right but are you saying that qemu 4.1 fix live migration of l1 vms that contin l2 vm or live migration of l2 vms | |
| 15:17:18 | MarkMielke | yes | |
| 15:17:26 | sean-k-mooney | ok that is different | |
| 15:17:38 | MarkMielke | the patches were added - and it passes for Oracle and others | |
| 15:17:40 | sean-k-mooney | you could previosly safely migrate the l2 guests | |
| 15:17:45 | MarkMielke | unfortunately i have a use case that it doesn't work for | |
| 15:17:55 | MarkMielke | but they've tried to solve all the edge cases | |
| 15:18:29 | MarkMielke | it uses several new kernel features only in Linux 4.19 and later | |
| 15:18:37 | MarkMielke | so RHEL 8 would be a candidate for some of it | |
| 15:18:50 | sean-k-mooney | rhel 8 is based on 4.18 | |
| 15:18:59 | sean-k-mooney | but ya it could be backported | |
| 15:19:11 | MarkMielke | yuck i thought it used 4.19 | |
| 15:19:22 | sean-k-mooney | nope which is annoying | |
| 15:19:34 | sean-k-mooney | but rhel kernel version dont really mena anything | |
| 15:19:37 | MarkMielke | Red Hat will just backport most of the patches anyways :-) | |
| 15:19:51 | sean-k-mooney | rhel 7 was 3.x and has patch from 4.15 | |
| 15:21:02 | sean-k-mooney | i work for redhat by the way which is why i said we could backport the oslo utils change if it was need becaue we stated shiping qemu 4.1 by default on rhel 7 with the advance vert stream | |
| 15:21:12 | MarkMielke | https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a | |
| 15:21:15 | sean-k-mooney | but i dont think we have or we would have got a bug report | |
| 15:21:21 | MarkMielke | there are a few patches in the set... but that's one of the important ones | |
| 15:24:26 | MarkMielke | I've been tracking these changes and the need for them since 2018, so it being almost usable is great. :-) | |
| 15:25:14 | MarkMielke | lots of nVMX patches in Linux 5.0, 5.1, and 5.2 though... so 4.18 will just be annoying to patch to for you | |
| 15:26:06 | sean-k-mooney | out of interest why to you need to migrate the l1 guests? are the l1 guest your tenant guest and they are using nested virt on your cloud? | |
| 15:26:40 | MarkMielke | in my particular use case - the users have "simulations" of network equipment running in guests | |
| 15:26:44 | MarkMielke | previously they were on physical workstations | |
| 15:26:53 | MarkMielke | now they are in virtual machines | |
| 15:27:10 | MarkMielke | could be an embedded Linux using KVM, or could be Vxworks | |
| 15:27:24 | MarkMielke | so those machines are basically not safe to live migration until now | |
| 15:27:37 | MarkMielke | i've found the only safe recipe is one by one: | |
| 15:27:52 | sean-k-mooney | yes but live migration is really only for mantaince | |
| 15:27:56 | MarkMielke | 1) confirm no L2 guests running in the L1 hypervisor | |
| 15:27:57 | sean-k-mooney | is that your usecase | |
| 15:28:00 | MarkMielke | 2) rmmod kvm_intel | |
| 15:28:04 | MarkMielke | 3) live migrate | |
| 15:28:07 | MarkMielke | 4) modprobe kvm_intel | |
| 15:28:13 | MarkMielke | it's terribly painful for 1700+ machines | |
| 15:28:32 | MarkMielke | yes, maintenance - basically to patch the hypervisors | |
| 15:28:44 | sean-k-mooney | if you do step 1 then why do you need 2 and 4 | |
| 15:28:49 | MarkMielke | or to balance load if necessary (although because of the limitations we basically ensure it is never necessary) | |
| 15:28:57 | sean-k-mooney | if there are no l2 guest it should be safe to migrate the l1 | |
| 15:28:58 | MarkMielke | because they might have used KVM in past | |
| 15:29:10 | MarkMielke | if they used it, and then not using it - it still transfers bad state | |
| 15:29:18 | MarkMielke | undefined memory where the KVM state should be | |
| 15:29:32 | MarkMielke | the qemu people say it's dangerous always | |
| 15:29:37 | MarkMielke | but i haven't hit one problem doing the above | |
| 15:30:05 | MarkMielke | i believe it triggers re-initialization of the memory | |
| 15:30:07 | sean-k-mooney | ok i was not aware it permently tainted the guest | |
| 15:30:17 | MarkMielke | yeah that's why they made live migration blocker | |
| 15:30:24 | MarkMielke | they don't want to support it at all after they realized how bad it was | |
| 15:30:44 | sean-k-mooney | they have not comunicated that well | |
| 15:30:48 | MarkMielke | nope :-) | |
| 15:30:59 | MarkMielke | they just went from it working sort of | |
| 15:31:10 | sean-k-mooney | if they did we might have considerd blocking it in nova | |
| 15:31:11 | MarkMielke | to live migration fails if VMX is enabled on the guest :-) | |
| 15:31:25 | MarkMielke | but Ubuntu, Fedora, etc. disable the live migration blocker | |
| 15:31:30 | MarkMielke | which I also did in my custom builds | |
| 15:31:40 | MarkMielke | because I still need to do it - even if I choose to almost never do it | |
| 15:32:02 | sean-k-mooney | ya i use nested vert all the time. and often migrate l1 guests | |
| 15:32:09 | sean-k-mooney | i have never had a problem | |
| 15:32:24 | sean-k-mooney | that said i have never migrated an l1 guest with l2 guests | |
| 15:32:59 | sean-k-mooney | although i normally end up migrating l2 guest more often the l1 | |
| 15:33:03 | sean-k-mooney | os that s proably why | |
| 15:33:24 | MarkMielke | i've never done l2 guest myself - but i know that the users do it on their own, because that's really their use case | |
| 15:33:47 | MarkMielke | they just know they want to move the guest from point A to point B | |
| 15:33:53 | MarkMielke | no idea it's L2 or not :-) | |
| 15:34:04 | sean-k-mooney | well on my dev systems im spinning up l1 guests to run devstack and then testing live migartion move l2 guests around | |
| 15:34:16 | MarkMielke | ah right | |
| 15:34:27 | MarkMielke | openstack on openstack :-) | |
| 15:34:42 | sean-k-mooney | so for me l1 is simulating a cloud and l2 is my test workload | |
| 15:34:51 | sean-k-mooney | and yes but not with tripleO | |
| 15:35:13 | MarkMielke | tripleO doesn't even mean tripleO anymore, right? doesn't it mean OpenStack on K8? haha | |
| 15:35:26 | sean-k-mooney | not yet | |
| 15:35:43 | sean-k-mooney | but yat they are slowlow dismanteling the undercloud | |
| 15:36:04 | sean-k-mooney | personally i much prefer kolla-ansible or OSA | |