| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-15 | |||
| 15:17:45 | MarkMielke | unfortunately i have a use case that it doesn't work for | |
| 15:17:55 | MarkMielke | but they've tried to solve all the edge cases | |
| 15:18:29 | MarkMielke | it uses several new kernel features only in Linux 4.19 and later | |
| 15:18:37 | MarkMielke | so RHEL 8 would be a candidate for some of it | |
| 15:18:50 | sean-k-mooney | rhel 8 is based on 4.18 | |
| 15:18:59 | sean-k-mooney | but ya it could be backported | |
| 15:19:11 | MarkMielke | yuck i thought it used 4.19 | |
| 15:19:22 | sean-k-mooney | nope which is annoying | |
| 15:19:34 | sean-k-mooney | but rhel kernel version dont really mena anything | |
| 15:19:37 | MarkMielke | Red Hat will just backport most of the patches anyways :-) | |
| 15:19:51 | sean-k-mooney | rhel 7 was 3.x and has patch from 4.15 | |
| 15:21:02 | sean-k-mooney | i work for redhat by the way which is why i said we could backport the oslo utils change if it was need becaue we stated shiping qemu 4.1 by default on rhel 7 with the advance vert stream | |
| 15:21:12 | MarkMielke | https://github.com/qemu/qemu/commit/ebbfef2f34cfc749c045a4569dedb4f748ec024a | |
| 15:21:15 | sean-k-mooney | but i dont think we have or we would have got a bug report | |
| 15:21:21 | MarkMielke | there are a few patches in the set... but that's one of the important ones | |
| 15:24:26 | MarkMielke | I've been tracking these changes and the need for them since 2018, so it being almost usable is great. :-) | |
| 15:25:14 | MarkMielke | lots of nVMX patches in Linux 5.0, 5.1, and 5.2 though... so 4.18 will just be annoying to patch to for you | |
| 15:26:06 | sean-k-mooney | out of interest why to you need to migrate the l1 guests? are the l1 guest your tenant guest and they are using nested virt on your cloud? | |
| 15:26:40 | MarkMielke | in my particular use case - the users have "simulations" of network equipment running in guests | |
| 15:26:44 | MarkMielke | previously they were on physical workstations | |
| 15:26:53 | MarkMielke | now they are in virtual machines | |
| 15:27:10 | MarkMielke | could be an embedded Linux using KVM, or could be Vxworks | |
| 15:27:24 | MarkMielke | so those machines are basically not safe to live migration until now | |
| 15:27:37 | MarkMielke | i've found the only safe recipe is one by one: | |
| 15:27:52 | sean-k-mooney | yes but live migration is really only for mantaince | |
| 15:27:56 | MarkMielke | 1) confirm no L2 guests running in the L1 hypervisor | |
| 15:27:57 | sean-k-mooney | is that your usecase | |
| 15:28:00 | MarkMielke | 2) rmmod kvm_intel | |
| 15:28:04 | MarkMielke | 3) live migrate | |
| 15:28:07 | MarkMielke | 4) modprobe kvm_intel | |
| 15:28:13 | MarkMielke | it's terribly painful for 1700+ machines | |
| 15:28:32 | MarkMielke | yes, maintenance - basically to patch the hypervisors | |
| 15:28:44 | sean-k-mooney | if you do step 1 then why do you need 2 and 4 | |
| 15:28:49 | MarkMielke | or to balance load if necessary (although because of the limitations we basically ensure it is never necessary) | |
| 15:28:57 | sean-k-mooney | if there are no l2 guest it should be safe to migrate the l1 | |
| 15:28:58 | MarkMielke | because they might have used KVM in past | |
| 15:29:10 | MarkMielke | if they used it, and then not using it - it still transfers bad state | |
| 15:29:18 | MarkMielke | undefined memory where the KVM state should be | |
| 15:29:32 | MarkMielke | the qemu people say it's dangerous always | |
| 15:29:37 | MarkMielke | but i haven't hit one problem doing the above | |
| 15:30:05 | MarkMielke | i believe it triggers re-initialization of the memory | |
| 15:30:07 | sean-k-mooney | ok i was not aware it permently tainted the guest | |
| 15:30:17 | MarkMielke | yeah that's why they made live migration blocker | |
| 15:30:24 | MarkMielke | they don't want to support it at all after they realized how bad it was | |
| 15:30:44 | sean-k-mooney | they have not comunicated that well | |
| 15:30:48 | MarkMielke | nope :-) | |
| 15:30:59 | MarkMielke | they just went from it working sort of | |
| 15:31:10 | sean-k-mooney | if they did we might have considerd blocking it in nova | |
| 15:31:11 | MarkMielke | to live migration fails if VMX is enabled on the guest :-) | |
| 15:31:25 | MarkMielke | but Ubuntu, Fedora, etc. disable the live migration blocker | |
| 15:31:30 | MarkMielke | which I also did in my custom builds | |
| 15:31:40 | MarkMielke | because I still need to do it - even if I choose to almost never do it | |
| 15:32:02 | sean-k-mooney | ya i use nested vert all the time. and often migrate l1 guests | |
| 15:32:09 | sean-k-mooney | i have never had a problem | |
| 15:32:24 | sean-k-mooney | that said i have never migrated an l1 guest with l2 guests | |
| 15:32:59 | sean-k-mooney | although i normally end up migrating l2 guest more often the l1 | |
| 15:33:03 | sean-k-mooney | os that s proably why | |
| 15:33:24 | MarkMielke | i've never done l2 guest myself - but i know that the users do it on their own, because that's really their use case | |
| 15:33:47 | MarkMielke | they just know they want to move the guest from point A to point B | |
| 15:33:53 | MarkMielke | no idea it's L2 or not :-) | |
| 15:34:04 | sean-k-mooney | well on my dev systems im spinning up l1 guests to run devstack and then testing live migartion move l2 guests around | |
| 15:34:16 | MarkMielke | ah right | |
| 15:34:27 | MarkMielke | openstack on openstack :-) | |
| 15:34:42 | sean-k-mooney | so for me l1 is simulating a cloud and l2 is my test workload | |
| 15:34:51 | sean-k-mooney | and yes but not with tripleO | |
| 15:35:13 | MarkMielke | tripleO doesn't even mean tripleO anymore, right? doesn't it mean OpenStack on K8? haha | |
| 15:35:26 | sean-k-mooney | not yet | |
| 15:35:43 | sean-k-mooney | but yat they are slowlow dismanteling the undercloud | |
| 15:36:04 | sean-k-mooney | personally i much prefer kolla-ansible or OSA | |
| 15:36:13 | MarkMielke | me too | |
| 15:36:14 | sean-k-mooney | triplo is too complicated | |
| 15:36:29 | sean-k-mooney | and has too much techdebt in all its layers | |
| 15:36:45 | MarkMielke | it's somewhat purist without regard to consequences | |
| 15:37:02 | MarkMielke | my view :-) | |
| 15:37:26 | sean-k-mooney | im not sure i would agree. i think it use to be more pure and simple | |
| 15:37:42 | sean-k-mooney | and now with contariation its less pure and more complicated | |
| 15:38:51 | sean-k-mooney | lucally while i have to debug customer systems ocationally (mainly via logs) i dont actuly need to use triplO day to day | |
| 15:39:16 | MarkMielke | Kayobe is the one that was appealing to me | |
| 15:39:22 | MarkMielke | Ironic in stand-alone mode | |
| 15:39:23 | MarkMielke | Kolla | |
| 15:39:26 | MarkMielke | Ansible | |
| 15:39:43 | sean-k-mooney | yep when i work at intel i pitched basically what they implemented internlaly | |
| 15:39:59 | sean-k-mooney | i wrote the original kolla biforst container | |
| 15:40:03 | MarkMielke | cool | |
| 15:40:50 | sean-k-mooney | i really wanted to see koll ansibel become the default openstack installer but i dont really work on the installer anymore | |
| 15:42:25 | MarkMielke | I think it will pick up. | |
| 15:42:48 | sean-k-mooney | if kaobe does then yes | |
| 15:42:59 | sean-k-mooney | *kayobe | |
| 15:43:00 | MarkMielke | It's so much simpler. | |
| 15:43:04 | sean-k-mooney | yep | |
| 15:43:12 | sean-k-mooney | its simple to debug and extend | |
| 15:43:34 | sean-k-mooney | the config overrde mechanisum is easy to use | |
| 15:43:40 | sean-k-mooney | and no puppet/ruby | |
| 15:43:46 | sean-k-mooney | which is defintly a win | |
| 15:43:48 | MarkMielke | Running Kubernetes on hypervisors seems overly complex to me personally. It's not really the right use case for it. | |
| 15:44:14 | MarkMielke | If you were running Docker containers - sure... but virtual machines from within Nova/libvirtd/qemu? I don't think so. | |
| 15:44:33 | sean-k-mooney | the last thing i was trying to push in kolla before i stope working on it was move to running the contaienr using systemd service file | |
| 15:44:50 | sean-k-mooney | so that tehre was a determinisitc order to them on host reboot | |
| 15:45:05 | MarkMielke | the move away from "Docker" may help with that :-) | |
| 15:45:20 | sean-k-mooney | what are they moving too | |