| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-15 | |||
| 15:27:24 | MarkMielke | so those machines are basically not safe to live migration until now | |
| 15:27:37 | MarkMielke | i've found the only safe recipe is one by one: | |
| 15:27:52 | sean-k-mooney | yes but live migration is really only for mantaince | |
| 15:27:56 | MarkMielke | 1) confirm no L2 guests running in the L1 hypervisor | |
| 15:27:57 | sean-k-mooney | is that your usecase | |
| 15:28:00 | MarkMielke | 2) rmmod kvm_intel | |
| 15:28:04 | MarkMielke | 3) live migrate | |
| 15:28:07 | MarkMielke | 4) modprobe kvm_intel | |
| 15:28:13 | MarkMielke | it's terribly painful for 1700+ machines | |
| 15:28:32 | MarkMielke | yes, maintenance - basically to patch the hypervisors | |
| 15:28:44 | sean-k-mooney | if you do step 1 then why do you need 2 and 4 | |
| 15:28:49 | MarkMielke | or to balance load if necessary (although because of the limitations we basically ensure it is never necessary) | |
| 15:28:57 | sean-k-mooney | if there are no l2 guest it should be safe to migrate the l1 | |
| 15:28:58 | MarkMielke | because they might have used KVM in past | |
| 15:29:10 | MarkMielke | if they used it, and then not using it - it still transfers bad state | |
| 15:29:18 | MarkMielke | undefined memory where the KVM state should be | |
| 15:29:32 | MarkMielke | the qemu people say it's dangerous always | |
| 15:29:37 | MarkMielke | but i haven't hit one problem doing the above | |
| 15:30:05 | MarkMielke | i believe it triggers re-initialization of the memory | |
| 15:30:07 | sean-k-mooney | ok i was not aware it permently tainted the guest | |
| 15:30:17 | MarkMielke | yeah that's why they made live migration blocker | |
| 15:30:24 | MarkMielke | they don't want to support it at all after they realized how bad it was | |
| 15:30:44 | sean-k-mooney | they have not comunicated that well | |
| 15:30:48 | MarkMielke | nope :-) | |
| 15:30:59 | MarkMielke | they just went from it working sort of | |
| 15:31:10 | sean-k-mooney | if they did we might have considerd blocking it in nova | |
| 15:31:11 | MarkMielke | to live migration fails if VMX is enabled on the guest :-) | |
| 15:31:25 | MarkMielke | but Ubuntu, Fedora, etc. disable the live migration blocker | |
| 15:31:30 | MarkMielke | which I also did in my custom builds | |
| 15:31:40 | MarkMielke | because I still need to do it - even if I choose to almost never do it | |
| 15:32:02 | sean-k-mooney | ya i use nested vert all the time. and often migrate l1 guests | |
| 15:32:09 | sean-k-mooney | i have never had a problem | |
| 15:32:24 | sean-k-mooney | that said i have never migrated an l1 guest with l2 guests | |
| 15:32:59 | sean-k-mooney | although i normally end up migrating l2 guest more often the l1 | |
| 15:33:03 | sean-k-mooney | os that s proably why | |
| 15:33:24 | MarkMielke | i've never done l2 guest myself - but i know that the users do it on their own, because that's really their use case | |
| 15:33:47 | MarkMielke | they just know they want to move the guest from point A to point B | |
| 15:33:53 | MarkMielke | no idea it's L2 or not :-) | |
| 15:34:04 | sean-k-mooney | well on my dev systems im spinning up l1 guests to run devstack and then testing live migartion move l2 guests around | |
| 15:34:16 | MarkMielke | ah right | |
| 15:34:27 | MarkMielke | openstack on openstack :-) | |
| 15:34:42 | sean-k-mooney | so for me l1 is simulating a cloud and l2 is my test workload | |
| 15:34:51 | sean-k-mooney | and yes but not with tripleO | |
| 15:35:13 | MarkMielke | tripleO doesn't even mean tripleO anymore, right? doesn't it mean OpenStack on K8? haha | |
| 15:35:26 | sean-k-mooney | not yet | |
| 15:35:43 | sean-k-mooney | but yat they are slowlow dismanteling the undercloud | |
| 15:36:04 | sean-k-mooney | personally i much prefer kolla-ansible or OSA | |
| 15:36:13 | MarkMielke | me too | |
| 15:36:14 | sean-k-mooney | triplo is too complicated | |
| 15:36:29 | sean-k-mooney | and has too much techdebt in all its layers | |
| 15:36:45 | MarkMielke | it's somewhat purist without regard to consequences | |
| 15:37:02 | MarkMielke | my view :-) | |
| 15:37:26 | sean-k-mooney | im not sure i would agree. i think it use to be more pure and simple | |
| 15:37:42 | sean-k-mooney | and now with contariation its less pure and more complicated | |
| 15:38:51 | sean-k-mooney | lucally while i have to debug customer systems ocationally (mainly via logs) i dont actuly need to use triplO day to day | |
| 15:39:16 | MarkMielke | Kayobe is the one that was appealing to me | |
| 15:39:22 | MarkMielke | Ironic in stand-alone mode | |
| 15:39:23 | MarkMielke | Kolla | |
| 15:39:26 | MarkMielke | Ansible | |
| 15:39:43 | sean-k-mooney | yep when i work at intel i pitched basically what they implemented internlaly | |
| 15:39:59 | sean-k-mooney | i wrote the original kolla biforst container | |
| 15:40:03 | MarkMielke | cool | |
| 15:40:50 | sean-k-mooney | i really wanted to see koll ansibel become the default openstack installer but i dont really work on the installer anymore | |
| 15:42:25 | MarkMielke | I think it will pick up. | |
| 15:42:48 | sean-k-mooney | if kaobe does then yes | |
| 15:42:59 | sean-k-mooney | *kayobe | |
| 15:43:00 | MarkMielke | It's so much simpler. | |
| 15:43:04 | sean-k-mooney | yep | |
| 15:43:12 | sean-k-mooney | its simple to debug and extend | |
| 15:43:34 | sean-k-mooney | the config overrde mechanisum is easy to use | |
| 15:43:40 | sean-k-mooney | and no puppet/ruby | |
| 15:43:46 | sean-k-mooney | which is defintly a win | |
| 15:43:48 | MarkMielke | Running Kubernetes on hypervisors seems overly complex to me personally. It's not really the right use case for it. | |
| 15:44:14 | MarkMielke | If you were running Docker containers - sure... but virtual machines from within Nova/libvirtd/qemu? I don't think so. | |
| 15:44:33 | sean-k-mooney | the last thing i was trying to push in kolla before i stope working on it was move to running the contaienr using systemd service file | |
| 15:44:50 | sean-k-mooney | so that tehre was a determinisitc order to them on host reboot | |
| 15:45:05 | MarkMielke | the move away from "Docker" may help with that :-) | |
| 15:45:20 | MarkMielke | podman, etc. | |
| 15:45:20 | sean-k-mooney | what are they moving too | |
| 15:45:24 | sean-k-mooney | ah ok | |
| 15:45:35 | sean-k-mooney | ya podman now create the unit file for you | |
| 15:45:38 | MarkMielke | RHEL 8 pushing that hard | |
| 15:45:40 | sean-k-mooney | in the lates version | |
| 15:45:47 | MarkMielke | doesn't come with Docker even in Extras. :-) | |
| 15:45:50 | sean-k-mooney | not as much as you would think | |
| 15:46:24 | sean-k-mooney | well docker yes | |
| 15:46:37 | MarkMielke | we didn't move up to RHEL 8 yes - will start to soon - but Docker being gone was a surprise to me | |
| 15:46:43 | sean-k-mooney | but redhat is not really encouraging use to work upstream in kolla | |
| 15:46:51 | sean-k-mooney | to move kolla ansible to podman | |
| 15:46:52 | MarkMielke | right | |
| 15:47:40 | sean-k-mooney | i was thinking about getting involved again to help with that but people internally didnt care that much so i focused on nova | |
| 15:47:56 | MarkMielke | only so much time in day :-) | |
| 15:48:16 | sean-k-mooney | they didnt disucrage me but suggested i look at getting involved in triplo instead | |
| 15:48:36 | sean-k-mooney | and i have no interst in that | |
| 15:48:44 | MarkMielke | at OpenStack conference, I thought it was amusing when they were talking about future of TripleO | |
| 15:48:57 | MarkMielke | and one of the Kayobe people spoke up "we've already done this" | |
| 15:49:35 | MarkMielke | it's fine though - community needs to develop ideas and then discover intersection of common objectives | |
| 15:49:54 | sean-k-mooney | well i mention trying to adopt kayobe when the topic of simplify triople o came up and it was more or less ignored | |
| 15:50:20 | sean-k-mooney | there is too much of a sunken cost falacy around triplo | |
| 15:50:32 | MarkMielke | yeah i see that | |