| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-25 | |||
| 14:23:43 | mriedem | well you don't even need the flavor to determine if it's the same host | |
| 14:23:53 | mriedem | we pass in the dest host, so you'd just check if dest_host == CONF.host | |
| 14:24:11 | mriedem | that doesn't tell you it's the same node... | |
| 14:24:47 | efried | okay, but they also need to be able to make sure that nothing has changed outside the limited subset of allowable changes | |
| 14:24:51 | efried | swhy they would need the flavors | |
| 14:25:41 | efried | but yeah, also some way to know it's same-node. Maybe that part wouldn't be wrong to do in the cond/api | |
| 14:25:45 | mriedem | so...gonna plumb this all up through the api and everything just to eventually have the ironic driver check if the only thing changing in the new flavor is the deploy template and fail if not? | |
| 14:26:00 | efried | well, except the plumbing is already there, no? | |
| 14:26:13 | efried | basically all they have to do is implement two virt driver methods to make it work the dumb way | |
| 14:26:15 | mriedem | i wouldn't assume so | |
| 14:26:17 | efried | and yeah, late fail | |
| 14:26:31 | efried | but that's no different than e.g. libvirt failing if you try to downsize the eph disk | |
| 14:26:44 | mriedem | the api / controller is pretty opinionated that during a move operation the host:node relationship is 1:1 b/c we don't support moving baremetal instances | |
| 14:27:01 | mriedem | and you've got the CONF.allow_resize_to_same_host issues | |
| 14:27:22 | mnaser | eh, it looks like there is no nested virt trait eh? | |
| 14:27:43 | mriedem | which reminds me of https://bugs.launchpad.net/nova/+bug/1819216 but that's for cold migration | |
| 14:27:44 | openstack | Launchpad bug 1819216 in OpenStack Compute (nova) "in devstack, "nova migrate <uuid>" will try to migrate to the same host (and then fail)" [Medium,Triaged] | |
| 14:27:52 | efried | mnaser: wha? | |
| 14:28:15 | mnaser | not all of our hypervisors have been updated to have nested virt enabled (its disabled by default on rhel based hosts) | |
| 14:28:26 | mnaser | so i was hoping to create a flavor for a user so they always got a machine that had nested virt | |
| 14:28:35 | mnaser | (rather than building a host aggregate and trying to keep it sync'd) | |
| 14:30:30 | mriedem | does HW_CPU_X86_VMX only show up on your nested virt enabled hosts? | |
| 14:30:44 | efried | sorry, this isn't going to be very helpful, but what does "nested virt" mean? | |
| 14:31:00 | mnaser | efried: oh sorry, it means enabling an option in kvm to expose the vmx flag to the virtual machine | |
| 14:31:11 | mnaser | so that it can start up kvm accelerated vms (instead of qemu emulated ones) | |
| 14:31:13 | kashyap | efried: It means you're running on "turtles all the way down" | |
| 14:31:30 | kashyap | It's only a part troll; as the original nested virt paper was titled that way | |
| 14:31:34 | mnaser | HW_CPU_X86_VMX will probably always be exposed on the host i assume (though libvirt can give you the answer if its enabled or not) | |
| 14:31:37 | dansmith | efried: it means... if random() % 10 < 4, segfault() | |
| 14:31:43 | kashyap | https://www.usenix.org/legacy/events/osdi10/tech/slides/ben-yehuda.pdf | |
| 14:31:47 | mnaser | lolol | |
| 14:32:04 | kashyap | Sorry, here's the paper: "The Turtles Project": https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Ben-Yehuda.pdf | |
| 14:32:06 | artom|gmtplus3 | mnaser, are flavors with more than 1 NUMA node also in the pipeline? | |
| 14:32:12 | mnaser | it's actually been pretty stable in kata's ci (in our nested virt hosts) and doing nested virt on newer kernels is pretty well | |
| 14:32:18 | mnaser | artom|gmtplus3: in this case, no | |
| 14:32:36 | mnaser | but we've looked into it in the past :> | |
| 14:32:54 | mnaser | but yeah, im gonna guess HW_CPU_X86_VMX is probably exposed in all kvm hosts | |
| 14:33:05 | dansmith | I would expect the same, FWIW | |
| 14:33:13 | dansmith | I think those come from host caps | |
| 14:33:19 | kashyap | mnaser: Nit-pick: "nested virt" can mean QEMU or KVM in level-1 :-). I use nested KVM to distinguish the "no emulator" part. | |
| 14:33:27 | artom|gmtplus3 | mnaser, because I will be eternally grateful when they come :) | |
| 14:33:47 | mnaser | kashyap: oh cool, good to know | |
| 14:33:53 | mnaser | artom|gmtplus3: i'll keep that in mind then :) | |
| 14:34:44 | artom|gmtplus3 | mnaser, heh, I don't need to tell you how badly we're missing NUMA test coverage with real hardware :) | |
| 14:35:11 | kashyap | mnaser: So, as mriedem said, assuming you have 'nested' as '1' on bare metal, and then set the flavor property: | |
| 14:35:14 | kashyap | openstack flavor set 1 --property trait:HW_CPU_X86_VMX=required` | |
| 14:35:30 | kashyap | Then launching the level-1 guest (or "guest hypervisor") should have VMX exposed | |
| 14:35:47 | mriedem | yonglihe: regarding https://review.opendev.org/#/c/627765/ i remember hearing at the ptg that there were concerns that it duplicated too much of _cleanup_running_deleted_instances and asked if it could just be integrated into _cleanup_running_deleted_instances with new choices for the running_deleted_instance_action option, but i'm not sure how feasible that is (or beneficial) | |
| 14:37:12 | kashyap | mnaser: dansmith: By the way GCE officially supports nested KVM; they've even done (in the community) extensive security audit, FWIW. | |
| 14:37:37 | dansmith | kashyap: omg, I'm convinced. | |
| 14:38:19 | kashyap | Good; otherwise, you need to show data for not being convinced ;-) | |
| 14:38:32 | dansmith | kashyap: you don't remember this, | |
| 14:38:50 | dansmith | but when I joined redhat almost six years ago, my first interaction with you was you saying this exact thing | |
| 14:39:06 | dansmith | that nested virt was 110% ready for prime time and you were fine if your bank ran it | |
| 14:39:17 | dansmith | don't make me dig up the logs, 'cause I'll do it! | |
| 14:40:02 | kashyap | I'm pretty confident I don't make such superlative remarks. Do dig them up | |
| 14:40:20 | kashyap | I used to systematically file kernel bugs since 2013 for nested KVM | |
| 14:40:21 | dansmith | superlatives are mine :) | |
| 14:40:32 | kashyap | And 98% of them are resolved (https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt) | |
| 14:40:51 | kashyap | Sure, there are cases where it doesn't work; but what exists already works for good number of cases | |
| 14:41:06 | kashyap | Heck, even upstream kernel itself now has 'nested' flag enabled by default for Intel | |
| 14:41:36 | mnaser | ^ yeah, i've seen this updated lately too | |
| 14:41:51 | sean-k-mooney | it was enabled in 4.19 ish | |
| 14:42:06 | sean-k-mooney | maybe a little earlier | |
| 14:42:20 | kashyap | [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e58e5e] | |
| 14:42:27 | kashyap | ["KVM: VMX: enable nested virtualization by default | |
| 14:42:29 | kashyap | ] | |
| 14:42:49 | sean-k-mooney | there were some bug in both the ubuntu kernel however until 4.20 | |
| 14:43:02 | sean-k-mooney | fedora was broken too | |
| 14:43:09 | sean-k-mooney | but the upstream on was fine | |
| 14:43:17 | dansmith | heh, Celebrated-By.. that's one I haven't seen | |
| 14:43:59 | kashyap | dansmith: Then you will enjoy this: https://lwn.net/Articles/503829/ | |
| 14:44:08 | kashyap | A sample from the KM-long list: | |
| 14:44:12 | kashyap | - Bitten-by-and-tested-by: | |
| 14:44:12 | kashyap | - Antagonized-by: | |
| 14:44:12 | kashyap | - Fatfingered-by: | |
| 14:44:12 | kashyap | - Signed-off-and-morning-tea-spilled-by: | |
| 14:44:38 | dansmith | yup, nice | |
| 14:45:11 | artom|gmtplus3 | Drive-by: Shoo Ting | |
| 14:45:31 | kashyap | mnaser: Live migrating level-1 guest (level-2 can be migrated fine) is the remaining gnarly problem that is being worked on (both in KVM and QEMU) | |
| 14:46:22 | sean-k-mooney | so Most-Definitely-Acked-by: | |
| 14:46:24 | sean-k-mooney | Mostly-acked-by: | |
| 14:46:26 | sean-k-mooney | NACKed-by: | |
| 14:46:33 | sean-k-mooney | are used as +2 +1 and -1 | |
| 14:46:46 | sean-k-mooney | also i like More-or-less-tested-by | |
| 14:47:35 | mriedem | kashyap: what are the chances GCE has forked everything to make nested KVM work for just their cloud? | |
| 14:48:14 | dansmith | *gasp* | |
| 14:48:19 | dansmith | they would NEVER do such a thing, surely | |
| 14:49:14 | kashyap | mriedem: The chances are 100% sunny :D I see what you mean, they do have their kernel; that said ... they contribute most KVM-related stuff back to the upstream, and post results of security audit, too | |
| 14:50:52 | sean-k-mooney | mriedem: from my past experience with kvm it works beeter if you use a vanial upstream kernel then a disto packaged one although that has change in the last 12-18 months or so as i think distors have got better at testing nested virt or just break it less | |
| 14:52:43 | mriedem | hmm, me checks notes...sean says things from a distro are better since about the time he started working for a distro...yup it checks out | |
| 14:53:11 | sean-k-mooney | goes back to his ubunut based laptop... | |
| 14:53:20 | dansmith | mmm, ubunut | |
| 14:53:47 | dansmith | that has to be about the best ubuntu type I've seen | |
| 14:53:54 | sean-k-mooney | my devstack vms are centos | |
| 14:53:55 | dansmith | *typo, hah | |
| 14:54:32 | mriedem | did you know a cashew is not a nut? | |
| 14:54:36 | dansmith | artom|gmtplus3: like a fancy brazilian walnut laptop, or the *really* fancy "ubunut" laptop | |
| 14:54:48 | sean-k-mooney | :) | |
| 14:55:50 | sean-k-mooney | mriedem: that was most based on the fact that centos finally started to wrok with nested vert although canonica broke it in 18.04 which made me sad | |