| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-25 | |||
| 14:26:15 | mriedem | i wouldn't assume so | |
| 14:26:17 | efried | and yeah, late fail | |
| 14:26:31 | efried | but that's no different than e.g. libvirt failing if you try to downsize the eph disk | |
| 14:26:44 | mriedem | the api / controller is pretty opinionated that during a move operation the host:node relationship is 1:1 b/c we don't support moving baremetal instances | |
| 14:27:01 | mriedem | and you've got the CONF.allow_resize_to_same_host issues | |
| 14:27:22 | mnaser | eh, it looks like there is no nested virt trait eh? | |
| 14:27:43 | mriedem | which reminds me of https://bugs.launchpad.net/nova/+bug/1819216 but that's for cold migration | |
| 14:27:44 | openstack | Launchpad bug 1819216 in OpenStack Compute (nova) "in devstack, "nova migrate <uuid>" will try to migrate to the same host (and then fail)" [Medium,Triaged] | |
| 14:27:52 | efried | mnaser: wha? | |
| 14:28:15 | mnaser | not all of our hypervisors have been updated to have nested virt enabled (its disabled by default on rhel based hosts) | |
| 14:28:26 | mnaser | so i was hoping to create a flavor for a user so they always got a machine that had nested virt | |
| 14:28:35 | mnaser | (rather than building a host aggregate and trying to keep it sync'd) | |
| 14:30:30 | mriedem | does HW_CPU_X86_VMX only show up on your nested virt enabled hosts? | |
| 14:30:44 | efried | sorry, this isn't going to be very helpful, but what does "nested virt" mean? | |
| 14:31:00 | mnaser | efried: oh sorry, it means enabling an option in kvm to expose the vmx flag to the virtual machine | |
| 14:31:11 | mnaser | so that it can start up kvm accelerated vms (instead of qemu emulated ones) | |
| 14:31:13 | kashyap | efried: It means you're running on "turtles all the way down" | |
| 14:31:30 | kashyap | It's only a part troll; as the original nested virt paper was titled that way | |
| 14:31:34 | mnaser | HW_CPU_X86_VMX will probably always be exposed on the host i assume (though libvirt can give you the answer if its enabled or not) | |
| 14:31:37 | dansmith | efried: it means... if random() % 10 < 4, segfault() | |
| 14:31:43 | kashyap | https://www.usenix.org/legacy/events/osdi10/tech/slides/ben-yehuda.pdf | |
| 14:31:47 | mnaser | lolol | |
| 14:32:04 | kashyap | Sorry, here's the paper: "The Turtles Project": https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Ben-Yehuda.pdf | |
| 14:32:06 | artom|gmtplus3 | mnaser, are flavors with more than 1 NUMA node also in the pipeline? | |
| 14:32:12 | mnaser | it's actually been pretty stable in kata's ci (in our nested virt hosts) and doing nested virt on newer kernels is pretty well | |
| 14:32:18 | mnaser | artom|gmtplus3: in this case, no | |
| 14:32:36 | mnaser | but we've looked into it in the past :> | |
| 14:32:54 | mnaser | but yeah, im gonna guess HW_CPU_X86_VMX is probably exposed in all kvm hosts | |
| 14:33:05 | dansmith | I would expect the same, FWIW | |
| 14:33:13 | dansmith | I think those come from host caps | |
| 14:33:19 | kashyap | mnaser: Nit-pick: "nested virt" can mean QEMU or KVM in level-1 :-). I use nested KVM to distinguish the "no emulator" part. | |
| 14:33:27 | artom|gmtplus3 | mnaser, because I will be eternally grateful when they come :) | |
| 14:33:47 | mnaser | kashyap: oh cool, good to know | |
| 14:33:53 | mnaser | artom|gmtplus3: i'll keep that in mind then :) | |
| 14:34:44 | artom|gmtplus3 | mnaser, heh, I don't need to tell you how badly we're missing NUMA test coverage with real hardware :) | |
| 14:35:11 | kashyap | mnaser: So, as mriedem said, assuming you have 'nested' as '1' on bare metal, and then set the flavor property: | |
| 14:35:14 | kashyap | openstack flavor set 1 --property trait:HW_CPU_X86_VMX=required` | |
| 14:35:30 | kashyap | Then launching the level-1 guest (or "guest hypervisor") should have VMX exposed | |
| 14:35:47 | mriedem | yonglihe: regarding https://review.opendev.org/#/c/627765/ i remember hearing at the ptg that there were concerns that it duplicated too much of _cleanup_running_deleted_instances and asked if it could just be integrated into _cleanup_running_deleted_instances with new choices for the running_deleted_instance_action option, but i'm not sure how feasible that is (or beneficial) | |
| 14:37:12 | kashyap | mnaser: dansmith: By the way GCE officially supports nested KVM; they've even done (in the community) extensive security audit, FWIW. | |
| 14:37:37 | dansmith | kashyap: omg, I'm convinced. | |
| 14:38:19 | kashyap | Good; otherwise, you need to show data for not being convinced ;-) | |
| 14:38:32 | dansmith | kashyap: you don't remember this, | |
| 14:38:50 | dansmith | but when I joined redhat almost six years ago, my first interaction with you was you saying this exact thing | |
| 14:39:06 | dansmith | that nested virt was 110% ready for prime time and you were fine if your bank ran it | |
| 14:39:17 | dansmith | don't make me dig up the logs, 'cause I'll do it! | |
| 14:40:02 | kashyap | I'm pretty confident I don't make such superlative remarks. Do dig them up | |
| 14:40:20 | kashyap | I used to systematically file kernel bugs since 2013 for nested KVM | |
| 14:40:21 | dansmith | superlatives are mine :) | |
| 14:40:32 | kashyap | And 98% of them are resolved (https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt) | |
| 14:40:51 | kashyap | Sure, there are cases where it doesn't work; but what exists already works for good number of cases | |
| 14:41:06 | kashyap | Heck, even upstream kernel itself now has 'nested' flag enabled by default for Intel | |
| 14:41:36 | mnaser | ^ yeah, i've seen this updated lately too | |
| 14:41:51 | sean-k-mooney | it was enabled in 4.19 ish | |
| 14:42:06 | sean-k-mooney | maybe a little earlier | |
| 14:42:20 | kashyap | [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e58e5e] | |
| 14:42:27 | kashyap | ["KVM: VMX: enable nested virtualization by default | |
| 14:42:29 | kashyap | ] | |
| 14:42:49 | sean-k-mooney | there were some bug in both the ubuntu kernel however until 4.20 | |
| 14:43:02 | sean-k-mooney | fedora was broken too | |
| 14:43:09 | sean-k-mooney | but the upstream on was fine | |
| 14:43:17 | dansmith | heh, Celebrated-By.. that's one I haven't seen | |
| 14:43:59 | kashyap | dansmith: Then you will enjoy this: https://lwn.net/Articles/503829/ | |
| 14:44:08 | kashyap | A sample from the KM-long list: | |
| 14:44:12 | kashyap | - Bitten-by-and-tested-by: | |
| 14:44:12 | kashyap | - Antagonized-by: | |
| 14:44:12 | kashyap | - Fatfingered-by: | |
| 14:44:12 | kashyap | - Signed-off-and-morning-tea-spilled-by: | |
| 14:44:38 | dansmith | yup, nice | |
| 14:45:11 | artom|gmtplus3 | Drive-by: Shoo Ting | |
| 14:45:31 | kashyap | mnaser: Live migrating level-1 guest (level-2 can be migrated fine) is the remaining gnarly problem that is being worked on (both in KVM and QEMU) | |
| 14:46:22 | sean-k-mooney | so Most-Definitely-Acked-by: | |
| 14:46:24 | sean-k-mooney | Mostly-acked-by: | |
| 14:46:26 | sean-k-mooney | NACKed-by: | |
| 14:46:33 | sean-k-mooney | are used as +2 +1 and -1 | |
| 14:46:46 | sean-k-mooney | also i like More-or-less-tested-by | |
| 14:47:35 | mriedem | kashyap: what are the chances GCE has forked everything to make nested KVM work for just their cloud? | |
| 14:48:14 | dansmith | *gasp* | |
| 14:48:19 | dansmith | they would NEVER do such a thing, surely | |
| 14:49:14 | kashyap | mriedem: The chances are 100% sunny :D I see what you mean, they do have their kernel; that said ... they contribute most KVM-related stuff back to the upstream, and post results of security audit, too | |
| 14:50:52 | sean-k-mooney | mriedem: from my past experience with kvm it works beeter if you use a vanial upstream kernel then a disto packaged one although that has change in the last 12-18 months or so as i think distors have got better at testing nested virt or just break it less | |
| 14:52:43 | mriedem | hmm, me checks notes...sean says things from a distro are better since about the time he started working for a distro...yup it checks out | |
| 14:53:11 | sean-k-mooney | goes back to his ubunut based laptop... | |
| 14:53:20 | dansmith | mmm, ubunut | |
| 14:53:47 | dansmith | that has to be about the best ubuntu type I've seen | |
| 14:53:54 | sean-k-mooney | my devstack vms are centos | |
| 14:53:55 | dansmith | *typo, hah | |
| 14:54:32 | mriedem | did you know a cashew is not a nut? | |
| 14:54:36 | dansmith | artom|gmtplus3: like a fancy brazilian walnut laptop, or the *really* fancy "ubunut" laptop | |
| 14:54:48 | sean-k-mooney | :) | |
| 14:55:50 | sean-k-mooney | mriedem: that was most based on the fact that centos finally started to wrok with nested vert although canonica broke it in 18.04 which made me sad | |
| 14:55:51 | mriedem | lyarwood: just so you're aware, stable/stein and rocky are essentially blocked until these devstack changes land https://review.opendev.org/#/q/I0d7f9f2baaee2836a719f199939156bd4f53f778 | |
| 14:56:24 | artom|gmtplus3 | Oh, not the whole laptop, just the CPU: Hazwellnut architecture :D | |
| 14:56:58 | sean-k-mooney | haha i would totally by that | |
| 14:57:44 | lyarwood | mriedem: was that behind tempest-slow etc timing out due to the network tests taking ages? | |
| 14:58:35 | lyarwood | ah nvm it doesn't look like it is | |
| 15:00:28 | mriedem | the agent on the tempest-slow jobs blows up | |
| 15:01:19 | lyarwood | ah I see | |
| 15:02:46 | lyarwood | well thanks for the heads up, I've had a tab open to look at that all day, I'll get on with some reviews once that has started to land in stable. | |
| 15:04:59 | sean-k-mooney | dansmith: im always amazed by how many grammer and spelling issue my brain just completely ignores | |