| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-25 | |||
| 14:20:53 | efried | yes | |
| 14:20:59 | mriedem | pretty sure that code flips the instance.flavor to new_flavor before calling the driver | |
| 14:21:12 | mriedem | i have this code mostly memorized by now | |
| 14:21:22 | dansmith | I thought it was after poweroff | |
| 14:21:42 | efried | I don't actually need to know the details right now; just wanted to make sure both flavors are available in some form to the virt driver at the early stage of the resize. See ML. | |
| 14:21:58 | mriedem | oh i was thinking of this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4686 | |
| 14:22:08 | mriedem | we do'nt update the instance.flavor to the new flavor until finish_resize on the dest | |
| 14:22:19 | dansmith | yeah, that's what I thought | |
| 14:22:29 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4550 | |
| 14:22:33 | mriedem | instance_type in that case is the new flavor | |
| 14:22:36 | mriedem | instance.flavor is still the old flavor | |
| 14:22:55 | efried | tldr: ironic could theoretically implement a super-restricted same-node-only 'resize' to twiddle tunables in their deploy template. | |
| 14:22:58 | mriedem | and instance.new_flavor is set before that https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4338 | |
| 14:23:43 | mriedem | well you don't even need the flavor to determine if it's the same host | |
| 14:23:53 | mriedem | we pass in the dest host, so you'd just check if dest_host == CONF.host | |
| 14:24:11 | mriedem | that doesn't tell you it's the same node... | |
| 14:24:47 | efried | okay, but they also need to be able to make sure that nothing has changed outside the limited subset of allowable changes | |
| 14:24:51 | efried | swhy they would need the flavors | |
| 14:25:41 | efried | but yeah, also some way to know it's same-node. Maybe that part wouldn't be wrong to do in the cond/api | |
| 14:25:45 | mriedem | so...gonna plumb this all up through the api and everything just to eventually have the ironic driver check if the only thing changing in the new flavor is the deploy template and fail if not? | |
| 14:26:00 | efried | well, except the plumbing is already there, no? | |
| 14:26:13 | efried | basically all they have to do is implement two virt driver methods to make it work the dumb way | |
| 14:26:15 | mriedem | i wouldn't assume so | |
| 14:26:17 | efried | and yeah, late fail | |
| 14:26:31 | efried | but that's no different than e.g. libvirt failing if you try to downsize the eph disk | |
| 14:26:44 | mriedem | the api / controller is pretty opinionated that during a move operation the host:node relationship is 1:1 b/c we don't support moving baremetal instances | |
| 14:27:01 | mriedem | and you've got the CONF.allow_resize_to_same_host issues | |
| 14:27:22 | mnaser | eh, it looks like there is no nested virt trait eh? | |
| 14:27:43 | mriedem | which reminds me of https://bugs.launchpad.net/nova/+bug/1819216 but that's for cold migration | |
| 14:27:44 | openstack | Launchpad bug 1819216 in OpenStack Compute (nova) "in devstack, "nova migrate <uuid>" will try to migrate to the same host (and then fail)" [Medium,Triaged] | |
| 14:27:52 | efried | mnaser: wha? | |
| 14:28:15 | mnaser | not all of our hypervisors have been updated to have nested virt enabled (its disabled by default on rhel based hosts) | |
| 14:28:26 | mnaser | so i was hoping to create a flavor for a user so they always got a machine that had nested virt | |
| 14:28:35 | mnaser | (rather than building a host aggregate and trying to keep it sync'd) | |
| 14:30:30 | mriedem | does HW_CPU_X86_VMX only show up on your nested virt enabled hosts? | |
| 14:30:44 | efried | sorry, this isn't going to be very helpful, but what does "nested virt" mean? | |
| 14:31:00 | mnaser | efried: oh sorry, it means enabling an option in kvm to expose the vmx flag to the virtual machine | |
| 14:31:11 | mnaser | so that it can start up kvm accelerated vms (instead of qemu emulated ones) | |
| 14:31:13 | kashyap | efried: It means you're running on "turtles all the way down" | |
| 14:31:30 | kashyap | It's only a part troll; as the original nested virt paper was titled that way | |
| 14:31:34 | mnaser | HW_CPU_X86_VMX will probably always be exposed on the host i assume (though libvirt can give you the answer if its enabled or not) | |
| 14:31:37 | dansmith | efried: it means... if random() % 10 < 4, segfault() | |
| 14:31:43 | kashyap | https://www.usenix.org/legacy/events/osdi10/tech/slides/ben-yehuda.pdf | |
| 14:31:47 | mnaser | lolol | |
| 14:32:04 | kashyap | Sorry, here's the paper: "The Turtles Project": https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Ben-Yehuda.pdf | |
| 14:32:06 | artom|gmtplus3 | mnaser, are flavors with more than 1 NUMA node also in the pipeline? | |
| 14:32:12 | mnaser | it's actually been pretty stable in kata's ci (in our nested virt hosts) and doing nested virt on newer kernels is pretty well | |
| 14:32:18 | mnaser | artom|gmtplus3: in this case, no | |
| 14:32:36 | mnaser | but we've looked into it in the past :> | |
| 14:32:54 | mnaser | but yeah, im gonna guess HW_CPU_X86_VMX is probably exposed in all kvm hosts | |
| 14:33:05 | dansmith | I would expect the same, FWIW | |
| 14:33:13 | dansmith | I think those come from host caps | |
| 14:33:19 | kashyap | mnaser: Nit-pick: "nested virt" can mean QEMU or KVM in level-1 :-). I use nested KVM to distinguish the "no emulator" part. | |
| 14:33:27 | artom|gmtplus3 | mnaser, because I will be eternally grateful when they come :) | |
| 14:33:47 | mnaser | kashyap: oh cool, good to know | |
| 14:33:53 | mnaser | artom|gmtplus3: i'll keep that in mind then :) | |
| 14:34:44 | artom|gmtplus3 | mnaser, heh, I don't need to tell you how badly we're missing NUMA test coverage with real hardware :) | |
| 14:35:11 | kashyap | mnaser: So, as mriedem said, assuming you have 'nested' as '1' on bare metal, and then set the flavor property: | |
| 14:35:14 | kashyap | openstack flavor set 1 --property trait:HW_CPU_X86_VMX=required` | |
| 14:35:30 | kashyap | Then launching the level-1 guest (or "guest hypervisor") should have VMX exposed | |
| 14:35:47 | mriedem | yonglihe: regarding https://review.opendev.org/#/c/627765/ i remember hearing at the ptg that there were concerns that it duplicated too much of _cleanup_running_deleted_instances and asked if it could just be integrated into _cleanup_running_deleted_instances with new choices for the running_deleted_instance_action option, but i'm not sure how feasible that is (or beneficial) | |
| 14:37:12 | kashyap | mnaser: dansmith: By the way GCE officially supports nested KVM; they've even done (in the community) extensive security audit, FWIW. | |
| 14:37:37 | dansmith | kashyap: omg, I'm convinced. | |
| 14:38:19 | kashyap | Good; otherwise, you need to show data for not being convinced ;-) | |
| 14:38:32 | dansmith | kashyap: you don't remember this, | |
| 14:38:50 | dansmith | but when I joined redhat almost six years ago, my first interaction with you was you saying this exact thing | |
| 14:39:06 | dansmith | that nested virt was 110% ready for prime time and you were fine if your bank ran it | |
| 14:39:17 | dansmith | don't make me dig up the logs, 'cause I'll do it! | |
| 14:40:02 | kashyap | I'm pretty confident I don't make such superlative remarks. Do dig them up | |
| 14:40:20 | kashyap | I used to systematically file kernel bugs since 2013 for nested KVM | |
| 14:40:21 | dansmith | superlatives are mine :) | |
| 14:40:32 | kashyap | And 98% of them are resolved (https://kashyapc.fedorapeople.org/kernel-kvm-bugs.txt) | |
| 14:40:51 | kashyap | Sure, there are cases where it doesn't work; but what exists already works for good number of cases | |
| 14:41:06 | kashyap | Heck, even upstream kernel itself now has 'nested' flag enabled by default for Intel | |
| 14:41:36 | mnaser | ^ yeah, i've seen this updated lately too | |
| 14:41:51 | sean-k-mooney | it was enabled in 4.19 ish | |
| 14:42:06 | sean-k-mooney | maybe a little earlier | |
| 14:42:20 | kashyap | [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1e58e5e] | |
| 14:42:27 | kashyap | ["KVM: VMX: enable nested virtualization by default | |
| 14:42:29 | kashyap | ] | |
| 14:42:49 | sean-k-mooney | there were some bug in both the ubuntu kernel however until 4.20 | |
| 14:43:02 | sean-k-mooney | fedora was broken too | |
| 14:43:09 | sean-k-mooney | but the upstream on was fine | |
| 14:43:17 | dansmith | heh, Celebrated-By.. that's one I haven't seen | |
| 14:43:59 | kashyap | dansmith: Then you will enjoy this: https://lwn.net/Articles/503829/ | |
| 14:44:08 | kashyap | A sample from the KM-long list: | |
| 14:44:12 | kashyap | - Bitten-by-and-tested-by: | |
| 14:44:12 | kashyap | - Antagonized-by: | |
| 14:44:12 | kashyap | - Fatfingered-by: | |
| 14:44:12 | kashyap | - Signed-off-and-morning-tea-spilled-by: | |
| 14:44:38 | dansmith | yup, nice | |
| 14:45:11 | artom|gmtplus3 | Drive-by: Shoo Ting | |
| 14:45:31 | kashyap | mnaser: Live migrating level-1 guest (level-2 can be migrated fine) is the remaining gnarly problem that is being worked on (both in KVM and QEMU) | |
| 14:46:22 | sean-k-mooney | so Most-Definitely-Acked-by: | |
| 14:46:24 | sean-k-mooney | Mostly-acked-by: | |
| 14:46:26 | sean-k-mooney | NACKed-by: | |
| 14:46:33 | sean-k-mooney | are used as +2 +1 and -1 | |
| 14:46:46 | sean-k-mooney | also i like More-or-less-tested-by | |
| 14:47:35 | mriedem | kashyap: what are the chances GCE has forked everything to make nested KVM work for just their cloud? | |
| 14:48:14 | dansmith | *gasp* | |