Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-12-03
14:19:29 dcapone2004 I found this fix here.... https://www.mess.org/2020/06/22/Hyper-V-in-KVM/#:~:text=Hyper%2DV%20on%20Windows%20requires,it%20is%...
14:19:46 sean-k-mooney so hyperv is need to run the linux continer in light weight vms
14:20:01 dcapone2004 so, I am trying to test what happens when the hypervisor=no parameter is passed
14:22:10 dcapone2004 I'm technically doing this using windows server 2019, but I found a different article that basically said the same thing for windows server 2019, but the article wasn't as short and sweet getting right to the config
14:24:06 mgariepy hello, is there someone who can help push this patch along for Ussuri please: https://review.opendev.org/c/openstack/nova/+/758761
14:25:26 lpetrut dcapone2004 it might be related to the virtual cpu flags. it's been a while since I last ran Hyper-V on top of KVM, but I was using those qemu parameters: -cpu Skylake-Client,hv-relaxed,hv-vapic,hv-time,hv-crash,hv-reset,hv-vpindex,hv-runtime,hv-synic,hv_spinlocks=0x1fff,+vmx
14:27:55 dcapone2004 ipetrut: those flags I have...they are added by nova with the os_type=windows properties....sean helped guide me to try that last night
14:31:25 lpetrut dcapone2004: it may be worth double checking, I'm not sure if Nova sets all the flags required for running Hyper-V on top
14:32:48 dcapone2004 the one I feel is missing based on that article I referenced and linked is the hypervisor=off option
14:33:10 dcapone2004 with your config changes, I have been able to add that via virsh, so enabling hyper-v now and about to reboot
14:39:25 dcapone2004 -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=1234567890ab,kvm=off is all that is generated by default, so there are definitely quite a few missing
14:48:05 dcapone2004 time and crash do not validate with virsh, but trying again with the rest of the options, -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-reset,hv-vendor-id=1234567890ab,kvm=off
14:56:54 dcapone2004 bingo, those options appear to have resolved the booting problem.....now only if nova could add those additional options with some type of image/flavor property ;-) ;-)
14:58:04 sean-k-mooney dcapone2004: so which options are needed that are missing?
14:58:41 sean-k-mooney look like you enabled a bunch of the enlightnments
14:59:22 sean-k-mooney i dont see hypervior=off in the list
14:59:36 sean-k-mooney so it looks like we jsut ened to enable the enlightements
14:59:54 dcapone2004 nope, hypervisor=off wasn't needed with all of the other enlightments....I don't know if there was a specific magic individual one that did it
14:59:56 sean-k-mooney lpetrut: do you know of any reason not to do that for all windows guests by default
15:00:34 sean-k-mooney lpetrut: e.g. if your qemu and libvirt is new enough to support a hyperv enlightment we would jsut enable it by defualt
15:01:05 sean-k-mooney lpetrut: that shoudl improve performace ingereall even without the nested virt or docker usecase
15:01:26 dcapone2004 hv-vpindex,hv-runtime,hv-synic,hv-reset those were the 4 additional that virsh would accept and I added
15:02:33 sean-k-mooney hv-synic might be the issue
15:02:35 stephenfin gibi: You mean the docs? Yes, we did, because it's still available on older releases
15:02:41 sean-k-mooney Enable Synthetic Interrupt Controller (SynIC)
15:02:48 gibi stephenfin: I mean the api ref docs.
15:02:48 stephenfin and the API reference is unversioned
15:03:10 gibi stephenfin: so we don't have victoria api ref
15:03:13 gibi :/
15:03:23 stephenfin We shouldn't have. There's just an API ref
15:03:44 lpetrut sean-k-mooney, dcapone2004: probably the qemu version was the main concern. then we'd also have to make sure that those flags don't impact the performance
15:03:45 stephenfin Because a request for e.g. 2.13 should look the exact same across all versions of nova that support that
15:04:33 sean-k-mooney lpetrut: our min qemu is proably new enought now
15:04:48 gibi stephenfin: but it is not true for fping as it is removed in rocky
15:05:12 gibi stephenfin: so fping request even if it is called with 2.13 is 404 since rocky
15:05:17 sean-k-mooney lpetrut: ok not quite
15:05:23 sean-k-mooney lpetrut: our min is 4.0.0
15:05:26 gibi or at least this is how I understand the api ref
15:05:32 sean-k-mooney and min livbirt is 5.0.0
15:05:58 stephenfin gibi: Yup, and for those cases we stick a note at the top saying "this has been removed in foo" and update the return codes to include HTTP 410
15:06:02 sean-k-mooney lpetrut: we will be going to livbirt 6.0.0 and qemu 4.2.0 this cycle
15:06:11 stephenfin https://docs.openstack.org/api-ref/compute/#ping-instances-os-fping
15:06:16 stephenfin "This API only works with nova-network which is deprecated. It should be avoided in any new applications. These will fail with a 404 starting from microversion 2.36. It was removed in the 18.0.0 Rocky release."
15:06:32 sean-k-mooney lpetrut: dcapone2004 so at that point we coudl enable all of them
15:07:22 lpetrut I think all the above mentioned flags should be supported by qemu 4.0.0, not sure if libvirt needs to be aware of those as well
15:07:36 sean-k-mooney https://libvirt.org/formatdomain.html#hypervisor-features
15:07:44 sean-k-mooney the versions are listed there
15:08:48 gibi stephenfin: OK, then what to do with this https://review.opendev.org/c/openstack/nova/+/765311/1/api-ref/source/parameters.yaml#396
15:09:24 stephenfin we should keep it but not update it
15:09:48 gibi stephenfin: keep it as we cannot update the already deleted code
15:09:50 gibi is it?
15:09:55 lpetrut sean-k-mooney: nice, then I guess Nova could set those flags. I'm still not 100% if that should be the default or if it should be opt-in
15:09:56 stephenfin yes
15:10:03 stephenfin it's just documentation
15:10:13 gibi stephenfin: OK, thanks. it is clear now
15:10:52 dcapone2004 ipetrut: an enable_hyperv_support property maybe?
15:12:01 lpetrut dcapone2004: yep, that would probably be the way to go unless it becomes the default for Windows guests
15:13:42 lpetrut we could take another look at qemu, maybe even ask the qemu guys if they think there's any potential issue (e.g. affecting perf, stability or older guests) in using those flags by default
15:15:19 dcapone2004 ipetrut: if you discuss, all fo the othe rflags are already added, the only ones that were added in addition were these 4: hv-vpindex,hv-runtime,hv-synic,hv-reset I know I mentioned that earlier, but figured I would just bring that up again
15:15:26 lpetrut I assume most of those features are related to Hyper-V VMBus (virtio equivalent), so older guests might not support those enlightments
15:16:16 lpetrut dcapone2004: right, this list is very helpful
15:16:21 sean-k-mooney so what we bacially will want is an image proerty
15:16:27 sean-k-mooney that is a list of enum values
15:16:47 sean-k-mooney and the image could declare what it supports
15:16:58 sean-k-mooney we can add the current allways on enlighemtn as the defualt for the value
15:17:29 sean-k-mooney the other way is to use traits
15:17:43 lpetrut sean-k-mooney: sounds good. fwiw, new hyper-v enlightments might be added in the future so it's probably important to be able to pick the ones that the image supports
15:18:52 lpetrut while having some safe defaults
15:19:08 sean-k-mooney dcapone2004: so this is basically evolving form a potenial bug to a feature request
15:19:09 noonedeadpunk hi! Any idea why I might be seing this http://paste.openstack.org/show/800707/ when `backend` is set to `oslo_cache.memcache_pool` in [cache]?
15:19:30 sean-k-mooney is this something you would have capsity to work on or just soemthjing you would like to see added
15:20:28 sean-k-mooney noonedeadpunk:that proably more of an oslo question
15:20:32 lpetrut sean-k-mooney: I can't remember, does the libvirt nova driver allow you to set arbitrary cpu flags?
15:21:21 sean-k-mooney noonedeadpunk: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/etcd3gw.py#L17
15:21:30 sean-k-mooney lpetrut: cpu flags yes but these are not cpu flags
15:21:41 sean-k-mooney lpetrut: and only via the nova config
15:21:59 sean-k-mooney noonedeadpunk: it look like it does an unconditonal import
15:22:02 noonedeadpunk it feels it tries to load all modules possible... so nova just passes [cache] section just directly to oslo? (I'm not very common with codebase still - it's huge)
15:22:38 sean-k-mooney noonedeadpunk: these config option are actully adde by oslo.cache
15:23:54 noonedeadpunk I'm jsut not sure why it loads etcd3gw backend at all...
15:26:05 sean-k-mooney im guessing its usign the stevado extention manager to load it unconditonally
15:26:11 sean-k-mooney looking for that currently
15:29:16 sean-k-mooney noonedeadpunk: where is that being raised
15:30:05 noonedeadpunk found it here https://zuul.opendev.org/t/openstack/build/0af9364cc9094b5096899ed7693a9f60/log/logs/host/nova-conductor.service.journal-10-50-46.log.txt#2431
15:31:55 sean-k-mooney we are not using sevedor to load this in oslo cache or in nova
15:32:33 noonedeadpunk um.....
15:32:56 sean-k-mooney im wondering if this is coming form oslo.config
15:33:22 noonedeadpunk can hardly imagine path of the imports tbh
15:33:54 noonedeadpunk But I think it appeared only in V since in U haven't seen that message
15:33:57 sean-k-mooney we are using oslo cache
15:34:20 sean-k-mooney but i dont see it directly using sevedor in its code
15:34:48 noonedeadpunk ah, yes, might be oslo.config that using it, right
15:34:54 sean-k-mooney https://codesearch.opendev.org/?q=stevedore&i=nope&files=&excludeFiles=&repos=openstack/nova,openstack/oslo.cache,openstack/oslo.config,openstack/oslo.privsep
15:34:58 sean-k-mooney that is what im looking at
15:35:44 sean-k-mooney we use stevador for 3 things in nova, extra spec validator, schudler dirvers and metric monitors
15:36:10 sean-k-mooney oslo cache and oslo privsep dont use it at all it just in ther lower constraits
15:36:20 sean-k-mooney because they depend on oslo.config which does use it
15:37:41 sean-k-mooney it might be https://github.com/openstack/oslo.cache/commit/308f49aaaad554573aeff8ddc68c39ff7fe57438
15:37:50 noonedeadpunk seem like https://opendev.org/openstack/oslo.config/src/branch/master/oslo_config/cfg.py#L2152-L2154
15:37:53 noonedeadpunk yeah

Earlier   Later