Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-12-03
14:48:05 dcapone2004 time and crash do not validate with virsh, but trying again with the rest of the options, -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-reset,hv-vendor-id=1234567890ab,kvm=off
14:56:54 dcapone2004 bingo, those options appear to have resolved the booting problem.....now only if nova could add those additional options with some type of image/flavor property ;-) ;-)
14:58:04 sean-k-mooney dcapone2004: so which options are needed that are missing?
14:58:41 sean-k-mooney look like you enabled a bunch of the enlightnments
14:59:22 sean-k-mooney i dont see hypervior=off in the list
14:59:36 sean-k-mooney so it looks like we jsut ened to enable the enlightements
14:59:54 dcapone2004 nope, hypervisor=off wasn't needed with all of the other enlightments....I don't know if there was a specific magic individual one that did it
14:59:56 sean-k-mooney lpetrut: do you know of any reason not to do that for all windows guests by default
15:00:34 sean-k-mooney lpetrut: e.g. if your qemu and libvirt is new enough to support a hyperv enlightment we would jsut enable it by defualt
15:01:05 sean-k-mooney lpetrut: that shoudl improve performace ingereall even without the nested virt or docker usecase
15:01:26 dcapone2004 hv-vpindex,hv-runtime,hv-synic,hv-reset those were the 4 additional that virsh would accept and I added
15:02:33 sean-k-mooney hv-synic might be the issue
15:02:35 stephenfin gibi: You mean the docs? Yes, we did, because it's still available on older releases
15:02:41 sean-k-mooney Enable Synthetic Interrupt Controller (SynIC)
15:02:48 stephenfin and the API reference is unversioned
15:02:48 gibi stephenfin: I mean the api ref docs.
15:03:10 gibi stephenfin: so we don't have victoria api ref
15:03:13 gibi :/
15:03:23 stephenfin We shouldn't have. There's just an API ref
15:03:44 lpetrut sean-k-mooney, dcapone2004: probably the qemu version was the main concern. then we'd also have to make sure that those flags don't impact the performance
15:03:45 stephenfin Because a request for e.g. 2.13 should look the exact same across all versions of nova that support that
15:04:33 sean-k-mooney lpetrut: our min qemu is proably new enought now
15:04:48 gibi stephenfin: but it is not true for fping as it is removed in rocky
15:05:12 gibi stephenfin: so fping request even if it is called with 2.13 is 404 since rocky
15:05:17 sean-k-mooney lpetrut: ok not quite
15:05:23 sean-k-mooney lpetrut: our min is 4.0.0
15:05:26 gibi or at least this is how I understand the api ref
15:05:32 sean-k-mooney and min livbirt is 5.0.0
15:05:58 stephenfin gibi: Yup, and for those cases we stick a note at the top saying "this has been removed in foo" and update the return codes to include HTTP 410
15:06:02 sean-k-mooney lpetrut: we will be going to livbirt 6.0.0 and qemu 4.2.0 this cycle
15:06:11 stephenfin https://docs.openstack.org/api-ref/compute/#ping-instances-os-fping
15:06:16 stephenfin "This API only works with nova-network which is deprecated. It should be avoided in any new applications. These will fail with a 404 starting from microversion 2.36. It was removed in the 18.0.0 Rocky release."
15:06:32 sean-k-mooney lpetrut: dcapone2004 so at that point we coudl enable all of them
15:07:22 lpetrut I think all the above mentioned flags should be supported by qemu 4.0.0, not sure if libvirt needs to be aware of those as well
15:07:36 sean-k-mooney https://libvirt.org/formatdomain.html#hypervisor-features
15:07:44 sean-k-mooney the versions are listed there
15:08:48 gibi stephenfin: OK, then what to do with this https://review.opendev.org/c/openstack/nova/+/765311/1/api-ref/source/parameters.yaml#396
15:09:24 stephenfin we should keep it but not update it
15:09:48 gibi stephenfin: keep it as we cannot update the already deleted code
15:09:50 gibi is it?
15:09:55 lpetrut sean-k-mooney: nice, then I guess Nova could set those flags. I'm still not 100% if that should be the default or if it should be opt-in
15:09:56 stephenfin yes
15:10:03 stephenfin it's just documentation
15:10:13 gibi stephenfin: OK, thanks. it is clear now
15:10:52 dcapone2004 ipetrut: an enable_hyperv_support property maybe?
15:12:01 lpetrut dcapone2004: yep, that would probably be the way to go unless it becomes the default for Windows guests
15:13:42 lpetrut we could take another look at qemu, maybe even ask the qemu guys if they think there's any potential issue (e.g. affecting perf, stability or older guests) in using those flags by default
15:15:19 dcapone2004 ipetrut: if you discuss, all fo the othe rflags are already added, the only ones that were added in addition were these 4: hv-vpindex,hv-runtime,hv-synic,hv-reset I know I mentioned that earlier, but figured I would just bring that up again
15:15:26 lpetrut I assume most of those features are related to Hyper-V VMBus (virtio equivalent), so older guests might not support those enlightments
15:16:16 lpetrut dcapone2004: right, this list is very helpful
15:16:21 sean-k-mooney so what we bacially will want is an image proerty
15:16:27 sean-k-mooney that is a list of enum values
15:16:47 sean-k-mooney and the image could declare what it supports
15:16:58 sean-k-mooney we can add the current allways on enlighemtn as the defualt for the value
15:17:29 sean-k-mooney the other way is to use traits
15:17:43 lpetrut sean-k-mooney: sounds good. fwiw, new hyper-v enlightments might be added in the future so it's probably important to be able to pick the ones that the image supports
15:18:52 lpetrut while having some safe defaults
15:19:08 sean-k-mooney dcapone2004: so this is basically evolving form a potenial bug to a feature request
15:19:09 noonedeadpunk hi! Any idea why I might be seing this http://paste.openstack.org/show/800707/ when `backend` is set to `oslo_cache.memcache_pool` in [cache]?
15:19:30 sean-k-mooney is this something you would have capsity to work on or just soemthjing you would like to see added
15:20:28 sean-k-mooney noonedeadpunk:that proably more of an oslo question
15:20:32 lpetrut sean-k-mooney: I can't remember, does the libvirt nova driver allow you to set arbitrary cpu flags?
15:21:21 sean-k-mooney noonedeadpunk: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/etcd3gw.py#L17
15:21:30 sean-k-mooney lpetrut: cpu flags yes but these are not cpu flags
15:21:41 sean-k-mooney lpetrut: and only via the nova config
15:21:59 sean-k-mooney noonedeadpunk: it look like it does an unconditonal import
15:22:02 noonedeadpunk it feels it tries to load all modules possible... so nova just passes [cache] section just directly to oslo? (I'm not very common with codebase still - it's huge)
15:22:38 sean-k-mooney noonedeadpunk: these config option are actully adde by oslo.cache
15:23:54 noonedeadpunk I'm jsut not sure why it loads etcd3gw backend at all...
15:26:05 sean-k-mooney im guessing its usign the stevado extention manager to load it unconditonally
15:26:11 sean-k-mooney looking for that currently
15:29:16 sean-k-mooney noonedeadpunk: where is that being raised
15:30:05 noonedeadpunk found it here https://zuul.opendev.org/t/openstack/build/0af9364cc9094b5096899ed7693a9f60/log/logs/host/nova-conductor.service.journal-10-50-46.log.txt#2431
15:31:55 sean-k-mooney we are not using sevedor to load this in oslo cache or in nova
15:32:33 noonedeadpunk um.....
15:32:56 sean-k-mooney im wondering if this is coming form oslo.config
15:33:22 noonedeadpunk can hardly imagine path of the imports tbh
15:33:54 noonedeadpunk But I think it appeared only in V since in U haven't seen that message
15:33:57 sean-k-mooney we are using oslo cache
15:34:20 sean-k-mooney but i dont see it directly using sevedor in its code
15:34:48 noonedeadpunk ah, yes, might be oslo.config that using it, right
15:34:54 sean-k-mooney https://codesearch.opendev.org/?q=stevedore&i=nope&files=&excludeFiles=&repos=openstack/nova,openstack/oslo.cache,openstack/oslo.config,openstack/oslo.privsep
15:34:58 sean-k-mooney that is what im looking at
15:35:44 sean-k-mooney we use stevador for 3 things in nova, extra spec validator, schudler dirvers and metric monitors
15:36:10 sean-k-mooney oslo cache and oslo privsep dont use it at all it just in ther lower constraits
15:36:20 sean-k-mooney because they depend on oslo.config which does use it
15:37:41 sean-k-mooney it might be https://github.com/openstack/oslo.cache/commit/308f49aaaad554573aeff8ddc68c39ff7fe57438
15:37:50 noonedeadpunk seem like https://opendev.org/openstack/oslo.config/src/branch/master/oslo_config/cfg.py#L2152-L2154
15:37:53 noonedeadpunk yeah
15:38:31 sean-k-mooney ya
15:40:26 sean-k-mooney anyway this seams to be an oslo issue in one form or another
15:41:02 noonedeadpunk yeah, agreed - thanks for helping out - will go and chase oslo folks
15:57:13 gibi nova weekly meeting starts in 3 minutes on #openstack-meeting-3
15:57:29 gmann gibi: sean-k-mooney we need to put API patch changes at the end which bump the microversion and all code cleanup as first in the series. and rebase is so hard for microversion race i think we end up doing this in every cycle. brin is not online I will check with him on policy stuff help
15:57:57 gmann ypo. * rebase is *not* so hard for microversion
15:59:55 gibi gmann: most of the changes needed for the tenant_id bp is actually changes in the API code so we cannot frontload tha backend change as there is none
16:00:50 gmann gibi: yeah but i think code cleanup also there which i suspect a large chunk but need to grep and see.
16:01:31 gmann I think having all changes up and zuul +1 then we can start merging.
16:01:36 gibi gmann: I thin the majority of the change line vise is api sample changes and test code, and both need to have the microversion bump already in place
16:02:02 gibi gmann: yes, that would be ideal, but Brin noted that this will be a lot of patches

Earlier   Later