| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-03 | |||
| 14:03:01 | dcapone2004 | just asked kolla to implement those changes, I'll report back in about 20 | |
| 14:03:35 | lpetrut | this is what you're seeing in the logs https://github.com/openstack/nova/blob/22.0.1/nova/compute/manager.py#L9622-L9627 | |
| 14:04:44 | lpetrut | unless those options are set, Nova will change the DB instance state when the instance is shut down through external actions. when it comes back up, nova enforces the db state and stops it again | |
| 14:08:03 | dcapone2004 | yeah except the message was confusing to me because it said "shutdown by itself", when in fact I just started it was libvirt | |
| 14:08:08 | dcapone2004 | with* | |
| 14:08:36 | dcapone2004 | so, I was interpreting the log entry to mean it recognized the start and then libvirt shut it down for some reason and now nova was syncing it | |
| 14:09:41 | sean-k-mooney | dcapone2004: that is liekly the power state sync | |
| 14:09:41 | dcapone2004 | oh i think i get what your saying....nova sees when I shut it down with libvirt....processes that change in change and updates the DB and shows that it is shutdown (I can see this happen in the horizon interface where it shows as shutdown) | |
| 14:09:51 | sean-k-mooney | if its off in the nova db nova will shut it down | |
| 14:10:01 | dcapone2004 | then when I try to start it again, nova is saying, this is supposed to be shutdown and powers it back off | |
| 14:10:09 | sean-k-mooney | if we see the vm stop we update the db to be shutdown | |
| 14:10:50 | sean-k-mooney | dcapone2004: as lpetrut noted you can disable this behavior if you want/need too | |
| 14:11:52 | dcapone2004 | yep, I am temporarily doing that now, waiting for kolla to reconfigure the changes, basically only need this for this test of the hypervisor option to see if that fixes the lock up with hyper-v on KVM | |
| 14:15:18 | lpetrut | is hyper-v hanging or crashing and rebooting? | |
| 14:17:06 | lpetrut | if you can't get any logs or crash dump, you can even attach a debugger to the serial port | |
| 14:18:51 | dcapone2004 | so, if I enable hyper-v, the system hangs on the boot | |
| 14:19:10 | dcapone2004 | after a hard reboot, windows loads itself into the startup recovery tool | |
| 14:19:24 | sean-k-mooney | lpetrut: dcapone2004 is trying to get docker on windows working on windows 10 for context | |
| 14:19:29 | dcapone2004 | I found this fix here.... https://www.mess.org/2020/06/22/Hyper-V-in-KVM/#:~:text=Hyper%2DV%20on%20Windows%20requires,it%20is%... | |
| 14:19:46 | sean-k-mooney | so hyperv is need to run the linux continer in light weight vms | |
| 14:20:01 | dcapone2004 | so, I am trying to test what happens when the hypervisor=no parameter is passed | |
| 14:22:10 | dcapone2004 | I'm technically doing this using windows server 2019, but I found a different article that basically said the same thing for windows server 2019, but the article wasn't as short and sweet getting right to the config | |
| 14:24:06 | mgariepy | hello, is there someone who can help push this patch along for Ussuri please: https://review.opendev.org/c/openstack/nova/+/758761 | |
| 14:25:26 | lpetrut | dcapone2004 it might be related to the virtual cpu flags. it's been a while since I last ran Hyper-V on top of KVM, but I was using those qemu parameters: -cpu Skylake-Client,hv-relaxed,hv-vapic,hv-time,hv-crash,hv-reset,hv-vpindex,hv-runtime,hv-synic,hv_spinlocks=0x1fff,+vmx | |
| 14:27:55 | dcapone2004 | ipetrut: those flags I have...they are added by nova with the os_type=windows properties....sean helped guide me to try that last night | |
| 14:31:25 | lpetrut | dcapone2004: it may be worth double checking, I'm not sure if Nova sets all the flags required for running Hyper-V on top | |
| 14:32:48 | dcapone2004 | the one I feel is missing based on that article I referenced and linked is the hypervisor=off option | |
| 14:33:10 | dcapone2004 | with your config changes, I have been able to add that via virsh, so enabling hyper-v now and about to reboot | |
| 14:39:25 | dcapone2004 | -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=1234567890ab,kvm=off is all that is generated by default, so there are definitely quite a few missing | |
| 14:48:05 | dcapone2004 | time and crash do not validate with virsh, but trying again with the rest of the options, -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-reset,hv-vendor-id=1234567890ab,kvm=off | |
| 14:56:54 | dcapone2004 | bingo, those options appear to have resolved the booting problem.....now only if nova could add those additional options with some type of image/flavor property ;-) ;-) | |
| 14:58:04 | sean-k-mooney | dcapone2004: so which options are needed that are missing? | |
| 14:58:41 | sean-k-mooney | look like you enabled a bunch of the enlightnments | |
| 14:59:22 | sean-k-mooney | i dont see hypervior=off in the list | |
| 14:59:36 | sean-k-mooney | so it looks like we jsut ened to enable the enlightements | |
| 14:59:54 | dcapone2004 | nope, hypervisor=off wasn't needed with all of the other enlightments....I don't know if there was a specific magic individual one that did it | |
| 14:59:56 | sean-k-mooney | lpetrut: do you know of any reason not to do that for all windows guests by default | |
| 15:00:34 | sean-k-mooney | lpetrut: e.g. if your qemu and libvirt is new enough to support a hyperv enlightment we would jsut enable it by defualt | |
| 15:01:05 | sean-k-mooney | lpetrut: that shoudl improve performace ingereall even without the nested virt or docker usecase | |
| 15:01:26 | dcapone2004 | hv-vpindex,hv-runtime,hv-synic,hv-reset those were the 4 additional that virsh would accept and I added | |
| 15:02:33 | sean-k-mooney | hv-synic might be the issue | |
| 15:02:35 | stephenfin | gibi: You mean the docs? Yes, we did, because it's still available on older releases | |
| 15:02:41 | sean-k-mooney | Enable Synthetic Interrupt Controller (SynIC) | |
| 15:02:48 | gibi | stephenfin: I mean the api ref docs. | |
| 15:02:48 | stephenfin | and the API reference is unversioned | |
| 15:03:10 | gibi | stephenfin: so we don't have victoria api ref | |
| 15:03:13 | gibi | :/ | |
| 15:03:23 | stephenfin | We shouldn't have. There's just an API ref | |
| 15:03:44 | lpetrut | sean-k-mooney, dcapone2004: probably the qemu version was the main concern. then we'd also have to make sure that those flags don't impact the performance | |
| 15:03:45 | stephenfin | Because a request for e.g. 2.13 should look the exact same across all versions of nova that support that | |
| 15:04:33 | sean-k-mooney | lpetrut: our min qemu is proably new enought now | |
| 15:04:48 | gibi | stephenfin: but it is not true for fping as it is removed in rocky | |
| 15:05:12 | gibi | stephenfin: so fping request even if it is called with 2.13 is 404 since rocky | |
| 15:05:17 | sean-k-mooney | lpetrut: ok not quite | |
| 15:05:23 | sean-k-mooney | lpetrut: our min is 4.0.0 | |
| 15:05:26 | gibi | or at least this is how I understand the api ref | |
| 15:05:32 | sean-k-mooney | and min livbirt is 5.0.0 | |
| 15:05:58 | stephenfin | gibi: Yup, and for those cases we stick a note at the top saying "this has been removed in foo" and update the return codes to include HTTP 410 | |
| 15:06:02 | sean-k-mooney | lpetrut: we will be going to livbirt 6.0.0 and qemu 4.2.0 this cycle | |
| 15:06:11 | stephenfin | https://docs.openstack.org/api-ref/compute/#ping-instances-os-fping | |
| 15:06:16 | stephenfin | "This API only works with nova-network which is deprecated. It should be avoided in any new applications. These will fail with a 404 starting from microversion 2.36. It was removed in the 18.0.0 Rocky release." | |
| 15:06:32 | sean-k-mooney | lpetrut: dcapone2004 so at that point we coudl enable all of them | |
| 15:07:22 | lpetrut | I think all the above mentioned flags should be supported by qemu 4.0.0, not sure if libvirt needs to be aware of those as well | |
| 15:07:36 | sean-k-mooney | https://libvirt.org/formatdomain.html#hypervisor-features | |
| 15:07:44 | sean-k-mooney | the versions are listed there | |
| 15:08:48 | gibi | stephenfin: OK, then what to do with this https://review.opendev.org/c/openstack/nova/+/765311/1/api-ref/source/parameters.yaml#396 | |
| 15:09:24 | stephenfin | we should keep it but not update it | |
| 15:09:48 | gibi | stephenfin: keep it as we cannot update the already deleted code | |
| 15:09:50 | gibi | is it? | |
| 15:09:55 | lpetrut | sean-k-mooney: nice, then I guess Nova could set those flags. I'm still not 100% if that should be the default or if it should be opt-in | |
| 15:09:56 | stephenfin | yes | |
| 15:10:03 | stephenfin | it's just documentation | |
| 15:10:13 | gibi | stephenfin: OK, thanks. it is clear now | |
| 15:10:52 | dcapone2004 | ipetrut: an enable_hyperv_support property maybe? | |
| 15:12:01 | lpetrut | dcapone2004: yep, that would probably be the way to go unless it becomes the default for Windows guests | |
| 15:13:42 | lpetrut | we could take another look at qemu, maybe even ask the qemu guys if they think there's any potential issue (e.g. affecting perf, stability or older guests) in using those flags by default | |
| 15:15:19 | dcapone2004 | ipetrut: if you discuss, all fo the othe rflags are already added, the only ones that were added in addition were these 4: hv-vpindex,hv-runtime,hv-synic,hv-reset I know I mentioned that earlier, but figured I would just bring that up again | |
| 15:15:26 | lpetrut | I assume most of those features are related to Hyper-V VMBus (virtio equivalent), so older guests might not support those enlightments | |
| 15:16:16 | lpetrut | dcapone2004: right, this list is very helpful | |
| 15:16:21 | sean-k-mooney | so what we bacially will want is an image proerty | |
| 15:16:27 | sean-k-mooney | that is a list of enum values | |
| 15:16:47 | sean-k-mooney | and the image could declare what it supports | |
| 15:16:58 | sean-k-mooney | we can add the current allways on enlighemtn as the defualt for the value | |
| 15:17:29 | sean-k-mooney | the other way is to use traits | |
| 15:17:43 | lpetrut | sean-k-mooney: sounds good. fwiw, new hyper-v enlightments might be added in the future so it's probably important to be able to pick the ones that the image supports | |
| 15:18:52 | lpetrut | while having some safe defaults | |
| 15:19:08 | sean-k-mooney | dcapone2004: so this is basically evolving form a potenial bug to a feature request | |
| 15:19:09 | noonedeadpunk | hi! Any idea why I might be seing this http://paste.openstack.org/show/800707/ when `backend` is set to `oslo_cache.memcache_pool` in [cache]? | |
| 15:19:30 | sean-k-mooney | is this something you would have capsity to work on or just soemthjing you would like to see added | |
| 15:20:28 | sean-k-mooney | noonedeadpunk:that proably more of an oslo question | |
| 15:20:32 | lpetrut | sean-k-mooney: I can't remember, does the libvirt nova driver allow you to set arbitrary cpu flags? | |
| 15:21:21 | sean-k-mooney | noonedeadpunk: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/backends/etcd3gw.py#L17 | |
| 15:21:30 | sean-k-mooney | lpetrut: cpu flags yes but these are not cpu flags | |
| 15:21:41 | sean-k-mooney | lpetrut: and only via the nova config | |
| 15:21:59 | sean-k-mooney | noonedeadpunk: it look like it does an unconditonal import | |
| 15:22:02 | noonedeadpunk | it feels it tries to load all modules possible... so nova just passes [cache] section just directly to oslo? (I'm not very common with codebase still - it's huge) | |
| 15:22:38 | sean-k-mooney | noonedeadpunk: these config option are actully adde by oslo.cache | |
| 15:23:54 | noonedeadpunk | I'm jsut not sure why it loads etcd3gw backend at all... | |
| 15:26:05 | sean-k-mooney | im guessing its usign the stevado extention manager to load it unconditonally | |
| 15:26:11 | sean-k-mooney | looking for that currently | |