| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-03 | |||
| 13:08:26 | gibi | brinzhang_: I know it is a pain to update such big series with a new api microversion number so I'm sorry but I still think we need to see the big picture and do that those rebases as we progress | |
| 13:09:36 | brinzhang_ | gibi: I said before, it's nothing | |
| 13:10:31 | gibi | brinzhang_: OK, then let's build up a series of tenant_id patches during the cycle, you can ping me to keep them reviewed, but we will not start merging the first one before we see more | |
| 13:10:33 | brinzhang_ | gibi: hope you all can review the patches, make it closes to merge except the microversion changes, maybe it's the smallest changes for me | |
| 13:11:28 | brinzhang_ | gibi: yeah, we can review but not merge it, I can accept this decision. | |
| 13:11:48 | gibi | brinzhang_: thank you for your understanding | |
| 13:12:28 | brinzhang_ | gibi, sean-k-mooney: NP, thanks again | |
| 13:13:52 | gibi | brinzhang_: o/ | |
| 13:52:41 | dcapone2004 | sean-k-mooney: I tried what you had suggested last night, and I think I need to pull the VM out and attempt to see what happens on a standalone libvirt kvm system....for some reason after start the VM, it would instantly automatically shut itself down | |
| 13:53:39 | gibi | stephenfin: when we removed os-fping in Rocky, did we intentionally kept the api doc for it in tree? | |
| 13:53:56 | sean-k-mooney | dcapone2004: ok i guess there might be a kernel panic or something | |
| 13:54:15 | sean-k-mooney | dcapone2004:in the windows guest | |
| 13:54:36 | dcapone2004 | sean-k-mooney: yeah not sure because I never am able to get to a console fast enough....Nova logs just show that instance stopped itself | |
| 13:55:21 | dcapone2004 | but the hypervisor=off parameter does get passed when it happens | |
| 13:55:44 | sean-k-mooney | ok so it still fails with hypervior=off | |
| 13:56:21 | sean-k-mooney | dcapone2004: you coudl also set hyperv to manual start in the windows service | |
| 13:56:30 | sean-k-mooney | that way when it crashs on start | |
| 13:56:36 | sean-k-mooney | you can check the system events | |
| 13:56:43 | sean-k-mooney | on the next boot in the vm | |
| 13:56:51 | sean-k-mooney | since it wont auto start | |
| 13:57:45 | dcapone2004 | I might try that but unsure if that will work either because the whole system immediately becomes unresponsive, so I dont know if anything even ends up logged | |
| 13:58:16 | lpetrut | hey, are instances automatically shutting down? nova might be doing it if db state says that it should be shut off | |
| 13:59:15 | dcapone2004 | ipetrut: it is a specific test case where I am trying to test something by shutting down and restarting an instance directly in libvirt instead of using the openstack api | |
| 13:59:37 | lpetrut | got it, that's expected behavior but it can be disabled | |
| 13:59:49 | lpetrut | just a sec, fetching the cfg opt | |
| 14:00:44 | dcapone2004 | original DB power_state: 4, current VM power_state: 1 | |
| 14:00:47 | lpetrut | you may want to use this so that Nova doesn't automatically turn the instance back off: http://paste.openstack.org/raw/800701/ | |
| 14:00:55 | dcapone2004 | help me with enum for the power states, that is what is showing in my logs | |
| 14:03:01 | dcapone2004 | just asked kolla to implement those changes, I'll report back in about 20 | |
| 14:03:35 | lpetrut | this is what you're seeing in the logs https://github.com/openstack/nova/blob/22.0.1/nova/compute/manager.py#L9622-L9627 | |
| 14:04:44 | lpetrut | unless those options are set, Nova will change the DB instance state when the instance is shut down through external actions. when it comes back up, nova enforces the db state and stops it again | |
| 14:08:03 | dcapone2004 | yeah except the message was confusing to me because it said "shutdown by itself", when in fact I just started it was libvirt | |
| 14:08:08 | dcapone2004 | with* | |
| 14:08:36 | dcapone2004 | so, I was interpreting the log entry to mean it recognized the start and then libvirt shut it down for some reason and now nova was syncing it | |
| 14:09:41 | dcapone2004 | oh i think i get what your saying....nova sees when I shut it down with libvirt....processes that change in change and updates the DB and shows that it is shutdown (I can see this happen in the horizon interface where it shows as shutdown) | |
| 14:09:41 | sean-k-mooney | dcapone2004: that is liekly the power state sync | |
| 14:09:51 | sean-k-mooney | if its off in the nova db nova will shut it down | |
| 14:10:01 | dcapone2004 | then when I try to start it again, nova is saying, this is supposed to be shutdown and powers it back off | |
| 14:10:09 | sean-k-mooney | if we see the vm stop we update the db to be shutdown | |
| 14:10:50 | sean-k-mooney | dcapone2004: as lpetrut noted you can disable this behavior if you want/need too | |
| 14:11:52 | dcapone2004 | yep, I am temporarily doing that now, waiting for kolla to reconfigure the changes, basically only need this for this test of the hypervisor option to see if that fixes the lock up with hyper-v on KVM | |
| 14:15:18 | lpetrut | is hyper-v hanging or crashing and rebooting? | |
| 14:17:06 | lpetrut | if you can't get any logs or crash dump, you can even attach a debugger to the serial port | |
| 14:18:51 | dcapone2004 | so, if I enable hyper-v, the system hangs on the boot | |
| 14:19:10 | dcapone2004 | after a hard reboot, windows loads itself into the startup recovery tool | |
| 14:19:24 | sean-k-mooney | lpetrut: dcapone2004 is trying to get docker on windows working on windows 10 for context | |
| 14:19:29 | dcapone2004 | I found this fix here.... https://www.mess.org/2020/06/22/Hyper-V-in-KVM/#:~:text=Hyper%2DV%20on%20Windows%20requires,it%20is%... | |
| 14:19:46 | sean-k-mooney | so hyperv is need to run the linux continer in light weight vms | |
| 14:20:01 | dcapone2004 | so, I am trying to test what happens when the hypervisor=no parameter is passed | |
| 14:22:10 | dcapone2004 | I'm technically doing this using windows server 2019, but I found a different article that basically said the same thing for windows server 2019, but the article wasn't as short and sweet getting right to the config | |
| 14:24:06 | mgariepy | hello, is there someone who can help push this patch along for Ussuri please: https://review.opendev.org/c/openstack/nova/+/758761 | |
| 14:25:26 | lpetrut | dcapone2004 it might be related to the virtual cpu flags. it's been a while since I last ran Hyper-V on top of KVM, but I was using those qemu parameters: -cpu Skylake-Client,hv-relaxed,hv-vapic,hv-time,hv-crash,hv-reset,hv-vpindex,hv-runtime,hv-synic,hv_spinlocks=0x1fff,+vmx | |
| 14:27:55 | dcapone2004 | ipetrut: those flags I have...they are added by nova with the os_type=windows properties....sean helped guide me to try that last night | |
| 14:31:25 | lpetrut | dcapone2004: it may be worth double checking, I'm not sure if Nova sets all the flags required for running Hyper-V on top | |
| 14:32:48 | dcapone2004 | the one I feel is missing based on that article I referenced and linked is the hypervisor=off option | |
| 14:33:10 | dcapone2004 | with your config changes, I have been able to add that via virsh, so enabling hyper-v now and about to reboot | |
| 14:39:25 | dcapone2004 | -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=1234567890ab,kvm=off is all that is generated by default, so there are definitely quite a few missing | |
| 14:48:05 | dcapone2004 | time and crash do not validate with virsh, but trying again with the rest of the options, -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-reset,hv-vendor-id=1234567890ab,kvm=off | |
| 14:56:54 | dcapone2004 | bingo, those options appear to have resolved the booting problem.....now only if nova could add those additional options with some type of image/flavor property ;-) ;-) | |
| 14:58:04 | sean-k-mooney | dcapone2004: so which options are needed that are missing? | |
| 14:58:41 | sean-k-mooney | look like you enabled a bunch of the enlightnments | |
| 14:59:22 | sean-k-mooney | i dont see hypervior=off in the list | |
| 14:59:36 | sean-k-mooney | so it looks like we jsut ened to enable the enlightements | |
| 14:59:54 | dcapone2004 | nope, hypervisor=off wasn't needed with all of the other enlightments....I don't know if there was a specific magic individual one that did it | |
| 14:59:56 | sean-k-mooney | lpetrut: do you know of any reason not to do that for all windows guests by default | |
| 15:00:34 | sean-k-mooney | lpetrut: e.g. if your qemu and libvirt is new enough to support a hyperv enlightment we would jsut enable it by defualt | |
| 15:01:05 | sean-k-mooney | lpetrut: that shoudl improve performace ingereall even without the nested virt or docker usecase | |
| 15:01:26 | dcapone2004 | hv-vpindex,hv-runtime,hv-synic,hv-reset those were the 4 additional that virsh would accept and I added | |
| 15:02:33 | sean-k-mooney | hv-synic might be the issue | |
| 15:02:35 | stephenfin | gibi: You mean the docs? Yes, we did, because it's still available on older releases | |
| 15:02:41 | sean-k-mooney | Enable Synthetic Interrupt Controller (SynIC) | |
| 15:02:48 | stephenfin | and the API reference is unversioned | |
| 15:02:48 | gibi | stephenfin: I mean the api ref docs. | |
| 15:03:10 | gibi | stephenfin: so we don't have victoria api ref | |
| 15:03:13 | gibi | :/ | |
| 15:03:23 | stephenfin | We shouldn't have. There's just an API ref | |
| 15:03:44 | lpetrut | sean-k-mooney, dcapone2004: probably the qemu version was the main concern. then we'd also have to make sure that those flags don't impact the performance | |
| 15:03:45 | stephenfin | Because a request for e.g. 2.13 should look the exact same across all versions of nova that support that | |
| 15:04:33 | sean-k-mooney | lpetrut: our min qemu is proably new enought now | |
| 15:04:48 | gibi | stephenfin: but it is not true for fping as it is removed in rocky | |
| 15:05:12 | gibi | stephenfin: so fping request even if it is called with 2.13 is 404 since rocky | |
| 15:05:17 | sean-k-mooney | lpetrut: ok not quite | |
| 15:05:23 | sean-k-mooney | lpetrut: our min is 4.0.0 | |
| 15:05:26 | gibi | or at least this is how I understand the api ref | |
| 15:05:32 | sean-k-mooney | and min livbirt is 5.0.0 | |
| 15:05:58 | stephenfin | gibi: Yup, and for those cases we stick a note at the top saying "this has been removed in foo" and update the return codes to include HTTP 410 | |
| 15:06:02 | sean-k-mooney | lpetrut: we will be going to livbirt 6.0.0 and qemu 4.2.0 this cycle | |
| 15:06:11 | stephenfin | https://docs.openstack.org/api-ref/compute/#ping-instances-os-fping | |
| 15:06:16 | stephenfin | "This API only works with nova-network which is deprecated. It should be avoided in any new applications. These will fail with a 404 starting from microversion 2.36. It was removed in the 18.0.0 Rocky release." | |
| 15:06:32 | sean-k-mooney | lpetrut: dcapone2004 so at that point we coudl enable all of them | |
| 15:07:22 | lpetrut | I think all the above mentioned flags should be supported by qemu 4.0.0, not sure if libvirt needs to be aware of those as well | |
| 15:07:36 | sean-k-mooney | https://libvirt.org/formatdomain.html#hypervisor-features | |
| 15:07:44 | sean-k-mooney | the versions are listed there | |
| 15:08:48 | gibi | stephenfin: OK, then what to do with this https://review.opendev.org/c/openstack/nova/+/765311/1/api-ref/source/parameters.yaml#396 | |
| 15:09:24 | stephenfin | we should keep it but not update it | |
| 15:09:48 | gibi | stephenfin: keep it as we cannot update the already deleted code | |
| 15:09:50 | gibi | is it? | |
| 15:09:55 | lpetrut | sean-k-mooney: nice, then I guess Nova could set those flags. I'm still not 100% if that should be the default or if it should be opt-in | |
| 15:09:56 | stephenfin | yes | |
| 15:10:03 | stephenfin | it's just documentation | |