| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-03 | |||
| 12:59:37 | vinay_m | sean-k-mooney: os i have used was ubuntu 18.0.4 | |
| 12:59:39 | sean-k-mooney | oh vepc is vEPC | |
| 13:00:03 | vinay_m | <sean-k-mooney> oh vepc is vEPC -> yes :-) | |
| 13:00:25 | brinzhang_ | From my current job, I may not be able to complete all of them, because I have to update the Cyborg Shelve /unshelve support patch, or the noVNC patch with my partner, and other jobs from my boss | |
| 13:00:33 | sean-k-mooney | i tought it was an image creation tool like disk image builder | |
| 13:00:49 | gibi | brinzhang_: does the tenant_id work has high priority from your perspective? | |
| 13:01:27 | sean-k-mooney | vinay_m: my best guess is there is someing in the vECP applcaition that is breakign the guest netorking | |
| 13:01:32 | brinzhang_ | gibi: is there a deadline line? maybe I can do some changes before the deadline | |
| 13:01:49 | gibi | brinzhang_: M3 is feature freeze | |
| 13:01:58 | sean-k-mooney | vinay_m: im not really sure we can help much unless you have a specific nova/libvirt errorr | |
| 13:01:59 | gibi | that is the deadline to land these changes | |
| 13:02:01 | gibi | in W | |
| 13:02:13 | brinzhang_ | gibi: it's lower than my boss's point | |
| 13:02:18 | brinzhang_ | when? | |
| 13:02:35 | sean-k-mooney | brinzhang_: m3 in march/april | |
| 13:02:37 | gibi | Marc 12 2021 | |
| 13:02:58 | sean-k-mooney | https://releases.openstack.org/wallaby/schedule.html yep | |
| 13:03:11 | gibi | brinzhang_: from my limited perspective the cyborg patches has higher prio than the tenant_id work as the cyborg patches finishing something we already started | |
| 13:03:44 | brinzhang_ | so if I want to complete this feature, I need to submit all changes before Marc 1 2021, otherwise there is no time to review and update | |
| 13:03:46 | vinay_m | sean-k-mooney: yes , i have thought same but i have tried this complete process in another devstack setup deployed in a ubuntu VM with same image , it worked over there | |
| 13:04:12 | sean-k-mooney | brinzhang_: ideally you would aim to finsih before that but ya | |
| 13:04:33 | brinzhang_ | gibi: yeah, the cyborg support patch is ready and it's closes to merge, hope you all to review | |
| 13:05:04 | gibi | brinzhang_: I suggest to focus on cyborg and the stuff that is higher on you boss's list | |
| 13:05:13 | brinzhang_ | sean-k-mooney: I am not sure I can complete, but I can try | |
| 13:05:16 | gibi | brinzhang_: as for me tenant_id can wait | |
| 13:05:48 | vinay_m | problem i'm thinking was with setup deployed on a bare metal .. will it ? | |
| 13:05:51 | brinzhang_ | gibi: Their priorities depend on my decision | |
| 13:06:34 | sean-k-mooney | vinay_m: if cirros/plain ubuntu is workign this is pointing to an issue with the vEPC config not with openstack | |
| 13:06:58 | brinzhang_ | gibi: yeah, after the shelve/unshelve patch merged, we will start the cyborg resume/paused support work, otherwise we will wait | |
| 13:07:01 | sean-k-mooney | brinzhang_: if you dont think you can complete the tenant work this cycle then we shoudl wait to merge it | |
| 13:07:19 | vinay_m | <sean-k-mooney> vinay_m: if cirros/plain ubuntu is workign this is pointing to an issue with the vEPC config not with openstack -> hoo ok | |
| 13:08:26 | gibi | brinzhang_: I know it is a pain to update such big series with a new api microversion number so I'm sorry but I still think we need to see the big picture and do that those rebases as we progress | |
| 13:09:36 | brinzhang_ | gibi: I said before, it's nothing | |
| 13:10:31 | gibi | brinzhang_: OK, then let's build up a series of tenant_id patches during the cycle, you can ping me to keep them reviewed, but we will not start merging the first one before we see more | |
| 13:10:33 | brinzhang_ | gibi: hope you all can review the patches, make it closes to merge except the microversion changes, maybe it's the smallest changes for me | |
| 13:11:28 | brinzhang_ | gibi: yeah, we can review but not merge it, I can accept this decision. | |
| 13:11:48 | gibi | brinzhang_: thank you for your understanding | |
| 13:12:28 | brinzhang_ | gibi, sean-k-mooney: NP, thanks again | |
| 13:13:52 | gibi | brinzhang_: o/ | |
| 13:52:41 | dcapone2004 | sean-k-mooney: I tried what you had suggested last night, and I think I need to pull the VM out and attempt to see what happens on a standalone libvirt kvm system....for some reason after start the VM, it would instantly automatically shut itself down | |
| 13:53:39 | gibi | stephenfin: when we removed os-fping in Rocky, did we intentionally kept the api doc for it in tree? | |
| 13:53:56 | sean-k-mooney | dcapone2004: ok i guess there might be a kernel panic or something | |
| 13:54:15 | sean-k-mooney | dcapone2004:in the windows guest | |
| 13:54:36 | dcapone2004 | sean-k-mooney: yeah not sure because I never am able to get to a console fast enough....Nova logs just show that instance stopped itself | |
| 13:55:21 | dcapone2004 | but the hypervisor=off parameter does get passed when it happens | |
| 13:55:44 | sean-k-mooney | ok so it still fails with hypervior=off | |
| 13:56:21 | sean-k-mooney | dcapone2004: you coudl also set hyperv to manual start in the windows service | |
| 13:56:30 | sean-k-mooney | that way when it crashs on start | |
| 13:56:36 | sean-k-mooney | you can check the system events | |
| 13:56:43 | sean-k-mooney | on the next boot in the vm | |
| 13:56:51 | sean-k-mooney | since it wont auto start | |
| 13:57:45 | dcapone2004 | I might try that but unsure if that will work either because the whole system immediately becomes unresponsive, so I dont know if anything even ends up logged | |
| 13:58:16 | lpetrut | hey, are instances automatically shutting down? nova might be doing it if db state says that it should be shut off | |
| 13:59:15 | dcapone2004 | ipetrut: it is a specific test case where I am trying to test something by shutting down and restarting an instance directly in libvirt instead of using the openstack api | |
| 13:59:37 | lpetrut | got it, that's expected behavior but it can be disabled | |
| 13:59:49 | lpetrut | just a sec, fetching the cfg opt | |
| 14:00:44 | dcapone2004 | original DB power_state: 4, current VM power_state: 1 | |
| 14:00:47 | lpetrut | you may want to use this so that Nova doesn't automatically turn the instance back off: http://paste.openstack.org/raw/800701/ | |
| 14:00:55 | dcapone2004 | help me with enum for the power states, that is what is showing in my logs | |
| 14:03:01 | dcapone2004 | just asked kolla to implement those changes, I'll report back in about 20 | |
| 14:03:35 | lpetrut | this is what you're seeing in the logs https://github.com/openstack/nova/blob/22.0.1/nova/compute/manager.py#L9622-L9627 | |
| 14:04:44 | lpetrut | unless those options are set, Nova will change the DB instance state when the instance is shut down through external actions. when it comes back up, nova enforces the db state and stops it again | |
| 14:08:03 | dcapone2004 | yeah except the message was confusing to me because it said "shutdown by itself", when in fact I just started it was libvirt | |
| 14:08:08 | dcapone2004 | with* | |
| 14:08:36 | dcapone2004 | so, I was interpreting the log entry to mean it recognized the start and then libvirt shut it down for some reason and now nova was syncing it | |
| 14:09:41 | dcapone2004 | oh i think i get what your saying....nova sees when I shut it down with libvirt....processes that change in change and updates the DB and shows that it is shutdown (I can see this happen in the horizon interface where it shows as shutdown) | |
| 14:09:41 | sean-k-mooney | dcapone2004: that is liekly the power state sync | |
| 14:09:51 | sean-k-mooney | if its off in the nova db nova will shut it down | |
| 14:10:01 | dcapone2004 | then when I try to start it again, nova is saying, this is supposed to be shutdown and powers it back off | |
| 14:10:09 | sean-k-mooney | if we see the vm stop we update the db to be shutdown | |
| 14:10:50 | sean-k-mooney | dcapone2004: as lpetrut noted you can disable this behavior if you want/need too | |
| 14:11:52 | dcapone2004 | yep, I am temporarily doing that now, waiting for kolla to reconfigure the changes, basically only need this for this test of the hypervisor option to see if that fixes the lock up with hyper-v on KVM | |
| 14:15:18 | lpetrut | is hyper-v hanging or crashing and rebooting? | |
| 14:17:06 | lpetrut | if you can't get any logs or crash dump, you can even attach a debugger to the serial port | |
| 14:18:51 | dcapone2004 | so, if I enable hyper-v, the system hangs on the boot | |
| 14:19:10 | dcapone2004 | after a hard reboot, windows loads itself into the startup recovery tool | |
| 14:19:24 | sean-k-mooney | lpetrut: dcapone2004 is trying to get docker on windows working on windows 10 for context | |
| 14:19:29 | dcapone2004 | I found this fix here.... https://www.mess.org/2020/06/22/Hyper-V-in-KVM/#:~:text=Hyper%2DV%20on%20Windows%20requires,it%20is%... | |
| 14:19:46 | sean-k-mooney | so hyperv is need to run the linux continer in light weight vms | |
| 14:20:01 | dcapone2004 | so, I am trying to test what happens when the hypervisor=no parameter is passed | |
| 14:22:10 | dcapone2004 | I'm technically doing this using windows server 2019, but I found a different article that basically said the same thing for windows server 2019, but the article wasn't as short and sweet getting right to the config | |
| 14:24:06 | mgariepy | hello, is there someone who can help push this patch along for Ussuri please: https://review.opendev.org/c/openstack/nova/+/758761 | |
| 14:25:26 | lpetrut | dcapone2004 it might be related to the virtual cpu flags. it's been a while since I last ran Hyper-V on top of KVM, but I was using those qemu parameters: -cpu Skylake-Client,hv-relaxed,hv-vapic,hv-time,hv-crash,hv-reset,hv-vpindex,hv-runtime,hv-synic,hv_spinlocks=0x1fff,+vmx | |
| 14:27:55 | dcapone2004 | ipetrut: those flags I have...they are added by nova with the os_type=windows properties....sean helped guide me to try that last night | |
| 14:31:25 | lpetrut | dcapone2004: it may be worth double checking, I'm not sure if Nova sets all the flags required for running Hyper-V on top | |
| 14:32:48 | dcapone2004 | the one I feel is missing based on that article I referenced and linked is the hypervisor=off option | |
| 14:33:10 | dcapone2004 | with your config changes, I have been able to add that via virsh, so enabling hyper-v now and about to reboot | |
| 14:39:25 | dcapone2004 | -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=1234567890ab,kvm=off is all that is generated by default, so there are definitely quite a few missing | |
| 14:48:05 | dcapone2004 | time and crash do not validate with virsh, but trying again with the rest of the options, -cpu Haswell,pcid=on,vmx=on,rtm=on,hle=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-runtime,hv-synic,hv-reset,hv-vendor-id=1234567890ab,kvm=off | |
| 14:56:54 | dcapone2004 | bingo, those options appear to have resolved the booting problem.....now only if nova could add those additional options with some type of image/flavor property ;-) ;-) | |
| 14:58:04 | sean-k-mooney | dcapone2004: so which options are needed that are missing? | |
| 14:58:41 | sean-k-mooney | look like you enabled a bunch of the enlightnments | |
| 14:59:22 | sean-k-mooney | i dont see hypervior=off in the list | |
| 14:59:36 | sean-k-mooney | so it looks like we jsut ened to enable the enlightements | |
| 14:59:54 | dcapone2004 | nope, hypervisor=off wasn't needed with all of the other enlightments....I don't know if there was a specific magic individual one that did it | |
| 14:59:56 | sean-k-mooney | lpetrut: do you know of any reason not to do that for all windows guests by default | |
| 15:00:34 | sean-k-mooney | lpetrut: e.g. if your qemu and libvirt is new enough to support a hyperv enlightment we would jsut enable it by defualt | |
| 15:01:05 | sean-k-mooney | lpetrut: that shoudl improve performace ingereall even without the nested virt or docker usecase | |
| 15:01:26 | dcapone2004 | hv-vpindex,hv-runtime,hv-synic,hv-reset those were the 4 additional that virsh would accept and I added | |
| 15:02:33 | sean-k-mooney | hv-synic might be the issue | |