| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 22:16:39 | openstack | Launchpad bug 1813165 in QEMU "KVM internal error. Suberror: 1 emulation failure" [Undecided,New] | |
| 22:16:39 | sean-k-mooney | https://bugs.launchpad.net/qemu/+bug/1813165 is a similar bug to yours but in there case they cpu was emulating System Management Mode | |
| 22:16:55 | sean-k-mooney | that is not the casue in your trace back as you have SMM=0 | |
| 22:17:05 | sean-k-mooney | but they mentione this happend in a nested case | |
| 22:17:14 | sean-k-mooney | *also happend | |
| 22:17:40 | sean-k-mooney | lucidguy: and know i just have been working with nested virt for year and have seeing it improve and break over that time | |
| 22:18:40 | lucidguy | Since I'm using HWE, I believe I'm using the same kernal as Ubuntu 18.04. So people with the latest LTS are still having this issue? | |
| 22:18:59 | sean-k-mooney | ya nested virt in the defaul 18.04 kernel is broken | |
| 22:19:39 | lucidguy | To be honest, I don't exactly know what "nested" means. | |
| 22:19:41 | sean-k-mooney | so if that is what the 16.04 hwe kerel is trackin then that is likely the cause | |
| 22:20:07 | sean-k-mooney | lucidguy: it allows you to run kvm in side the vm to have 2+ layers of vms | |
| 22:20:48 | lucidguy | So should I not use nested? | |
| 22:21:02 | lucidguy | I don't think we have users runing vms within their vms | |
| 22:21:25 | sean-k-mooney | with that specific kernel proably not | |
| 22:21:40 | sean-k-mooney | but in general its on by default now but only from 4.19 on i think | |
| 22:21:43 | lucidguy | Hmm, now to find out where that is set. | |
| 22:21:49 | sean-k-mooney | sean@pop-os:~$ cat /etc/modprobe.d/qemu-system-x86.conf | |
| 22:21:51 | sean-k-mooney | options kvm_intel nested=1 | |
| 22:21:58 | sean-k-mooney | change 1 to 0 | |
| 22:22:44 | sean-k-mooney | you could also downgrae your kernel to the 16.04 non hwe kerenl | |
| 22:22:50 | lucidguy | I don't recall setting this anywhere, you think its on by default? | |
| 22:23:05 | lucidguy | We have hardware requiring that kernel | |
| 22:23:11 | sean-k-mooney | it should not be on by defualt on ubuntu 16.04 | |
| 22:23:27 | lucidguy | Hmm, set in nova somewhere? | |
| 22:23:32 | sean-k-mooney | no | |
| 22:23:38 | sean-k-mooney | nova does not set this | |
| 22:23:42 | lucidguy | odd | |
| 22:24:04 | sean-k-mooney | it can be set in two places | |
| 22:24:13 | sean-k-mooney | either in a file in /etc/modeprobe.d | |
| 22:24:18 | sean-k-mooney | or on the kernel command line | |
| 22:24:27 | lucidguy | sean-k-mooney: I have to leave soon, before I forget I just want to thank you for this info | |
| 22:24:55 | sean-k-mooney | no worries. nested virt would be my best guess but i dont know its defintly that | |
| 22:25:11 | sean-k-mooney | it is definetly a kenrel bug | |
| 22:25:14 | lucidguy | Still lots more to think about | |
| 22:26:51 | lucidguy | have to run. Thanks again. | |
| 22:26:58 | sean-k-mooney | no worries o/ | |
| 22:31:08 | sean-k-mooney | does anyone remember how to enable debug loggin in functinal tests? | |
| 22:43:04 | sean-k-mooney | nevermind i think i found the issue | |
| 22:43:11 | sean-k-mooney | yep | |
| 22:43:50 | sean-k-mooney | before train you should use NUMAHostInfo not HostInfo when you want it to constuct a numa toplogy form the kwargs | |
| 22:43:59 | sean-k-mooney | i even fixed that in one of the previous patches. | |
| 22:56:29 | lucidguy | back for a few minutes | |
| 22:56:49 | lucidguy | sean-k-mooney: You supporting a large OpenStack production environment? | |
| 22:57:44 | sean-k-mooney | not directly. i work at redhat on the comptue team so i mainly work upstream but i also support customer when they report bugs downstream | |
| 22:58:13 | sean-k-mooney | so we have some large cloud deployment that i support indirely but im on the enginerring team rather then support | |
| 22:58:30 | lucidguy | I'm supporting a 600+ hypervisor Queens deployment. | |
| 22:59:05 | lucidguy | I'm betting my boss is going to ask me to upgrade to Train. Lord help me. | |
| 22:59:43 | sean-k-mooney | what do you use for your installer. charms? or do you use something like kolla ansible or osa? | |
| 23:00:15 | sean-k-mooney | queens to train is a bit leap but its doable | |
| 23:00:50 | lucidguy | We install everything from scratch, using custom ansible playbooks | |
| 23:01:02 | sean-k-mooney | oh ok | |
| 23:01:10 | lucidguy | Are we crazy? | |
| 23:01:13 | sean-k-mooney | ya that makes it more challanging in some ways | |
| 23:01:23 | sean-k-mooney | that depends | |
| 23:01:28 | lucidguy | But this way you trully understand how things work. | |
| 23:01:40 | sean-k-mooney | yep that is the advantage | |
| 23:01:52 | sean-k-mooney | i would suggest looking at the ansibel roles form OSA | |
| 23:02:04 | sean-k-mooney | osa being openstack ansibel | |
| 23:02:11 | sean-k-mooney | to so see if they are of use too you | |
| 23:02:34 | lucidguy | OpenStack ansible popular ehh? | |
| 23:02:42 | sean-k-mooney | yes | |
| 23:03:01 | sean-k-mooney | it is the install that vexhost use | |
| 23:03:10 | lucidguy | vexhost? | |
| 23:03:33 | sean-k-mooney | they are a large public cloud operator that donate ci resouce to the openstck comunity | |
| 23:03:45 | lucidguy | interesting | |
| 23:04:00 | sean-k-mooney | lucidguy: you shoudl reach out to mnaser | |
| 23:04:23 | sean-k-mooney | i think he is still the PTL of OSA but they have done a lot to make upgrade simpler | |
| 23:04:59 | lucidguy | Doesn't canonical support OpenStack? | |
| 23:05:19 | sean-k-mooney | yes they do, they have an installer base on opesntack charms deployed with juju | |
| 23:05:58 | sean-k-mooney | so if you pay for support with them that is the install they woudl use | |
| 23:05:59 | lucidguy | interesting | |
| 23:06:21 | lucidguy | I would think large organizations would go with our approach. Avoid wrappers. | |
| 23:06:28 | sean-k-mooney | redhat has an openstack distobution that build on top of the triplo project | |
| 23:06:43 | sean-k-mooney | you would be surprised | |
| 23:07:03 | sean-k-mooney | some do if they have the devepolment staff to mainatin it | |
| 23:07:25 | sean-k-mooney | but its more common to see large companies contibuie and addopt the comuntiy installer | |
| 23:07:37 | sean-k-mooney | *installers | |
| 23:08:41 | lucidguy | Just sounds like a pain to have to troubleshoot another layer of software. | |
| 23:10:28 | sean-k-mooney | well when that software is built by a group of operators that have really clouds in produciton you would be suprised at how helpful it can be | |
| 23:10:53 | sean-k-mooney | that said i agree if its a complex installer it can be a pain to debug | |
| 23:11:24 | sean-k-mooney | so i tend to prefer the simpler ones like devstack and kolla-ansible | |
| 23:11:37 | sean-k-mooney | just never run devstack in production... | |
| 23:14:59 | lucidguy | ugg stupid instance stuck in deleting state.. and virsh on hv not responding FML | |
| 23:15:44 | sean-k-mooney | that sounds like the host kernel might have hung | |
| 23:16:18 | lucidguy | this is the box I disabled nesting | |
| 23:16:19 | sean-k-mooney | by the way i hope if you are runing 1TB VM you are using Hugepages for the memory | |
| 23:16:36 | lucidguy | Whatever the default option is. | |
| 23:16:44 | sean-k-mooney | so no you are not | |
| 23:17:12 | sean-k-mooney | the reason i mentioned that is it can take a long time for qemu/kvm to release all the mapped memory when you kill it | |
| 23:17:16 | lucidguy | I was just reading those options in nova.conf | |
| 23:17:40 | sean-k-mooney | so if you are using the decault 4k memory then it might take a while to fully delete the vm | |
| 23:18:19 | lucidguy | Would I not see a load on the system anywhere? | |
| 23:19:37 | lucidguy | I don't even see a qemu process running | |
| 23:21:05 | sean-k-mooney | you would see that the memory has not been freed with free -m | |
| 23:21:24 | sean-k-mooney | but in that case if virsh is not working i would check to see if libvirt is still running | |
| 23:21:31 | lucidguy | Its free | |
| 23:21:50 | lucidguy | libvirtError: failed to connect to monitor socket: No such process | |
| 23:22:16 | sean-k-mooney | so that mean libvirt tried to connecct to qemu but it failed because it was not running | |
| 23:22:23 | lucidguy | oooo its gone. So it did take forever. | |
| 23:22:44 | lucidguy | Now you're going to make me research how Hugepages works. Thanks alot. heh | |