| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 21:59:35 | lucidguy | sean-k-mooney: Suggestions? | |
| 22:01:11 | sean-k-mooney | ubutun 16.04 is getting near the end of its life have you updated to the latest 16.04 kernel available on the host. | |
| 22:01:18 | sean-k-mooney | its possible this is already fixed if not | |
| 22:01:26 | sean-k-mooney | this is not an openstack related bug | |
| 22:02:00 | sean-k-mooney | so your best bet is to ralk to the ubuntu kernel folks or reach our to the kvm comunity on irc | |
| 22:02:53 | sean-k-mooney | personlly if you are stuck on 16.04 but can upgrade your kernel i would consider using there hardware enabling kernel which tends to be more up to date | |
| 22:05:39 | lucidguy | sean-k-mooney: We are already using the hwe kernel | |
| 22:07:17 | sean-k-mooney | actully just re reading this | |
| 22:07:28 | sean-k-mooney | if you launch the vm with 1TB of ram it start fine | |
| 22:07:43 | sean-k-mooney | and if you launch it with 1.2TB it fails with this error | |
| 22:08:50 | lucidguy | Thats correct | |
| 22:10:20 | sean-k-mooney | im just wondering if you are hitting a qemu or kvm memory limit | |
| 22:10:47 | lucidguy | sean-k-mooney: That's what I'm thinking but I can't find any documentation stating that | |
| 22:12:01 | jroll | efried: did you have a question for me or want my comments on how we might do that CI? | |
| 22:12:28 | jroll | (or neither?) | |
| 22:12:36 | sean-k-mooney | lucidguy: are you using nested virt by the way | |
| 22:13:01 | lucidguy | sean-k-mooney: How do I confirm that? | |
| 22:14:00 | sean-k-mooney | cat /sys/module/kvm_intel/parameters/nested | |
| 22:14:42 | lucidguy | Y | |
| 22:15:16 | sean-k-mooney | right so in 4.15 nested virt is disable by default in the kernel because tehre were still a few edgcaces that did not work correctly | |
| 22:15:54 | sean-k-mooney | by kernel 4.19 or so they had been fixed and the upstream kernel default it to Y | |
| 22:16:12 | lucidguy | Are you reading this somewhere? | |
| 22:16:39 | sean-k-mooney | https://bugs.launchpad.net/qemu/+bug/1813165 is a similar bug to yours but in there case they cpu was emulating System Management Mode | |
| 22:16:39 | openstack | Launchpad bug 1813165 in QEMU "KVM internal error. Suberror: 1 emulation failure" [Undecided,New] | |
| 22:16:55 | sean-k-mooney | that is not the casue in your trace back as you have SMM=0 | |
| 22:17:05 | sean-k-mooney | but they mentione this happend in a nested case | |
| 22:17:14 | sean-k-mooney | *also happend | |
| 22:17:40 | sean-k-mooney | lucidguy: and know i just have been working with nested virt for year and have seeing it improve and break over that time | |
| 22:18:40 | lucidguy | Since I'm using HWE, I believe I'm using the same kernal as Ubuntu 18.04. So people with the latest LTS are still having this issue? | |
| 22:18:59 | sean-k-mooney | ya nested virt in the defaul 18.04 kernel is broken | |
| 22:19:39 | lucidguy | To be honest, I don't exactly know what "nested" means. | |
| 22:19:41 | sean-k-mooney | so if that is what the 16.04 hwe kerel is trackin then that is likely the cause | |
| 22:20:07 | sean-k-mooney | lucidguy: it allows you to run kvm in side the vm to have 2+ layers of vms | |
| 22:20:48 | lucidguy | So should I not use nested? | |
| 22:21:02 | lucidguy | I don't think we have users runing vms within their vms | |
| 22:21:25 | sean-k-mooney | with that specific kernel proably not | |
| 22:21:40 | sean-k-mooney | but in general its on by default now but only from 4.19 on i think | |
| 22:21:43 | lucidguy | Hmm, now to find out where that is set. | |
| 22:21:49 | sean-k-mooney | sean@pop-os:~$ cat /etc/modprobe.d/qemu-system-x86.conf | |
| 22:21:51 | sean-k-mooney | options kvm_intel nested=1 | |
| 22:21:58 | sean-k-mooney | change 1 to 0 | |
| 22:22:44 | sean-k-mooney | you could also downgrae your kernel to the 16.04 non hwe kerenl | |
| 22:22:50 | lucidguy | I don't recall setting this anywhere, you think its on by default? | |
| 22:23:05 | lucidguy | We have hardware requiring that kernel | |
| 22:23:11 | sean-k-mooney | it should not be on by defualt on ubuntu 16.04 | |
| 22:23:27 | lucidguy | Hmm, set in nova somewhere? | |
| 22:23:32 | sean-k-mooney | no | |
| 22:23:38 | sean-k-mooney | nova does not set this | |
| 22:23:42 | lucidguy | odd | |
| 22:24:04 | sean-k-mooney | it can be set in two places | |
| 22:24:13 | sean-k-mooney | either in a file in /etc/modeprobe.d | |
| 22:24:18 | sean-k-mooney | or on the kernel command line | |
| 22:24:27 | lucidguy | sean-k-mooney: I have to leave soon, before I forget I just want to thank you for this info | |
| 22:24:55 | sean-k-mooney | no worries. nested virt would be my best guess but i dont know its defintly that | |
| 22:25:11 | sean-k-mooney | it is definetly a kenrel bug | |
| 22:25:14 | lucidguy | Still lots more to think about | |
| 22:26:51 | lucidguy | have to run. Thanks again. | |
| 22:26:58 | sean-k-mooney | no worries o/ | |
| 22:31:08 | sean-k-mooney | does anyone remember how to enable debug loggin in functinal tests? | |
| 22:43:04 | sean-k-mooney | nevermind i think i found the issue | |
| 22:43:11 | sean-k-mooney | yep | |
| 22:43:50 | sean-k-mooney | before train you should use NUMAHostInfo not HostInfo when you want it to constuct a numa toplogy form the kwargs | |
| 22:43:59 | sean-k-mooney | i even fixed that in one of the previous patches. | |
| 22:56:29 | lucidguy | back for a few minutes | |
| 22:56:49 | lucidguy | sean-k-mooney: You supporting a large OpenStack production environment? | |
| 22:57:44 | sean-k-mooney | not directly. i work at redhat on the comptue team so i mainly work upstream but i also support customer when they report bugs downstream | |
| 22:58:13 | sean-k-mooney | so we have some large cloud deployment that i support indirely but im on the enginerring team rather then support | |
| 22:58:30 | lucidguy | I'm supporting a 600+ hypervisor Queens deployment. | |
| 22:59:05 | lucidguy | I'm betting my boss is going to ask me to upgrade to Train. Lord help me. | |
| 22:59:43 | sean-k-mooney | what do you use for your installer. charms? or do you use something like kolla ansible or osa? | |
| 23:00:15 | sean-k-mooney | queens to train is a bit leap but its doable | |
| 23:00:50 | lucidguy | We install everything from scratch, using custom ansible playbooks | |
| 23:01:02 | sean-k-mooney | oh ok | |
| 23:01:10 | lucidguy | Are we crazy? | |
| 23:01:13 | sean-k-mooney | ya that makes it more challanging in some ways | |
| 23:01:23 | sean-k-mooney | that depends | |
| 23:01:28 | lucidguy | But this way you trully understand how things work. | |
| 23:01:40 | sean-k-mooney | yep that is the advantage | |
| 23:01:52 | sean-k-mooney | i would suggest looking at the ansibel roles form OSA | |
| 23:02:04 | sean-k-mooney | osa being openstack ansibel | |
| 23:02:11 | sean-k-mooney | to so see if they are of use too you | |
| 23:02:34 | lucidguy | OpenStack ansible popular ehh? | |
| 23:02:42 | sean-k-mooney | yes | |
| 23:03:01 | sean-k-mooney | it is the install that vexhost use | |
| 23:03:10 | lucidguy | vexhost? | |
| 23:03:33 | sean-k-mooney | they are a large public cloud operator that donate ci resouce to the openstck comunity | |
| 23:03:45 | lucidguy | interesting | |
| 23:04:00 | sean-k-mooney | lucidguy: you shoudl reach out to mnaser | |
| 23:04:23 | sean-k-mooney | i think he is still the PTL of OSA but they have done a lot to make upgrade simpler | |
| 23:04:59 | lucidguy | Doesn't canonical support OpenStack? | |
| 23:05:19 | sean-k-mooney | yes they do, they have an installer base on opesntack charms deployed with juju | |
| 23:05:58 | sean-k-mooney | so if you pay for support with them that is the install they woudl use | |
| 23:05:59 | lucidguy | interesting | |
| 23:06:21 | lucidguy | I would think large organizations would go with our approach. Avoid wrappers. | |
| 23:06:28 | sean-k-mooney | redhat has an openstack distobution that build on top of the triplo project | |
| 23:06:43 | sean-k-mooney | you would be surprised | |
| 23:07:03 | sean-k-mooney | some do if they have the devepolment staff to mainatin it | |
| 23:07:25 | sean-k-mooney | but its more common to see large companies contibuie and addopt the comuntiy installer | |
| 23:07:37 | sean-k-mooney | *installers | |
| 23:08:41 | lucidguy | Just sounds like a pain to have to troubleshoot another layer of software. | |