| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-01-22 | |||
| 19:17:38 | melwitt | ok, now to find where this thing is failing LM | |
| 19:18:41 | sean-k-mooney | i have been doing " curl <log url> | zcat | lnav -q" | |
| 19:19:45 | melwitt | that's helpful, thanks | |
| 19:21:56 | sean-k-mooney | melwitt: artom was say you can also to "curl <url> | zless" but i like using lnav to browse logs | |
| 19:23:13 | melwitt | I still haven't gotten around to trying lnav so I'll use zless for now | |
| 19:23:47 | sean-k-mooney | ya if you do try it by default lnav dumps all the logs to std out so the -q is there to stop that | |
| 19:24:16 | melwitt | \:| [instance: 5bca844b-4ca6-4c63-a579-8f00316da019] Instance spawn was interrupted before instance_claim, setting instance to ERROR state | |
| 19:24:51 | sean-k-mooney | am that is new | |
| 19:24:59 | melwitt | yeah, it's new to me | |
| 19:25:18 | sean-k-mooney | does that mean qemu crashed or something? | |
| 19:25:25 | melwitt | no idea | |
| 19:25:51 | sean-k-mooney | well that makes two of us | |
| 19:26:34 | melwitt | I don't expect it's related to the patch, and the patch has been sitting around forever so I wouldn't be surprised if it needs a rebase by now. but the nova-live-migration job is failing pretty consistently on it and so far, for unknown reasons | |
| 19:26:58 | melwitt | rechecked it like 3 times and always fail on nova-live-migration | |
| 19:27:59 | sean-k-mooney | i can take a look at it a bit later. what is the review? | |
| 19:28:27 | sean-k-mooney | once i fix the osp 15 backport im doing ill take a quick look before moveing to osp 13 | |
| 19:28:30 | melwitt | sean-k-mooney: it's this one https://review.opendev.org/683008 | |
| 19:28:56 | sean-k-mooney | cool i have it open. | |
| 19:28:58 | melwitt | note that it is very old, not rebased since september | |
| 19:29:00 | artom | sean-k-mooney, huh, that's the same thing we saw in whitebox | |
| 19:29:25 | artom | We thought it was because we were restarting nova-compute to change configs | |
| 19:29:31 | sean-k-mooney | melwitt: if you do a rebase via the ui i should not hurt | |
| 19:29:39 | sean-k-mooney | but i dont think that is the issue | |
| 19:29:53 | sean-k-mooney | zuul will do a merge with master so it should basicaly be the same | |
| 19:29:56 | melwitt | yeah, probably worth it to try. it's gonna have to be rebased to merge anyway | |
| 19:30:06 | melwitt | let me do it now | |
| 19:30:39 | sean-k-mooney | well its more or less the same as a recheck and will avoid the merge commit so it wont hurt | |
| 19:30:49 | sean-k-mooney | artom: did you fix it in whitebox? | |
| 19:31:07 | artom | sean-k-mooney, by extending the service restart wait delay | |
| 19:31:21 | artom | Which apparently has nothing to do with it, as we're now seeing the same thing in Nova? | |
| 19:31:27 | sean-k-mooney | we are not restart services in this job | |
| 19:31:31 | sean-k-mooney | actully | |
| 19:31:35 | sean-k-mooney | we might be | |
| 19:31:39 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: Avoid redundant initialize_connection on source post live migration https://review.opendev.org/683008 | |
| 19:31:44 | artom | Wait, n-l-m? | |
| 19:31:46 | sean-k-mooney | i think we do | |
| 19:31:52 | sean-k-mooney | ya | |
| 19:32:03 | artom | We're changing some configs IIRC | |
| 19:32:04 | sean-k-mooney | i think there is a hack that swaps the storage to ceph or something | |
| 19:32:05 | artom | So yeah, we are | |
| 19:32:41 | sean-k-mooney | i think its in the post run playbook | |
| 19:33:07 | sean-k-mooney | ok its not there | |
| 19:33:15 | sean-k-mooney | its proably in the test hook | |
| 19:34:56 | sean-k-mooney | ya so here https://github.com/openstack/nova/blob/master/gate/live_migration/hooks/run_tests.sh#L55-L67 | |
| 19:35:14 | artom | Ah, yeah, configure_and_start_nova | |
| 19:35:53 | artom | Are... are we just going to hax a sleep in there? Before the run_tempest call? | |
| 19:35:54 | sean-k-mooney | https://github.com/openstack/nova/blob/master/gate/live_migration/hooks/ceph.sh#L71-L100 | |
| 19:36:33 | sean-k-mooney | we could hack a sleep here https://github.com/openstack/nova/blob/master/gate/live_migration/hooks/ceph.sh#L114 | |
| 19:37:26 | artom | Why in ceph? | |
| 19:37:48 | sean-k-mooney | this is where we restart nova after reconfiguring to use ceph | |
| 19:38:36 | sean-k-mooney | so in the ligration job we deploy withour ceph intially run tempest then after that we reconfivure the deploymetn for ceph and do more testing | |
| 19:40:11 | sean-k-mooney | so we only restart the serivce in the ceph teting at the end here https://github.com/openstack/nova/blob/master/gate/live_migration/hooks/run_tests.sh#L64 | |
| 19:40:43 | sean-k-mooney | that function get loaded form the ceph.sh file here https://github.com/openstack/nova/blob/master/gate/live_migration/hooks/run_tests.sh#L21 | |
| 19:53:26 | stephenfin | sean-k-mooney: Seeing as you're still here, does my comment here make sense https://review.opendev.org/#/c/662522/13/nova/compute/api.py@3637 | |
| 19:53:40 | stephenfin | checking it tomorrow is fine too. I am outta here | |
| 19:56:18 | sean-k-mooney | ill check | |
| 19:58:06 | sean-k-mooney | am the conducor calls the schduer | |
| 19:58:30 | sean-k-mooney | so it could modify the numa toplogy before it calls the select destination function i think | |
| 19:59:51 | sean-k-mooney | i would have to check but you are right in that you definetly need to recalulate the numa toployg based on the new flavor before the numa toplogy filter runs | |
| 20:00:20 | sean-k-mooney | meaining either in the api or in the conductor if the conductor si what is invoking the scheduler. | |
| 20:01:27 | sean-k-mooney | i dont have that part of code loaded up in my brain right now hence vague answer | |
| 21:54:31 | lucidguy | I'll be very very appreciative if someone can assist me with this, i've spent hours googling etc. https://paste.ubuntu.com/p/6wqhRX68tr/ | |
| 21:56:01 | sean-k-mooney | you are booting vms with 1TB of ram | |
| 21:56:30 | sean-k-mooney | that is fun | |
| 21:57:12 | sean-k-mooney | this looks like a kvm kernel bug | |
| 21:59:35 | lucidguy | sean-k-mooney: Suggestions? | |
| 22:01:11 | sean-k-mooney | ubutun 16.04 is getting near the end of its life have you updated to the latest 16.04 kernel available on the host. | |
| 22:01:18 | sean-k-mooney | its possible this is already fixed if not | |
| 22:01:26 | sean-k-mooney | this is not an openstack related bug | |
| 22:02:00 | sean-k-mooney | so your best bet is to ralk to the ubuntu kernel folks or reach our to the kvm comunity on irc | |
| 22:02:53 | sean-k-mooney | personlly if you are stuck on 16.04 but can upgrade your kernel i would consider using there hardware enabling kernel which tends to be more up to date | |
| 22:05:39 | lucidguy | sean-k-mooney: We are already using the hwe kernel | |
| 22:07:17 | sean-k-mooney | actully just re reading this | |
| 22:07:28 | sean-k-mooney | if you launch the vm with 1TB of ram it start fine | |
| 22:07:43 | sean-k-mooney | and if you launch it with 1.2TB it fails with this error | |
| 22:08:50 | lucidguy | Thats correct | |
| 22:10:20 | sean-k-mooney | im just wondering if you are hitting a qemu or kvm memory limit | |
| 22:10:47 | lucidguy | sean-k-mooney: That's what I'm thinking but I can't find any documentation stating that | |
| 22:12:01 | jroll | efried: did you have a question for me or want my comments on how we might do that CI? | |
| 22:12:28 | jroll | (or neither?) | |
| 22:12:36 | sean-k-mooney | lucidguy: are you using nested virt by the way | |
| 22:13:01 | lucidguy | sean-k-mooney: How do I confirm that? | |
| 22:14:00 | sean-k-mooney | cat /sys/module/kvm_intel/parameters/nested | |
| 22:14:42 | lucidguy | Y | |
| 22:15:16 | sean-k-mooney | right so in 4.15 nested virt is disable by default in the kernel because tehre were still a few edgcaces that did not work correctly | |
| 22:15:54 | sean-k-mooney | by kernel 4.19 or so they had been fixed and the upstream kernel default it to Y | |
| 22:16:12 | lucidguy | Are you reading this somewhere? | |
| 22:16:39 | sean-k-mooney | https://bugs.launchpad.net/qemu/+bug/1813165 is a similar bug to yours but in there case they cpu was emulating System Management Mode | |
| 22:16:39 | openstack | Launchpad bug 1813165 in QEMU "KVM internal error. Suberror: 1 emulation failure" [Undecided,New] | |
| 22:16:55 | sean-k-mooney | that is not the casue in your trace back as you have SMM=0 | |
| 22:17:05 | sean-k-mooney | but they mentione this happend in a nested case | |
| 22:17:14 | sean-k-mooney | *also happend | |
| 22:17:40 | sean-k-mooney | lucidguy: and know i just have been working with nested virt for year and have seeing it improve and break over that time | |
| 22:18:40 | lucidguy | Since I'm using HWE, I believe I'm using the same kernal as Ubuntu 18.04. So people with the latest LTS are still having this issue? | |
| 22:18:59 | sean-k-mooney | ya nested virt in the defaul 18.04 kernel is broken | |
| 22:19:39 | lucidguy | To be honest, I don't exactly know what "nested" means. | |
| 22:19:41 | sean-k-mooney | so if that is what the 16.04 hwe kerel is trackin then that is likely the cause | |
| 22:20:07 | sean-k-mooney | lucidguy: it allows you to run kvm in side the vm to have 2+ layers of vms | |
| 22:20:48 | lucidguy | So should I not use nested? | |
| 22:21:02 | lucidguy | I don't think we have users runing vms within their vms | |