| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-12 | |||
| 14:06:08 | bauzas | which kernel version is currently running on our CI jobs ? | |
| 14:06:10 | dansmith | the image in our CI gets refreshed periodically for that reason | |
| 14:06:14 | bauzas | because I have the choices | |
| 14:06:20 | sean-k-mooney | i allways do an apt dist-upgrade and reboot before running devstack for what its worth | |
| 14:06:30 | sean-k-mooney | nightly for upstream ci | |
| 14:06:46 | dansmith | bauzas: https://zuul.opendev.org/t/openstack/build/3c70446c0452434a87166d336dc0dbc5/log/job-output.txt#22580 | |
| 14:06:55 | sean-k-mooney | bauzas: just install the generic kernel | |
| 14:06:57 | dansmith | linux-headers-5.15.0-86-generic | |
| 14:07:18 | bauzas | I'll then install linux-image-5.15.0-86-generic | |
| 14:07:28 | dansmith | that's what I'm running locally too | |
| 14:07:43 | sean-k-mooney | same | |
| 14:11:39 | dansmith | hmm, actually linux-headers-kvm resolves to linux-headers-$ver-kvm as well, so I'm not quite sure why this isn't just working, actually | |
| 14:12:24 | dansmith | oh but there's also linux-kvm-headers-$ver .. weird | |
| 14:15:14 | bauzas | ok, changing the kernel gave me some kernel tain at boot | |
| 14:15:17 | bauzas | taint* | |
| 14:15:25 | dansmith | at boot? | |
| 14:15:31 | bauzas | yeah | |
| 14:15:40 | bauzas | I'll just upload a fucking plain ubuntu image | |
| 14:15:51 | dansmith | lol | |
| 14:16:07 | dansmith | can you pastebin the taint message? really curious | |
| 14:16:16 | dansmith | unless your provider is inserting something of their own, I can't imagine... | |
| 14:21:44 | bauzas | argh, sorry, again already rebuilt | |
| 14:22:06 | bauzas | IIRC, it was about getting the disk | |
| 14:25:08 | bauzas | ok I need to go parenting my kid | |
| 14:25:10 | bauzas | bbiab | |
| 14:50:04 | dansmith | bauzas: fwiw, I can reproduce your issue if I install the kvm variant, so I'll see if I can figure out why that's not working | |
| 14:50:16 | bauzas | ack | |
| 14:50:45 | bauzas | that's crazy, I don't know yet why but the variant is automatically installed in my cloud env :facepalm | |
| 14:51:00 | dansmith | yeah | |
| 14:52:50 | dansmith | oh neither of the headers packages for that kernel contain the module symbols it seems | |
| 14:54:17 | dansmith | yeah totally different, weird | |
| 14:54:46 | bauzas | dansmith: yup and apparently I can't avoid this kernel variant to be installed in my cloud | |
| 14:55:11 | bauzas | I can try to use a bare kernel like the last time but my guess is that it won't boot | |
| 14:55:18 | dansmith | bauzas: can't avoid installing it or can't avoid booting to it? | |
| 14:55:25 | bauzas | both | |
| 14:55:31 | dansmith | how does it fail? | |
| 14:55:43 | bauzas | it's internal thing, so slack | |
| 15:19:48 | dansmith | bauzas: sean-k-mooney: ah, I got it.. the -kvm kernel has none of the vfio or mdev stuff enabled in config (because it's for virtual only) and so we have module symbols, but not symbols for the vfio or mdev base infrastructure | |
| 15:19:51 | dansmith | which makes sense of course | |
| 15:20:16 | dansmith | so we can't compile those mdev drivers because the infra they depend on is missing | |
| 15:26:12 | bauzas | dansmith: I eventually was able to create a new instance w/o that variant thanks to another image | |
| 15:26:41 | dansmith | cool, I think that's the way | |
| 15:26:42 | bauzas | but fwiw https://cloud-images.ubuntu.com/jammy/20231012/ provides a specific QEMU image with the kvm variant | |
| 15:27:01 | dansmith | yep, but we're simulating hardware here so we need a kernel with hardware stuff in it | |
| 15:27:04 | bauzas | so that's something so we can't support | |
| 15:27:15 | bauzas | agreed | |
| 15:58:10 | opendevreview | Dan Smith proposed openstack/nova master: Compile mdev samples for nova-next https://review.opendev.org/c/openstack/nova/+/897708 | |
| 16:01:58 | dansmith | bauzas: this ^ hard fails on the kvm variant to make sure it's obvious | |
| 16:02:03 | dansmith | and also addresses some of sean-k-mooney's feedback | |
| 16:02:10 | bauzas | cool | |
| 16:08:05 | bauzas | my devstack node is still deploying the stack but I have good hopes | |
| 16:08:55 | bauzas | actually, my hope was wrong : E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 30066 (apt) | |
| 16:09:00 | bauzas | shit, need to respin | |
| 16:09:16 | dansmith | is auto upgrades running in the background? | |
| 16:09:29 | dansmith | that's a common thing | |
| 16:17:49 | sean-k-mooney | thats why i always do sudo apt update; sudo apt dist-upgrade -y; sudo reboot | |
| 16:17:57 | sean-k-mooney | before i run devstack for the first time | |
| 16:19:29 | sean-k-mooney | ok i also do "sudo apt install python3-dev libffi-dev libssl-dev gcc make git" too | |
| 16:19:51 | sean-k-mooney | devstack does pull those in as needed but it causes less issues if you do it upfront | |
| 16:21:00 | sean-k-mooney | thats just making sure that we can compile python c modules for pip deps | |
| 16:43:21 | bauzas | fwiw, was able to build the kernel modules with the plain ubuntu kernel | |
| 16:43:32 | bauzas | so I'll +2 dansmith's patch | |
| 16:43:52 | bauzas | still fighting with a last devstack issue, but unrelated to dan's | |
| 16:43:57 | dansmith | cool | |
| 17:21:14 | bauzas | dansmith: for some reason, I'm hitting hard https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1523 but it's just the first devstack start | |
| 17:21:24 | bauzas | so I wonder how to fix it | |
| 17:21:58 | dansmith | uh, leftover database? | |
| 17:22:25 | bauzas | probably | |
| 17:22:42 | bauzas | so, do you prefer that I directly write the file or removing the node ? | |
| 17:24:14 | dansmith | up to you, either way it's detecting the right thing | |
| 17:24:31 | bauzas | actually yeah, I recloned the host but this isn't a AIO | |
| 17:24:40 | bauzas | so the DB still exists | |
| 17:24:44 | bauzas | DB record* | |
| 17:24:49 | bauzas | okay, will write the file | |
| 17:27:12 | dansmith | right which is exactly what that check is for :P | |
| 17:54:29 | bauzas | dansmith: actually, I have to doublecheck but since state_path is /opt/stack/data/nova, this probably goes away when you unstack | |
| 17:54:48 | dansmith | only when you clean I think | |
| 17:54:55 | dansmith | but you said it's not an AIO right? | |
| 17:55:07 | bauzas | yeah, just a single compute connecting to another instance | |
| 17:55:10 | dansmith | oh you mean nova's state path gets deleted | |
| 17:55:22 | bauzas | yeah, my theory is when unstacking | |
| 17:55:45 | dansmith | right, but that's the thing it's checking for.. that you basically rebuilt a node with the same name in place | |
| 17:56:00 | dansmith | if you don't want that, copy the uuid file to /etc/nova and it will re-use the same thing over and over | |
| 17:56:01 | bauzas | right, on purpose :) | |
| 17:56:13 | dansmith | (assuming you don't delete /etc/nova I guess unstack might) | |
| 17:56:33 | dansmith | maybe we need to let devstack force a uuid so we can keep it for subsequent stack runs | |
| 17:56:39 | dansmith | I always do AIO so I never hit that | |
| 17:56:56 | bauzas | anyway, not sure we need to change anything, but that's a bit complicating a story of a 2-node devstack install with the n-cpu failing once | |
| 17:57:48 | bauzas | maybe, for the moment, my brain is doing concurrency between a devstack install and a customer bug | |
| 17:58:08 | bauzas | so I don't have the energy to think about any potential solution | |
| 18:01:31 | dansmith | it's doing the thing it's supposed to which is catching that you've recreated a compute node with the same name without deleting it from the database | |
| 18:01:48 | dansmith | so either delete it from the DB, or slap the uuid file into place | |
| 18:02:06 | dansmith | I'll add something to devstack to basically disable that behavior by letting you force the uuid to be the same on each stack | |
| 18:10:13 | dansmith | bauzas: https://review.opendev.org/c/openstack/devstack/+/898134 | |
| 18:10:28 | dansmith | put a uuid in NOVA_CPU_UUID= in your localrc and you should be good | |
| 18:11:02 | bauzas | yup | |
| 18:11:06 | bauzas | excellent thing | |
| 18:11:08 | bauzas | +1 from me | |
| 19:20:55 | opendevreview | Dan Smith proposed openstack/nova master: Do not manage CPU0's state https://review.opendev.org/c/openstack/nova/+/898137 | |
| 19:21:08 | dansmith | bauzas: yw ^ | |