| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-12 | |||
| 13:54:12 | bauzas | and yeah, if the symbols are only on the plain kernel, that explains | |
| 13:54:36 | bauzas | not sure how I got this variant, probably because of the image I used in my cloud | |
| 13:54:42 | dansmith | so linux-headers-$kver won't work for the variants, because it needs to be linux-headers-kvm I guess | |
| 13:55:01 | dansmith | bauzas: so you can switch to the normal one? | |
| 13:55:03 | bauzas | yeah that's my guess | |
| 13:55:12 | bauzas | dansmith: too late, I rebuilt it | |
| 13:55:32 | dansmith | what does that mean, you installed the right headers package and built the modules? | |
| 13:55:46 | sean-k-mooney | i think they mean reinstalled the vm | |
| 13:56:11 | dansmith | that's what I said.. switch to the normal kernel, however that needs to happen :) | |
| 13:56:19 | dansmith | either with apt or rm :) | |
| 13:56:22 | sean-k-mooney | bauzas: are you using a cloud imave or similar? i have never seen the kvm kernel actully installed in anything | |
| 13:56:37 | dansmith | yeah I'm not sure what it's for either | |
| 13:56:55 | sean-k-mooney | well its a striped down kernel without any hard specific drivers | |
| 13:56:59 | dansmith | "Targeting KVM instance usage. This carries the minimum support required for a guest kernel." | |
| 13:57:07 | sean-k-mooney | but they dont use it for the cloud image | |
| 13:57:18 | sean-k-mooney | so i dont know who the audiance for that is | |
| 13:57:25 | dansmith | yeah, probably for ultra size-conscious situations or something | |
| 13:57:28 | bauzas | sean-k-mooney: I used some internal cloud at my employer | |
| 13:58:25 | bauzas | stack@sbauza-dev2:~$ uname -r | |
| 13:58:25 | bauzas | 5.15.0-1017-kvm | |
| 13:58:32 | bauzas | okay, coming from the image itsefl | |
| 13:58:52 | bauzas | fun | |
| 13:59:09 | bauzas | I'm tempted to pick another image or download another kernel | |
| 13:59:25 | sean-k-mooney | yes just install a diffent kernel and reboot | |
| 13:59:37 | sean-k-mooney | or use the ubuntu image i upoloaded | |
| 13:59:41 | bauzas | that will be the quickiest | |
| 14:00:09 | dansmith | I can see if I can make it work either way, but yeah do the workaround for the moment | |
| 14:00:25 | sean-k-mooney | the offical ubuntu cloud images have 5.15.0-48-generic | |
| 14:01:00 | sean-k-mooney | well thats an old image so its probaly outdated but i just confirmed that so you booted form a customized image | |
| 14:02:34 | dansmith | yeah so you can't even get the support packages for very old kernels, so you need to be running something current anyway | |
| 14:02:39 | dansmith | like they remove them | |
| 14:05:44 | bauzas | hum | |
| 14:06:08 | bauzas | which kernel version is currently running on our CI jobs ? | |
| 14:06:10 | dansmith | the image in our CI gets refreshed periodically for that reason | |
| 14:06:14 | bauzas | because I have the choices | |
| 14:06:20 | sean-k-mooney | i allways do an apt dist-upgrade and reboot before running devstack for what its worth | |
| 14:06:30 | sean-k-mooney | nightly for upstream ci | |
| 14:06:46 | dansmith | bauzas: https://zuul.opendev.org/t/openstack/build/3c70446c0452434a87166d336dc0dbc5/log/job-output.txt#22580 | |
| 14:06:55 | sean-k-mooney | bauzas: just install the generic kernel | |
| 14:06:57 | dansmith | linux-headers-5.15.0-86-generic | |
| 14:07:18 | bauzas | I'll then install linux-image-5.15.0-86-generic | |
| 14:07:28 | dansmith | that's what I'm running locally too | |
| 14:07:43 | sean-k-mooney | same | |
| 14:11:39 | dansmith | hmm, actually linux-headers-kvm resolves to linux-headers-$ver-kvm as well, so I'm not quite sure why this isn't just working, actually | |
| 14:12:24 | dansmith | oh but there's also linux-kvm-headers-$ver .. weird | |
| 14:15:14 | bauzas | ok, changing the kernel gave me some kernel tain at boot | |
| 14:15:17 | bauzas | taint* | |
| 14:15:25 | dansmith | at boot? | |
| 14:15:31 | bauzas | yeah | |
| 14:15:40 | bauzas | I'll just upload a fucking plain ubuntu image | |
| 14:15:51 | dansmith | lol | |
| 14:16:07 | dansmith | can you pastebin the taint message? really curious | |
| 14:16:16 | dansmith | unless your provider is inserting something of their own, I can't imagine... | |
| 14:21:44 | bauzas | argh, sorry, again already rebuilt | |
| 14:22:06 | bauzas | IIRC, it was about getting the disk | |
| 14:25:08 | bauzas | ok I need to go parenting my kid | |
| 14:25:10 | bauzas | bbiab | |
| 14:50:04 | dansmith | bauzas: fwiw, I can reproduce your issue if I install the kvm variant, so I'll see if I can figure out why that's not working | |
| 14:50:16 | bauzas | ack | |
| 14:50:45 | bauzas | that's crazy, I don't know yet why but the variant is automatically installed in my cloud env :facepalm | |
| 14:51:00 | dansmith | yeah | |
| 14:52:50 | dansmith | oh neither of the headers packages for that kernel contain the module symbols it seems | |
| 14:54:17 | dansmith | yeah totally different, weird | |
| 14:54:46 | bauzas | dansmith: yup and apparently I can't avoid this kernel variant to be installed in my cloud | |
| 14:55:11 | bauzas | I can try to use a bare kernel like the last time but my guess is that it won't boot | |
| 14:55:18 | dansmith | bauzas: can't avoid installing it or can't avoid booting to it? | |
| 14:55:25 | bauzas | both | |
| 14:55:31 | dansmith | how does it fail? | |
| 14:55:43 | bauzas | it's internal thing, so slack | |
| 15:19:48 | dansmith | bauzas: sean-k-mooney: ah, I got it.. the -kvm kernel has none of the vfio or mdev stuff enabled in config (because it's for virtual only) and so we have module symbols, but not symbols for the vfio or mdev base infrastructure | |
| 15:19:51 | dansmith | which makes sense of course | |
| 15:20:16 | dansmith | so we can't compile those mdev drivers because the infra they depend on is missing | |
| 15:26:12 | bauzas | dansmith: I eventually was able to create a new instance w/o that variant thanks to another image | |
| 15:26:41 | dansmith | cool, I think that's the way | |
| 15:26:42 | bauzas | but fwiw https://cloud-images.ubuntu.com/jammy/20231012/ provides a specific QEMU image with the kvm variant | |
| 15:27:01 | dansmith | yep, but we're simulating hardware here so we need a kernel with hardware stuff in it | |
| 15:27:04 | bauzas | so that's something so we can't support | |
| 15:27:15 | bauzas | agreed | |
| 15:58:10 | opendevreview | Dan Smith proposed openstack/nova master: Compile mdev samples for nova-next https://review.opendev.org/c/openstack/nova/+/897708 | |
| 16:01:58 | dansmith | bauzas: this ^ hard fails on the kvm variant to make sure it's obvious | |
| 16:02:03 | dansmith | and also addresses some of sean-k-mooney's feedback | |
| 16:02:10 | bauzas | cool | |
| 16:08:05 | bauzas | my devstack node is still deploying the stack but I have good hopes | |
| 16:08:55 | bauzas | actually, my hope was wrong : E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 30066 (apt) | |
| 16:09:00 | bauzas | shit, need to respin | |
| 16:09:16 | dansmith | is auto upgrades running in the background? | |
| 16:09:29 | dansmith | that's a common thing | |
| 16:17:49 | sean-k-mooney | thats why i always do sudo apt update; sudo apt dist-upgrade -y; sudo reboot | |
| 16:17:57 | sean-k-mooney | before i run devstack for the first time | |
| 16:19:29 | sean-k-mooney | ok i also do "sudo apt install python3-dev libffi-dev libssl-dev gcc make git" too | |
| 16:19:51 | sean-k-mooney | devstack does pull those in as needed but it causes less issues if you do it upfront | |
| 16:21:00 | sean-k-mooney | thats just making sure that we can compile python c modules for pip deps | |
| 16:43:21 | bauzas | fwiw, was able to build the kernel modules with the plain ubuntu kernel | |
| 16:43:32 | bauzas | so I'll +2 dansmith's patch | |
| 16:43:52 | bauzas | still fighting with a last devstack issue, but unrelated to dan's | |
| 16:43:57 | dansmith | cool | |
| 17:21:14 | bauzas | dansmith: for some reason, I'm hitting hard https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1523 but it's just the first devstack start | |
| 17:21:24 | bauzas | so I wonder how to fix it | |
| 17:21:58 | dansmith | uh, leftover database? | |
| 17:22:25 | bauzas | probably | |