| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-12 | |||
| 13:39:59 | bauzas | yeah | |
| 13:40:15 | bauzas | if that fails again, I'll run the plugin itself | |
| 13:40:35 | bauzas | dansmith: fwiw, I started to provide the mtty change on nova | |
| 13:41:21 | bauzas | I could have missed something (and I know I did, since I haven't modified the method that checks the mdevs when we restart, but meh not a problem for testing) but I think it's a start | |
| 13:42:14 | dansmith | cool | |
| 13:42:50 | bauzas | dansmith: failing again https://paste.opendev.org/show/bFTPQR1VBSJNQj2jrknG/ | |
| 13:43:10 | bauzas | this time, this is linking the modules | |
| 13:43:18 | bauzas | but still failing on the definitions | |
| 13:43:42 | dansmith | bauzas: show me "dpkg -l | grep linux" and "ls /usr/src" | |
| 13:43:49 | bauzas | cool | |
| 13:44:20 | dansmith | also uname -r and "ls -l /lib/modules/*/build" | |
| 13:44:45 | dansmith | worked fine for me and sean-k-mooney, but it gets weird if you have a bunch of mismatched kernels and things installed sometimes | |
| 13:45:54 | bauzas | dansmith: there you go https://paste.opendev.org/show/bSXwvpbc6Z96awnWuIRS/ | |
| 13:46:27 | bauzas | I can reclone the machine and reinstall devstack again | |
| 13:46:41 | bauzas | I mean, if my env is dirty, meh | |
| 13:46:50 | bauzas | probably better to start with fresh things | |
| 13:48:18 | dansmith | yeah that' a ton of kernel distraction, although it seems like it all lines up | |
| 13:48:19 | dansmith | however, | |
| 13:48:43 | dansmith | there are lots of reports out there about having linux-headers installed for the right kernel, but needing to reinstall it if it was installed by a previous version | |
| 13:49:07 | dansmith | not sure why, because it should be all separate | |
| 13:49:20 | dansmith | did you already have linux-headers for your kernel installed or did the first run of this install it for you? | |
| 13:52:09 | sean-k-mooney | you have a kvm variant of the kernl | |
| 13:52:19 | sean-k-mooney | so maybe thats the issue | |
| 13:52:32 | sean-k-mooney | i have Linux upstream-devstack 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | |
| 13:52:57 | dansmith | oh is there a separate headers package for the kvm variant? if so that's definitely it | |
| 13:53:28 | bauzas | I just rebuilt the instance I have as devstack2 | |
| 13:53:32 | dansmith | ah yep | |
| 13:53:40 | dansmith | that's the problem | |
| 13:54:12 | bauzas | and yeah, if the symbols are only on the plain kernel, that explains | |
| 13:54:36 | bauzas | not sure how I got this variant, probably because of the image I used in my cloud | |
| 13:54:42 | dansmith | so linux-headers-$kver won't work for the variants, because it needs to be linux-headers-kvm I guess | |
| 13:55:01 | dansmith | bauzas: so you can switch to the normal one? | |
| 13:55:03 | bauzas | yeah that's my guess | |
| 13:55:12 | bauzas | dansmith: too late, I rebuilt it | |
| 13:55:32 | dansmith | what does that mean, you installed the right headers package and built the modules? | |
| 13:55:46 | sean-k-mooney | i think they mean reinstalled the vm | |
| 13:56:11 | dansmith | that's what I said.. switch to the normal kernel, however that needs to happen :) | |
| 13:56:19 | dansmith | either with apt or rm :) | |
| 13:56:22 | sean-k-mooney | bauzas: are you using a cloud imave or similar? i have never seen the kvm kernel actully installed in anything | |
| 13:56:37 | dansmith | yeah I'm not sure what it's for either | |
| 13:56:55 | sean-k-mooney | well its a striped down kernel without any hard specific drivers | |
| 13:56:59 | dansmith | "Targeting KVM instance usage. This carries the minimum support required for a guest kernel." | |
| 13:57:07 | sean-k-mooney | but they dont use it for the cloud image | |
| 13:57:18 | sean-k-mooney | so i dont know who the audiance for that is | |
| 13:57:25 | dansmith | yeah, probably for ultra size-conscious situations or something | |
| 13:57:28 | bauzas | sean-k-mooney: I used some internal cloud at my employer | |
| 13:58:25 | bauzas | 5.15.0-1017-kvm | |
| 13:58:25 | bauzas | stack@sbauza-dev2:~$ uname -r | |
| 13:58:32 | bauzas | okay, coming from the image itsefl | |
| 13:58:52 | bauzas | fun | |
| 13:59:09 | bauzas | I'm tempted to pick another image or download another kernel | |
| 13:59:25 | sean-k-mooney | yes just install a diffent kernel and reboot | |
| 13:59:37 | sean-k-mooney | or use the ubuntu image i upoloaded | |
| 13:59:41 | bauzas | that will be the quickiest | |
| 14:00:09 | dansmith | I can see if I can make it work either way, but yeah do the workaround for the moment | |
| 14:00:25 | sean-k-mooney | the offical ubuntu cloud images have 5.15.0-48-generic | |
| 14:01:00 | sean-k-mooney | well thats an old image so its probaly outdated but i just confirmed that so you booted form a customized image | |
| 14:02:34 | dansmith | yeah so you can't even get the support packages for very old kernels, so you need to be running something current anyway | |
| 14:02:39 | dansmith | like they remove them | |
| 14:05:44 | bauzas | hum | |
| 14:06:08 | bauzas | which kernel version is currently running on our CI jobs ? | |
| 14:06:10 | dansmith | the image in our CI gets refreshed periodically for that reason | |
| 14:06:14 | bauzas | because I have the choices | |
| 14:06:20 | sean-k-mooney | i allways do an apt dist-upgrade and reboot before running devstack for what its worth | |
| 14:06:30 | sean-k-mooney | nightly for upstream ci | |
| 14:06:46 | dansmith | bauzas: https://zuul.opendev.org/t/openstack/build/3c70446c0452434a87166d336dc0dbc5/log/job-output.txt#22580 | |
| 14:06:55 | sean-k-mooney | bauzas: just install the generic kernel | |
| 14:06:57 | dansmith | linux-headers-5.15.0-86-generic | |
| 14:07:18 | bauzas | I'll then install linux-image-5.15.0-86-generic | |
| 14:07:28 | dansmith | that's what I'm running locally too | |
| 14:07:43 | sean-k-mooney | same | |
| 14:11:39 | dansmith | hmm, actually linux-headers-kvm resolves to linux-headers-$ver-kvm as well, so I'm not quite sure why this isn't just working, actually | |
| 14:12:24 | dansmith | oh but there's also linux-kvm-headers-$ver .. weird | |
| 14:15:14 | bauzas | ok, changing the kernel gave me some kernel tain at boot | |
| 14:15:17 | bauzas | taint* | |
| 14:15:25 | dansmith | at boot? | |
| 14:15:31 | bauzas | yeah | |
| 14:15:40 | bauzas | I'll just upload a fucking plain ubuntu image | |
| 14:15:51 | dansmith | lol | |
| 14:16:07 | dansmith | can you pastebin the taint message? really curious | |
| 14:16:16 | dansmith | unless your provider is inserting something of their own, I can't imagine... | |
| 14:21:44 | bauzas | argh, sorry, again already rebuilt | |
| 14:22:06 | bauzas | IIRC, it was about getting the disk | |
| 14:25:08 | bauzas | ok I need to go parenting my kid | |
| 14:25:10 | bauzas | bbiab | |
| 14:50:04 | dansmith | bauzas: fwiw, I can reproduce your issue if I install the kvm variant, so I'll see if I can figure out why that's not working | |
| 14:50:16 | bauzas | ack | |
| 14:50:45 | bauzas | that's crazy, I don't know yet why but the variant is automatically installed in my cloud env :facepalm | |
| 14:51:00 | dansmith | yeah | |
| 14:52:50 | dansmith | oh neither of the headers packages for that kernel contain the module symbols it seems | |
| 14:54:17 | dansmith | yeah totally different, weird | |
| 14:54:46 | bauzas | dansmith: yup and apparently I can't avoid this kernel variant to be installed in my cloud | |
| 14:55:11 | bauzas | I can try to use a bare kernel like the last time but my guess is that it won't boot | |
| 14:55:18 | dansmith | bauzas: can't avoid installing it or can't avoid booting to it? | |
| 14:55:25 | bauzas | both | |
| 14:55:31 | dansmith | how does it fail? | |
| 14:55:43 | bauzas | it's internal thing, so slack | |
| 15:19:48 | dansmith | bauzas: sean-k-mooney: ah, I got it.. the -kvm kernel has none of the vfio or mdev stuff enabled in config (because it's for virtual only) and so we have module symbols, but not symbols for the vfio or mdev base infrastructure | |
| 15:19:51 | dansmith | which makes sense of course | |
| 15:20:16 | dansmith | so we can't compile those mdev drivers because the infra they depend on is missing | |