| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-12 | |||
| 12:46:38 | bauzas | sean-k-mooney: ^ | |
| 12:47:53 | bauzas | actually the whole stack is here https://paste.opendev.org/show/bLirDAMuGmxhLLod0S9p/ | |
| 12:49:19 | bauzas | DISTRIB_CODENAME=jammy | |
| 12:49:19 | bauzas | DISTRIB_RELEASE=22.04 | |
| 12:49:19 | bauzas | DISTRIB_ID=Ubuntu | |
| 12:49:19 | bauzas | stack@sbauza-dev2:~/devstack$ cat /etc/lsb-release | |
| 12:49:20 | bauzas | DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" | |
| 12:49:35 | bauzas | 5.15.0-1037-kvm | |
| 12:49:35 | bauzas | stack@sbauza-dev2:~/devstack$ uname -r | |
| 13:19:21 | frickler | bauzas: is there some linux-modules-extra pkg for that kernel? | |
| 13:22:22 | dansmith | bauzas: looks like maybe it has module symbols from a different kernel | |
| 13:22:37 | dansmith | since it failed to create /opt/stack/kernel .. did you run it once, upgrade kernel, run again? | |
| 13:36:29 | bauzas | dansmith: it didn't failed to create kernel | |
| 13:36:37 | bauzas | (sorry was on a meeting) | |
| 13:36:39 | dansmith | didn't fail? | |
| 13:36:52 | dansmith | mkdir: cannot create directory ‘/opt/stack/kernel’: File exists | |
| 13:36:52 | dansmith | ++/opt/stack/nova/devstack/lib/mdev_samples:compile_mdev_samples:19 mkdir /opt/stack/kernel | |
| 13:36:53 | bauzas | the kernel modules, yeah | |
| 13:36:57 | bauzas | ah | |
| 13:37:14 | bauzas | probably because devstack failed before on OVS | |
| 13:37:25 | bauzas | but I hadn't upgraded my kernel | |
| 13:37:38 | dansmith | that's what I just said yeah :) | |
| 13:37:51 | dansmith | blow that away and try again, I'll add cleaning of that to clean.sh | |
| 13:37:55 | bauzas | want me to unstack, delete the kernel dir and stack ? | |
| 13:38:02 | bauzas | ok | |
| 13:38:04 | bauzas | I can try | |
| 13:38:24 | bauzas | it's a pure compute node, no controllers on it so should be small and quick | |
| 13:39:33 | dansmith | well, you could just run the plugin itself again (after cleaning the kernel dir) but re-stacking would be better | |
| 13:39:59 | bauzas | yeah | |
| 13:40:15 | bauzas | if that fails again, I'll run the plugin itself | |
| 13:40:35 | bauzas | dansmith: fwiw, I started to provide the mtty change on nova | |
| 13:41:21 | bauzas | I could have missed something (and I know I did, since I haven't modified the method that checks the mdevs when we restart, but meh not a problem for testing) but I think it's a start | |
| 13:42:14 | dansmith | cool | |
| 13:42:50 | bauzas | dansmith: failing again https://paste.opendev.org/show/bFTPQR1VBSJNQj2jrknG/ | |
| 13:43:10 | bauzas | this time, this is linking the modules | |
| 13:43:18 | bauzas | but still failing on the definitions | |
| 13:43:42 | dansmith | bauzas: show me "dpkg -l | grep linux" and "ls /usr/src" | |
| 13:43:49 | bauzas | cool | |
| 13:44:20 | dansmith | also uname -r and "ls -l /lib/modules/*/build" | |
| 13:44:45 | dansmith | worked fine for me and sean-k-mooney, but it gets weird if you have a bunch of mismatched kernels and things installed sometimes | |
| 13:45:54 | bauzas | dansmith: there you go https://paste.opendev.org/show/bSXwvpbc6Z96awnWuIRS/ | |
| 13:46:27 | bauzas | I can reclone the machine and reinstall devstack again | |
| 13:46:41 | bauzas | I mean, if my env is dirty, meh | |
| 13:46:50 | bauzas | probably better to start with fresh things | |
| 13:48:18 | dansmith | yeah that' a ton of kernel distraction, although it seems like it all lines up | |
| 13:48:19 | dansmith | however, | |
| 13:48:43 | dansmith | there are lots of reports out there about having linux-headers installed for the right kernel, but needing to reinstall it if it was installed by a previous version | |
| 13:49:07 | dansmith | not sure why, because it should be all separate | |
| 13:49:20 | dansmith | did you already have linux-headers for your kernel installed or did the first run of this install it for you? | |
| 13:52:09 | sean-k-mooney | you have a kvm variant of the kernl | |
| 13:52:19 | sean-k-mooney | so maybe thats the issue | |
| 13:52:32 | sean-k-mooney | i have Linux upstream-devstack 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | |
| 13:52:57 | dansmith | oh is there a separate headers package for the kvm variant? if so that's definitely it | |
| 13:53:28 | bauzas | I just rebuilt the instance I have as devstack2 | |
| 13:53:32 | dansmith | ah yep | |
| 13:53:40 | dansmith | that's the problem | |
| 13:54:12 | bauzas | and yeah, if the symbols are only on the plain kernel, that explains | |
| 13:54:36 | bauzas | not sure how I got this variant, probably because of the image I used in my cloud | |
| 13:54:42 | dansmith | so linux-headers-$kver won't work for the variants, because it needs to be linux-headers-kvm I guess | |
| 13:55:01 | dansmith | bauzas: so you can switch to the normal one? | |
| 13:55:03 | bauzas | yeah that's my guess | |
| 13:55:12 | bauzas | dansmith: too late, I rebuilt it | |
| 13:55:32 | dansmith | what does that mean, you installed the right headers package and built the modules? | |
| 13:55:46 | sean-k-mooney | i think they mean reinstalled the vm | |
| 13:56:11 | dansmith | that's what I said.. switch to the normal kernel, however that needs to happen :) | |
| 13:56:19 | dansmith | either with apt or rm :) | |
| 13:56:22 | sean-k-mooney | bauzas: are you using a cloud imave or similar? i have never seen the kvm kernel actully installed in anything | |
| 13:56:37 | dansmith | yeah I'm not sure what it's for either | |
| 13:56:55 | sean-k-mooney | well its a striped down kernel without any hard specific drivers | |
| 13:56:59 | dansmith | "Targeting KVM instance usage. This carries the minimum support required for a guest kernel." | |
| 13:57:07 | sean-k-mooney | but they dont use it for the cloud image | |
| 13:57:18 | sean-k-mooney | so i dont know who the audiance for that is | |
| 13:57:25 | dansmith | yeah, probably for ultra size-conscious situations or something | |
| 13:57:28 | bauzas | sean-k-mooney: I used some internal cloud at my employer | |
| 13:58:25 | bauzas | 5.15.0-1017-kvm | |
| 13:58:25 | bauzas | stack@sbauza-dev2:~$ uname -r | |
| 13:58:32 | bauzas | okay, coming from the image itsefl | |
| 13:58:52 | bauzas | fun | |
| 13:59:09 | bauzas | I'm tempted to pick another image or download another kernel | |
| 13:59:25 | sean-k-mooney | yes just install a diffent kernel and reboot | |
| 13:59:37 | sean-k-mooney | or use the ubuntu image i upoloaded | |
| 13:59:41 | bauzas | that will be the quickiest | |
| 14:00:09 | dansmith | I can see if I can make it work either way, but yeah do the workaround for the moment | |
| 14:00:25 | sean-k-mooney | the offical ubuntu cloud images have 5.15.0-48-generic | |
| 14:01:00 | sean-k-mooney | well thats an old image so its probaly outdated but i just confirmed that so you booted form a customized image | |
| 14:02:34 | dansmith | yeah so you can't even get the support packages for very old kernels, so you need to be running something current anyway | |
| 14:02:39 | dansmith | like they remove them | |
| 14:05:44 | bauzas | hum | |
| 14:06:08 | bauzas | which kernel version is currently running on our CI jobs ? | |
| 14:06:10 | dansmith | the image in our CI gets refreshed periodically for that reason | |
| 14:06:14 | bauzas | because I have the choices | |
| 14:06:20 | sean-k-mooney | i allways do an apt dist-upgrade and reboot before running devstack for what its worth | |
| 14:06:30 | sean-k-mooney | nightly for upstream ci | |
| 14:06:46 | dansmith | bauzas: https://zuul.opendev.org/t/openstack/build/3c70446c0452434a87166d336dc0dbc5/log/job-output.txt#22580 | |
| 14:06:55 | sean-k-mooney | bauzas: just install the generic kernel | |
| 14:06:57 | dansmith | linux-headers-5.15.0-86-generic | |
| 14:07:18 | bauzas | I'll then install linux-image-5.15.0-86-generic | |
| 14:07:28 | dansmith | that's what I'm running locally too | |
| 14:07:43 | sean-k-mooney | same | |
| 14:11:39 | dansmith | hmm, actually linux-headers-kvm resolves to linux-headers-$ver-kvm as well, so I'm not quite sure why this isn't just working, actually | |