| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-12 | |||
| 09:08:20 | tobias-urdin | sean-k-mooney: when you have some seconds, based on https://review.opendev.org/c/openstack/nova/+/824048 and the issue from libvirt down expressed in https://gitlab.com/libvirt/libvirt/-/issues/161 I assume it would be thrown upon implementing the workaround in the nova layer by checking for cgroupsv1/cgroupsv2 and reintroducing the default | |
| 09:08:20 | tobias-urdin | cputune.shares value based on that (lets ignore the live migration part of it now and just focus on "having the correct value" based on the crazy decision made in libvirt).. any thoughts on that? | |
| 09:40:08 | opendevreview | Sylvain Bauza proposed openstack/nova master: WIP: add mtty support for vgpus https://review.opendev.org/c/openstack/nova/+/898100 | |
| 12:46:31 | bauzas | dansmith: I know that's too early for you but my devstack fails on your mtty patch https://paste.opendev.org/show/bwpHZPMuePifFk0JWi7v/ (just reply when you want) | |
| 12:46:38 | bauzas | sean-k-mooney: ^ | |
| 12:47:53 | bauzas | actually the whole stack is here https://paste.opendev.org/show/bLirDAMuGmxhLLod0S9p/ | |
| 12:49:19 | bauzas | stack@sbauza-dev2:~/devstack$ cat /etc/lsb-release | |
| 12:49:19 | bauzas | DISTRIB_ID=Ubuntu | |
| 12:49:19 | bauzas | DISTRIB_RELEASE=22.04 | |
| 12:49:19 | bauzas | DISTRIB_CODENAME=jammy | |
| 12:49:20 | bauzas | DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" | |
| 12:49:35 | bauzas | stack@sbauza-dev2:~/devstack$ uname -r | |
| 12:49:35 | bauzas | 5.15.0-1037-kvm | |
| 13:19:21 | frickler | bauzas: is there some linux-modules-extra pkg for that kernel? | |
| 13:22:22 | dansmith | bauzas: looks like maybe it has module symbols from a different kernel | |
| 13:22:37 | dansmith | since it failed to create /opt/stack/kernel .. did you run it once, upgrade kernel, run again? | |
| 13:36:29 | bauzas | dansmith: it didn't failed to create kernel | |
| 13:36:37 | bauzas | (sorry was on a meeting) | |
| 13:36:39 | dansmith | didn't fail? | |
| 13:36:52 | dansmith | ++/opt/stack/nova/devstack/lib/mdev_samples:compile_mdev_samples:19 mkdir /opt/stack/kernel | |
| 13:36:52 | dansmith | mkdir: cannot create directory ‘/opt/stack/kernel’: File exists | |
| 13:36:53 | bauzas | the kernel modules, yeah | |
| 13:36:57 | bauzas | ah | |
| 13:37:14 | bauzas | probably because devstack failed before on OVS | |
| 13:37:25 | bauzas | but I hadn't upgraded my kernel | |
| 13:37:38 | dansmith | that's what I just said yeah :) | |
| 13:37:51 | dansmith | blow that away and try again, I'll add cleaning of that to clean.sh | |
| 13:37:55 | bauzas | want me to unstack, delete the kernel dir and stack ? | |
| 13:38:02 | bauzas | ok | |
| 13:38:04 | bauzas | I can try | |
| 13:38:24 | bauzas | it's a pure compute node, no controllers on it so should be small and quick | |
| 13:39:33 | dansmith | well, you could just run the plugin itself again (after cleaning the kernel dir) but re-stacking would be better | |
| 13:39:59 | bauzas | yeah | |
| 13:40:15 | bauzas | if that fails again, I'll run the plugin itself | |
| 13:40:35 | bauzas | dansmith: fwiw, I started to provide the mtty change on nova | |
| 13:41:21 | bauzas | I could have missed something (and I know I did, since I haven't modified the method that checks the mdevs when we restart, but meh not a problem for testing) but I think it's a start | |
| 13:42:14 | dansmith | cool | |
| 13:42:50 | bauzas | dansmith: failing again https://paste.opendev.org/show/bFTPQR1VBSJNQj2jrknG/ | |
| 13:43:10 | bauzas | this time, this is linking the modules | |
| 13:43:18 | bauzas | but still failing on the definitions | |
| 13:43:42 | dansmith | bauzas: show me "dpkg -l | grep linux" and "ls /usr/src" | |
| 13:43:49 | bauzas | cool | |
| 13:44:20 | dansmith | also uname -r and "ls -l /lib/modules/*/build" | |
| 13:44:45 | dansmith | worked fine for me and sean-k-mooney, but it gets weird if you have a bunch of mismatched kernels and things installed sometimes | |
| 13:45:54 | bauzas | dansmith: there you go https://paste.opendev.org/show/bSXwvpbc6Z96awnWuIRS/ | |
| 13:46:27 | bauzas | I can reclone the machine and reinstall devstack again | |
| 13:46:41 | bauzas | I mean, if my env is dirty, meh | |
| 13:46:50 | bauzas | probably better to start with fresh things | |
| 13:48:18 | dansmith | yeah that' a ton of kernel distraction, although it seems like it all lines up | |
| 13:48:19 | dansmith | however, | |
| 13:48:43 | dansmith | there are lots of reports out there about having linux-headers installed for the right kernel, but needing to reinstall it if it was installed by a previous version | |
| 13:49:07 | dansmith | not sure why, because it should be all separate | |
| 13:49:20 | dansmith | did you already have linux-headers for your kernel installed or did the first run of this install it for you? | |
| 13:52:09 | sean-k-mooney | you have a kvm variant of the kernl | |
| 13:52:19 | sean-k-mooney | so maybe thats the issue | |
| 13:52:32 | sean-k-mooney | i have Linux upstream-devstack 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20 08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux | |
| 13:52:57 | dansmith | oh is there a separate headers package for the kvm variant? if so that's definitely it | |
| 13:53:28 | bauzas | I just rebuilt the instance I have as devstack2 | |
| 13:53:32 | dansmith | ah yep | |
| 13:53:40 | dansmith | that's the problem | |
| 13:54:12 | bauzas | and yeah, if the symbols are only on the plain kernel, that explains | |
| 13:54:36 | bauzas | not sure how I got this variant, probably because of the image I used in my cloud | |
| 13:54:42 | dansmith | so linux-headers-$kver won't work for the variants, because it needs to be linux-headers-kvm I guess | |
| 13:55:01 | dansmith | bauzas: so you can switch to the normal one? | |
| 13:55:03 | bauzas | yeah that's my guess | |
| 13:55:12 | bauzas | dansmith: too late, I rebuilt it | |
| 13:55:32 | dansmith | what does that mean, you installed the right headers package and built the modules? | |
| 13:55:46 | sean-k-mooney | i think they mean reinstalled the vm | |
| 13:56:11 | dansmith | that's what I said.. switch to the normal kernel, however that needs to happen :) | |
| 13:56:19 | dansmith | either with apt or rm :) | |
| 13:56:22 | sean-k-mooney | bauzas: are you using a cloud imave or similar? i have never seen the kvm kernel actully installed in anything | |
| 13:56:37 | dansmith | yeah I'm not sure what it's for either | |
| 13:56:55 | sean-k-mooney | well its a striped down kernel without any hard specific drivers | |
| 13:56:59 | dansmith | "Targeting KVM instance usage. This carries the minimum support required for a guest kernel." | |
| 13:57:07 | sean-k-mooney | but they dont use it for the cloud image | |
| 13:57:18 | sean-k-mooney | so i dont know who the audiance for that is | |
| 13:57:25 | dansmith | yeah, probably for ultra size-conscious situations or something | |
| 13:57:28 | bauzas | sean-k-mooney: I used some internal cloud at my employer | |
| 13:58:25 | bauzas | stack@sbauza-dev2:~$ uname -r | |
| 13:58:25 | bauzas | 5.15.0-1017-kvm | |
| 13:58:32 | bauzas | okay, coming from the image itsefl | |
| 13:58:52 | bauzas | fun | |
| 13:59:09 | bauzas | I'm tempted to pick another image or download another kernel | |
| 13:59:25 | sean-k-mooney | yes just install a diffent kernel and reboot | |
| 13:59:37 | sean-k-mooney | or use the ubuntu image i upoloaded | |
| 13:59:41 | bauzas | that will be the quickiest | |
| 14:00:09 | dansmith | I can see if I can make it work either way, but yeah do the workaround for the moment | |
| 14:00:25 | sean-k-mooney | the offical ubuntu cloud images have 5.15.0-48-generic | |
| 14:01:00 | sean-k-mooney | well thats an old image so its probaly outdated but i just confirmed that so you booted form a customized image | |
| 14:02:34 | dansmith | yeah so you can't even get the support packages for very old kernels, so you need to be running something current anyway | |
| 14:02:39 | dansmith | like they remove them | |
| 14:05:44 | bauzas | hum | |
| 14:06:08 | bauzas | which kernel version is currently running on our CI jobs ? | |
| 14:06:10 | dansmith | the image in our CI gets refreshed periodically for that reason | |
| 14:06:14 | bauzas | because I have the choices | |
| 14:06:20 | sean-k-mooney | i allways do an apt dist-upgrade and reboot before running devstack for what its worth | |
| 14:06:30 | sean-k-mooney | nightly for upstream ci | |
| 14:06:46 | dansmith | bauzas: https://zuul.opendev.org/t/openstack/build/3c70446c0452434a87166d336dc0dbc5/log/job-output.txt#22580 | |
| 14:06:55 | sean-k-mooney | bauzas: just install the generic kernel | |
| 14:06:57 | dansmith | linux-headers-5.15.0-86-generic | |