| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-25 | |||
| 13:51:22 | dansmith | eharney: because right now the ceph jobs are all totally wedged because a bunch of packages have just banned anything python<=3.8 | |
| 13:51:40 | eharney | dansmith: not sure who but i think some manila folks have been more active in devstack-plugin-ceph recently | |
| 13:51:41 | dansmith | reverts are in place or in flight for some of them, but we need to get it resolved | |
| 13:51:43 | eharney | ahh | |
| 13:51:53 | dansmith | because focal is not in our PTI right now... | |
| 13:52:06 | dansmith | eharney: so maybe gouthamr ? | |
| 13:57:07 | sean-k-mooney | dansmith: did you notice the ceph nfs job passed? https://zuul.opendev.org/t/openstack/build/3f0ec49fa9e048f6a73e6c03833fecc9/logs | |
| 13:57:31 | dansmith | sean-k-mooney: I assume that's not using the qemu block driver | |
| 13:57:51 | sean-k-mooney | perhaps although that is an odd issue | |
| 13:57:53 | dansmith | sean-k-mooney: the actual setup of ceph seemed to work on jammy, it's just that qemu doesn't have the block driver to load to talk to it (or something) | |
| 13:58:05 | sean-k-mooney | they may have chagned that to a weak dep in the qemu packaging | |
| 13:58:13 | sean-k-mooney | so we might need to install it in devstack explictly | |
| 13:58:19 | dansmith | it seemed like maybe those were provided in the ceph packages instead of qemu or something | |
| 13:58:28 | dansmith | sean-k-mooney: well, I looked and couldn't find any package in jammy for it | |
| 13:58:37 | dansmith | sean-k-mooney: but yes, hopefully it's something simple | |
| 13:58:48 | sean-k-mooney | i assumed it was complied in if im being honest but didnt look | |
| 13:59:01 | dansmith | me too | |
| 13:59:08 | dansmith | until I saw the logs.. let me get you a link | |
| 13:59:20 | dansmith | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/1435ec8b57be4b159c3d373e23673ab5/log/controller/logs/screen-n-cpu.txt#8437 | |
| 14:00:04 | dansmith | it's also weirdly named "block-block-rbd".. perhaps there's some change in jammy with something and someone is prefixing an extra... prefix/ | |
| 14:00:23 | dansmith | although it says "unknown driver rbd" on the next line, which looks right | |
| 14:00:32 | sean-k-mooney | there nova package just depend on qemu-system https://packages.ubuntu.com/jammy/nova-compute-qemu | |
| 14:00:44 | bauzas | dansmith: sean-k-mooney: so, to clarify, once we merge https://review.opendev.org/c/openstack/nova/+/881409 | |
| 14:01:06 | bauzas | we will still have a problem with ceph-multistore due to keystone + cephadm right? | |
| 14:01:12 | dansmith | bauzas: yes | |
| 14:01:16 | sean-k-mooney | hum that is not linked agains librbd | |
| 14:01:19 | sean-k-mooney | https://packages.ubuntu.com/jammy/qemu-system-x86 | |
| 14:01:28 | dansmith | bauzas: not "due to cephadm" | |
| 14:01:34 | sean-k-mooney | ah | |
| 14:01:36 | dansmith | bauzas: just the ceph job | |
| 14:01:36 | sean-k-mooney | https://packages.ubuntu.com/jammy/qemu-block-extra | |
| 14:01:44 | sean-k-mooney | dansmith: we are misisng qemu-block-extra | |
| 14:01:46 | bauzas | because of keystone ? | |
| 14:01:55 | sean-k-mooney | that is what provides the rbd supprot | |
| 14:02:06 | dansmith | sean-k-mooney: ack, nice | |
| 14:02:19 | sean-k-mooney | and just checked its a recommended package | |
| 14:02:26 | sean-k-mooney | not a dep | |
| 14:02:26 | eharney | ahh, cool, was just looking for that missing link myself | |
| 14:02:31 | dansmith | sean-k-mooney: so we likely need a devstack ceph plugin change, let me push that up and try a dep | |
| 14:03:12 | sean-k-mooney | ya so we coudl also do this via bindep in nova as an optional dep . perhaps we shoudl as a follow up | |
| 14:03:26 | sean-k-mooney | i was just going to look at devstack to see how it installes qemu/libvirt | |
| 14:03:44 | sean-k-mooney | and add it there but doing it in the devstack plugin proably makes more sense | |
| 14:03:45 | eharney | it can go in devstack-plugin-ceph/devstack/files/debs i think | |
| 14:03:47 | dansmith | sean-k-mooney: is that package also in focal? | |
| 14:04:16 | sean-k-mooney | yep | |
| 14:04:27 | sean-k-mooney | well im waiting fo the page to refrsh | |
| 14:04:32 | dansmith | it is | |
| 14:04:35 | sean-k-mooney | but it looks like ti shoudl be there form bionic | |
| 14:04:38 | dansmith | I confirmed | |
| 14:05:31 | dansmith | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/881479 | |
| 14:05:44 | dansmith | I'll fix up the revert-focal devstack patch to depend on it | |
| 14:05:58 | sean-k-mooney | so they moved it form suggetes to recommends from focal to jammy | |
| 14:06:21 | dansmith | ah | |
| 14:07:37 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/nova_plugins/functions-libvirt#L72 | |
| 14:07:47 | sean-k-mooney | so i was thinkign of just adding hit there to be honest | |
| 14:08:16 | sean-k-mooney | just alwasy install qemu-block-extra in base devstack | |
| 14:08:29 | dansmith | the ceph plugin already has a list of specific-to-it deb packages.. so it needs to go there anyway, IMHO | |
| 14:08:48 | sean-k-mooney | sure | |
| 14:09:03 | sean-k-mooney | im just wonderign what else is ther | |
| 14:09:11 | sean-k-mooney | it look like libisci is also there | |
| 14:09:14 | dansmith | xfsprogs | |
| 14:09:19 | dansmith | although I dunno why | |
| 14:09:27 | dansmith | anyway, let's see if this works | |
| 14:09:31 | sean-k-mooney | aho ok we dont need it for isci because we host mount | |
| 14:09:34 | sean-k-mooney | for multipath | |
| 14:09:57 | dansmith | ah, iscsi is in block-extra as well? | |
| 14:10:24 | sean-k-mooney | yes but only if you use qemu to directly conenct to the iscsi backend which we nolonger do. i dont think we use gluserfs either | |
| 14:10:28 | dansmith | right | |
| 14:35:12 | dansmith | sean-k-mooney: eharney okay it installed the qemu-block-extra package successfully, so that's good.. we'll see if it can actually boot instances once it gets there :) | |
| 14:49:46 | dansmith | looks like it might be booting servers ... | |
| 15:25:08 | bauzas | reminder : nova meeting in 35 mins | |
| 15:34:14 | bauzas | sean-k-mooney: dansmith: I'm about to communicate back to the ML saying that the gate is unblocked, amirite ? | |
| 15:34:24 | dansmith | no | |
| 15:34:25 | sean-k-mooney | no | |
| 15:34:30 | dansmith | we have a ways to go before that | |
| 15:34:30 | sean-k-mooney | none fo the patches are merged | |
| 15:34:42 | dansmith | I have to squash my fix into the main one, since it can't merge on its own | |
| 15:34:56 | dansmith | and we'll have to recheck the main one anyway since it failed more volume attach things | |
| 15:35:02 | dansmith | so we still got a bit | |
| 15:35:13 | bauzas | because other libs are capping >=3.9 ? | |
| 15:35:30 | bauzas | I thought neutron did the revert | |
| 15:36:13 | dansmith | bauzas: so many more problems than that dude :) | |
| 15:36:27 | dansmith | now pysaml2 has gone >3.8 which is breaking everyone | |
| 15:36:32 | bauzas | oh | |
| 15:36:41 | dansmith | that revert may land and we might be okay, but there might be others too | |
| 15:36:52 | bauzas | yeah, I got the memo but I didn't read it correctly | |
| 15:37:08 | bauzas | okay, then I'll write some status email then | |
| 15:37:31 | sean-k-mooney | i think we are just goign to mvoe everytin to jammy and drop the 3.8 testing | |
| 15:37:36 | bauzas | saying we can't guarantee that any of the deps are still able to use py3.8 | |
| 15:37:44 | sean-k-mooney | assumign the ceph job evenutally passes without the detach error | |
| 15:37:53 | bauzas | sean-k-mooney: that was my thought | |
| 15:38:11 | bauzas | https://review.opendev.org/c/openstack/nova/+/881409 should help | |
| 15:39:56 | sean-k-mooney | https://review.opendev.org/q/topic:drop-py38 is the full set of fixes | |
| 15:40:25 | sean-k-mooney | we basically need to revert the py39 change out of https://review.opendev.org/c/openstack/nova/+/881339 and incopreate the ceph job fixes instead | |
| 15:41:23 | sean-k-mooney | so https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/881479 and https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 need to be squased | |
| 15:41:46 | sean-k-mooney | and https://review.opendev.org/c/openstack/nova/+/881339 need to depend on the squahsed commit | |
| 15:42:30 | sean-k-mooney | dansmith: those were the two patches you planned to squash right? | |
| 15:42:40 | bauzas | sean-k-mooney: thanks, I'll explain it | |
| 15:42:50 | dansmith | yes | |
| 15:42:59 | dansmith | I'm waiting for the run to finish so I can look at the failures | |