| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-25 | |||
| 13:19:08 | dansmith | yeah that ^ | |
| 13:19:19 | dansmith | that's what I failed on | |
| 13:19:24 | dansmith | cripes | |
| 13:19:49 | ykarel | revert patch https://review.opendev.org/c/openstack/requirements/+/881466 | |
| 13:21:11 | bauzas | ykarel: last time I looked, this was due to the fact that py39 wanted UUIDs | |
| 13:35:38 | ykarel | bauzas, sorry which issue? /me can't relate to UUIDs | |
| 13:39:13 | dansmith | yeah not sure what the uuid thing is | |
| 13:41:11 | bauzas | it was when elodilles tried to update the py version for ceph-multinode | |
| 13:41:21 | bauzas | but I can be wrong | |
| 13:41:32 | dansmith | oh, unrelated to pysaml I see | |
| 13:41:47 | ykarel | ack | |
| 13:41:57 | bauzas | speaking of https://1742443592df6307c50f-d94079506d616d2cb38d2b70f09b441e.ssl.cf2.rackcdn.com/881339/3/check/nova-ceph-multistore/5f269ed/controller/logs/devstacklog.txt | |
| 13:42:15 | bauzas | 2023-04-24 11:48:22.720 | [01;33mWARNING py.warnings [[01;36mNone req-518808ec-0a14-444d-b6d3-fb6383d03e9c [00;36mNone None[01;33m] [01;35m[01;33m/usr/local/lib/python3.9/dist-packages/pycadf/identifier.py:71: UserWarning: Invalid uuid: RegionOne. To ensure interoperability, identifiers should be a valid uuid. | |
| 13:43:34 | dansmith | eharney: are you around to talk about the ceph job stuff? I think you were merging some of those cephadm changes | |
| 13:43:44 | ykarel | ack got it, it's seperate thing | |
| 13:45:12 | eharney | dansmith: yes | |
| 13:45:41 | eharney | dansmith: what's going on there? | |
| 13:45:45 | dansmith | eharney: so, is that how we should be moving the upstream jobs at this point? jammy has quincy ceph packages, as I understand it | |
| 13:45:53 | dansmith | and the cephadm jobs don't work either | |
| 13:47:06 | dansmith | and they're nonvoting so I assumed that meant they weren't "ready" | |
| 13:47:25 | dansmith | eharney: anyway, the context is, we desperately need to get the ceph jobs running on jammy | |
| 13:48:03 | eharney | i was trying to understand the situation with these the other day, i guess they were pinned to focal before because there weren't jammy packages, but maybe now there are? | |
| 13:49:26 | dansmith | eharney: yeah that's the reason for the pinning, but indeed now there seem to be base packages in jammy for quincy | |
| 13:49:40 | dansmith | I rechecked the unpin patch yesterday and it failed with qemu not having the rbd block driver | |
| 13:49:53 | dansmith | eharney: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315?tab=change-view-tab-header-zuul-results-summary | |
| 13:50:47 | eharney | hmm, looking, i'm not really up to speed on this | |
| 13:50:57 | dansmith | eharney: okay is there someone else we should bug? | |
| 13:51:22 | dansmith | eharney: because right now the ceph jobs are all totally wedged because a bunch of packages have just banned anything python<=3.8 | |
| 13:51:40 | eharney | dansmith: not sure who but i think some manila folks have been more active in devstack-plugin-ceph recently | |
| 13:51:41 | dansmith | reverts are in place or in flight for some of them, but we need to get it resolved | |
| 13:51:43 | eharney | ahh | |
| 13:51:53 | dansmith | because focal is not in our PTI right now... | |
| 13:52:06 | dansmith | eharney: so maybe gouthamr ? | |
| 13:57:07 | sean-k-mooney | dansmith: did you notice the ceph nfs job passed? https://zuul.opendev.org/t/openstack/build/3f0ec49fa9e048f6a73e6c03833fecc9/logs | |
| 13:57:31 | dansmith | sean-k-mooney: I assume that's not using the qemu block driver | |
| 13:57:51 | sean-k-mooney | perhaps although that is an odd issue | |
| 13:57:53 | dansmith | sean-k-mooney: the actual setup of ceph seemed to work on jammy, it's just that qemu doesn't have the block driver to load to talk to it (or something) | |
| 13:58:05 | sean-k-mooney | they may have chagned that to a weak dep in the qemu packaging | |
| 13:58:13 | sean-k-mooney | so we might need to install it in devstack explictly | |
| 13:58:19 | dansmith | it seemed like maybe those were provided in the ceph packages instead of qemu or something | |
| 13:58:28 | dansmith | sean-k-mooney: well, I looked and couldn't find any package in jammy for it | |
| 13:58:37 | dansmith | sean-k-mooney: but yes, hopefully it's something simple | |
| 13:58:48 | sean-k-mooney | i assumed it was complied in if im being honest but didnt look | |
| 13:59:01 | dansmith | me too | |
| 13:59:08 | dansmith | until I saw the logs.. let me get you a link | |
| 13:59:20 | dansmith | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/1435ec8b57be4b159c3d373e23673ab5/log/controller/logs/screen-n-cpu.txt#8437 | |
| 14:00:04 | dansmith | it's also weirdly named "block-block-rbd".. perhaps there's some change in jammy with something and someone is prefixing an extra... prefix/ | |
| 14:00:23 | dansmith | although it says "unknown driver rbd" on the next line, which looks right | |
| 14:00:32 | sean-k-mooney | there nova package just depend on qemu-system https://packages.ubuntu.com/jammy/nova-compute-qemu | |
| 14:00:44 | bauzas | dansmith: sean-k-mooney: so, to clarify, once we merge https://review.opendev.org/c/openstack/nova/+/881409 | |
| 14:01:06 | bauzas | we will still have a problem with ceph-multistore due to keystone + cephadm right? | |
| 14:01:12 | dansmith | bauzas: yes | |
| 14:01:16 | sean-k-mooney | hum that is not linked agains librbd | |
| 14:01:19 | sean-k-mooney | https://packages.ubuntu.com/jammy/qemu-system-x86 | |
| 14:01:28 | dansmith | bauzas: not "due to cephadm" | |
| 14:01:34 | sean-k-mooney | ah | |
| 14:01:36 | sean-k-mooney | https://packages.ubuntu.com/jammy/qemu-block-extra | |
| 14:01:36 | dansmith | bauzas: just the ceph job | |
| 14:01:44 | sean-k-mooney | dansmith: we are misisng qemu-block-extra | |
| 14:01:46 | bauzas | because of keystone ? | |
| 14:01:55 | sean-k-mooney | that is what provides the rbd supprot | |
| 14:02:06 | dansmith | sean-k-mooney: ack, nice | |
| 14:02:19 | sean-k-mooney | and just checked its a recommended package | |
| 14:02:26 | eharney | ahh, cool, was just looking for that missing link myself | |
| 14:02:26 | sean-k-mooney | not a dep | |
| 14:02:31 | dansmith | sean-k-mooney: so we likely need a devstack ceph plugin change, let me push that up and try a dep | |
| 14:03:12 | sean-k-mooney | ya so we coudl also do this via bindep in nova as an optional dep . perhaps we shoudl as a follow up | |
| 14:03:26 | sean-k-mooney | i was just going to look at devstack to see how it installes qemu/libvirt | |
| 14:03:44 | sean-k-mooney | and add it there but doing it in the devstack plugin proably makes more sense | |
| 14:03:45 | eharney | it can go in devstack-plugin-ceph/devstack/files/debs i think | |
| 14:03:47 | dansmith | sean-k-mooney: is that package also in focal? | |
| 14:04:16 | sean-k-mooney | yep | |
| 14:04:27 | sean-k-mooney | well im waiting fo the page to refrsh | |
| 14:04:32 | dansmith | it is | |
| 14:04:35 | sean-k-mooney | but it looks like ti shoudl be there form bionic | |
| 14:04:38 | dansmith | I confirmed | |
| 14:05:31 | dansmith | https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/881479 | |
| 14:05:44 | dansmith | I'll fix up the revert-focal devstack patch to depend on it | |
| 14:05:58 | sean-k-mooney | so they moved it form suggetes to recommends from focal to jammy | |
| 14:06:21 | dansmith | ah | |
| 14:07:37 | sean-k-mooney | https://github.com/openstack/devstack/blob/master/lib/nova_plugins/functions-libvirt#L72 | |
| 14:07:47 | sean-k-mooney | so i was thinkign of just adding hit there to be honest | |
| 14:08:16 | sean-k-mooney | just alwasy install qemu-block-extra in base devstack | |
| 14:08:29 | dansmith | the ceph plugin already has a list of specific-to-it deb packages.. so it needs to go there anyway, IMHO | |
| 14:08:48 | sean-k-mooney | sure | |
| 14:09:03 | sean-k-mooney | im just wonderign what else is ther | |
| 14:09:11 | sean-k-mooney | it look like libisci is also there | |
| 14:09:14 | dansmith | xfsprogs | |
| 14:09:19 | dansmith | although I dunno why | |
| 14:09:27 | dansmith | anyway, let's see if this works | |
| 14:09:31 | sean-k-mooney | aho ok we dont need it for isci because we host mount | |
| 14:09:34 | sean-k-mooney | for multipath | |
| 14:09:57 | dansmith | ah, iscsi is in block-extra as well? | |
| 14:10:24 | sean-k-mooney | yes but only if you use qemu to directly conenct to the iscsi backend which we nolonger do. i dont think we use gluserfs either | |
| 14:10:28 | dansmith | right | |
| 14:35:12 | dansmith | sean-k-mooney: eharney okay it installed the qemu-block-extra package successfully, so that's good.. we'll see if it can actually boot instances once it gets there :) | |
| 14:49:46 | dansmith | looks like it might be booting servers ... | |
| 15:25:08 | bauzas | reminder : nova meeting in 35 mins | |
| 15:34:14 | bauzas | sean-k-mooney: dansmith: I'm about to communicate back to the ML saying that the gate is unblocked, amirite ? | |
| 15:34:24 | dansmith | no | |