| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-08-26 | |||
| 08:38:32 | auniyal | bauzas, frickler, yes, so I was not sure of https://review.opendev.org/c/openstack/releases/+/926768 | |
| 08:38:47 | auniyal | so was waiting for bauzas | |
| 09:01:25 | bauzas | no worries | |
| 10:46:04 | gibi | It seems we have a gate block on stable/2024.1 | |
| 10:46:19 | gibi | curl -f -O https://download.ceph.com/rpm-/el9/noarch/cephadm returns 404 making the ceph job fail | |
| 10:46:28 | gibi | example https://zuul.opendev.org/t/openstack/build/3be807e5dbf843ff995fbe1fdcb7fed2 | |
| 10:46:42 | gibi | elodilles: is it something known? | |
| 10:47:36 | elodilles | gibi: well, i haven't seen that yet :-o | |
| 10:48:01 | gibi | sorry not 2024.1 but 2023.2 | |
| 10:48:29 | elodilles | the link is definitely missing something after "rpm-" | |
| 10:49:30 | gibi | hm, good point | |
| 10:50:20 | elodilles | and it doesn't seem to be a constant error, at least if we look at the job history: https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack%2Fnova&branch=stable%2F2023.2&skip=0 | |
| 10:51:15 | sean-k-mooney | it should be https://download.ceph.com/rpm-quincy/ | |
| 10:51:21 | sean-k-mooney | or https://download.ceph.com/rpm-reef/ | |
| 10:51:26 | elodilles | it looks like a CEPH_RELEASE (?) variable or something like that was not set | |
| 10:51:34 | sean-k-mooney | yep | |
| 10:51:55 | sean-k-mooney | although that shoudl not block the gate just the c9s jobs | |
| 10:52:03 | sean-k-mooney | which should not be voting | |
| 10:52:23 | gibi | https://review.opendev.org/c/openstack/nova/+/926555 this reproduced the error twice in a row | |
| 10:52:49 | gibi | it happening in the nova-multi-cell jo | |
| 10:52:50 | gibi | b | |
| 10:53:12 | sean-k-mooney | why is that not using ubuntu? | |
| 10:53:59 | sean-k-mooney | oh it is | |
| 10:54:06 | sean-k-mooney | Ubuntu 22.04 | |
| 10:54:24 | sean-k-mooney | so for some reason its trying to pull the rpm version when it should be pulling the deb package | |
| 10:56:17 | gibi | hm there was error in the script before that | |
| 10:56:19 | gibi | Error: writing blob: storing blob to file "/var/tmp/storage2783954232/1": happened during read: unexpected EOF | |
| 10:56:30 | gibi | Error: writing blob: storing blob to file "/var/tmp/storage2783954232/1": happened during read: unexpected EOF | |
| 10:56:36 | gibi | https://zuul.opendev.org/t/openstack/build/3be807e5dbf843ff995fbe1fdcb7fed2/log/job-output.txt#11462 | |
| 10:57:09 | elodilles | gibi: note that the 1st error is from nova-ceph-multistore and the second error is from nova-multi-cell job | |
| 10:57:50 | elodilles | gibi: and the nova-multi-cell job failure is 2 test case failure | |
| 10:58:04 | elodilles | so it's not the same | |
| 10:58:21 | sean-k-mooney | ok cause i was looking at multi cell and not seeing any references to ceph :) | |
| 10:59:09 | elodilles | :] | |
| 11:00:46 | sean-k-mooney | oh it is | |
| 11:01:06 | sean-k-mooney | so looking at the multi store josb local.conf https://3e402c0e76741e83fc60-d00ff4f1a74cdbc5ea9d8044145b77c0.ssl.cf2.rackcdn.com/926555/2/check/nova-ceph-multistore/3be807e/controller/logs/_.localrc_auto.txt | |
| 11:01:34 | sean-k-mooney | we do not seam to be settign a release version in the job so we are using the defaults form the plugin | |
| 11:01:38 | sean-k-mooney | which should be ok | |
| 11:04:15 | sean-k-mooney | that is defiend here https://github.com/openstack/devstack-plugin-ceph/blob/stable/2023.2/devstack/lib/ceph#L31 | |
| 11:04:48 | sean-k-mooney | this is in the lib module meaing that it for internal use only otherwise it shoudl be in settings at the top of the plugin | |
| 11:05:29 | sean-k-mooney | it should still be set but lateter so its probaly fine as is | |
| 11:06:46 | sean-k-mooney | so this is just trhe podman pull that is failing an di have seen that fail at least once in the last 2 weeks | |
| 11:07:04 | sean-k-mooney | but i didnt look into why so im not sure this is jobs specific | |
| 11:08:13 | sean-k-mooney | i think the way this is coded we download the container that ceph adm would use | |
| 11:08:32 | sean-k-mooney | then we parse teh version form that and download the corresponding cephadm binary directly | |
| 11:08:35 | sean-k-mooney | and then use that | |
| 11:09:59 | sean-k-mooney | so the fact its using el9 on ubuntu does not matter https://github.com/openstack/devstack-plugin-ceph/blob/stable/2023.2/devstack/lib/cephadm#L132 | |
| 11:10:45 | sean-k-mooney | so ya this look like a generic podman pull issue with quay.io | |
| 11:11:01 | sean-k-mooney | we could perhaps retry that pull to mitigate it | |
| 11:12:12 | sean-k-mooney | although i think ideally we would mirror that or precache the container and cephadm in the base image at least for the default release | |
| 11:25:18 | frickler | iiuc we do have a caching proxy for quay.io, so someone just would have to make that job use it (cc gouthamr) | |
| 11:25:49 | sean-k-mooney | ah ok is that by configuing http_proxy ? or something else? | |
| 11:26:09 | sean-k-mooney | we have a generic setup mirrors role in the openstack-ci-jobs repo i think | |
| 11:26:23 | sean-k-mooney | i kind of assumed that also configured the http proxies | |
| 11:27:14 | frickler | there was some discussion about this in some other channel last week, iirc that work never completed. but kolla does somehow use that proxy already | |
| 11:27:15 | sean-k-mooney | ah i was thinking of https://github.com/openstack/openstack-zuul-jobs/tree/master/roles but i guess the morrior setup is in the generic zuul-jobs repo | |
| 11:28:00 | gibi | sorry for the mixup | |
| 11:28:58 | sean-k-mooney | i guess that does not handel the systm proxies | |
| 11:29:00 | sean-k-mooney | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/defaults/main.yaml | |
| 11:30:09 | frickler | sean-k-mooney: yes, afaict this is still missing for general zuul jobs. cf. https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/templates/globals-default.j2#L82-L88 | |
| 11:30:40 | sean-k-mooney | so that could be a good change to put in the base devstack job | |
| 11:30:58 | sean-k-mooney | do we know why that has not been done before? | |
| 11:32:05 | sean-k-mooney | oh this is doing it via explicly using a difent registry | |
| 11:32:06 | frickler | sean-k-mooney: iiuc this was started by some tripleo folks but never finished. https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2024-08-21.log.html#t2024-08-21T11:06:16 was the discussion I was referring to | |
| 11:32:40 | sean-k-mooney | right i tought this might be settign the generic http proxy but its acatully podman/docker speciric config | |
| 11:33:16 | frickler | yes, we cannot do a generic proxy since we could not properly secure it | |
| 11:33:24 | sean-k-mooney | i think the kolla approch need to have all usage of podman be parmatiarised by the registry | |
| 11:33:46 | sean-k-mooney | so we would need to modify the ceph devstack plugin | |
| 11:33:49 | sean-k-mooney | but that should not be hard | |
| 11:34:43 | frickler | yes, I don't think podman is currently being used anywhere else in devstack? even when it was, that should not be a blocker | |
| 11:35:35 | sean-k-mooney | it might have been used for something else but we can always do it one plugin at a time | |
| 11:35:58 | sean-k-mooney | i dont think we use it in core devstack | |
| 11:41:34 | opendevreview | Balazs Gibizer proposed openstack/nova stable/2024.1: hardware: Correct log https://review.opendev.org/c/openstack/nova/+/927110 | |
| 11:50:14 | sean-k-mooney | frickler: looks like we can either configure a mirror which will take precidnece and then fall back or we can remap prefixes | |
| 11:50:18 | sean-k-mooney | https://www.redhat.com/sysadmin/manage-container-registries | |
| 11:50:53 | sean-k-mooney | i think the mirror approch is prefered as it will try the direct connection if the mirror does not work | |
| 11:51:22 | sean-k-mooney | i can see if i can find time to code that up in core devstack | |
| 14:22:11 | opendevreview | Dan Smith proposed openstack/nova master: Use format_inspector from oslo https://review.opendev.org/c/openstack/nova/+/925025 | |
| 14:55:59 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 14:55:59 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 14:56:00 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 14:56:00 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 14:56:01 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 14:56:01 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 14:56:02 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 14:56:03 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 14:56:03 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 14:56:05 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 14:56:05 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 14:56:07 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 14:56:07 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 14:56:09 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 14:56:09 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 14:56:11 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 14:56:11 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 14:56:13 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 14:56:13 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 14:56:15 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 14:56:15 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 14:56:17 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |