| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-30 | |||
| 18:41:59 | gouthamr | melwitt: how did you arrive at that ceph orch being the problem is i guess my question | |
| 18:42:40 | melwitt | gouthamr: sorry, I didn't include info info in the bug, I did it in haste. I don't think so -- my assumption is that things failed during ceph installation, well before any tests were run. this is an example run https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f | |
| 18:42:48 | melwitt | *include enough info | |
| 18:43:26 | gouthamr | i see, it's happening on the migration job, but, that's unrelated | |
| 18:43:53 | gouthamr | melwitt: thanks.. that was my initial thought; maybe that part of the handling is just .. flaky | |
| 18:44:47 | gouthamr | melwitt: the job history is pretty decent: https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration-ceph | |
| 18:45:12 | melwitt | gouthamr: yeah I wondered if maybe something is just not waiting for 'ceph mgr module enable orchestrator' to finish or something like that? | |
| 18:45:45 | melwitt | yeah. it is definitely a "sometimes" bug but I have personally seen it a lot enough to ask someone if they know something to stop it haha | |
| 18:46:32 | gouthamr | so we don't bail out early.. | |
| 18:46:32 | gouthamr | melwitt: ack, i've added some retry logic here: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 | |
| 18:47:00 | gouthamr | disabling cephadm isn't necessary.. just something that'll help save resources if we have nothing more to setup/change on the ceph cluster | |
| 18:47:52 | melwitt | ah gotcha, makes sense | |
| 18:49:37 | gouthamr | sean-k-mooney headed in the same direction.. so i guess we can see if this helps with stability. thanks for flagging this | |
| 18:50:15 | melwitt | thanks you for looking at it :) | |
| 18:50:31 | sean-k-mooney | so i looked at the plugin breifly but didn know which job failed | |
| 18:50:41 | sean-k-mooney | so i didnt knwo which method was actully the failure | |
| 18:51:03 | sean-k-mooney | gouthamr: i saw you patch to add the retry on disable but i suspect that wasint the issue | |
| 18:51:32 | sean-k-mooney | perhaps it was but i was assomign like mel that it was happenign durign the install | |
| 18:52:15 | melwitt | I added another comment on https://bugs.launchpad.net/nova/+bug/2149965 to explain a bit more, for future reference | |
| 18:52:54 | gouthamr | sean-k-mooney: the restart you stated is an internal ceph thingy - we don't explicitly ask to restart.. the manager module reloads things each time and during that reload the API is down.. so "orch" just throws a random error: "set a backend | |
| 18:53:07 | sean-k-mooney | + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:210 | |
| 18:53:08 | gouthamr | " - that's just catching a giant exception in the client | |
| 18:53:59 | sean-k-mooney | right os we need to be more robot to the internal errors | |
| 18:54:04 | gouthamr | yes | |
| 18:54:54 | sean-k-mooney | in any case it faillign here https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/cephadm#L207 | |
| 18:55:20 | melwitt | thanks to both of you for looking despite the major lack of info on the bug 😬 | |
| 18:55:54 | sean-k-mooney | melwitt: i definlly have not seen that error several time sin the last month and not looked at it because fo tiem... | |
| 18:56:23 | sean-k-mooney | definlty not a thing that woudl happen | |
| 18:57:02 | sean-k-mooney | anyway https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 wont fix it because that is not what is failign | |
| 18:57:09 | sean-k-mooney | btu the approch might work if we use it else hwere | |
| 18:57:28 | sean-k-mooney | i dont knwo if tis just safe to invoke the same command again when we are in this state | |
| 18:57:36 | sean-k-mooney | i.e. will it just fail again | |
| 18:57:57 | melwitt | hm ok | |
| 18:58:08 | sean-k-mooney | or is that the right thing to do possibel with a sleep ro query ot see fi the thing that restarted is up | |
| 18:58:45 | sean-k-mooney | this honestly feels like a cphadm bug that should eventully get fixed there | |
| 18:59:58 | sean-k-mooney | gouthamr: i assume you wrapped https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818/1/devstack/lib/cephadm because that is the only place we call set backend | |
| 19:00:23 | sean-k-mooney | but you will noteice in teh trace back its failing when we are settting the backend to cephadm and there we are not setting it ot anythin | |
| 19:00:42 | sean-k-mooney | cli(['orch', 'set', 'backend', 'cephadm']) | |
| 19:01:00 | melwitt | so you don't think it's something to do with commands the devstack plugin is running, like it could wait for a thing to be ready first | |
| 19:01:36 | gouthamr | we'll just be yelling at it to disable orch, what's wrong with that? | |
| 19:01:41 | gouthamr | it should be idempotent | |
| 19:01:57 | sean-k-mooney | its https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f/log/job-output.txt#12495 | |
| 19:01:58 | gouthamr | https://docs.ceph.com/en/reef/mgr/orchestrator/#disable-the-orchestrator | |
| 19:02:23 | sean-k-mooney | so if you look at the logs we are failign in start_ceph | |
| 19:02:44 | sean-k-mooney | so the failign call is all internal to that initall call to cephadm | |
| 19:03:04 | sean-k-mooney | 2026-04-29 21:16:02.191617 | controller | + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:1 : exit_trap | |
| 19:03:07 | gouthamr | AH! see that's why i needed the full log :D | |
| 19:03:19 | sean-k-mooney | we trap on the execode form that call | |
| 19:04:11 | sean-k-mooney | so this is very much a race in cephadm rather then our devstack plugin | |
| 19:06:01 | gouthamr | lemme see if there's something new they invented to prevent a "bootstrap" failure | |
| 19:06:36 | sean-k-mooney | ack, we should not need to disable the orc module by the way we definlly want to use it because that is how real cluster work | |
| 19:06:51 | gouthamr | no we had to do this just for CI | |
| 19:07:45 | gouthamr | when we're spinning up single-node ceph and openstack compute on the flavor we use, we ran into OOMs | |
| 19:08:33 | gouthamr | locally, i give it a beefy-enough node, and for jobs that do more, we use multi-node and spin up instances on compute nodes | |
| 19:09:53 | sean-k-mooney | right in this case i dont think its an OOM | |
| 19:10:05 | gouthamr | yeah | |
| 19:12:11 | sean-k-mooney | https://paste.opendev.org/show/bf4uNdKhVfHSw5XRwFXS/ | |
| 19:12:41 | sean-k-mooney | well a littel more https://paste.opendev.org/show/bClxtqx7PviIQSfdnEkq/ | |
| 19:12:53 | gouthamr | okay, we can just retry the bootstrap until it succeeds... there are some opts that can help it be idempotent; check if mon is up, --allow-overwrite so we can be okay with cruft | |
| 19:13:35 | gouthamr | yeah, that message isn't helpful there too :/ we can't enable orch, we're trying to bootstrap $machine ¯\_(ツ)_/¯ | |
| 19:13:57 | gouthamr | it's as good as "something unexpected happened" | |
| 19:14:24 | gouthamr | i can report a ceph tracker if they don't already know about this issue | |
| 19:14:37 | sean-k-mooney | but is it as good as "no valid host found" | |
| 19:14:48 | sean-k-mooney | nova most useful of uesful errors | |
| 19:15:04 | gouthamr | :D | |
| 21:55:09 | opendevreview | Merged openstack/nova master: Use python-native keyword-only arguments https://review.opendev.org/c/openstack/nova/+/969050 | |
| #openstack-nova - 2026-05-01 | |||
| 07:05:09 | opendevreview | Alokedip Choudhuri proposed openstack/nova master: Nova driver changes for InfoScale Cinder Driver https://review.opendev.org/c/openstack/nova/+/986071 | |
| 10:09:35 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 14:24:28 | opendevreview | Merged openstack/os-vif master: Drop support for Python 3.9 and add 3.13 https://review.opendev.org/c/openstack/os-vif/+/976272 | |
| 14:27:23 | opendevreview | Stephen Finucane proposed openstack/os-vif master: pre-commit: Bump versions https://review.opendev.org/c/openstack/os-vif/+/976358 | |
| 14:27:24 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/os-vif/+/976359 | |
| 14:27:25 | opendevreview | Stephen Finucane proposed openstack/os-vif master: tests: Remove errant parameters https://review.opendev.org/c/openstack/os-vif/+/976360 | |
| 14:27:26 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Use bit shifting https://review.opendev.org/c/openstack/os-vif/+/976361 | |
| 14:27:28 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add initial type annotations https://review.opendev.org/c/openstack/os-vif/+/745162 | |
| 14:27:30 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_noop module https://review.opendev.org/c/openstack/os-vif/+/976362 | |
| 14:27:32 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 14:27:34 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Return Self from PluginBase.load https://review.opendev.org/c/openstack/os-vif/+/986890 | |
| 14:27:36 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Inline add_keepalives decorator https://review.opendev.org/c/openstack/os-vif/+/986891 | |
| 14:27:38 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Specify args to OvsPlugin._create_vif_port https://review.opendev.org/c/openstack/os-vif/+/986892 | |
| 15:04:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | |
| 15:57:29 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 15:57:29 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 15:57:37 | opendevreview | Antonin Ruan proposed openstack/nova-specs master: Temp URL download spec https://review.opendev.org/c/openstack/nova-specs/+/975883 | |
| 16:51:11 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 16:51:11 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 16:51:52 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 16:51:52 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 17:07:34 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Convert add_keepalives to function https://review.opendev.org/c/openstack/os-vif/+/986891 | |
| 17:07:35 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Specify args to OvsPlugin._create_vif_port https://review.opendev.org/c/openstack/os-vif/+/986892 | |
| 17:07:36 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 19:38:48 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Update policy scope enforcement testing https://review.opendev.org/c/openstack/nova/+/986946 | |
| 19:39:57 | opendevreview | Ghanshyam Maan proposed openstack/nova master: DNM: Test removal of enforce_scope https://review.opendev.org/c/openstack/nova/+/986479 | |
| 20:35:27 | opendevreview | Merged openstack/nova master: Replace deprecated glanceclient exception https://review.opendev.org/c/openstack/nova/+/914761 | |
| 21:12:24 | opendevreview | Merged openstack/nova master: [py313-threading]Skip eventlet specific tests https://review.opendev.org/c/openstack/nova/+/984608 | |
| #openstack-nova - 2026-05-02 | |||
| 00:21:06 | opendevreview | Merged openstack/os-vif master: pre-commit: Bump versions https://review.opendev.org/c/openstack/os-vif/+/976358 | |
| 13:36:31 | opendevreview | Merged openstack/os-vif master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/os-vif/+/976359 | |
| 15:37:18 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 15:37:39 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 16:21:48 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 16:35:41 | opendevreview | Takashi Kajinami proposed openstack/nova master: Do not use format string in logs https://review.opendev.org/c/openstack/nova/+/987016 | |