| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-30 | |||
| 18:54:04 | gouthamr | yes | |
| 18:54:54 | sean-k-mooney | in any case it faillign here https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/cephadm#L207 | |
| 18:55:20 | melwitt | thanks to both of you for looking despite the major lack of info on the bug 😬 | |
| 18:55:54 | sean-k-mooney | melwitt: i definlly have not seen that error several time sin the last month and not looked at it because fo tiem... | |
| 18:56:23 | sean-k-mooney | definlty not a thing that woudl happen | |
| 18:57:02 | sean-k-mooney | anyway https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 wont fix it because that is not what is failign | |
| 18:57:09 | sean-k-mooney | btu the approch might work if we use it else hwere | |
| 18:57:28 | sean-k-mooney | i dont knwo if tis just safe to invoke the same command again when we are in this state | |
| 18:57:36 | sean-k-mooney | i.e. will it just fail again | |
| 18:57:57 | melwitt | hm ok | |
| 18:58:08 | sean-k-mooney | or is that the right thing to do possibel with a sleep ro query ot see fi the thing that restarted is up | |
| 18:58:45 | sean-k-mooney | this honestly feels like a cphadm bug that should eventully get fixed there | |
| 18:59:58 | sean-k-mooney | gouthamr: i assume you wrapped https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818/1/devstack/lib/cephadm because that is the only place we call set backend | |
| 19:00:23 | sean-k-mooney | but you will noteice in teh trace back its failing when we are settting the backend to cephadm and there we are not setting it ot anythin | |
| 19:00:42 | sean-k-mooney | cli(['orch', 'set', 'backend', 'cephadm']) | |
| 19:01:00 | melwitt | so you don't think it's something to do with commands the devstack plugin is running, like it could wait for a thing to be ready first | |
| 19:01:36 | gouthamr | we'll just be yelling at it to disable orch, what's wrong with that? | |
| 19:01:41 | gouthamr | it should be idempotent | |
| 19:01:57 | sean-k-mooney | its https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f/log/job-output.txt#12495 | |
| 19:01:58 | gouthamr | https://docs.ceph.com/en/reef/mgr/orchestrator/#disable-the-orchestrator | |
| 19:02:23 | sean-k-mooney | so if you look at the logs we are failign in start_ceph | |
| 19:02:44 | sean-k-mooney | so the failign call is all internal to that initall call to cephadm | |
| 19:03:04 | sean-k-mooney | 2026-04-29 21:16:02.191617 | controller | + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:1 : exit_trap | |
| 19:03:07 | gouthamr | AH! see that's why i needed the full log :D | |
| 19:03:19 | sean-k-mooney | we trap on the execode form that call | |
| 19:04:11 | sean-k-mooney | so this is very much a race in cephadm rather then our devstack plugin | |
| 19:06:01 | gouthamr | lemme see if there's something new they invented to prevent a "bootstrap" failure | |
| 19:06:36 | sean-k-mooney | ack, we should not need to disable the orc module by the way we definlly want to use it because that is how real cluster work | |
| 19:06:51 | gouthamr | no we had to do this just for CI | |
| 19:07:45 | gouthamr | when we're spinning up single-node ceph and openstack compute on the flavor we use, we ran into OOMs | |
| 19:08:33 | gouthamr | locally, i give it a beefy-enough node, and for jobs that do more, we use multi-node and spin up instances on compute nodes | |
| 19:09:53 | sean-k-mooney | right in this case i dont think its an OOM | |
| 19:10:05 | gouthamr | yeah | |
| 19:12:11 | sean-k-mooney | https://paste.opendev.org/show/bf4uNdKhVfHSw5XRwFXS/ | |
| 19:12:41 | sean-k-mooney | well a littel more https://paste.opendev.org/show/bClxtqx7PviIQSfdnEkq/ | |
| 19:12:53 | gouthamr | okay, we can just retry the bootstrap until it succeeds... there are some opts that can help it be idempotent; check if mon is up, --allow-overwrite so we can be okay with cruft | |
| 19:13:35 | gouthamr | yeah, that message isn't helpful there too :/ we can't enable orch, we're trying to bootstrap $machine ¯\_(ツ)_/¯ | |
| 19:13:57 | gouthamr | it's as good as "something unexpected happened" | |
| 19:14:24 | gouthamr | i can report a ceph tracker if they don't already know about this issue | |
| 19:14:37 | sean-k-mooney | but is it as good as "no valid host found" | |
| 19:14:48 | sean-k-mooney | nova most useful of uesful errors | |
| 19:15:04 | gouthamr | :D | |
| 21:55:09 | opendevreview | Merged openstack/nova master: Use python-native keyword-only arguments https://review.opendev.org/c/openstack/nova/+/969050 | |
| #openstack-nova - 2026-05-01 | |||
| 07:05:09 | opendevreview | Alokedip Choudhuri proposed openstack/nova master: Nova driver changes for InfoScale Cinder Driver https://review.opendev.org/c/openstack/nova/+/986071 | |
| 10:09:35 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 14:24:28 | opendevreview | Merged openstack/os-vif master: Drop support for Python 3.9 and add 3.13 https://review.opendev.org/c/openstack/os-vif/+/976272 | |
| 14:27:23 | opendevreview | Stephen Finucane proposed openstack/os-vif master: pre-commit: Bump versions https://review.opendev.org/c/openstack/os-vif/+/976358 | |
| 14:27:24 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/os-vif/+/976359 | |
| 14:27:25 | opendevreview | Stephen Finucane proposed openstack/os-vif master: tests: Remove errant parameters https://review.opendev.org/c/openstack/os-vif/+/976360 | |
| 14:27:26 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Use bit shifting https://review.opendev.org/c/openstack/os-vif/+/976361 | |
| 14:27:28 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add initial type annotations https://review.opendev.org/c/openstack/os-vif/+/745162 | |
| 14:27:30 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_noop module https://review.opendev.org/c/openstack/os-vif/+/976362 | |
| 14:27:32 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 14:27:34 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Return Self from PluginBase.load https://review.opendev.org/c/openstack/os-vif/+/986890 | |
| 14:27:36 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Inline add_keepalives decorator https://review.opendev.org/c/openstack/os-vif/+/986891 | |
| 14:27:38 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Specify args to OvsPlugin._create_vif_port https://review.opendev.org/c/openstack/os-vif/+/986892 | |
| 15:04:03 | opendevreview | Balazs Gibizer proposed openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | |
| 15:57:29 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 15:57:29 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 15:57:37 | opendevreview | Antonin Ruan proposed openstack/nova-specs master: Temp URL download spec https://review.opendev.org/c/openstack/nova-specs/+/975883 | |
| 16:51:11 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 16:51:11 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 16:51:52 | opendevreview | Stephen Finucane proposed openstack/nova master: zuul: Bump functional job to py314 https://review.opendev.org/c/openstack/nova/+/986942 | |
| 16:51:52 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop Python 3.10 https://review.opendev.org/c/openstack/nova/+/986881 | |
| 17:07:34 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Convert add_keepalives to function https://review.opendev.org/c/openstack/os-vif/+/986891 | |
| 17:07:35 | opendevreview | Stephen Finucane proposed openstack/os-vif master: trivial: Specify args to OvsPlugin._create_vif_port https://review.opendev.org/c/openstack/os-vif/+/986892 | |
| 17:07:36 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 19:38:48 | opendevreview | Ghanshyam Maan proposed openstack/nova master: Update policy scope enforcement testing https://review.opendev.org/c/openstack/nova/+/986946 | |
| 19:39:57 | opendevreview | Ghanshyam Maan proposed openstack/nova master: DNM: Test removal of enforce_scope https://review.opendev.org/c/openstack/nova/+/986479 | |
| 20:35:27 | opendevreview | Merged openstack/nova master: Replace deprecated glanceclient exception https://review.opendev.org/c/openstack/nova/+/914761 | |
| 21:12:24 | opendevreview | Merged openstack/nova master: [py313-threading]Skip eventlet specific tests https://review.opendev.org/c/openstack/nova/+/984608 | |
| #openstack-nova - 2026-05-02 | |||
| 00:21:06 | opendevreview | Merged openstack/os-vif master: pre-commit: Bump versions https://review.opendev.org/c/openstack/os-vif/+/976358 | |
| 13:36:31 | opendevreview | Merged openstack/os-vif master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/os-vif/+/976359 | |
| 15:37:18 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 15:37:39 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 16:21:48 | opendevreview | Takashi Kajinami proposed openstack/nova master: AMD SEV: omit iommu='on' for virtio devices https://review.opendev.org/c/openstack/nova/+/909635 | |
| 16:35:41 | opendevreview | Takashi Kajinami proposed openstack/nova master: Do not use format string in logs https://review.opendev.org/c/openstack/nova/+/987016 | |
| #openstack-nova - 2026-05-03 | |||
| 05:33:58 | opendevreview | Takashi Kajinami proposed openstack/nova master: Replace OVN Metadata agent with OVN agent https://review.opendev.org/c/openstack/nova/+/987042 | |
| 05:34:25 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Replace OVN Metadata agent with OVN agent https://review.opendev.org/c/openstack/os-vif/+/987043 | |
| 05:36:13 | opendevreview | Takashi Kajinami proposed openstack/os-vif master: Replace OVN Metadata agent with OVN agent https://review.opendev.org/c/openstack/os-vif/+/987043 | |
| 05:36:19 | opendevreview | Takashi Kajinami proposed openstack/nova master: Replace OVN Metadata agent with OVN agent https://review.opendev.org/c/openstack/nova/+/987042 | |
| 06:53:44 | opendevreview | Merged openstack/os-vif master: tests: Remove errant parameters https://review.opendev.org/c/openstack/os-vif/+/976360 | |
| 10:59:04 | opendevreview | Merged openstack/os-vif master: trivial: Use bit shifting https://review.opendev.org/c/openstack/os-vif/+/976361 | |
| 10:59:37 | opendevreview | Merged openstack/os-vif master: typing: Add initial type annotations https://review.opendev.org/c/openstack/os-vif/+/745162 | |
| 11:04:22 | opendevreview | Merged openstack/os-vif master: typing: Add typing to vif_plug_noop module https://review.opendev.org/c/openstack/os-vif/+/976362 | |
| 16:57:22 | opendevreview | Stephen Finucane proposed openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 23:07:52 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 23:07:53 | opendevreview | Ashish Gupta proposed openstack/nova master: testing https://review.opendev.org/c/openstack/nova/+/987110 | |
| 23:07:53 | opendevreview | Ashish Gupta proposed openstack/nova master: Disable greenlet-based notifier sender check under native threading temporarily https://review.opendev.org/c/openstack/nova/+/985621 | |
| #openstack-nova - 2026-05-04 | |||
| 03:58:09 | opendevreview | Ashish Gupta proposed openstack/nova master: Disable greenlet-based notifier sender check under native threading temporarily https://review.opendev.org/c/openstack/nova/+/985621 | |
| 03:58:09 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 03:58:10 | opendevreview | Ashish Gupta proposed openstack/nova master: testing https://review.opendev.org/c/openstack/nova/+/987110 | |
| 05:43:51 | opendevreview | Merged openstack/python-novaclient master: Remove deprecation warning workaround https://review.opendev.org/c/openstack/python-novaclient/+/985993 | |
| 06:15:39 | opendevreview | Ashish Gupta proposed openstack/nova master: Run functional test with threading mode https://review.opendev.org/c/openstack/nova/+/979850 | |
| 06:15:40 | opendevreview | Ashish Gupta proposed openstack/nova master: testing https://review.opendev.org/c/openstack/nova/+/987110 | |
| 06:15:40 | opendevreview | Ashish Gupta proposed openstack/nova master: Disable greenlet-based notifier sender check under native threading temporarily https://review.opendev.org/c/openstack/nova/+/985621 | |
| 13:09:26 | opendevreview | Khoi proposed openstack/nova master: Allow removing stale aggregate hosts https://review.opendev.org/c/openstack/nova/+/987168 | |
| 13:11:16 | opendevreview | Khoi proposed openstack/nova master: Allow removing stale aggregate hosts https://review.opendev.org/c/openstack/nova/+/987168 | |
| 13:11:43 | opendevreview | Khoi proposed openstack/nova master: Allow removing stale aggregate hosts https://review.opendev.org/c/openstack/nova/+/987168 | |
| 13:12:18 | opendevreview | Khoi proposed openstack/nova master: Allow removing stale aggregate hosts https://review.opendev.org/c/openstack/nova/+/987168 | |