| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-04-30 | |||
| 16:27:10 | gibi | OK that is fair | |
| 16:27:35 | gibi | key works fo rme | |
| 16:27:38 | gibi | *for me | |
| 16:27:51 | gibi | we even save a char in each def :D | |
| 16:29:33 | dansmith | ack, I'm not too concerned over the char of course, but the meaning | |
| 16:30:22 | gibi | I know, I just entertain myself with the saving of chars :D | |
| 16:31:13 | dansmith | and I guess we could have more than one alias reference a type of group? I'm not sure why really, but I assume that's "desirable" | |
| 16:31:43 | dansmith | ugh, I'm getting 403 from docs.o.o | |
| 16:32:52 | gibi | yes two alias can refer to the same group_type but can asks for different traits, and two groups in the same group_type can have two different set of traits. I have no good example why this would be useful but the current alias syntax allows it today already | |
| 16:33:06 | gibi | ohh I have a good example | |
| 16:33:18 | gibi | two alias can differ in numa_policy | |
| 16:33:39 | dansmith | yeah okay | |
| 16:33:41 | gibi | so you want two GPUs but one alias give with with NUMA affinity required the other just preferres | |
| 16:33:45 | gibi | * preferred | |
| 16:35:03 | dansmith | I wish that we could have alias reference device_spec or group_spec things instead of the devices (by address or vendor/model) and was just thinking of other ways to collapse that a bit | |
| 16:35:06 | dansmith | but probably not possible | |
| 16:35:35 | dansmith | alias can find things by address, vendor/model, and now group_type.. just feels far too undefined | |
| 16:35:51 | gibi | yeah I know | |
| 16:36:28 | gibi | at least alias with a group_type is better in my eyes than using vendor/product | |
| 16:37:14 | dansmith | yes definitely, and kinda where I was going with "group with one device".. I don't want to make things more complicated, but having alias reference only a group, which references device_specs would be a nicer hierarchy | |
| 16:37:26 | dansmith | *group_type I should say | |
| 16:37:35 | gibi | hm | |
| 16:38:18 | gibi | so we i) add group_type to alias ii)allow one device per group in the group_spec 3) deprecate vendor/product in alias 4)... 5) profit | |
| 16:38:47 | dansmith | 4) annoy people with more typing for no reason | |
| 16:38:55 | dansmith | it would be a better diagram but not a better UX I imagine | |
| 16:39:20 | dansmith | if we could imply the auto-generated single-device group type maybe, but idk | |
| 16:39:23 | gibi | nah, people we use LLMs to generate the config :D | |
| 16:39:39 | dansmith | lol, of course | |
| 16:39:43 | dansmith | silly me | |
| 16:40:47 | dansmith | maybe then 4) sell the users an LLM as the only way to actually generate this config so that 5) profit | |
| 16:41:53 | gibi | I need to start frontier model company, easy-peasy | |
| 16:42:22 | dansmith | that's the obvious solution to any problem in 2026 | |
| 16:43:29 | gibi | tomorrow is public holiday here so I will have time to *plan* :D | |
| 17:03:41 | opendevreview | Dan Smith proposed openstack/nova-specs master: Add unpin-az spec https://review.opendev.org/c/openstack/nova-specs/+/986539 | |
| 18:39:16 | gouthamr | hey melwitt: question regarding the ML post and this bug: https://bugs.launchpad.net/nova/+bug/2149965 | |
| 18:40:22 | gouthamr | i was curious if the migration flow in any way _need_ cephadm? i.e., does anything run "ceph orch" commands? i wasn't able to find that info and was wondering | |
| 18:41:59 | gouthamr | melwitt: how did you arrive at that ceph orch being the problem is i guess my question | |
| 18:42:40 | melwitt | gouthamr: sorry, I didn't include info info in the bug, I did it in haste. I don't think so -- my assumption is that things failed during ceph installation, well before any tests were run. this is an example run https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f | |
| 18:42:48 | melwitt | *include enough info | |
| 18:43:26 | gouthamr | i see, it's happening on the migration job, but, that's unrelated | |
| 18:43:53 | gouthamr | melwitt: thanks.. that was my initial thought; maybe that part of the handling is just .. flaky | |
| 18:44:47 | gouthamr | melwitt: the job history is pretty decent: https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration-ceph | |
| 18:45:12 | melwitt | gouthamr: yeah I wondered if maybe something is just not waiting for 'ceph mgr module enable orchestrator' to finish or something like that? | |
| 18:45:45 | melwitt | yeah. it is definitely a "sometimes" bug but I have personally seen it a lot enough to ask someone if they know something to stop it haha | |
| 18:46:32 | gouthamr | so we don't bail out early.. | |
| 18:46:32 | gouthamr | melwitt: ack, i've added some retry logic here: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 | |
| 18:47:00 | gouthamr | disabling cephadm isn't necessary.. just something that'll help save resources if we have nothing more to setup/change on the ceph cluster | |
| 18:47:52 | melwitt | ah gotcha, makes sense | |
| 18:49:37 | gouthamr | sean-k-mooney headed in the same direction.. so i guess we can see if this helps with stability. thanks for flagging this | |
| 18:50:15 | melwitt | thanks you for looking at it :) | |
| 18:50:31 | sean-k-mooney | so i looked at the plugin breifly but didn know which job failed | |
| 18:50:41 | sean-k-mooney | so i didnt knwo which method was actully the failure | |
| 18:51:03 | sean-k-mooney | gouthamr: i saw you patch to add the retry on disable but i suspect that wasint the issue | |
| 18:51:32 | sean-k-mooney | perhaps it was but i was assomign like mel that it was happenign durign the install | |
| 18:52:15 | melwitt | I added another comment on https://bugs.launchpad.net/nova/+bug/2149965 to explain a bit more, for future reference | |
| 18:52:54 | gouthamr | sean-k-mooney: the restart you stated is an internal ceph thingy - we don't explicitly ask to restart.. the manager module reloads things each time and during that reload the API is down.. so "orch" just throws a random error: "set a backend | |
| 18:53:07 | sean-k-mooney | + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:210 | |
| 18:53:08 | gouthamr | " - that's just catching a giant exception in the client | |
| 18:53:59 | sean-k-mooney | right os we need to be more robot to the internal errors | |
| 18:54:04 | gouthamr | yes | |
| 18:54:54 | sean-k-mooney | in any case it faillign here https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/cephadm#L207 | |
| 18:55:20 | melwitt | thanks to both of you for looking despite the major lack of info on the bug 😬 | |
| 18:55:54 | sean-k-mooney | melwitt: i definlly have not seen that error several time sin the last month and not looked at it because fo tiem... | |
| 18:56:23 | sean-k-mooney | definlty not a thing that woudl happen | |
| 18:57:02 | sean-k-mooney | anyway https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 wont fix it because that is not what is failign | |
| 18:57:09 | sean-k-mooney | btu the approch might work if we use it else hwere | |
| 18:57:28 | sean-k-mooney | i dont knwo if tis just safe to invoke the same command again when we are in this state | |
| 18:57:36 | sean-k-mooney | i.e. will it just fail again | |
| 18:57:57 | melwitt | hm ok | |
| 18:58:08 | sean-k-mooney | or is that the right thing to do possibel with a sleep ro query ot see fi the thing that restarted is up | |
| 18:58:45 | sean-k-mooney | this honestly feels like a cphadm bug that should eventully get fixed there | |
| 18:59:58 | sean-k-mooney | gouthamr: i assume you wrapped https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818/1/devstack/lib/cephadm because that is the only place we call set backend | |
| 19:00:23 | sean-k-mooney | but you will noteice in teh trace back its failing when we are settting the backend to cephadm and there we are not setting it ot anythin | |
| 19:00:42 | sean-k-mooney | cli(['orch', 'set', 'backend', 'cephadm']) | |
| 19:01:00 | melwitt | so you don't think it's something to do with commands the devstack plugin is running, like it could wait for a thing to be ready first | |
| 19:01:36 | gouthamr | we'll just be yelling at it to disable orch, what's wrong with that? | |
| 19:01:41 | gouthamr | it should be idempotent | |
| 19:01:57 | sean-k-mooney | its https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f/log/job-output.txt#12495 | |
| 19:01:58 | gouthamr | https://docs.ceph.com/en/reef/mgr/orchestrator/#disable-the-orchestrator | |
| 19:02:23 | sean-k-mooney | so if you look at the logs we are failign in start_ceph | |
| 19:02:44 | sean-k-mooney | so the failign call is all internal to that initall call to cephadm | |
| 19:03:04 | sean-k-mooney | 2026-04-29 21:16:02.191617 | controller | + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:1 : exit_trap | |
| 19:03:07 | gouthamr | AH! see that's why i needed the full log :D | |
| 19:03:19 | sean-k-mooney | we trap on the execode form that call | |
| 19:04:11 | sean-k-mooney | so this is very much a race in cephadm rather then our devstack plugin | |
| 19:06:01 | gouthamr | lemme see if there's something new they invented to prevent a "bootstrap" failure | |
| 19:06:36 | sean-k-mooney | ack, we should not need to disable the orc module by the way we definlly want to use it because that is how real cluster work | |
| 19:06:51 | gouthamr | no we had to do this just for CI | |
| 19:07:45 | gouthamr | when we're spinning up single-node ceph and openstack compute on the flavor we use, we ran into OOMs | |
| 19:08:33 | gouthamr | locally, i give it a beefy-enough node, and for jobs that do more, we use multi-node and spin up instances on compute nodes | |
| 19:09:53 | sean-k-mooney | right in this case i dont think its an OOM | |
| 19:10:05 | gouthamr | yeah | |
| 19:12:11 | sean-k-mooney | https://paste.opendev.org/show/bf4uNdKhVfHSw5XRwFXS/ | |
| 19:12:41 | sean-k-mooney | well a littel more https://paste.opendev.org/show/bClxtqx7PviIQSfdnEkq/ | |
| 19:12:53 | gouthamr | okay, we can just retry the bootstrap until it succeeds... there are some opts that can help it be idempotent; check if mon is up, --allow-overwrite so we can be okay with cruft | |
| 19:13:35 | gouthamr | yeah, that message isn't helpful there too :/ we can't enable orch, we're trying to bootstrap $machine ¯\_(ツ)_/¯ | |
| 19:13:57 | gouthamr | it's as good as "something unexpected happened" | |
| 19:14:24 | gouthamr | i can report a ceph tracker if they don't already know about this issue | |
| 19:14:37 | sean-k-mooney | but is it as good as "no valid host found" | |
| 19:14:48 | sean-k-mooney | nova most useful of uesful errors | |