Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-04-30
16:10:41 gibi one_time_use is an edge case that does not need to be per device today
16:11:08 gibi but logically if I group an NVME and a NIC and I want to define that the NVME in the group the one that needs the cleaning then
16:11:31 gibi it would be better to model one_time_use on the device instead of on the group
16:11:37 dansmith live_migratable is per group though right? you can't have a group with one migratable device and one not
16:12:20 dansmith I guess I'm not sure why you would group a nic and a nvme together, but fair enough - for the cleaning it makes more sense to account for them individually
16:12:44 gibi imagine that we start grouping a "normally" live_migratable GPU with a NIC that is stateless and should not prevent the live migration but instead we will allow saying for the NIC that live_migratable=unplug
16:13:05 dansmith I know we _should_ be able to group any two devices for any reason, but it's also okay for this to be minimally useful and leave complex scenarios to cyborg :)
16:13:59 dansmith that unplug case could be a cyborg-only too I think, but yeah fair as well
16:15:04 gibi the whole grouping could be / should be cybog only :D
16:15:34 dansmith right, which is why I guess I'm (perhaps too) willing to have some limitations :D
16:15:50 dansmith but okay so sounds like we have to keep the device_spec lines per device
16:16:36 dansmith the "name" is required for each group so we know which provider in placement goes with each set of devices right?
16:19:59 gibi yeah the name is an ugly leak of how we implement resource tracking
16:20:08 gibi we need to name the RP in placement with a stable name
16:20:30 dansmith I _hate_ that
16:20:32 gibi I thought about generating it from the hash of the PCI addresses in the group but that is still not stable
16:20:36 gibi I hate it too
16:21:05 dansmith yeah, I also thought .. even just the one/first pci address would be enough, no?
16:21:16 dansmith but it makes it a bit messy on the backend in case the order changes, etc
16:21:41 gibi so far I thought we allow reconfiguring the content of a *not allocated* group
16:21:51 gibi that breaks naming by address
16:22:03 gibi I thought about just indexing group-type-1 -2 -3
16:22:17 gibi but then we will be config line order dependent
16:22:47 dansmith sorted(group.addresses)[0] would be somewhat stable against rerders
16:22:53 dansmith but if they add a new earlier one it won't of course
16:23:30 dansmith either way, I think "name" is the wrong label for this thing.. it's an identifier, a key, a ... something
16:23:57 dansmith "name" is really most appropriate for the alias, IMHO, the rest are basically FK's
16:27:10 gibi OK that is fair
16:27:35 gibi key works fo rme
16:27:38 gibi *for me
16:27:51 gibi we even save a char in each def :D
16:29:33 dansmith ack, I'm not too concerned over the char of course, but the meaning
16:30:22 gibi I know, I just entertain myself with the saving of chars :D
16:31:13 dansmith and I guess we could have more than one alias reference a type of group? I'm not sure why really, but I assume that's "desirable"
16:31:43 dansmith ugh, I'm getting 403 from docs.o.o
16:32:52 gibi yes two alias can refer to the same group_type but can asks for different traits, and two groups in the same group_type can have two different set of traits. I have no good example why this would be useful but the current alias syntax allows it today already
16:33:06 gibi ohh I have a good example
16:33:18 gibi two alias can differ in numa_policy
16:33:39 dansmith yeah okay
16:33:41 gibi so you want two GPUs but one alias give with with NUMA affinity required the other just preferres
16:33:45 gibi * preferred
16:35:03 dansmith I wish that we could have alias reference device_spec or group_spec things instead of the devices (by address or vendor/model) and was just thinking of other ways to collapse that a bit
16:35:06 dansmith but probably not possible
16:35:35 dansmith alias can find things by address, vendor/model, and now group_type.. just feels far too undefined
16:35:51 gibi yeah I know
16:36:28 gibi at least alias with a group_type is better in my eyes than using vendor/product
16:37:14 dansmith yes definitely, and kinda where I was going with "group with one device".. I don't want to make things more complicated, but having alias reference only a group, which references device_specs would be a nicer hierarchy
16:37:26 dansmith *group_type I should say
16:37:35 gibi hm
16:38:18 gibi so we i) add group_type to alias ii)allow one device per group in the group_spec 3) deprecate vendor/product in alias 4)... 5) profit
16:38:47 dansmith 4) annoy people with more typing for no reason
16:38:55 dansmith it would be a better diagram but not a better UX I imagine
16:39:20 dansmith if we could imply the auto-generated single-device group type maybe, but idk
16:39:23 gibi nah, people we use LLMs to generate the config :D
16:39:39 dansmith lol, of course
16:39:43 dansmith silly me
16:40:47 dansmith maybe then 4) sell the users an LLM as the only way to actually generate this config so that 5) profit
16:41:53 gibi I need to start frontier model company, easy-peasy
16:42:22 dansmith that's the obvious solution to any problem in 2026
16:43:29 gibi tomorrow is public holiday here so I will have time to *plan* :D
17:03:41 opendevreview Dan Smith proposed openstack/nova-specs master: Add unpin-az spec https://review.opendev.org/c/openstack/nova-specs/+/986539
18:39:16 gouthamr hey melwitt: question regarding the ML post and this bug: https://bugs.launchpad.net/nova/+bug/2149965
18:40:22 gouthamr i was curious if the migration flow in any way _need_ cephadm? i.e., does anything run "ceph orch" commands? i wasn't able to find that info and was wondering
18:41:59 gouthamr melwitt: how did you arrive at that ceph orch being the problem is i guess my question
18:42:40 melwitt gouthamr: sorry, I didn't include info info in the bug, I did it in haste. I don't think so -- my assumption is that things failed during ceph installation, well before any tests were run. this is an example run https://zuul.opendev.org/t/openstack/build/2199510cc500444b9638763287ebc62f
18:42:48 melwitt *include enough info
18:43:26 gouthamr i see, it's happening on the migration job, but, that's unrelated
18:43:53 gouthamr melwitt: thanks.. that was my initial thought; maybe that part of the handling is just .. flaky
18:44:47 gouthamr melwitt: the job history is pretty decent: https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration-ceph
18:45:12 melwitt gouthamr: yeah I wondered if maybe something is just not waiting for 'ceph mgr module enable orchestrator' to finish or something like that?
18:45:45 melwitt yeah. it is definitely a "sometimes" bug but I have personally seen it a lot enough to ask someone if they know something to stop it haha
18:46:32 gouthamr melwitt: ack, i've added some retry logic here: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818
18:46:32 gouthamr so we don't bail out early..
18:47:00 gouthamr disabling cephadm isn't necessary.. just something that'll help save resources if we have nothing more to setup/change on the ceph cluster
18:47:52 melwitt ah gotcha, makes sense
18:49:37 gouthamr sean-k-mooney headed in the same direction.. so i guess we can see if this helps with stability. thanks for flagging this
18:50:15 melwitt thanks you for looking at it :)
18:50:31 sean-k-mooney so i looked at the plugin breifly but didn know which job failed
18:50:41 sean-k-mooney so i didnt knwo which method was actully the failure
18:51:03 sean-k-mooney gouthamr: i saw you patch to add the retry on disable but i suspect that wasint the issue
18:51:32 sean-k-mooney perhaps it was but i was assomign like mel that it was happenign durign the install
18:52:15 melwitt I added another comment on https://bugs.launchpad.net/nova/+bug/2149965 to explain a bit more, for future reference
18:52:54 gouthamr sean-k-mooney: the restart you stated is an internal ceph thingy - we don't explicitly ask to restart.. the manager module reloads things each time and during that reload the API is down.. so "orch" just throws a random error: "set a backend
18:53:07 sean-k-mooney + /opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:start_ceph:210
18:53:08 gouthamr " - that's just catching a giant exception in the client
18:53:59 sean-k-mooney right os we need to be more robot to the internal errors
18:54:04 gouthamr yes
18:54:54 sean-k-mooney in any case it faillign here https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/cephadm#L207
18:55:20 melwitt thanks to both of you for looking despite the major lack of info on the bug 😬
18:55:54 sean-k-mooney melwitt: i definlly have not seen that error several time sin the last month and not looked at it because fo tiem...
18:56:23 sean-k-mooney definlty not a thing that woudl happen
18:57:02 sean-k-mooney anyway https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818 wont fix it because that is not what is failign
18:57:09 sean-k-mooney btu the approch might work if we use it else hwere
18:57:28 sean-k-mooney i dont knwo if tis just safe to invoke the same command again when we are in this state
18:57:36 sean-k-mooney i.e. will it just fail again
18:57:57 melwitt hm ok
18:58:08 sean-k-mooney or is that the right thing to do possibel with a sleep ro query ot see fi the thing that restarted is up
18:58:45 sean-k-mooney this honestly feels like a cphadm bug that should eventully get fixed there
18:59:58 sean-k-mooney gouthamr: i assume you wrapped https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/986818/1/devstack/lib/cephadm because that is the only place we call set backend
19:00:23 sean-k-mooney but you will noteice in teh trace back its failing when we are settting the backend to cephadm and there we are not setting it ot anythin

Earlier   Later