| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-06-02 | |||
| 14:01:01 | jgilaber | While we gather, feel free to add topics to the agenda https://etherpad.opendev.org/p/openstack-cyborg-irc-meeting#L48 | |
| 14:01:13 | jgilaber | courtesy ping: sean-k-mooney amoralej bogdando rlandy chandankumar | |
| 14:01:22 | chandankumar | o/ | |
| 14:01:30 | sean-k-mooney | o/ | |
| 14:02:20 | jgilaber | let's give folks a minute to join and then we can start | |
| 14:03:23 | jgilaber | ok let's get started | |
| 14:03:30 | jgilaber | we have a topic from chandankumar | |
| 14:03:39 | jgilaber | #topic nvme cleanup stages spec proposal discussion | |
| 14:03:45 | jgilaber | #link nvme cleanup stages spec proposal discussion | |
| 14:03:49 | jgilaber | go ahead chandankumar | |
| 14:04:02 | chandankumar | sure | |
| 14:04:14 | chandankumar | Here is the current spec for nvme secure cleanup https://review.opendev.org/c/openstack/cyborg-specs/+/985349/10/specs/2026.2/approved/generic-nvme-driver-with-secure-cleanup.rst | |
| 14:04:20 | chandankumar | Thank you everyone for reviewing the spec. | |
| 14:04:27 | chandankumar | Below is the cleanup flow for nvme device once user deletes an instance: | |
| 14:04:33 | chandankumar | 1. Instance deletion and unbind process goes seperately. | |
| 14:04:42 | chandankumar | 2. During unbind process, cyborg will disable the nvme device with maintaining status by default and set resevered=total in placement | |
| 14:04:48 | chandankumar | 3. Unbind process will finish and cleanup will run async. | |
| 14:04:53 | chandankumar | 4. if the cleanup finishes successfully, we set reserved=0 in placement and add a new flag cleanup_fail to false and enable the device | |
| 14:04:59 | chandankumar | 5. if cleanup fails, keep the device disabled with maintaing state and set cleanup_fail to true and reserved=total in placement. | |
| 14:05:06 | chandankumar | 6. Operator can list the device with cleanup_failed and run cleanup manually on those devices. | |
| 14:05:17 | chandankumar | Earlier in previous patch iteration, I went with cleaning and cleaning_failed status message but there are too many state. | |
| 14:05:23 | chandankumar | So I sticked with | |
| 14:05:25 | chandankumar | default maintaining state and cleanup_failed flag to keep the flow simpler. | |
| 14:05:36 | chandankumar | Do we want to add a cleaning state to know the exact state? and if cleanup failed, we will stick with maintaining? | |
| 14:05:37 | sean-k-mooney | so im not ok with reusing disabled for cleaning failed | |
| 14:05:51 | chandankumar | How do we want to handle that? | |
| 14:05:52 | sean-k-mooney | that was one of the feedack i gave on irc when you pushed the sepc initally | |
| 14:05:56 | amoralej | o/ | |
| 14:06:21 | chandankumar | sean-k-mooney: sorry I am not getting | |
| 14:06:51 | sean-k-mooney | i want to intoduce a sperate device_state filed which will taransation between avilable -> allcoated -> cleaning and hten to eitehr error or aviabel dependign on if cleanign succeeded | |
| 14:07:34 | chandankumar | ok | |
| 14:07:49 | sean-k-mooney | we also need to set reserved=total durign arq bind not durign unbind | |
| 14:08:18 | sean-k-mooney | so on bind we woudl transiation the device to allcoated and set reserved=total | |
| 14:08:55 | sean-k-mooney | on unbind it moved to cleaning keeping reserved=total | |
| 14:09:14 | chandankumar | above device state sounds good, I was focus earloer on cleanup | |
| 14:09:16 | sean-k-mooney | and ether end in error (reserved=total) or aviable (reseting reserved=0) | |
| 14:10:01 | chandankumar | yes that approach seems better | |
| 14:10:33 | chandankumar | One more thing, Does operator can toggle the device state manually? once clenaup finishes successfully | |
| 14:10:42 | sean-k-mooney | im also wondering if we want to add a /device/<uuid>/clean endpoint ot manually trigger cleaning | |
| 14:10:51 | sean-k-mooney | that would be an admin operator similar to program | |
| 14:11:03 | sean-k-mooney | chandankumar: no | |
| 14:11:07 | sean-k-mooney | the admin cannot | |
| 14:11:22 | sean-k-mooney | if we add clean as a deivce action | |
| 14:11:30 | sean-k-mooney | that is how they would recover it | |
| 14:11:47 | sean-k-mooney | we coudl perhaps consider a way to force it | |
| 14:12:02 | jgilaber | that endpoint would only work for devices that are in error? | |
| 14:12:04 | sean-k-mooney | but this si really internal stant that an admin shoudl not have to change in a normal workflow | |
| 14:12:31 | sean-k-mooney | jgilaber: i would say error or aviableale woudl be ok | |
| 14:12:47 | sean-k-mooney | but a 409 for allcoated/cleaning | |
| 14:13:06 | jgilaber | right, available would be fine, but redundant | |
| 14:13:17 | sean-k-mooney | we coudl dicusssi fi cleanign an allcoated fpga before repogrammign it shoudl eb allowed or not in the spec | |
| 14:14:12 | sean-k-mooney | the other approch here for erroed device woudl eb a cyborg-manage command or simialr for operator to manualy update the state | |
| 14:14:28 | sean-k-mooney | im a little reluctant to mirror nova's reset-state api | |
| 14:14:31 | jgilaber | that is currently proposed in the spec, right chandankumar? | |
| 14:14:48 | sean-k-mooney | nova and cinder allows admin to reset the state of instnace after you have fixed the instance/volume | |
| 14:15:12 | chandankumar | currently If a nvme device with cleanup_failed to true, we need to run cleanup manually | |
| 14:15:23 | sean-k-mooney | but we have had a lot fo issues with that in the past with customers or support causing more damabge by using it then if we didnt provide it | |
| 14:16:12 | sean-k-mooney | chandankumar: jgilaber lets follow up in the spec on the exact mechanics | |
| 14:16:30 | sean-k-mooney | i do want to capature both the happy and error paths and ensure we have a documented workflow for both | |
| 14:17:44 | jgilaber | ack | |
| 14:17:53 | chandankumar | I alsoanual recovery:** Operators retry failed cleanups using | |
| 14:17:55 | chandankumar | ``cyborg-nvme-cleanup --device <uuid>`` (requires admin credentials). The CLI | |
| 14:17:57 | chandankumar | tool re-triggers cleanup and resets ``cleanup_failed=False`` on success. | |
| 14:18:06 | chandankumar | that I have proposed in the spec | |
| 14:18:31 | chandankumar | anyway let me address the current comments and new design based on above discussion | |
| 14:18:38 | chandankumar | we can follow up on spec | |
| 14:19:41 | sean-k-mooney | im not really a fan of provideing a cil for the cleaning | |
| 14:20:10 | sean-k-mooney | but ack i have some pednign coeemt form when i first lookd but i stoped at the problem description after my inital skim pass | |
| 14:20:30 | sean-k-mooney | ill review it in detail this week ideally today or tomrorow | |
| 14:20:54 | sean-k-mooney | to be clear im not really a fan of having any dirver speciric clis | |
| 14:21:18 | chandankumar | sean-k-mooney: I will ping you tomorrow for review once I update it based on above design, | |
| 14:22:02 | sean-k-mooney | cyborgs core role is too provied a hardware indepent common api over the acclerator it manges so you shoudl nto need nvme specific clis but a generic way to triger cleaning or programmign is ok | |
| 14:22:43 | sean-k-mooney | ack | |
| 14:23:48 | jgilaber | ack, is that all for this topic? | |
| 14:23:49 | chandankumar | sure | |
| 14:23:54 | chandankumar | thank you jgilaber sean-k-mooney ! | |
| 14:24:10 | jgilaber | thanks Chandan, let's move to reviews | |
| 14:24:17 | jgilaber | #topic Reviews | |
| 14:24:23 | jgilaber | we have one | |
| 14:24:29 | jgilaber | #link https://review.opendev.org/c/openstack/cyborg/+/991027 | |
| 14:24:37 | jgilaber | with tempest tests https://review.opendev.org/c/openstack/cyborg-tempest-plugin/+/991081 | |
| 14:24:56 | sean-k-mooney | i have 2 to add later | |
| 14:24:57 | chandankumar | I was working on dropping the image verification code for device program functionality | |
| 14:25:52 | sean-k-mooney | so the fake driver already supprot program | |
| 14:25:53 | chandankumar | It is dropping the code and marking existing verify_glance_signatures config as deprecated | |
| 14:26:00 | sean-k-mooney | https://review.opendev.org/c/openstack/cyborg/+/991027/2/cyborg/accelerator/drivers/fake.py#127 | |
| 14:26:03 | chandankumar | it has a update method not program one | |
| 14:26:04 | sean-k-mooney | that what update is | |
| 14:26:14 | chandankumar | will I rename that? | |
| 14:26:17 | sean-k-mooney | no update is what prgram is called in teh drier interface | |
| 14:26:24 | chandankumar | ok | |
| 14:26:36 | sean-k-mooney | so that a sperat question | |
| 14:26:45 | sean-k-mooney | we could perhas rename it but that is a large change | |
| 14:26:56 | sean-k-mooney | sicne it woudl be changing the public api of the drivers | |
| 14:27:02 | chandankumar | program interface is currently used in fpga driver only not in generic one | |
| 14:27:16 | sean-k-mooney | again that is but true and untrue | |
| 14:27:41 | sean-k-mooney | so form an api perspective we shoudl not have any driver specifc apis | |
| 14:28:11 | sean-k-mooney | so its more correct to say that the update api si a noop for driver other then the fpg driver | |
| 14:28:24 | sean-k-mooney | rahter then teh progarm api is only for fpga | |
| 14:28:43 | sean-k-mooney | the ablity to program or update a device with a glance image | |