| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-10 | |||
| 14:30:29 | sean-k-mooney | i suspect self._update is reusted for cold migration/resize | |
| 14:30:46 | sean-k-mooney | ya it is here https://github.com/openstack/nova/blob/276685b3db6e8f2ad59c33bc254461c255700ff8/nova/compute/resource_tracker.py#L378 | |
| 14:31:08 | sean-k-mooney | so i think that is fine too | |
| 14:31:24 | sean-k-mooney | live migration is out os scope for this this cycle? | |
| 14:33:19 | sean-k-mooney | live migration does use a move claim but the claiming of the device in the nova db happens a littel earlier | |
| 14:34:06 | sean-k-mooney | i think its fine to still do the reserved=total in the same place, im just asking for the scope fo testing/support | |
| 14:34:53 | dansmith | I'll track down the live case, but yeah the cold move case is also good | |
| 14:35:32 | sean-k-mooney | so for cold migration i think we just need to document that the device is burned once we have got to verify_resize | |
| 14:35:36 | dansmith | I think I'm fine with live being out of scope here, especially since we can mark a device as live_migratable=false now, but there's no reason this shouldn't work the same I think, even if we have to do it in a slightly different place on the dest | |
| 14:35:51 | dansmith | sean-k-mooney: yeah once you've touched both devices on either end, both are burned :) | |
| 14:36:21 | sean-k-mooney | exactly, some opt might assuem that happens at confirm | |
| 14:36:25 | sean-k-mooney | so we just need to document that | |
| 14:36:38 | opendevreview | Merged openstack/nova master: Update conductor and filters allowing migration with SR-IOV devices https://review.opendev.org/c/openstack/nova/+/942146 | |
| 14:37:46 | dansmith | sure | |
| 14:40:20 | gibi | dansmith: yeah by default flavor based PCI devices are not live migratable, only exception is via the recently added live_migratable flag in the dev_spec and alias but we don't expect that will be set for OTU devices | |
| 14:40:46 | dansmith | oh, okay well, that's even easier I guess | |
| 14:41:00 | sean-k-mooney | ya the default helps | |
| 14:41:06 | gibi | only neutron based PCI devs are live migrated by default via unplug and plug | |
| 14:41:18 | sean-k-mooney | right but we dont support them in placment yet | |
| 14:41:22 | gibi | yes | |
| 14:41:25 | sean-k-mooney | so we can also ignor them | |
| 14:41:29 | gibi | yes | |
| 14:41:29 | sean-k-mooney | we could add a check | |
| 14:41:39 | sean-k-mooney | that you dont set otu and physical_network | |
| 14:41:56 | sean-k-mooney | since we know thats invalid today | |
| 14:42:09 | sean-k-mooney | but its not strictly required | |
| 14:42:11 | opendevreview | Gaudenz Steinlin proposed openstack/nova master: Make SSH compression for remote copy configurable https://review.opendev.org/c/openstack/nova/+/943377 | |
| 14:44:41 | dansmith | Okay well, I'll add some notes to the spec for this to cover the above details | |
| 15:06:03 | opendevreview | Sylvain Bauza proposed openstack/nova master: doc: mark the maximum microversion for 2025.1 Epoxy https://review.opendev.org/c/openstack/nova/+/943948 | |
| 15:10:22 | opendevreview | Dan Smith proposed openstack/nova-specs master: Add one-time-use-devices spec https://review.opendev.org/c/openstack/nova-specs/+/943486 | |
| 15:11:37 | gibi | dansmith: left couple of comments but I think the direction looks good | |
| 15:11:49 | dansmith | gibi: thanks! | |
| 15:12:34 | gibi | the only think I'm a bit hesitant about it is the long term effect of the decision that we go with placement to solve this while and therefore we are forced not to support OTU VFs. | |
| 15:13:28 | dansmith | gibi: tbh, I'm not really sure when VFs would need to use this, as I suspect any virt-knowing device would be sensitive to exposure between uses (perhaps not) | |
| 15:14:07 | dansmith | that said, I think that from the operator point of view, it doesn't necessarily matter that it's happening in placement (other than the steps to unreserve after cleaning I guess).. but the devspec wouldn't need to change if we started accounting for the one-time-use in, say, PCI tracker | |
| 15:14:20 | gibi | yeah that is a valid point. This is why I'm not against going with the placement route, just hesitant that our udnerstanding of SRIOV VFs are correct | |
| 15:14:53 | gibi | dansmith: yeah the unreserve is the problematic step if we go with the PCI tracker route | |
| 15:14:57 | dansmith | right | |
| 15:15:01 | gibi | nova has no API for that | |
| 15:15:16 | dansmith | an SRIOV NIC is safe for use without cleaning, because you can't update firmware I think | |
| 15:15:46 | dansmith | not sure about a VGPU and memory exposure I guess, but .. they have to know that the point of that is sharing, so I would expect it's okay | |
| 15:15:52 | gibi | I also think that, but I never checked what happens with GPU VFs or NVMe VFs | |
| 15:16:27 | dansmith | an SRIOV NVMe would be a problem I guess, but if you're de- and re-allocating them they should be safe (like new namespaces on a regular one) | |
| 15:17:03 | gibi | true, if the VF is mapped to a namespace then it is OK | |
| 15:17:06 | dansmith | yeah | |
| 15:17:26 | gibi | OK. I'm less hesitant now. We have good reasons to believe that this feature is not needed for VFs | |
| 15:17:46 | gibi | a sane VF implementor needs to handle the sharing issue | |
| 15:17:51 | dansmith | I think we just make sure it's clear in the doc that this is only for whole devices because they're not virt-aware, and we can also say "for more comprehensive situations you need cyborg anyway" :) | |
| 15:17:55 | dansmith | yeah | |
| 15:18:21 | gibi | OK | |
| 15:18:36 | gibi | and :) for cyborg as a fallback | |
| 15:19:42 | dansmith | I really don't want to blur the line of nova-vs-cyborg much, so I think keeping this limited to specific uses is helpful to avoid people trying to push nova to do more | |
| 15:19:49 | opendevreview | Sylvain Bauza proposed openstack/nova master: Update compute rpc alias for epoxy https://review.opendev.org/c/openstack/nova/+/943952 | |
| 15:19:57 | sean-k-mooney | dansmith: gibi you cant update teh frimware vai the vf from an nic i have worked with | |
| 15:20:23 | dansmith | right | |
| 15:20:24 | sean-k-mooney | dansmith: you can flash ab bitrstream via a specific VF on an fpga that change the behvior of other VFs | |
| 15:20:35 | sean-k-mooney | but that a very different usecasue | |
| 15:20:46 | sean-k-mooney | to me that is not what you are creating this for | |
| 15:20:52 | dansmith | definitely not | |
| 15:20:59 | sean-k-mooney | if we add pci groups in the future | |
| 15:21:07 | sean-k-mooney | we could extend the concpet to the group | |
| 15:21:17 | sean-k-mooney | i.e. a one time group | |
| 15:21:52 | dansmith | tbh, the pci-in-placement approach seems to me to be a bit of a stretch of the model, since placement really assumes resources of a given class are fungible and does not track actual assignment | |
| 15:22:23 | sean-k-mooney | dansmith: well in general tehy are fungible | |
| 15:22:24 | dansmith | so anything specific to a given VF probably needs a new model and new API (which cyborg is well-positioned to handle, IMHO) | |
| 15:22:46 | dansmith | sean-k-mooney: VFs are, but not whole PCI devices | |
| 15:23:17 | sean-k-mooney | maybe you could argue it either way | |
| 15:23:21 | sean-k-mooney | i get where your coming form | |
| 15:23:48 | dansmith | the fact that we have a custom resource type for each device, even if its identical to another, with a total=1 is, I think, the stretch of the model | |
| 15:23:59 | sean-k-mooney | on the cyborg side do we watn to reach out to them during the ptg on any of these topics? | |
| 15:24:02 | dansmith | we're calling two devices that are identical but have different PCI addresses "different" :) | |
| 15:24:08 | gibi | yeah we made the assumption in PCI in Placement that the same RC inventory on an RP means fungible devices | |
| 15:24:20 | sean-k-mooney | dansmith: we are not | |
| 15:24:34 | sean-k-mooney | they have the saem resouce class if they have the same vendor_id and product id | |
| 15:24:37 | gibi | and we opted in to represent a PF as RC=1 while VFs as RC=N | |
| 15:25:11 | sean-k-mooney | the reason they are diffent RPs is because fo VFs | |
| 15:25:30 | gibi | two RPs with the same RC=1 inventory are non fungible due to the RP name encoding the PCI address | |
| 15:25:32 | dansmith | sean-k-mooney: okay, sorry, terminology.. same resource class, but separate providers.. | |
| 15:25:40 | dansmith | right ^ | |
| 15:26:12 | sean-k-mooney | gibi: yep and that because of other factors like numa or phsynet associativity | |
| 15:26:43 | sean-k-mooney | as it stands to day we could have an invetlory of PFs directly off the compute RP | |
| 15:26:43 | gibi | yeah exactly numa and physnet granurality is also aligned with that | |
| 15:27:04 | sean-k-mooney | the reason we dont was to allow numa and physnets to be modleed in the future | |
| 15:27:17 | dansmith | I guess it's not unlike if we had two storage providers on a host that you can't just arbitrarily sum together.. however, the inventory within those are still fungible, but it's not within a PCI provider, hence the only one inventory | |
| 15:27:21 | sean-k-mooney | we only need 1 RP per device in the VF case today | |
| 15:28:46 | sean-k-mooney | dansmith: we could in some cases coalese multipel PCI invetories into one provider | |
| 15:29:18 | sean-k-mooney | it just seamed harder to have it sometiems on the root provider and sometimes on its own | |
| 15:29:25 | sean-k-mooney | depending on what you set in the devspec | |
| 15:29:43 | dansmith | sean-k-mooney: I'm just saying, I think we're bordering on breaking/abusing the model... it still fits nicely, but only just | |
| 15:29:45 | sean-k-mooney | and what we discoverd (vendor/product id) | |
| 15:30:00 | dansmith | and tbc, I'm not complaining :) | |
| 15:30:27 | sean-k-mooney | dansmith: to b efair we have a design for numa in palcment and neutorn port in placment | |
| 15:30:41 | sean-k-mooney | im also hesitant to proceed with the numa part for similar reasons | |
| 15:31:17 | sean-k-mooney | i think part of the orginal selling point for palcemetn (moveing all or almost all the schduling work to it) didnt really pan out | |
| 15:31:41 | sean-k-mooney | if we are beign pargmatic and say the nova schduler will always be needed to do find grained choices liek numa | |
| 15:31:42 | dansmith | I have no mental space to consider that now, but a numa node with fungible inventory inside is a closer fit to the original goal I think | |
| 15:31:51 | sean-k-mooney | then there is an argrument to say palcment should never know about it | |
| 15:32:02 | sean-k-mooney | it being numa | |
| 15:32:25 | sean-k-mooney | ack lets leave it there for now | |
| 15:32:26 | dansmith | I wonder if we could introduce another data type to placement that represents actual physical allocations where needed | |