| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-10 | |||
| 15:13:28 | dansmith | gibi: tbh, I'm not really sure when VFs would need to use this, as I suspect any virt-knowing device would be sensitive to exposure between uses (perhaps not) | |
| 15:14:07 | dansmith | that said, I think that from the operator point of view, it doesn't necessarily matter that it's happening in placement (other than the steps to unreserve after cleaning I guess).. but the devspec wouldn't need to change if we started accounting for the one-time-use in, say, PCI tracker | |
| 15:14:20 | gibi | yeah that is a valid point. This is why I'm not against going with the placement route, just hesitant that our udnerstanding of SRIOV VFs are correct | |
| 15:14:53 | gibi | dansmith: yeah the unreserve is the problematic step if we go with the PCI tracker route | |
| 15:14:57 | dansmith | right | |
| 15:15:01 | gibi | nova has no API for that | |
| 15:15:16 | dansmith | an SRIOV NIC is safe for use without cleaning, because you can't update firmware I think | |
| 15:15:46 | dansmith | not sure about a VGPU and memory exposure I guess, but .. they have to know that the point of that is sharing, so I would expect it's okay | |
| 15:15:52 | gibi | I also think that, but I never checked what happens with GPU VFs or NVMe VFs | |
| 15:16:27 | dansmith | an SRIOV NVMe would be a problem I guess, but if you're de- and re-allocating them they should be safe (like new namespaces on a regular one) | |
| 15:17:03 | gibi | true, if the VF is mapped to a namespace then it is OK | |
| 15:17:06 | dansmith | yeah | |
| 15:17:26 | gibi | OK. I'm less hesitant now. We have good reasons to believe that this feature is not needed for VFs | |
| 15:17:46 | gibi | a sane VF implementor needs to handle the sharing issue | |
| 15:17:51 | dansmith | I think we just make sure it's clear in the doc that this is only for whole devices because they're not virt-aware, and we can also say "for more comprehensive situations you need cyborg anyway" :) | |
| 15:17:55 | dansmith | yeah | |
| 15:18:21 | gibi | OK | |
| 15:18:36 | gibi | and :) for cyborg as a fallback | |
| 15:19:42 | dansmith | I really don't want to blur the line of nova-vs-cyborg much, so I think keeping this limited to specific uses is helpful to avoid people trying to push nova to do more | |
| 15:19:49 | opendevreview | Sylvain Bauza proposed openstack/nova master: Update compute rpc alias for epoxy https://review.opendev.org/c/openstack/nova/+/943952 | |
| 15:19:57 | sean-k-mooney | dansmith: gibi you cant update teh frimware vai the vf from an nic i have worked with | |
| 15:20:23 | dansmith | right | |
| 15:20:24 | sean-k-mooney | dansmith: you can flash ab bitrstream via a specific VF on an fpga that change the behvior of other VFs | |
| 15:20:35 | sean-k-mooney | but that a very different usecasue | |
| 15:20:46 | sean-k-mooney | to me that is not what you are creating this for | |
| 15:20:52 | dansmith | definitely not | |
| 15:20:59 | sean-k-mooney | if we add pci groups in the future | |
| 15:21:07 | sean-k-mooney | we could extend the concpet to the group | |
| 15:21:17 | sean-k-mooney | i.e. a one time group | |
| 15:21:52 | dansmith | tbh, the pci-in-placement approach seems to me to be a bit of a stretch of the model, since placement really assumes resources of a given class are fungible and does not track actual assignment | |
| 15:22:23 | sean-k-mooney | dansmith: well in general tehy are fungible | |
| 15:22:24 | dansmith | so anything specific to a given VF probably needs a new model and new API (which cyborg is well-positioned to handle, IMHO) | |
| 15:22:46 | dansmith | sean-k-mooney: VFs are, but not whole PCI devices | |
| 15:23:17 | sean-k-mooney | maybe you could argue it either way | |
| 15:23:21 | sean-k-mooney | i get where your coming form | |
| 15:23:48 | dansmith | the fact that we have a custom resource type for each device, even if its identical to another, with a total=1 is, I think, the stretch of the model | |
| 15:23:59 | sean-k-mooney | on the cyborg side do we watn to reach out to them during the ptg on any of these topics? | |
| 15:24:02 | dansmith | we're calling two devices that are identical but have different PCI addresses "different" :) | |
| 15:24:08 | gibi | yeah we made the assumption in PCI in Placement that the same RC inventory on an RP means fungible devices | |
| 15:24:20 | sean-k-mooney | dansmith: we are not | |
| 15:24:34 | sean-k-mooney | they have the saem resouce class if they have the same vendor_id and product id | |
| 15:24:37 | gibi | and we opted in to represent a PF as RC=1 while VFs as RC=N | |
| 15:25:11 | sean-k-mooney | the reason they are diffent RPs is because fo VFs | |
| 15:25:30 | gibi | two RPs with the same RC=1 inventory are non fungible due to the RP name encoding the PCI address | |
| 15:25:32 | dansmith | sean-k-mooney: okay, sorry, terminology.. same resource class, but separate providers.. | |
| 15:25:40 | dansmith | right ^ | |
| 15:26:12 | sean-k-mooney | gibi: yep and that because of other factors like numa or phsynet associativity | |
| 15:26:43 | gibi | yeah exactly numa and physnet granurality is also aligned with that | |
| 15:26:43 | sean-k-mooney | as it stands to day we could have an invetlory of PFs directly off the compute RP | |
| 15:27:04 | sean-k-mooney | the reason we dont was to allow numa and physnets to be modleed in the future | |
| 15:27:17 | dansmith | I guess it's not unlike if we had two storage providers on a host that you can't just arbitrarily sum together.. however, the inventory within those are still fungible, but it's not within a PCI provider, hence the only one inventory | |
| 15:27:21 | sean-k-mooney | we only need 1 RP per device in the VF case today | |
| 15:28:46 | sean-k-mooney | dansmith: we could in some cases coalese multipel PCI invetories into one provider | |
| 15:29:18 | sean-k-mooney | it just seamed harder to have it sometiems on the root provider and sometimes on its own | |
| 15:29:25 | sean-k-mooney | depending on what you set in the devspec | |
| 15:29:43 | dansmith | sean-k-mooney: I'm just saying, I think we're bordering on breaking/abusing the model... it still fits nicely, but only just | |
| 15:29:45 | sean-k-mooney | and what we discoverd (vendor/product id) | |
| 15:30:00 | dansmith | and tbc, I'm not complaining :) | |
| 15:30:27 | sean-k-mooney | dansmith: to b efair we have a design for numa in palcment and neutorn port in placment | |
| 15:30:41 | sean-k-mooney | im also hesitant to proceed with the numa part for similar reasons | |
| 15:31:17 | sean-k-mooney | i think part of the orginal selling point for palcemetn (moveing all or almost all the schduling work to it) didnt really pan out | |
| 15:31:41 | sean-k-mooney | if we are beign pargmatic and say the nova schduler will always be needed to do find grained choices liek numa | |
| 15:31:42 | dansmith | I have no mental space to consider that now, but a numa node with fungible inventory inside is a closer fit to the original goal I think | |
| 15:31:51 | sean-k-mooney | then there is an argrument to say palcment should never know about it | |
| 15:32:02 | sean-k-mooney | it being numa | |
| 15:32:25 | sean-k-mooney | ack lets leave it there for now | |
| 15:32:26 | dansmith | I wonder if we could introduce another data type to placement that represents actual physical allocations where needed | |
| 15:32:38 | sean-k-mooney | maybe | |
| 15:32:51 | dansmith | something that ties an allocation uuid to that other thing that represents an actual thing with an address | |
| 15:33:02 | sean-k-mooney | im happy to disucss that at some point just also dont have the brain power to context switch to this in detail right now | |
| 15:33:03 | gibi | lets add more metadata :) | |
| 15:33:16 | dansmith | memory has no address or serial number, but a pci device does, a disk might | |
| 15:33:32 | sean-k-mooney | memory kind of does | |
| 15:33:43 | dansmith | gibi: it would only be good if we can drop something else in favor, of course :) | |
| 15:33:54 | sean-k-mooney | its at least got a numa node | |
| 15:34:08 | sean-k-mooney | it might not be a unique idetifier across all host | |
| 15:34:18 | sean-k-mooney | but there are ways to talk about fungible pools of it | |
| 15:34:23 | dansmith | sean-k-mooney: if the numa node is a nested provider, then the allocation within has no (further) addressing | |
| 15:34:27 | dansmith | right | |
| 15:34:58 | dansmith | anyway, we're way off in the weeds, didn't mean to start digging a hole | |
| 15:35:36 | sean-k-mooney | https://collection.nam.ac.uk/images/960/204000-204999/204324.jpg | |
| 15:36:06 | dansmith | heh yeah | |
| 15:36:11 | sean-k-mooney | thats not actuly the meme i was loogkign for but it will do | |
| 15:36:33 | gibi | :) | |
| 15:50:49 | bauzas | folks I just saw your discussion, will review again dansmith's spec tomorrow | |
| 16:52:15 | opendevreview | Masahito Muroi proposed openstack/nova master: Use dict object for request_specs_dict in the _list_view https://review.opendev.org/c/openstack/nova/+/939658 | |
| 16:54:37 | masahito | sean-k-mooney: gibi: please review this api bug fix again https://review.opendev.org/c/openstack/nova/+/939658 thank you. | |
| 18:25:05 | gmann | sean-k-mooney: do you remember the some missing package for pcre.h in bindep.txt for doc job running on noble. this one https://review.opendev.org/c/openstack/nova/+/935577 | |
| 18:25:20 | sean-k-mooney | yes | |
| 18:25:30 | sean-k-mooney | didnt htat go away? | |
| 18:25:43 | sean-k-mooney | or is that still a thing | |
| 18:25:45 | gmann | sean-k-mooney: no, I tired some but still error | |
| 18:26:07 | sean-k-mooney | what job was failing | |
| 18:26:12 | gmann | libpcre3-dev should include it but that is already in bindep file | |
| 18:26:21 | sean-k-mooney | docs and apiref it seams | |
| 18:26:37 | sean-k-mooney | i thinki have noble somewhere | |
| 18:26:39 | gmann | openstack-tox-docs this one https://zuul.opendev.org/t/openstack/build/a1a1fbda6cf24246a3ed9912c5e952b3 | |
| 18:26:42 | gmann | yeah | |
| 18:27:27 | gmann | I am not able to produce it locally I think I installed some extra package there | |
| 18:28:11 | sean-k-mooney | if you install devstack on the know it fixes the porblem if i recall | |