| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-25 | |||
| 14:17:39 | jaypipes | cdent, efried: yeah, I remember chatting with you about it in Boston, though I think I said I would be happy to just stop using the whitelist for two purposes (filtering stuff that guests can use vs. inventory management of devices) | |
| 14:17:59 | cdent | it’s rather limiting that it is doing /sys file stuff ... | |
| 14:18:37 | efried | Yeah, those are two examples where I would like to be able to have e.g. "is_physical_function" be a compute driver override whose default impl could be the /sys/bus business, but in the powervm driver I can do it my way. | |
| 14:19:36 | efried | jaypipes Right, so the inventory is actually being done by get_available_resource (which will eventually become get_inventory when all the plumbing is ready). | |
| 14:19:47 | efried | The whitelist is how you limit which devices are allowed to be assigned. | |
| 14:20:12 | efried | And then the alias list is how you nickname devices or device classes so you can specify them easily in a flavor. | |
| 14:20:19 | jaypipes | ya | |
| 14:20:20 | efried | So really, the conf stuff isn't doing inventorying. | |
| 14:20:46 | jaypipes | if we can just have the whitelist do the former thing and not the latter, I'd be happy | |
| 14:20:53 | jaypipes | efried: zactly. | |
| 14:21:03 | efried | which former/latter? | |
| 14:21:11 | jaypipes | efried: my only question is why haven't you gotten it all done yet? WAITING! | |
| 14:21:18 | mriedem | (filtering stuff that guests can use vs. inventory management of devices) | |
| 14:21:21 | mriedem | "(filtering stuff that guests can use vs. inventory management of devices)" | |
| 14:21:27 | mriedem | you turkeys | |
| 14:21:30 | jaypipes | efried: former == filtering for guests. | |
| 14:21:54 | efried | hm, how is the current impl of the whitelist doing inventory management? | |
| 14:22:16 | jaypipes | would be nice if IBM would just hop on the "newfangled" PCI bus. | |
| 14:22:23 | efried | Hah! | |
| 14:22:30 | claudiub | stephenfin: o/ | |
| 14:22:33 | jaypipes | efried: :P | |
| 14:22:58 | claudiub | stephenfin: yeah, we do support pci passthrough since ocata. | |
| 14:23:22 | efried | and jaypipes, I actually have a prototype/PoC that works for PowerVM. But I very carefully have to bypass PCI address handling in some interesting ways. | |
| 14:23:42 | stephenfin | claudiub: Cool. How do you manage whitelisting of those devices? I assume you're not indexing devices in domain:bus:slot.func format | |
| 14:23:46 | jaypipes | efried: yes, I can imagine. | |
| 14:23:51 | efried | https://review.openstack.org/#/c/496434/ | |
| 14:24:01 | claudiub | stephenfin: we are. | |
| 14:24:55 | claudiub | stephenfin: we're currently only reporting devices which have the domain:bus:slot.func format | |
| 14:25:05 | efried | ...with e.g. [pci]alias = {"name": "USB", "product_id": "8241", "vendor_id": "104c", "device_type": "type-PCI"} and [pci]passthrough_whitelist = {"product_id": "8241", "vendor_id": "104c"} | |
| 14:26:08 | efried | claudiub Those devices show up on the compute node? | |
| 14:26:21 | claudiub | stephenfin: but I've mainly whitelisted devices using product_id / vendor_id | |
| 14:26:42 | claudiub | efried: if they're are passthrough-able, yes. | |
| 14:26:56 | claudiub | efried: they have to be prepared for passthrough first | |
| 14:27:06 | efried | claudiub How? | |
| 14:27:38 | claudiub | efried: https://github.com/openstack/nova/blob/master/releasenotes/notes/hyper-v-pci-passthrough-babf104d6bc2baa6.yaml | |
| 14:28:41 | jaypipes | efried: reviewed. | |
| 14:29:02 | efried | Oh, thanks jaypipes :) | |
| 14:29:22 | claudiub | stephenfin: anyways. it would be nice to be also be able to whitelist devices using other ways than product_id / vendor_id, or domain:bus:slot.func. | |
| 14:29:59 | efried | jaypipes Nice. None of that was worthy of a -1? Really? | |
| 14:30:10 | stephenfin | claudiub: What kind of IDs would you expect, e.g. how else can devices be identified in Hyper-V? | |
| 14:30:45 | jaypipes | efried: :) | |
| 14:30:59 | jaypipes | stephenfin: by green card. | |
| 14:31:00 | efried | jaypipes I really didn't expect you to review it, but I would like to point out the spoof_pci_address method (https://review.openstack.org/#/c/496434/3/nova_powervm/virt/powervm/vm.py@884) | |
| 14:31:21 | claudiub | stephenfin: for example, I've had this issue when working on the sr-iov support. apparenty, different NIC vendors have different PCI device ID formats. for example, Intel NICs contain the vendor_id and product_id, which can be extracted and reported. But other NICs, like Mellanox or Chelsio, do not. | |
| 14:31:37 | claudiub | stephenfin: a simple device_id would do, IMO. | |
| 14:31:53 | jaypipes | efried: oh, trust me, I saw it :) | |
| 14:33:22 | efried | claudiub Right, so I would like my [pci]passthrough_whitelist entry to identify devices in whatever way my compute deems appropriate, and have the logic for doing that whitelist filtering live in my compute driver. | |
| 14:36:46 | claudiub | efried: well, the compute driver is already reporting "all" the PCI devices it sees, right? after which the PCI resource tracker filters those compute driver reported PCI devices according to the configured whitelist. is there anything that doesn't fit your usecase? | |
| 14:37:33 | efried | claudiub Yeah: the ability to specify individual devices in the whitelist (vs. just vendor/prod ID classes) | |
| 14:37:51 | efried | Because as soon as I start chucking addresses around, nova tries to get at them under /sys/bus/... | |
| 14:38:13 | stephenfin | efried, jaypipes: Would we still have per-compute-node filtering in the resource provider world? | |
| 14:38:32 | claudiub | efried: ah yes, i see. that would make sense, IMO. the pci resource tracker should be updated to allow other filters to be configured, imo. | |
| 14:39:19 | efried | stephenfin So the way I would think it would work in a RP world is... | |
| 14:39:46 | efried | Your RP (in this case the compute node) inventories the devices it has available, *already* filtered by whitelist. | |
| 14:40:25 | cdent | meaning filtering as a descendant of get_inventory? | |
| 14:40:36 | jaypipes | stephenfin: are you asking whether there'd be a need for a whitelist conf option when we manage PCI devices in placement? | |
| 14:40:48 | stephenfin | jaypipes: Yup | |
| 14:41:01 | efried | cdent Within get_inventory, yeah. | |
| 14:41:08 | cdent | cool, good | |
| 14:41:14 | jaypipes | stephenfin: yes, there would still be a need for a way to filter out host devices that should not be allocatable to a consumer. | |
| 14:42:00 | jaypipes | stephenfin: I just want the pci_passthrough_whitelist CONF option to do a single thing. I'd like to see a separate CONF option (or YAML file even) that contains device inventory information that cannot be auto-discovered. | |
| 14:42:05 | claudiub | jaypipes: isn't that already done at the api level? | |
| 14:42:14 | jaypipes | claudiub: hmm? | |
| 14:42:21 | jaypipes | claudiub: not sure I understand you | |
| 14:42:22 | claudiub | with pci device alias config opion | |
| 14:42:55 | claudiub | afaik, you request a pci device through an alias. if a pci device is not aliased, i don't think it's allocatable | |
| 14:43:11 | efried | Correct. passthrough_whitelist is intersected with alias to get the list of claimable devs. | |
| 14:43:12 | jaypipes | claudiub: well, that's yet another thing that the pci_passthrough_whitelist CONF option does :( alias things... | |
| 14:43:32 | efried | jaypipes Well, [pci]alias aliases things. | |
| 14:43:44 | efried | AFAICT that's in place to make it easier to specify them in the flavor | |
| 14:43:51 | efried | extra_specs pci_passthrough:alias=<alias>:<count> | |
| 14:44:10 | jaypipes | there's still alias stuff in the whitelist option. | |
| 14:44:17 | efried | ? | |
| 14:44:35 | efried | no lo veo | |
| 14:44:37 | claudiub | so, then, you can still request PCI devices through other means, like vendor_id / product_id, or other fields like this? i didn't see this documented. | |
| 14:44:58 | claudiub | not counting sr-iov. | |
| 14:45:19 | claudiub | that's requested whenever a port is vnic_type direct or something similar | |
| 14:45:34 | efried | oh, that. | |
| 14:45:54 | claudiub | that's requested automatically * | |
| 14:45:56 | efried | That's done based on the physical_network tag in the passthrough_whitelist, I believe. Aliases are not involved at all in that flow. | |
| 14:46:11 | claudiub | efried: yeah | |
| 14:46:12 | efried | which is... goofy. | |
| 14:46:38 | efried | Hey, it's Friday. | |
| 14:47:19 | fried_rice | Yeah, handling SR-IOV is a separate can of worms. | |
| 14:47:48 | fried_rice | And btw, I think SR-IOV is where a lot of the /sys/bus business comes into play. Or rather, why it's there in the first place. | |
| 14:47:55 | claudiub | oh yeah. which reminds me. i didn't find a proper way to "associate" different SR-IOV NICs to different "physical_networks" | |
| 14:48:28 | fried_rice | claudiub How so? | |
| 14:48:44 | fried_rice | oh, the thing where you can only actually have one physical network | |
| 14:49:20 | fried_rice | Yeah, there's a limitation in the binding metadata in neutron. We tried to "fix" it in I think ocata, but armax shot us down. | |
| 14:49:38 | armax | fried_rice: what did I do? | |
| 14:49:49 | fried_rice | armax :) Hold on, let me see if I can dig it up. | |
| 14:50:24 | fried_rice | armax I think it was this one: https://review.openstack.org/#/c/358125/ | |
| 14:50:25 | armax | I don’t typically shoot people down without proper justification :) | |
| 14:50:58 | fried_rice | https://bugs.launchpad.net/neutron/+bug/1615128 | |
| 14:50:59 | openstack | Launchpad bug 1615128 in neutron "Custom binding:profile values not coming through" [Undecided,Invalid] - Assigned to Eric Fried (efried) | |
| 14:51:04 | fried_rice | Oh, you provided justification | |
| 14:51:08 | fried_rice | And we backed off accordingly. | |
| 14:51:37 | jaypipes | fried_rice: apologies, I was wrong on the pci_passthrough + alias thing. :( | |
| 14:51:54 | armax | fried_rice: right, I remember this now…I suppose the next step from the -2 was to follow up with an RFE that never came | |
| 14:52:05 | armax | iirc | |