Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-05-21
15:01:45 bauzas dansmith: I guess I understood your case
15:02:00 bauzas you wanna request a group of arbitrary pci devices by traits
15:02:37 bauzas that's why I'm saying you shouldn't provide vendor and product IDs in the alias definition or you would restrict that list
15:03:00 gibi yeah so if you have two different devices with different product id then you can whitelist them via address (with full or partial match on the address) and connect them the the same resource class in the whitelist, you can add traits there as well. Then you can request that resource class in the alias
15:03:06 bauzas but IIRC, when we wanted to do that thing, we found some limitation that we fixed somehow I can't recall (maybe a star or something else)
15:03:38 Uggla Yes I think in the alias section we can skip product / vendor and only use ressource class or trait.
15:03:39 gibi the key is that you can use the same RC in multiple device_spec lines
15:05:49 Uggla Yep I updated the documentation for the alias section to highlight that product_id / vendor_id are optional.
15:07:14 dansmith I don't see that
15:07:34 dansmith ...that they are optional.. I do see the class example
15:08:02 Uggla dansmith, https://docs.openstack.org/nova/latest/configuration/config.html --> alias --> A JSON dictionary which describe a PCI device.
15:08:18 dansmith gibi: traits would be better for me that RC.. if they use the same RC, then I can't do inventory planning based on RC, and also, I might want to still request via RC for specific devices, or traits for generic "just give me a 256G nvme"
15:08:20 gibi https://github.com/openstack/nova/blob/221a3e89e8988bc664298106ee691a4e41ca71f9/nova/tests/functional/libvirt/test_pci_in_placement.py#L1759-L1772
15:08:57 dansmith Uggla: yeah I said I see the example, just not that vendor/product is optional (as it says for traits and rc)
15:09:04 gibi dansmith: if you don't define the RC then the RC will be based on the product_id so you cannot have a single alias requesting it as a single alias cannot request two different RC
15:09:27 dansmith gibi: ... but I can request just a trait no?
15:10:14 Uggla dansmith, --> Note that [...] indicates optional field.
15:10:16 gibi you can request a set of traits
15:10:36 gibi but you need to eithe provide an RC or a product/vendor id in the alias
15:10:49 gibi otherwise we cannot createa placement allocation candidate query
15:11:56 dansmith Uggla: fine, but very not as obvious as: "resource_class: The **optional** Placement resource class name"
15:12:21 dansmith gibi: okay if that's the case then that's definitely not very well explained in the docs here
15:12:33 dansmith lemme try with just the trait so I can see what happens
15:12:36 gibi lets improve the doc
15:13:18 gibi if you provides just traits in the alias and no RC or vendor/product id then you will get that the alias is invalid
15:13:27 gibi (I hope : )
15:15:07 dansmith I do not, on service startup at least
15:15:13 Uggla dansmith, I agree it could be better.
15:18:52 gibi dansmith: it is validated when the alias is used
15:19:12 gibi it was true for before PCI in Placement time
15:19:43 dansmith gibi: so, I might be missing something but it looks to me like I was able to create a server and it just silently ignored my pci request
15:20:30 dansmith https://paste.opendev.org/show/bkADIKJZwFGSAv3UcXCF/
15:21:38 gibi alias = {"traits": "CUSTOM_NVME256G", "device_type":"type-PCI", "name": "nvme256g"}
15:21:52 gibi this should have been rejected at server create. if not that is a bug
15:22:44 gibi it cannot work as we cannot generate a proper allocation candidate query as the resource class is mandatory there
15:22:46 dansmith you see it's running and you can confirm the embedded flavor attempted to request it right?
15:22:51 gibi yes I see
15:22:56 gibi so it is a bug
15:24:10 gibi dansmith: do you use the same nova-pci.conf for both compute and api?
15:24:39 dansmith both computes, api and scheduler yeah
15:24:45 dansmith that's why it's on mnt :)
15:25:08 gibi ack, I tend to make that mistake that I update the alias in the nova-compute conf and not in the nova-api conf
15:25:26 dansmith yep, tried to set myself up to avoid that this time
15:26:03 gibi (the whole above discussion shows that this is the first time we really started using the PCI in Placement feature so we are finding the real bugs now)
15:26:06 dansmith this is a just-rebuilt setup so let me test with the other aliases to make sure pci is working in general
15:28:16 dansmith hmm, no actually
15:28:55 dansmith I wonder if my pci filter is not getting set, hmm
15:29:33 dansmith ah, it's not
15:29:42 dansmith so that's probably why it worked before too, hrm
15:33:38 dansmith gibi: so, I had [scheduler] instead of [filter_scheduler]
15:33:40 Uggla dansmith, not sure but my pci_in_placement is in the filter_scheduler section.
15:33:51 Uggla :)
15:33:52 dansmith but, still works.. what all reads that? more than n-sch?
15:34:04 dansmith conductor maybe?
15:36:43 gibi if pci_in_placement is read by the api
15:36:49 gibi s/if//
15:37:04 gibi repot_in_placement is read by the compute
15:37:49 gibi (yes it is not optimal I know)
15:38:40 dansmith okay api is seeing the new value, still not stopping me
15:39:18 gibi I can try to reproduce the issue on my side...
15:39:41 bauzas what's the issue ?
15:40:13 dansmith but, even using a good alias isn't getting me the device or a failure, so something else must be broken. here
15:41:47 Uggla bauzas, my understanding is if you only specify an trait in the alias section, pci in placement should block that as it needs RC too. And it is not the case.
15:42:09 bauzas yeah, that could be the case
15:42:23 bauzas dansmith doesn't see anything in placement ?
15:42:33 bauzas about the PCI RPs
15:42:49 bauzas (just rebooted due to a wifi crash, <3 F42)
15:47:05 dansmith gibi: okay I had a typo in my flavor property so it was just being ignored. but, fixing that, here's what I see as a user:
15:47:06 dansmith https://paste.opendev.org/show/bI7lakvj0atefW6KPNkw/
15:47:26 dansmith the log does have something in it, but it's not quite obvious I suspect
15:47:50 dansmith I suspect no operator will know what "resources={CUSTOM_PCI_NONE_NONE=1}" means
15:49:36 gibi yepp we need to add an explicit validation
15:49:50 gibi that either RC or product/vendor needs to be provided in the alias
15:50:13 dansmith yeah
15:51:07 gibi I can file a bug and link it to the downstream PCI in Placement Jira as well so we will have time for it to fix
15:51:08 dansmith gibi: so ... why again can we not support just the trait? You can ask for all providers with that trait, no?
15:51:30 gibi you cannot have an allocation candidate without telling placement what resource you are allocating
15:51:51 dansmith because you're asking for inventory not a provider?
15:52:09 dansmith (and providers have the traits)
15:52:40 gibi I'm asking for things I can allocate, I cannot allocate a trait or a provider I can only allocate a piece of inventory
15:53:06 dansmith right, that's what I mean
15:53:09 gibi yepp
15:54:13 gibi in our oversimplified case where a single PF RP only have a single RC with a single inventory allocation the RP or allocating one piece of that inventory is the same.
15:54:38 gibi so it is easy to make that mental jump that we want to allocate RPs
15:56:42 gibi this also shows the effect of the design decisions that we connect traits to RPs not to an inventory of the RP
15:57:05 gibi we allocate inventories but we filter on traits that are not on the inventories
15:57:20 gibi it is a bit of a mixup
15:57:35 gibi so we tend to namespace traits to somehow refer to inventories :)
15:58:04 gibi like with the OTU trait, we put PCI in the name to signify it is only related to PCI inventories of the RP
15:58:18 gibi anyhow
15:58:23 gibi I will going to file a bug
15:58:32 gibi based on your paste
16:01:42 Uggla gibi, yes but finally the code has blocked the invalid settings. So not so bad.
16:02:00 gibi https://bugs.launchpad.net/nova/+bug/2111440
16:02:20 gibi Uggla: yep, but we should consider pre-validating the alias at sevice startup
16:02:30 gibi and also be a bit more instructive in the lgs
16:02:31 gibi logs
16:02:46 dansmith gibi: yeah there have been a few things in this journey where traits applied to providers to say something about inventory has been a bit weird
16:03:44 dansmith gibi: Uggla I also had a typo in my pci alias config (unparseable json) and it didn't get noticed until the first time I tried to boot with a pci-enabled flavor, and we returned json parse complaints to the user in the error :(
16:04:00 dansmith which of course is "line 1, character X" which makes no sense since they're all one line

Earlier   Later