| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-19 | |||
| 17:24:32 | sean-k-mooney | no | |
| 17:24:36 | bauzas | because the flavors wouldn't express those | |
| 17:24:58 | bauzas | pci in placement is a good strawman, let's continue to use it for the example | |
| 17:25:00 | sean-k-mooney | the problem is with the placment api | |
| 17:25:19 | sean-k-mooney | well it also apples to cyborg or neturon prots with resouce requests | |
| 17:25:31 | bauzas | say I have VGPUS and PCI | |
| 17:25:43 | sean-k-mooney | sure | |
| 17:25:47 | bauzas | the problem arises when I want to ask for both, right? | |
| 17:25:54 | sean-k-mooney | yes | |
| 17:26:07 | bauzas | and what if I'm asking for a single resource ? | |
| 17:26:24 | bauzas | (resource *class) | |
| 17:26:33 | sean-k-mooney | yes | |
| 17:27:05 | sean-k-mooney | so pci in placeemnt is a less good exampel because we dont suppot that with neturon port yet. but ill give you an example | |
| 17:27:08 | bauzas | you agree that's not a problem if I'm only asking for VGPU *or* PCI, right ? | |
| 17:27:26 | sean-k-mooney | openstack flavor set vgpu_2 --property "resources1:VGPU=1" \ | |
| 17:27:28 | sean-k-mooney | --property "resources2:VGPU=1" \ | |
| 17:27:30 | sean-k-mooney | --property "group_policy=isolate" | |
| 17:28:07 | sean-k-mooney | if we add --property "hw:pci_ailias=my_vf:2" | |
| 17:28:36 | sean-k-mooney | then if we only have one defivce with my_vf aviaable | |
| 17:28:45 | bauzas | I got it | |
| 17:28:48 | bauzas | hence my warning | |
| 17:28:48 | sean-k-mooney | then isolate will break that | |
| 17:28:50 | bauzas | .. warning:: Be careful when using ``group_policy`` as this policy is global to all the resources request. If you ask for other resources but only VGPUs in your flavor, that ``isolate`` policy will also apply for other | |
| 17:28:52 | bauzas | ResourceProviders. | |
| 17:29:08 | bauzas | that will be written in solid red | |
| 17:29:26 | sean-k-mooney | and it also breadk AggregateInstanceExtraSpecsFilter and computecapaitlies filter | |
| 17:29:33 | bauzas | written too | |
| 17:29:37 | bauzas | .. note:: If you use ``AggregateInstanceExtraSpecsFilter`` filter, you also need to configure your aggregates metadata with the ``group_policy`` values. | |
| 17:29:53 | bauzas | (well, I can add the mention to computecapabilitiesFilter for sure) | |
| 17:30:04 | sean-k-mooney | and neutron QOS prots and cyborg integration if the cyborg request profile has multiple devices | |
| 17:30:16 | bauzas | that's covered by the warning section | |
| 17:30:43 | sean-k-mooney | can you see why an approch that break 5 differnt feature is not something i want to recomemnd people to use as the first option | |
| 17:30:45 | bauzas | I'm ready to bet that a very small fraction of users and ops don't care a bit about any of those stuff | |
| 17:31:20 | sean-k-mooney | sure so we can docuemnt it for them but not the first way we recomemnd | |
| 17:31:40 | bauzas | this isn't breaking the feature, this is a limitation saying "you can't and shouldn't request multiple vgpus per instance if you want to create a flavor that asks for more than vgpus" | |
| 17:31:57 | sean-k-mooney | all im asking you to do is move lines 132-156 after lines 158-181 | |
| 17:32:04 | bauzas | if you really want to mix resources, do other ways | |
| 17:32:16 | bauzas | let's talk about the alternatives | |
| 17:32:19 | bauzas | custom traits | |
| 17:32:54 | bauzas | what you're proposing requires ops to define distinct traits between GPUs | |
| 17:33:00 | bauzas | even if they share the same type | |
| 17:33:31 | bauzas | the only difference would be "I'm GPU A" while the other would be "I'm GPU B" | |
| 17:33:46 | bauzas | that's not great, right? | |
| 17:33:54 | sean-k-mooney | well since this bugfix is really a feature then yes | |
| 17:34:19 | sean-k-mooney | without intoducing a new feature to expclity supprot selecting vgpus form diffent parent pysical devices | |
| 17:34:52 | sean-k-mooney | and makign that work transparly you whave to reuse one of our supproted features to acive the same goal | |
| 17:35:12 | bauzas | the real fix is is to drop the silly warning https://review.opendev.org/c/openstack/nova/+/845757/4/nova/virt/libvirt/driver.py#b8541 and just provide a loop | |
| 17:35:34 | bauzas | we could improve that for sure in a blueprint | |
| 17:36:02 | bauzas | we discussed that years ago and we said that it would require some better placement syntax | |
| 17:36:21 | bauzas | before that, there is a quickwin | |
| 17:36:22 | sean-k-mooney | we dont supprot booting a vm with multiple gpus upstream today. it can be done ebcasue we dont block it but this is a new feature | |
| 17:37:07 | bauzas | today, there a gap that prevents you to request two allocations | |
| 17:37:23 | sean-k-mooney | right and its not supported as a result | |
| 17:37:35 | sean-k-mooney | upstream and downstream | |
| 17:38:15 | bauzas | pas-ha[m]: this is Friday but I'd appreciate your thoughts https://review.opendev.org/c/openstack/nova/+/845757/ | |
| 17:38:32 | bauzas | pas-ha[m]: we can talk on Monday | |
| 17:39:05 | bauzas | sean-k-mooney: in the meantime, I'll split https://review.opendev.org/c/openstack/nova/+/845757/ in twice | |
| 17:39:16 | bauzas | and just leave the doc discussion aside | |
| 17:44:36 | sean-k-mooney | sure im ok with merging the repoducer and fix then following up with the doc. can you also add a short release note to the fix patch | |
| 17:44:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 17:44:40 | opendevreview | Sylvain Bauza proposed openstack/nova master: document how to ask for more a single vGPU https://review.opendev.org/c/openstack/nova/+/906151 | |
| 17:45:33 | bauzas | sean-k-mooney: if I go with a relnotes that would be a fixes section just mentioning that now you can request vGPUs using multiple resource groups | |
| 17:45:34 | sean-k-mooney | bauzas: so another way to do this which we can disucss monday | |
| 17:45:56 | sean-k-mooney | is we could have a weigher that perferve allocatoin candates that had resouce form diffent RPs | |
| 17:46:15 | sean-k-mooney | bauzas: yep exactly just short and sweet | |
| 17:46:32 | bauzas | cool | |
| 17:46:57 | spatel | sean-k-mooney hey! | |
| 17:47:28 | spatel | quick question, does snapshot include memory footprint also ? | |
| 17:47:57 | _colby | bauzas: Thank you for your help. So I removed all the resource providers except those listed in the nova config for the device types and now nova is spinning up vgpu instances correctly. Its even creating the mdevs on its own. So I guess it was tying to create an mdev for a device not listed in the config | |
| 17:48:41 | bauzas | _colby: good to know | |
| 17:48:56 | bauzas | _colby: while you're here, I'd like your opinion | |
| 17:49:00 | sean-k-mooney | spatel: no | |
| 17:49:09 | sean-k-mooney | spatel: snaptions are of the root disk only | |
| 17:49:22 | sean-k-mooney | for image baccked innstances | |
| 17:49:37 | bauzas | _colby: I provided a patch that would allow you to request VGPU allocations with different resource groups https://review.opendev.org/c/openstack/nova/+/906151/1 | |
| 17:50:22 | bauzas | _colby: the main concern is how to correctly express the request and my personal take is that in general VGPU flavors only ask for VGPUs | |
| 17:51:09 | bauzas | _colby: do you confirm that you're not interested in mixing vgpu resources and other resources like pci passthrough devices in the same flavor ? | |
| 17:56:18 | spatel | sean-k-mooney good to know :) thanks | |
| 17:58:52 | _colby | bauzas: we have a hypervisor that is strictly for passthrough and some that use vGPU. On the vGPU host we offer a vGPU option with 100% of the GPU though. | |
| 17:59:45 | _colby | Is it even possible to do passthrough with vgpu? With passthrough the nvidia drivers on the hypervisor need to be disabled | |
| 18:00:04 | sean-k-mooney | _colby: not passthogu of a vgpu | |
| 18:00:16 | bauzas | _colby: and you wouldn't be interested in requesting qos-bandwidth ports or cyborg ? | |
| 18:00:18 | _colby | ah I see | |
| 18:00:23 | sean-k-mooney | we mean booting a vm with a sriov nic + a vgpu | |
| 18:00:26 | _colby | not on our system no | |
| 18:00:54 | bauzas | _colby: the context is that I'm documenting some usage | |
| 18:01:04 | bauzas | https://review.opendev.org/c/openstack/nova/+/906151/1/doc/source/admin/virtual-gpu.rst#136 | |
| 18:01:35 | bauzas | if you start using some extra spec parameter called 'group_policy', you may not be able to request other things | |
| 18:01:51 | bauzas | but you can continue to request for a single vGPU *and* other things tho | |
| 18:02:30 | bauzas | the alternative is to do a lot of configuration by privoding distinct traits https://review.opendev.org/c/openstack/nova/+/906151/1/doc/source/admin/virtual-gpu.rst#158 | |
| 18:02:42 | bauzas | but that sounds nearly impracticable for ops to me | |
| 18:05:01 | _colby | bauzas: interesting. Our use case this should be fin. We have not actually had many requests for multiple vgpus, and we dont offer any special NIC options at this time. | |
| 18:05:43 | _colby | We have HPC systems for uses who need things like that. This is more for research groups to use for their projects. | |
| 18:06:11 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 18:06:11 | opendevreview | Sylvain Bauza proposed openstack/nova master: document how to ask for more a single vGPU https://review.opendev.org/c/openstack/nova/+/906151 | |
| 18:06:18 | bauzas | sean-k-mooney: added the relnote ^ | |
| 18:06:34 | bauzas | sean-k-mooney: _colby also provided some insights about their VGPU usages | |
| 18:08:26 | bauzas | _colby: thanks for your valuable feedback | |
| 18:08:45 | bauzas | _colby: fwiw, we have a couple of improvments in flight this cycle and you may be interested in getting those | |
| 18:09:22 | sean-k-mooney | yep i saw | |