| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-01-19 | |||
| 17:31:20 | sean-k-mooney | sure so we can docuemnt it for them but not the first way we recomemnd | |
| 17:31:40 | bauzas | this isn't breaking the feature, this is a limitation saying "you can't and shouldn't request multiple vgpus per instance if you want to create a flavor that asks for more than vgpus" | |
| 17:31:57 | sean-k-mooney | all im asking you to do is move lines 132-156 after lines 158-181 | |
| 17:32:04 | bauzas | if you really want to mix resources, do other ways | |
| 17:32:16 | bauzas | let's talk about the alternatives | |
| 17:32:19 | bauzas | custom traits | |
| 17:32:54 | bauzas | what you're proposing requires ops to define distinct traits between GPUs | |
| 17:33:00 | bauzas | even if they share the same type | |
| 17:33:31 | bauzas | the only difference would be "I'm GPU A" while the other would be "I'm GPU B" | |
| 17:33:46 | bauzas | that's not great, right? | |
| 17:33:54 | sean-k-mooney | well since this bugfix is really a feature then yes | |
| 17:34:19 | sean-k-mooney | without intoducing a new feature to expclity supprot selecting vgpus form diffent parent pysical devices | |
| 17:34:52 | sean-k-mooney | and makign that work transparly you whave to reuse one of our supproted features to acive the same goal | |
| 17:35:12 | bauzas | the real fix is is to drop the silly warning https://review.opendev.org/c/openstack/nova/+/845757/4/nova/virt/libvirt/driver.py#b8541 and just provide a loop | |
| 17:35:34 | bauzas | we could improve that for sure in a blueprint | |
| 17:36:02 | bauzas | we discussed that years ago and we said that it would require some better placement syntax | |
| 17:36:21 | bauzas | before that, there is a quickwin | |
| 17:36:22 | sean-k-mooney | we dont supprot booting a vm with multiple gpus upstream today. it can be done ebcasue we dont block it but this is a new feature | |
| 17:37:07 | bauzas | today, there a gap that prevents you to request two allocations | |
| 17:37:23 | sean-k-mooney | right and its not supported as a result | |
| 17:37:35 | sean-k-mooney | upstream and downstream | |
| 17:38:15 | bauzas | pas-ha[m]: this is Friday but I'd appreciate your thoughts https://review.opendev.org/c/openstack/nova/+/845757/ | |
| 17:38:32 | bauzas | pas-ha[m]: we can talk on Monday | |
| 17:39:05 | bauzas | sean-k-mooney: in the meantime, I'll split https://review.opendev.org/c/openstack/nova/+/845757/ in twice | |
| 17:39:16 | bauzas | and just leave the doc discussion aside | |
| 17:44:36 | sean-k-mooney | sure im ok with merging the repoducer and fix then following up with the doc. can you also add a short release note to the fix patch | |
| 17:44:39 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 17:44:40 | opendevreview | Sylvain Bauza proposed openstack/nova master: document how to ask for more a single vGPU https://review.opendev.org/c/openstack/nova/+/906151 | |
| 17:45:33 | bauzas | sean-k-mooney: if I go with a relnotes that would be a fixes section just mentioning that now you can request vGPUs using multiple resource groups | |
| 17:45:34 | sean-k-mooney | bauzas: so another way to do this which we can disucss monday | |
| 17:45:56 | sean-k-mooney | is we could have a weigher that perferve allocatoin candates that had resouce form diffent RPs | |
| 17:46:15 | sean-k-mooney | bauzas: yep exactly just short and sweet | |
| 17:46:32 | bauzas | cool | |
| 17:46:57 | spatel | sean-k-mooney hey! | |
| 17:47:28 | spatel | quick question, does snapshot include memory footprint also ? | |
| 17:47:57 | _colby | bauzas: Thank you for your help. So I removed all the resource providers except those listed in the nova config for the device types and now nova is spinning up vgpu instances correctly. Its even creating the mdevs on its own. So I guess it was tying to create an mdev for a device not listed in the config | |
| 17:48:41 | bauzas | _colby: good to know | |
| 17:48:56 | bauzas | _colby: while you're here, I'd like your opinion | |
| 17:49:00 | sean-k-mooney | spatel: no | |
| 17:49:09 | sean-k-mooney | spatel: snaptions are of the root disk only | |
| 17:49:22 | sean-k-mooney | for image baccked innstances | |
| 17:49:37 | bauzas | _colby: I provided a patch that would allow you to request VGPU allocations with different resource groups https://review.opendev.org/c/openstack/nova/+/906151/1 | |
| 17:50:22 | bauzas | _colby: the main concern is how to correctly express the request and my personal take is that in general VGPU flavors only ask for VGPUs | |
| 17:51:09 | bauzas | _colby: do you confirm that you're not interested in mixing vgpu resources and other resources like pci passthrough devices in the same flavor ? | |
| 17:56:18 | spatel | sean-k-mooney good to know :) thanks | |
| 17:58:52 | _colby | bauzas: we have a hypervisor that is strictly for passthrough and some that use vGPU. On the vGPU host we offer a vGPU option with 100% of the GPU though. | |
| 17:59:45 | _colby | Is it even possible to do passthrough with vgpu? With passthrough the nvidia drivers on the hypervisor need to be disabled | |
| 18:00:04 | sean-k-mooney | _colby: not passthogu of a vgpu | |
| 18:00:16 | bauzas | _colby: and you wouldn't be interested in requesting qos-bandwidth ports or cyborg ? | |
| 18:00:18 | _colby | ah I see | |
| 18:00:23 | sean-k-mooney | we mean booting a vm with a sriov nic + a vgpu | |
| 18:00:26 | _colby | not on our system no | |
| 18:00:54 | bauzas | _colby: the context is that I'm documenting some usage | |
| 18:01:04 | bauzas | https://review.opendev.org/c/openstack/nova/+/906151/1/doc/source/admin/virtual-gpu.rst#136 | |
| 18:01:35 | bauzas | if you start using some extra spec parameter called 'group_policy', you may not be able to request other things | |
| 18:01:51 | bauzas | but you can continue to request for a single vGPU *and* other things tho | |
| 18:02:30 | bauzas | the alternative is to do a lot of configuration by privoding distinct traits https://review.opendev.org/c/openstack/nova/+/906151/1/doc/source/admin/virtual-gpu.rst#158 | |
| 18:02:42 | bauzas | but that sounds nearly impracticable for ops to me | |
| 18:05:01 | _colby | bauzas: interesting. Our use case this should be fin. We have not actually had many requests for multiple vgpus, and we dont offer any special NIC options at this time. | |
| 18:05:43 | _colby | We have HPC systems for uses who need things like that. This is more for research groups to use for their projects. | |
| 18:06:11 | opendevreview | Sylvain Bauza proposed openstack/nova master: Support multiple allocations for vGPUs https://review.opendev.org/c/openstack/nova/+/845757 | |
| 18:06:11 | opendevreview | Sylvain Bauza proposed openstack/nova master: document how to ask for more a single vGPU https://review.opendev.org/c/openstack/nova/+/906151 | |
| 18:06:18 | bauzas | sean-k-mooney: added the relnote ^ | |
| 18:06:34 | bauzas | sean-k-mooney: _colby also provided some insights about their VGPU usages | |
| 18:08:26 | bauzas | _colby: thanks for your valuable feedback | |
| 18:08:45 | bauzas | _colby: fwiw, we have a couple of improvments in flight this cycle and you may be interested in getting those | |
| 18:09:22 | sean-k-mooney | yep i saw | |
| 18:09:47 | bauzas | https://review.opendev.org/q/topic:%22bug/2041519%22+AND+(is:open+OR+is:merged) | |
| 18:09:47 | sean-k-mooney | _colby even with the patch that bauzas is providing to fix this current bug i would really treat multipel vgpus as experimental | |
| 18:10:29 | bauzas | _colby: the above link would give you the SRIOV fix | |
| 18:10:43 | bauzas | sean-k-mooney: I'm cool with documenting that | |
| 18:10:56 | sean-k-mooney | bauzas: have you tested the move operatios with multipel vgpus. like cold migrate and or your future live migration supprot | |
| 18:11:01 | bauzas | (the fact that's more an unsupported feature than a real thing) | |
| 18:11:26 | bauzas | sean-k-mooney: no tbc | |
| 18:11:50 | bauzas | although I don't see any reason why it wouldn't work | |
| 18:11:59 | sean-k-mooney | i woudl certenly feel more comfortabel saying "multiple vgpus in the same vm is experimatal, please report bugs" | |
| 18:12:21 | bauzas | sean-k-mooney: that's why I'm offering to drop the doc | |
| 18:12:31 | bauzas | the relnote seems enough to me | |
| 18:12:44 | bauzas | if people want to use it, cool | |
| 18:13:09 | sean-k-mooney | lets discuss that next week. you shoudl go start your weekend | |
| 18:13:26 | bauzas | the patches are all ready now | |
| 18:13:36 | bauzas | let's see what CI is saying | |
| 18:13:56 | bauzas | btw. it occured to me that our jobs become again flakey on weird things | |
| 18:20:10 | sean-k-mooney | ya a little | |
| 18:20:14 | sean-k-mooney | any thing in particalr | |
| 18:20:23 | sean-k-mooney | i have seen some nova-lvm failures | |
| 21:11:58 | opendevreview | Jay Faulkner proposed openstack/nova stable/2023.2: [ironic] Partition & use cache for list_instance* https://review.opendev.org/c/openstack/nova/+/906155 | |
| #openstack-nova - 2024-01-20 | |||
| 03:22:04 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Fail when file-backed memory is used with reservation https://review.opendev.org/c/openstack/nova/+/906161 | |
| 07:14:31 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Fail when file-backed memory is used with reservation https://review.opendev.org/c/openstack/nova/+/906161 | |
| #openstack-nova - 2024-01-21 | |||
| 01:22:07 | opendevreview | sean mooney proposed openstack/nova-specs master: enable nova to dynamically partion cpus https://review.opendev.org/c/openstack/nova-specs/+/906205 | |
| #openstack-nova - 2024-01-22 | |||
| 06:34:24 | songwenping | -----------------++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ | |
| 09:57:53 | opendevreview | Sylvain Bauza proposed openstack/nova master: document how to ask for more a single vGPU https://review.opendev.org/c/openstack/nova/+/906151 | |
| 10:14:21 | songwenping_ | hi, team, any body can help to look at the problem about the tempest failed:https://zuul.opendev.org/t/openstack/build/5ea2c69c3b0c4a4d882944be580ba27f | |
| 10:25:04 | bauzas | songwenping_: something is wrong with the cyborg tempest plugin | |
| 10:25:06 | bauzas | 2024-01-22 09:51:08.246168 | controller | ++ /opt/stack/cyborg/devstack/plugin.sh:source:8 : source /opt/stack/cyborg/devstack/lib/cyborg 2024-01-22 09:51:08.248710 | controller | /opt/stack/cyborg/devstack/lib/cyborg: line 293: syntax error: unexpected end of file | |
| 10:25:27 | bauzas | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ea/906246/1/check/cyborg-tempest/5ea2c69/job-output.txt | |
| 10:26:25 | bauzas | that's seen again in the devstack install logs : | |
| 10:26:27 | bauzas | 2024-01-22 09:51:08.231 | + functions-common:run_plugins:1820 : local dir=/opt/stack/cyborg 2024-01-22 09:51:08.233 | + functions-common:run_plugins:1821 : [[ -f /opt/stack/cyborg/devstack/plugin.sh ]] 2024-01-22 09:51:08.236 | + functions-common:run_plugins:1822 : source /opt/stack/cyborg/devstack/plugin.sh stack pre-install 2024-01-22 09:51:08.239 | ++ /opt/stack/cyborg/devstack/plugin.sh:source:7 : | |
| 10:26:29 | bauzas | echo_summary 'cyborg devstack plugin.sh called: stack/pre-install' 2024-01-22 09:51:08.241 | ++ ./stack.sh:echo_summary:468 : [[ -t 3 ]] 2024-01-22 09:51:08.243 | ++ ./stack.sh:echo_summary:474 : echo -e cyborg devstack plugin.sh called: stack/pre-install 2024-01-22 09:51:08.246 | ++ /opt/stack/cyborg/devstack/plugin.sh:source:8 : source /opt/stack/cyborg/devstack/lib/cyborg 2024-01-22 09:51:08.2 | |
| 10:26:31 | bauzas | 48 | /opt/stack/cyborg/devstack/lib/cyborg: line 293: syntax error: unexpected end of file 2024-01-22 09:51:08.250 | ++++ /opt/stack/cyborg/devstack/lib/cyborg:source:1 : exit_trap | |