| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-19 | |||
| 16:31:24 | tkajinam | gibi, thanks for reviewing these (sorry for my late reply but I wanted to avoid congestion) ! | |
| 16:31:50 | tkajinam | gibi, and may I ask you a favor, to review https://review.opendev.org/c/openstack/nova/+/930726 , too? | |
| 16:31:55 | luca_ | hello, we are experimenting with GPU passtrough with nova and placement, we incurred in a strange issue, we have 2 hosts with 8 GPUs each, we managed to run VMs with 5 GPU, but as soon as we try to schedule a VM with 6 or more GPUs placement times out because of the many candidates yields (40320), i was wondering if you can help checking our configuration for fixes. | |
| 16:32:58 | gibi | tkajinam: sure | |
| 16:33:04 | tkajinam | gibi, thanks ! | |
| 16:33:07 | tkajinam | luca_, which version of nova/placement are you using ? | |
| 16:34:31 | luca_ | we are using kolla-ansible 2024.1 quay.io/openstack.kolla/placement-api:2024.1-ubuntu-jammy and quay.io/openstack.kolla/nova-api:2024.1-ubuntu-jammy | |
| 16:34:51 | gibi | tkajinam: done. Thanks for all the cleanups. I really appreciate your commitment to make nova a cleaner place | |
| 16:35:28 | gibi | luca_: this is a known issue, let me find the bug | |
| 16:35:38 | bauzas | do you have T4s ? | |
| 16:35:55 | sean-k-mooney | t4s would not really impact this | |
| 16:36:01 | gibi | luca_: https://review.opendev.org/c/openstack/nova/+/855885 | |
| 16:36:07 | tkajinam | gibi, honor to hear it :-) | |
| 16:36:21 | sean-k-mooney | bauzas: this is related to the combinitorial explaiton that happens with many nestted resouce providers | |
| 16:36:48 | opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 16:36:49 | opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 16:36:49 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 16:36:50 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 16:36:50 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 16:36:51 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 16:36:51 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 16:36:52 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 16:36:52 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 16:36:54 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 16:36:54 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 16:36:56 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 16:36:56 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 16:36:58 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 16:36:58 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 16:37:00 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 16:37:00 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 16:37:02 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 16:37:02 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 16:37:04 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 16:37:04 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 16:37:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 16:37:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 16:37:08 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 16:37:08 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 16:37:10 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 16:37:10 | opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | |
| 16:37:18 | bauzas | sean-k-mooney: ah I was doing too many things at once, I just briefly looked and saw GPU passthrough" | |
| 16:37:43 | sean-k-mooney | luca_: gibi: i tought we had a diffent one but https://bugs.launchpad.net/placement/+bug/2070257 is the same porblem | |
| 16:37:50 | opendevreview | Merged openstack/nova stable/2024.2: Route shared storage RPC to evac dest at startup https://review.opendev.org/c/openstack/nova/+/934302 | |
| 16:37:52 | sean-k-mooney | there is an older one on the nova side somewhere | |
| 16:38:18 | sean-k-mooney | related to amd cpus? and numa? | |
| 16:38:23 | gibi | luca_: sean-k-mooney: yeah that bug seems to report the same thing | |
| 16:38:56 | luca_ | i see, by any chanche using the old PciPasstroughFilter in the scheduler instead of using placement will solve this issue? | |
| 16:39:11 | sean-k-mooney | it will work around it yes | |
| 16:39:26 | sean-k-mooney | at the cost of the inablity to have quota | |
| 16:39:36 | sean-k-mooney | to do do some more advanced schduilgn | |
| 16:39:57 | sean-k-mooney | if you disable pci in placmenement then it will remove the nested resouce providers | |
| 16:40:09 | gibi | this is GPU so GPU is using placement without the PCI in Placement feature enabled | |
| 16:40:10 | sean-k-mooney | and the allcoation candiate epxlotion will be mitigated | |
| 16:40:19 | gibi | so this logic cannot be turned on I think | |
| 16:40:29 | gibi | turned off | |
| 16:40:34 | sean-k-mooney | is it gpu or vgpu | |
| 16:40:40 | luca_ | GPU | |
| 16:40:42 | sean-k-mooney | gpu passthough is just pci passthough | |
| 16:40:52 | luca_ | yeah i am passing pci-devices | |
| 16:41:01 | tkajinam | gibi, sorry for asking a favor again but do you mind adding +a, too ? | |
| 16:41:09 | gibi | luca_: did you enabled pci_in_placement ? | |
| 16:41:13 | luca_ | yes | |
| 16:41:17 | gibi | luca_: ahh I see. | |
| 16:41:18 | tkajinam | I wonder if adding an option to limit maximum candidates to placement resolves the issue ? | |
| 16:41:29 | gibi | luca_: then yes, turning off pci_in_placement will help | |
| 16:41:29 | luca_ | wwhich i read is not possible to disable after enabling it :D | |
| 16:41:34 | tkajinam | (I don't have full context, though) | |
| 16:41:39 | sean-k-mooney | luca_: this is the main reason that is tech preview downstream for what its worth | |
| 16:42:00 | sean-k-mooney | luca_: kind of not with workload at least | |
| 16:42:01 | luca_ | iyeah no problem this is a testing enviroment we can rebuild it if it is needed | |
| 16:42:09 | gibi | luca_: not out of the box, you probably need to hack around I'm affraid | |
| 16:42:16 | gibi | tkajinam: done, sorry I missed it | |
| 16:42:24 | tkajinam | gibi, np and thanks again ! | |
| 16:42:31 | sean-k-mooney | luca_: you can do a few things | |
| 16:42:41 | gibi | luca_: wondering about the limit tkajinam suggests | |
| 16:43:02 | sean-k-mooney | gibi: that kind of works but not well | |
| 16:43:22 | tkajinam | one downside of that limit is that it may not work well with randomize_allocation_candidates | |
| 16:43:23 | sean-k-mooney | if you reduce the limit you will likely only get one or two chosts | |
| 16:43:44 | tkajinam | yeah | |
| 16:43:49 | sean-k-mooney | tkajinam: i think by defualt we generate all the permutiaon on a per host basis | |
| 16:44:06 | sean-k-mooney | we have a downstream issue to optimise this in the future where we woudl do a barbar pole stirp | |
| 16:44:20 | gibi | luca_: this is the limit config https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results but maybe it limits after the candidates are generated :/ | |
| 16:44:26 | sean-k-mooney | i.e. take the first candate for each host until we get to the limit in a round robbing fashion | |
| 16:45:01 | sean-k-mooney | the default limti is 1000 canidates i belive | |
| 16:45:15 | tkajinam | gibi, it is used to determine the limit query parameter in HTTP Request | |
| 16:45:27 | tkajinam | so the limit may be applied in placement (if I read the code correctly) | |
| 16:45:45 | gibi | tkajinam: yeah applied in placement but maybe after all the permutation is generated :/ | |
| 16:45:49 | tkajinam | yeah | |
| 16:46:56 | sean-k-mooney | luca_: so what i would do is the following if this is a test env, deleate any insances usign pci devices, remove all pci devices form the pci devspec, restart the comptue agents and then disble the config oprtion for pci in placmeent after they have remove the nested resouce providers | |
| 16:47:04 | sean-k-mooney | then add the pci devspecs back | |
| 16:47:10 | gibi | tkajinam: I don't see the limit applied directly in the permutation generaton in https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/allocation_candidate.py#L371 | |
| 16:47:11 | sean-k-mooney | and try again | |
| 16:47:46 | luca_ | since i am asking, is it normale that a single resource provider is generated for each GPU? | |
| 16:48:10 | sean-k-mooney | gibi i think its later | |
| 16:48:11 | luca_ | sean-k-mooney_ thanks for the possible solution i will try it | |
| 16:48:23 | sean-k-mooney | but to fix this properly we need to adjust the nestign of those loops | |
| 16:48:26 | ishanwar | Hi Team, | |
| 16:49:12 | tkajinam | gibi, yeah, I see limit_results applies the limit but it is applied to the full list... | |
| 16:49:14 | sean-k-mooney | luca_: yes becasue we create 1 RP per PF | |