| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-11-19 | |||
| 16:28:56 | bauzas | sean-k-mooney: please do so we respect the current policy | |
| 16:29:00 | sean-k-mooney | i just said i would ask ot see if there was any objection | |
| 16:29:22 | bauzas | I don't expect any controversy but I'd like to continue doing things the normal way | |
| 16:29:39 | sean-k-mooney | ok thats all form me then | |
| 16:29:55 | bauzas | cool then we can end this meeting | |
| 16:30:01 | bauzas | thanks all ! | |
| 16:30:06 | bauzas | #endmeeting | |
| 16:30:06 | opendevmeet | Meeting ended Tue Nov 19 16:30:06 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:30:06 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-11-19-16.00.html | |
| 16:30:06 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-11-19-16.00.txt | |
| 16:30:06 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-11-19-16.00.log.html | |
| 16:30:14 | elodilles | thanks o/ | |
| 16:30:24 | tkajinam | thank you | |
| 16:30:27 | gibi | thanks folks | |
| 16:30:54 | s3rj1k | thanks | |
| 16:31:24 | tkajinam | gibi, thanks for reviewing these (sorry for my late reply but I wanted to avoid congestion) ! | |
| 16:31:50 | tkajinam | gibi, and may I ask you a favor, to review https://review.opendev.org/c/openstack/nova/+/930726 , too? | |
| 16:31:55 | luca_ | hello, we are experimenting with GPU passtrough with nova and placement, we incurred in a strange issue, we have 2 hosts with 8 GPUs each, we managed to run VMs with 5 GPU, but as soon as we try to schedule a VM with 6 or more GPUs placement times out because of the many candidates yields (40320), i was wondering if you can help checking our configuration for fixes. | |
| 16:32:58 | gibi | tkajinam: sure | |
| 16:33:04 | tkajinam | gibi, thanks ! | |
| 16:33:07 | tkajinam | luca_, which version of nova/placement are you using ? | |
| 16:34:31 | luca_ | we are using kolla-ansible 2024.1 quay.io/openstack.kolla/placement-api:2024.1-ubuntu-jammy and quay.io/openstack.kolla/nova-api:2024.1-ubuntu-jammy | |
| 16:34:51 | gibi | tkajinam: done. Thanks for all the cleanups. I really appreciate your commitment to make nova a cleaner place | |
| 16:35:28 | gibi | luca_: this is a known issue, let me find the bug | |
| 16:35:38 | bauzas | do you have T4s ? | |
| 16:35:55 | sean-k-mooney | t4s would not really impact this | |
| 16:36:01 | gibi | luca_: https://review.opendev.org/c/openstack/nova/+/855885 | |
| 16:36:07 | tkajinam | gibi, honor to hear it :-) | |
| 16:36:21 | sean-k-mooney | bauzas: this is related to the combinitorial explaiton that happens with many nestted resouce providers | |
| 16:36:48 | opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 16:36:49 | opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 16:36:49 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 16:36:50 | opendevreview | ribaudr proposed openstack/nova master: Use client token when talking to manila https://review.opendev.org/c/openstack/nova/+/925277 | |
| 16:36:50 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 16:36:51 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 16:36:51 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 16:36:52 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 16:36:52 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 16:36:54 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 16:36:54 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 16:36:56 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 16:36:56 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 16:36:58 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287 | |
| 16:36:58 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 16:37:00 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 16:37:00 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 16:37:02 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 16:37:02 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 16:37:04 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 16:37:04 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 16:37:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 16:37:06 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 16:37:08 | opendevreview | ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088 | |
| 16:37:08 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 16:37:10 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 16:37:10 | opendevreview | ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642 | |
| 16:37:18 | bauzas | sean-k-mooney: ah I was doing too many things at once, I just briefly looked and saw GPU passthrough" | |
| 16:37:43 | sean-k-mooney | luca_: gibi: i tought we had a diffent one but https://bugs.launchpad.net/placement/+bug/2070257 is the same porblem | |
| 16:37:50 | opendevreview | Merged openstack/nova stable/2024.2: Route shared storage RPC to evac dest at startup https://review.opendev.org/c/openstack/nova/+/934302 | |
| 16:37:52 | sean-k-mooney | there is an older one on the nova side somewhere | |
| 16:38:18 | sean-k-mooney | related to amd cpus? and numa? | |
| 16:38:23 | gibi | luca_: sean-k-mooney: yeah that bug seems to report the same thing | |
| 16:38:56 | luca_ | i see, by any chanche using the old PciPasstroughFilter in the scheduler instead of using placement will solve this issue? | |
| 16:39:11 | sean-k-mooney | it will work around it yes | |
| 16:39:26 | sean-k-mooney | at the cost of the inablity to have quota | |
| 16:39:36 | sean-k-mooney | to do do some more advanced schduilgn | |
| 16:39:57 | sean-k-mooney | if you disable pci in placmenement then it will remove the nested resouce providers | |
| 16:40:09 | gibi | this is GPU so GPU is using placement without the PCI in Placement feature enabled | |
| 16:40:10 | sean-k-mooney | and the allcoation candiate epxlotion will be mitigated | |
| 16:40:19 | gibi | so this logic cannot be turned on I think | |
| 16:40:29 | gibi | turned off | |
| 16:40:34 | sean-k-mooney | is it gpu or vgpu | |
| 16:40:40 | luca_ | GPU | |
| 16:40:42 | sean-k-mooney | gpu passthough is just pci passthough | |
| 16:40:52 | luca_ | yeah i am passing pci-devices | |
| 16:41:01 | tkajinam | gibi, sorry for asking a favor again but do you mind adding +a, too ? | |
| 16:41:09 | gibi | luca_: did you enabled pci_in_placement ? | |
| 16:41:13 | luca_ | yes | |
| 16:41:17 | gibi | luca_: ahh I see. | |
| 16:41:18 | tkajinam | I wonder if adding an option to limit maximum candidates to placement resolves the issue ? | |
| 16:41:29 | gibi | luca_: then yes, turning off pci_in_placement will help | |
| 16:41:29 | luca_ | wwhich i read is not possible to disable after enabling it :D | |
| 16:41:34 | tkajinam | (I don't have full context, though) | |
| 16:41:39 | sean-k-mooney | luca_: this is the main reason that is tech preview downstream for what its worth | |
| 16:42:00 | sean-k-mooney | luca_: kind of not with workload at least | |
| 16:42:01 | luca_ | iyeah no problem this is a testing enviroment we can rebuild it if it is needed | |
| 16:42:09 | gibi | luca_: not out of the box, you probably need to hack around I'm affraid | |
| 16:42:16 | gibi | tkajinam: done, sorry I missed it | |
| 16:42:24 | tkajinam | gibi, np and thanks again ! | |
| 16:42:31 | sean-k-mooney | luca_: you can do a few things | |
| 16:42:41 | gibi | luca_: wondering about the limit tkajinam suggests | |
| 16:43:02 | sean-k-mooney | gibi: that kind of works but not well | |
| 16:43:22 | tkajinam | one downside of that limit is that it may not work well with randomize_allocation_candidates | |
| 16:43:23 | sean-k-mooney | if you reduce the limit you will likely only get one or two chosts | |
| 16:43:44 | tkajinam | yeah | |
| 16:43:49 | sean-k-mooney | tkajinam: i think by defualt we generate all the permutiaon on a per host basis | |
| 16:44:06 | sean-k-mooney | we have a downstream issue to optimise this in the future where we woudl do a barbar pole stirp | |
| 16:44:20 | gibi | luca_: this is the limit config https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results but maybe it limits after the candidates are generated :/ | |
| 16:44:26 | sean-k-mooney | i.e. take the first candate for each host until we get to the limit in a round robbing fashion | |
| 16:45:01 | sean-k-mooney | the default limti is 1000 canidates i belive | |