Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-11
15:23:01 sean-k-mooney second i belive in newer release of nova that is just a warning not an error
15:23:25 sean-k-mooney 2024.1 should not error out for example
15:25:32 zigo Mickael is already running Caracal.
15:25:35 sean-k-mooney moot: one other thing to note is that vGPU is not the same thing as pci passthough and today we do not supprot using the VFs form nvidia vGPUs directly
15:26:42 sean-k-mooney intel and amd gpus supprot sriov and that can be used witn nova generic pci passhtough supprot
15:27:26 sean-k-mooney but nvida's vGPU support uses mdevs today, we are lookign at addign supprot for the vfio-variant drivers in 2025.1 but that work has not reached the POC stage yet
15:29:32 moot device_addresses = 0000:41:00.0
15:29:32 moot [mdev_nvidia-746]
15:29:32 moot enabled_mdev_types = nvidia-746
15:29:32 moot [devices]
15:29:32 moot but if I have :
15:29:34 moot in nova.conf as told in https://docs.openstack.org/nova/latest/admin/virtual-gpu.html
15:29:36 moot a ressource provider should apear right ?
15:30:30 sean-k-mooney it should yes. are you cactully getting an errro in the logs
15:30:54 sean-k-mooney the code you are refernceign here https://github.com/openstack/nova/blob/stable/2024.1/nova/virt/libvirt/host.py#L1487-L1489 is not related to vgpus
15:35:07 moot It is not an error nor a warning just a DEBUG log
15:35:07 sean-k-mooney the _get_sriov_netdev_details funciton has a gracefull fallback in the case that the VF is not a nic
15:35:21 sean-k-mooney right this one https://github.com/openstack/nova/blob/stable/2024.1/nova/virt/libvirt/host.py#L1455
15:35:26 sean-k-mooney you can ignore that
15:36:02 sean-k-mooney its not ment of operators its ment for developer to know which code path was taken
15:37:24 moot okay but the RP is never created, I enabled DEBUG hoping I would find some clue
15:37:52 sean-k-mooney ack well we can proably help you debug that but that not related to that log :)
15:38:33 sean-k-mooney can you post the ouput of "ls /sys/class/mdev_bus/*/mdev_supported_types" somewhere
15:38:55 sean-k-mooney lets see if your host is properly configure to be capable of supproting mdevs on that device and with that type
15:40:14 moot https://paste.opendev.org/show/buohdB4juV5JTLRA0mna/
15:40:14 moot sure !
15:40:35 sean-k-mooney ah i see the problem
15:40:50 sean-k-mooney so 0000:41:00.0 is the PF correct
15:40:54 sean-k-mooney the physical gpu
15:41:09 moot yes
15:41:17 sean-k-mooney you have enabled mig mode on the gpu which moves the mdevs form the PF to the VF
15:41:31 sean-k-mooney or VFs plural
15:41:53 sean-k-mooney so you need to update device_addresses with the pci adress of the vfs
15:42:33 sean-k-mooney specificly you should only list a number of adresses equal to the number of nvidia-746 instance that mdev type can create
15:43:46 sean-k-mooney if you do not enable mig mode with "sriov-manage -e" then all the mdevs will work in time slicing mode and will be reported on the PF
15:45:21 sean-k-mooney in otherword wither you list the PF address of VF adddress depend on if your using timesliced vgpus or mig more vgpus
15:48:46 moot I don't think I enabled the ig mode I enable my VF using : /usr/lib/nvidia/sriov-manage -e 0000:41:00.0
15:48:46 moot I tried to provide the pci addresses of the VFs to nova.conf device_address field and it worked but I wanted to let nova manage the whole PF.
15:48:46 moot A2 nvidia GPU should not support mig mode if I am not mistaken ?
15:51:51 moot I think I am already in the time slicing mode
15:54:16 opendevreview Takashi Kajinami proposed openstack/nova master: Skip functional tests on pre-commit config update https://review.opendev.org/c/openstack/nova/+/933192
15:56:40 moot I am double checking about the mig mode that you talked about to be sure
15:59:42 moot Unable to enable MIG Mode for GPU 00000000:41:00.0: Not Supported
15:59:42 moot >> sudo nvidia-smi -mig 1
15:59:42 moot Unable to disable MIG Mode for GPU 00000000:41:00.0: Not Supported
15:59:42 moot >> sudo nvidia-smi -mig 0
15:59:42 moot I get :
16:15:48 sean-k-mooney so i think you are only ment to run "/usr/lib/nvidia/sriov-manage -e" if you dont want timesilcing
16:15:56 sean-k-mooney with that said nvidia keep changing how that works
16:16:06 sean-k-mooney so even if that was ture once it may not be anymore
16:16:52 moot MIG mode is not enabled on my gpu I can see that using nvidia-smi -q
16:16:59 sean-k-mooney moot: if you dont have any usecase for usign the mdevs on the host you can ommit the device address and instead define the maxum mdevs i think
16:18:12 sean-k-mooney oh thats missing form our docs entirly
16:18:47 sean-k-mooney that might be becasue its an option in an dynmic config section
16:19:27 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/conf/devices.py#L108-L115
16:20:04 sean-k-mooney that is not invoked as part of our docs generation. not really sure how to fix that
16:20:45 moot I will try that but it does not work in a senario with multiple PF right ?
16:21:22 sean-k-mooney it does provided you do not want to use diffent mdev typs per pf
16:21:34 sean-k-mooney the second you want to do that you need to staticly partion them
16:22:08 moot even if I am using time sliced gpus ?
16:22:43 sean-k-mooney so in the past for timesliced vgpu you did not use sriov-manage -e <addres>
16:23:02 sean-k-mooney so it might be worth testing without doing that and see fi the mdevs move back to the pf
16:23:22 sean-k-mooney that is how it used to work before nvidia relased card that supprot sriov
16:23:54 moot you mean testing out without enabling VF so no sriov-manage -e <addres> ?
16:23:56 sean-k-mooney nviida have unfortually changed this behvior on the same card with diffent verions fo there driver/firmware
16:24:04 sean-k-mooney moot: correct
16:24:43 sean-k-mooney if you dont enabel sriov i would expect the mdev capablity to be adversed on the PF isntead and then in nova you could list jut the pf adress
16:24:57 moot I can do that, I will look into contributing to the doc if you think it needs some updates
16:25:12 sean-k-mooney thats how it worked on the T and older generation of cards
16:25:31 sean-k-mooney this is kind of why we just say "look at the vendor docs" for that part
16:26:03 sean-k-mooney i.e. https://docs.nvidia.com/vgpu/5.0/pdf/grid-vgpu-user-guide.pdf
16:26:26 moot well nvidia doc just tell to enable the VF if am not mistaken
16:29:20 moot btw I tried to remove the pci address from the nova conf add a ressource provider was created for each VF
16:29:45 moot 'If not set, it implies that we use the maximum allowed by the type."
16:32:58 moot and not add *
16:45:49 zigo sean-k-mooney: Thanks for helping my (intern) colleague that was kind of lost! :)
16:46:50 sean-k-mooney zigo: vgpus are not a kind thing to expose interns too :P happy to help
16:47:16 sean-k-mooney this is one of the case where nova is kind of helpless to document proeprly becasue it not only depends on the vendor but also the card and the driver used
16:47:51 sean-k-mooney and you know the phase of the moon, your choice in wiskey and if you are feelign lucky today
16:47:53 zigo sean-k-mooney: Yeah, but that one intern is special and knows his stuff ... :P
16:48:29 sean-k-mooney moot: if you do see a way to make the docs better please feel free too
16:55:10 zigo I'll beat him until all is right! :P
17:02:22 moot Any way thank you for your help I will try my best to find a way to add this information to the docs!
17:02:22 moot when I do not specifies either the device id nor the mdev max number nova does seem to manage well I did not test extensivly I will do that tomorrow!
17:02:22 moot sean-k-mooney: I tried not to enable SRIOV as discussed but the PF does not seems to be mdev capable, so no RP is created.
21:29:19 opendevreview Ghanshyam proposed openstack/nova master: Remove default override for config options policy_file https://review.opendev.org/c/openstack/nova/+/934578
#openstack-nova - 2024-11-12
03:55:06 opendevreview melanie witt proposed openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110
04:45:46 opendevreview melanie witt proposed openstack/nova master: nova-manage: Add flavor scanning to migrate_to_unified_limits https://review.opendev.org/c/openstack/nova/+/924110
06:47:12 opendevreview Tony Breeds proposed openstack/nova stable/2023.1: Critical change for 2023.1/Antelope https://review.opendev.org/c/openstack/nova/+/934720
06:56:02 opendevreview Rajesh Tailor proposed openstack/nova master: Add support for showing finish_time https://review.opendev.org/c/openstack/nova/+/928933
08:22:16 opendevreview melanie witt proposed openstack/nova master: Add [quota]unified_limits_required_resources list https://review.opendev.org/c/openstack/nova/+/924025
09:35:30 opendevreview Amit Uniyal proposed openstack/nova master: Update Nova bdm with updated swap info https://review.opendev.org/c/openstack/nova/+/929858
12:22:08 opendevreview Merged openstack/nova master: docs: Change note:: to warning:: for service user token https://review.opendev.org/c/openstack/nova/+/928308
12:22:14 opendevreview Merged openstack/nova master: docs: Correct unified limits CLI commands https://review.opendev.org/c/openstack/nova/+/924888
13:07:25 ratailor melwitt, Could you please review https://review.opendev.org/c/openstack/nova/+/873901
13:17:34 mina Hi everyone, I've just sent a merge request for Nova and was wondering if someone can have a look when that's possible. The MR link is https://review.opendev.org/c/openstack/nova/+/933311. Thank you very much in advance!
13:21:51 opendevreview Rajesh Tailor proposed openstack/nova master: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/873901
13:23:38 opendevreview Rajesh Tailor proposed openstack/nova master: Fix concurrent migration of vms with multiattach volume failure https://review.opendev.org/c/openstack/nova/+/872278
13:41:01 opendevreview Merged openstack/nova-specs master: Repropose per-process-healthchecks for Epoxy https://review.opendev.org/c/openstack/nova-specs/+/934028
14:23:41 sean-k-mooney mina: im sorry to say that i dont think that change is inlien with how nova should operate

Earlier   Later