Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-11
13:05:28 sean-k-mooney so to me that should be the last patch
13:06:26 ratailor gibi, yes, that's not useful.
13:07:21 ratailor gibi, sean-k-mooney ack. I will prepare one patch for any api and see if that works. And if it requires all patches, then I will add as many as required for other apis.
13:08:41 ratailor gibi, sean-k-mooney Thanks for your suggestions. Also please provide your reviews on other patches.
13:18:33 gibi ratailor: I think the action list tests creates a VM and stops it. If you fix those first the the api sample test will probably work.
13:19:12 ratailor gibi, ack. sure.
13:21:21 ratailor gibi, I will then try to add finish call to stop api first and see how it goes.
13:21:50 ratailor gibi, Thanks for the hint. :)
13:50:26 opendevreview Rajesh Tailor proposed openstack/nova master: Fix instance vm_state during shelve https://review.opendev.org/c/openstack/nova/+/934294
14:24:53 dansmith gibi: agree, no point until it's useful
14:35:15 gibi sean-k-mooney, dansmith: thanks for chiming in, that was my immediate feeling as well
15:06:18 opendevreview Merged openstack/nova-specs master: fix footnote refernces https://review.opendev.org/c/openstack/nova-specs/+/934493
15:20:59 moot "Cannot get MAC address of the PF 0000:41:00.0. It is probably attached to a guest already _get_pf_details /var/lib/kolla/venv/lib/python3.11/site-packages/nova/virt/libvirt/host.py:1318"
15:20:59 moot "Could not get a PF mac for 0000:41:00.0 _get_sriov_netdev_details /var/lib/kolla/venv/lib/python3.11/site-packages/nova/virt/libvirt/host.py:1455"
15:20:59 moot If there is not, I am really not sure about what can cause thoses errors :
15:20:59 moot When trying to understand what could have gone wrong I stumbled across this bit of code and I am wondering if there is an indentation error : https://github.com/openstack/nova/blob/stable/2024.1/nova/virt/libvirt/host.py#L1487-L1489
15:20:59 moot Hello ! I am having trouble with my GPU being detected as netdev when trying to setup vgpu managed by nova, I am on caracal.
15:22:33 sean-k-mooney so there are two factors, first in the pci dev-spec its imporant to make sure you have not set physical_netork on teh entry for the gpus
15:23:01 sean-k-mooney second i belive in newer release of nova that is just a warning not an error
15:23:25 sean-k-mooney 2024.1 should not error out for example
15:25:32 zigo Mickael is already running Caracal.
15:25:35 sean-k-mooney moot: one other thing to note is that vGPU is not the same thing as pci passthough and today we do not supprot using the VFs form nvidia vGPUs directly
15:26:42 sean-k-mooney intel and amd gpus supprot sriov and that can be used witn nova generic pci passhtough supprot
15:27:26 sean-k-mooney but nvida's vGPU support uses mdevs today, we are lookign at addign supprot for the vfio-variant drivers in 2025.1 but that work has not reached the POC stage yet
15:29:32 moot device_addresses = 0000:41:00.0
15:29:32 moot [mdev_nvidia-746]
15:29:32 moot enabled_mdev_types = nvidia-746
15:29:32 moot [devices]
15:29:32 moot but if I have :
15:29:34 moot in nova.conf as told in https://docs.openstack.org/nova/latest/admin/virtual-gpu.html
15:29:36 moot a ressource provider should apear right ?
15:30:30 sean-k-mooney it should yes. are you cactully getting an errro in the logs
15:30:54 sean-k-mooney the code you are refernceign here https://github.com/openstack/nova/blob/stable/2024.1/nova/virt/libvirt/host.py#L1487-L1489 is not related to vgpus
15:35:07 moot It is not an error nor a warning just a DEBUG log
15:35:07 sean-k-mooney the _get_sriov_netdev_details funciton has a gracefull fallback in the case that the VF is not a nic
15:35:21 sean-k-mooney right this one https://github.com/openstack/nova/blob/stable/2024.1/nova/virt/libvirt/host.py#L1455
15:35:26 sean-k-mooney you can ignore that
15:36:02 sean-k-mooney its not ment of operators its ment for developer to know which code path was taken
15:37:24 moot okay but the RP is never created, I enabled DEBUG hoping I would find some clue
15:37:52 sean-k-mooney ack well we can proably help you debug that but that not related to that log :)
15:38:33 sean-k-mooney can you post the ouput of "ls /sys/class/mdev_bus/*/mdev_supported_types" somewhere
15:38:55 sean-k-mooney lets see if your host is properly configure to be capable of supproting mdevs on that device and with that type
15:40:14 moot https://paste.opendev.org/show/buohdB4juV5JTLRA0mna/
15:40:14 moot sure !
15:40:35 sean-k-mooney ah i see the problem
15:40:50 sean-k-mooney so 0000:41:00.0 is the PF correct
15:40:54 sean-k-mooney the physical gpu
15:41:09 moot yes
15:41:17 sean-k-mooney you have enabled mig mode on the gpu which moves the mdevs form the PF to the VF
15:41:31 sean-k-mooney or VFs plural
15:41:53 sean-k-mooney so you need to update device_addresses with the pci adress of the vfs
15:42:33 sean-k-mooney specificly you should only list a number of adresses equal to the number of nvidia-746 instance that mdev type can create
15:43:46 sean-k-mooney if you do not enable mig mode with "sriov-manage -e" then all the mdevs will work in time slicing mode and will be reported on the PF
15:45:21 sean-k-mooney in otherword wither you list the PF address of VF adddress depend on if your using timesliced vgpus or mig more vgpus
15:48:46 moot I don't think I enabled the ig mode I enable my VF using : /usr/lib/nvidia/sriov-manage -e 0000:41:00.0
15:48:46 moot I tried to provide the pci addresses of the VFs to nova.conf device_address field and it worked but I wanted to let nova manage the whole PF.
15:48:46 moot A2 nvidia GPU should not support mig mode if I am not mistaken ?
15:51:51 moot I think I am already in the time slicing mode
15:54:16 opendevreview Takashi Kajinami proposed openstack/nova master: Skip functional tests on pre-commit config update https://review.opendev.org/c/openstack/nova/+/933192
15:56:40 moot I am double checking about the mig mode that you talked about to be sure
15:59:42 moot Unable to enable MIG Mode for GPU 00000000:41:00.0: Not Supported
15:59:42 moot >> sudo nvidia-smi -mig 1
15:59:42 moot Unable to disable MIG Mode for GPU 00000000:41:00.0: Not Supported
15:59:42 moot >> sudo nvidia-smi -mig 0
15:59:42 moot I get :
16:15:48 sean-k-mooney so i think you are only ment to run "/usr/lib/nvidia/sriov-manage -e" if you dont want timesilcing
16:15:56 sean-k-mooney with that said nvidia keep changing how that works
16:16:06 sean-k-mooney so even if that was ture once it may not be anymore
16:16:52 moot MIG mode is not enabled on my gpu I can see that using nvidia-smi -q
16:16:59 sean-k-mooney moot: if you dont have any usecase for usign the mdevs on the host you can ommit the device address and instead define the maxum mdevs i think
16:18:12 sean-k-mooney oh thats missing form our docs entirly
16:18:47 sean-k-mooney that might be becasue its an option in an dynmic config section
16:19:27 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/conf/devices.py#L108-L115
16:20:04 sean-k-mooney that is not invoked as part of our docs generation. not really sure how to fix that
16:20:45 moot I will try that but it does not work in a senario with multiple PF right ?
16:21:22 sean-k-mooney it does provided you do not want to use diffent mdev typs per pf
16:21:34 sean-k-mooney the second you want to do that you need to staticly partion them
16:22:08 moot even if I am using time sliced gpus ?
16:22:43 sean-k-mooney so in the past for timesliced vgpu you did not use sriov-manage -e <addres>
16:23:02 sean-k-mooney so it might be worth testing without doing that and see fi the mdevs move back to the pf
16:23:22 sean-k-mooney that is how it used to work before nvidia relased card that supprot sriov
16:23:54 moot you mean testing out without enabling VF so no sriov-manage -e <addres> ?
16:23:56 sean-k-mooney nviida have unfortually changed this behvior on the same card with diffent verions fo there driver/firmware
16:24:04 sean-k-mooney moot: correct
16:24:43 sean-k-mooney if you dont enabel sriov i would expect the mdev capablity to be adversed on the PF isntead and then in nova you could list jut the pf adress
16:24:57 moot I can do that, I will look into contributing to the doc if you think it needs some updates
16:25:12 sean-k-mooney thats how it worked on the T and older generation of cards
16:25:31 sean-k-mooney this is kind of why we just say "look at the vendor docs" for that part
16:26:03 sean-k-mooney i.e. https://docs.nvidia.com/vgpu/5.0/pdf/grid-vgpu-user-guide.pdf
16:26:26 moot well nvidia doc just tell to enable the VF if am not mistaken
16:29:20 moot btw I tried to remove the pci address from the nova conf add a ressource provider was created for each VF
16:29:45 moot 'If not set, it implies that we use the maximum allowed by the type."
16:32:58 moot and not add *
16:45:49 zigo sean-k-mooney: Thanks for helping my (intern) colleague that was kind of lost! :)
16:46:50 sean-k-mooney zigo: vgpus are not a kind thing to expose interns too :P happy to help
16:47:16 sean-k-mooney this is one of the case where nova is kind of helpless to document proeprly becasue it not only depends on the vendor but also the card and the driver used
16:47:51 sean-k-mooney and you know the phase of the moon, your choice in wiskey and if you are feelign lucky today
16:47:53 zigo sean-k-mooney: Yeah, but that one intern is special and knows his stuff ... :P
16:48:29 sean-k-mooney moot: if you do see a way to make the docs better please feel free too
16:55:10 zigo I'll beat him until all is right! :P

Earlier   Later