Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-11-08
10:05:19 bauzas again, my rant wasn't directed to your change
10:05:52 sean-k-mooney bauzas: i know
10:06:12 sean-k-mooney i just feel like i have to play devels avocate and defend the python core team
10:13:45 jrosser talking of gpu company starting with n, A100/A30 removal from the vGPU driver v16 is causing us a massive headache right now
10:14:03 sean-k-mooney jrosser: yep...
10:14:05 jrosser anyone with ideas what to do would be interesting
10:14:28 sean-k-mooney other then keepign the old driver and pinnign your kernel till you retire the hardware
10:14:34 sean-k-mooney there isnt much you can do
10:14:47 sean-k-mooney jrosser: what os are you on
10:14:54 jrosser ubuntu
10:15:18 sean-k-mooney cool well i think you can continue to use the 15.x driver for the lifetime of 22.04 right
10:15:31 sean-k-mooney although obviorulsy wihtout any future security/preformance improvements
10:15:44 jrosser i don't believe we are going to be able to get any more vCS licences once ours expire
10:16:22 sean-k-mooney oh right the lisciense have an experation date...
10:16:31 jrosser yup
10:16:45 sean-k-mooney they have a diffent driver that the created for some of there skus
10:16:53 sean-k-mooney do you know fi the a100 moved to the new dirver
10:17:21 sean-k-mooney they are spliting there product skus between enterpirse and ai/ml usecases i belive
10:18:28 sean-k-mooney i havent really looked at the detail but i know the grid driver is being split in two for two diffent classes of hardware/workloads but i dont knwo where the a100 lands in that regard
10:30:57 jrosser sean-k-mooney: we have been attempting to PCI passthrough the MIG sriov VF (this might be pointless and never work, but it is at the advice of someone from nv) and get this https://paste.opendev.org/show/bTDDSu1EaNMhflc12ROB/ - do you see anything like that before with PCI passthrough?
10:32:57 sean-k-mooney jrosser: my understanding is while that could work the nvida driver in the guest does not curretnly work on the vf
10:33:17 jrosser i could totally believe that
10:33:46 sean-k-mooney header type 127 does seam famialr for some reason
10:34:08 sean-k-mooney i belive its one fo the extended pci capablity headers
10:34:09 jrosser and that certainly the case for vGPU/GRID driver on the host and free DataCenter driver in the guest which would be the most obvious licencing bypass you could do, so I expect that not to work
10:34:32 jrosser the advice we got was to try DataCenter driver on the host and in the guest
10:35:10 sean-k-mooney https://forum.level1techs.com/t/unknown-pci-header-type-127-for-device-000000-0/180125/4
10:35:27 sean-k-mooney appreently its somethign to do with the graphsics card sleep modes
10:35:46 opendevreview Merged openstack/nova-specs master: Re-propose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/897502
10:35:48 opendevreview Merged openstack/nova-specs master: Re-propose spec for ephemeral encryption for libvirt https://review.opendev.org/c/openstack/nova-specs/+/897503
10:37:13 sean-k-mooney jrosser: so its apprently because the card is not reseting properly when attached/detached form the vm
10:37:41 sean-k-mooney jrosser: thats a pretty common thing on consumer graphics cards that are pci passhtough to a vm
10:37:47 jrosser hmm
10:38:09 sean-k-mooney im suprised thats a problem for a100 but since they never intended the vfs to be passed to the guest its not that surprising
10:41:03 sean-k-mooney https://forum.level1techs.com/t/solved-gpu-passthrough-suddently-stopped-working-on-windows-10-vm/171273/5
10:41:42 sean-k-mooney i have not tired this but apprently you can disbable the d3 idel state for that device
10:41:43 sean-k-mooney options vfio-pci ids=10de:2486,10de:228b disable_vga=1 disable_idle_d3=1
10:42:24 opendevreview Merged openstack/nova-specs master: Re-propose "Allow Manila shares to be directly attached to an instance when using libvirt" for Caracal https://review.opendev.org/c/openstack/nova-specs/+/898319
10:42:51 sean-k-mooney jrosser: if it is just a sleep issue i woudl give that a try for the mig vfs
10:43:23 jrosser ah thats interesting, thanks
10:43:46 bauzas jrosser: fwiw, please know that you could be hit with https://bugs.launchpad.net/nova/+bug/2041519 if you use A100
10:44:17 bauzas jrosser: and yeah, I send hugs to you about vGPU GRID16 EOL support for AI
10:44:42 jrosser bauzas: yeah i saw that discussion the other day - the whole thing is a total disaster /o\
10:44:50 bauzas you now need to buy a fancy expensive AIE license : https://docs.nvidia.com/ai-enterprise/4.0/release-notes/index.html
10:45:04 jrosser indeed, for 50 to 100% of the cost of the hardware
10:45:40 sean-k-mooney i recently bought an intel a750 apprently you can flash that to run there datacenter firmware
10:45:50 sean-k-mooney and get sriov supprot on the consumer card
10:45:54 bauzas jrosser: IMO, now working with some n guys, I can say that they're excellent hardware vendors but terrible software engineers
10:46:01 sean-k-mooney i have been debating about doing that
10:46:51 bauzas sean-k-mooney: btw. when you and dansmith have time, I'd like to discuss on what direction we should do with https://bugs.launchpad.net/nova/+bug/2041519
10:47:21 bauzas the alternative series I wrote is not helping as it doesn't delete existing RPs https://review.opendev.org/q/topic:bug%252F2041519-new
10:47:59 bauzas (the above patch is just a nice try to pass a PF in device_addr instead of VFs, but that's not mandatory)
10:49:00 sean-k-mooney ok ill need to load context on this again so let either set up a meeting ro choose a time to discuss it but it woudl be good to sumemrise what you have tried and what did/didnt work
10:49:20 bauzas sean-k-mooney: and you know, I haven't darken my view of the world, I'm just mattman as matthew perry said
10:49:23 sean-k-mooney bauzas: im currently reviewing dans spec
10:49:30 sean-k-mooney but i can take a look at the patches after
10:49:42 bauzas sean-k-mooney: sure, let's do a gmeet with dan when he's up
10:49:46 zigo jrosser: Am I reading well that the A100 removal is from the Linux kernel ?!?
10:49:49 bauzas sean-k-mooney: no urgency
10:50:14 sean-k-mooney zigo: i dont think soo i think its just being droped form the nvidia binary driver they ship via there portal
10:50:14 bauzas zigo: no, A100 are no longer supported in GRID, that's the point
10:50:26 bauzas zigo: starting with GRID16
10:50:44 sean-k-mooney it would work as a normal graphics card with teh opensouce neuvous driver
10:50:44 bauzas https://docs.nvidia.com/grid/16.0/whats-new-vgpu/index.html
10:50:54 dvo-plv Hello, All. I would like to discuss results of yesterday debate regarding packed ring. Currently we have tow way traits and prefilter based on the release version. How we properly can solve this problem, Should be ask gibi for his ?
10:51:04 sean-k-mooney but none of there vGPU stuff is supproted in the the opensouce dirver
10:51:15 bauzas dvo-plv: I said yesterday that eventually, I'll let the spec sail
10:51:24 bauzas so ask gibi for a review
10:52:15 bauzas dvo-plv: sean-k-mooney had a point on the fact we shouldn't conditionnally provide the trait, so update your spec and add testing and I'll let sean-k-mooney +2 it
10:52:42 dvo-plv okay, I see your point, thank you
10:52:45 bauzas for me, I'll pass as I don't want to give an approval for some direction I have arguments against, but on the other way, I don't wanna hold the series, so I defer to other cores
10:53:02 bauzas I'll just not vote on the spec
10:53:02 sean-k-mooney dvo-plv: tl;dr we bumped libvirt min to 7.0.0 last cycle so no need to do a check on the version anymore
10:53:09 sean-k-mooney either for the trait or in the driver
10:53:20 sean-k-mooney you can just assume its new enough
10:54:02 dvo-plv yes, i remember your comment under the nova patch, that i have to update it, because this https://review.opendev.org/c/openstack/nova/+/887255 was merge first
10:54:19 bauzas correct
10:54:26 sean-k-mooney yep
10:54:47 sean-k-mooney it should simplfy your patch although you had already added the checks
10:55:06 dvo-plv sure, i will update spec first and then nova. Thank you for your time
10:55:38 bauzas dvo-plv: fwiw, even if I want to abstain on the spec, I can review the implementation
10:56:06 bauzas now that I have the whole context, I surely can review it quickly once the design is approved by other cores but me
10:56:14 dvo-plv sure
10:56:35 dvo-plv it sounds critical:)
10:57:01 zigo stephenfin: There are many more services that fail with SQLA 2.x, that haven't been addressed: https://qa.debian.org/excuses.php?experimental=1&package=sqlalchemy
10:57:01 zigo Namely, cloudkitty, trove, ironic-inspector, masakari, sahara, senlin, vitrage, zaqar ...
10:57:19 bauzas your spec will allow operators to not care about mixing hypervisors, which is good, it just doesn't solve our general problem on libvirt knobs we wanna handle
10:57:45 bauzas sean-k-mooney: see, I'm not that dark
10:58:30 sean-k-mooney bauzas: by the way i think we can do a better job on the adress size spec. i -1 with a request to supprot it in image properties adn enchanment to automatically use triats and enchnce the image propperties filter
10:59:07 sean-k-mooney its a minimal expantionin the scope but will basically make it work out of the box for most peopel
10:59:48 bauzas ack, I'll review that one
11:00:16 bauzas tbh, it was a straight reproposal without much additions, I didn't put my brain on it like I did for some other spec :)
11:00:33 sean-k-mooney ya im not sure why i didnt raise these point last cycle
11:00:38 bauzas sean-k-mooney: I also reconsidered the debt we put on ourselves by adding traits
11:00:41 sean-k-mooney which i said in the spec review
11:01:15 bauzas since traits aren't allocations, we can remove compute traits whenever we want, provided we support N-1 or 2 computes
11:01:30 bauzas without need of a reshape
11:01:36 sean-k-mooney you mean requesting and or reporting them?
11:01:40 bauzas both
11:01:45 sean-k-mooney in general we shoudl never remove standard traits

Earlier   Later