Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-03-10
10:50:57 noonedeadpunk with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/
10:51:54 noonedeadpunk so basically on nova-compute start we need to ensure that devices exist same way we do during migration I guess?
10:52:29 noonedeadpunk it's on V just in case, so not sure maybe it's already fixed on later branches
11:08:19 gibi noonedeadpunk: I think it is an open bug https://bugs.launchpad.net/nova/+bug/1900800
11:10:38 noonedeadpunk oh, mdevctl define, nice, thanks!
11:11:38 gibi noonedeadpunk: happy to help :)
11:18:15 noonedeadpunk gibi: I wonder if it's worth to mention it on https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats ?
11:19:02 gibi noonedeadpunk: good point. I think it would be good to list it there. If you have time please push a small doc patch.
11:23:52 sean-k-mooney[m] we dont currenlty use mdevctl and if we wer too we would have to basically re write how we do mdev management
11:24:14 sean-k-mooney[m] today we expect nova to creat the medevs after a host reboot
11:24:20 sean-k-mooney[m] when it recreates the vms
11:24:52 noonedeadpunk but it doesn't?
11:24:52 sean-k-mooney[m] if we want to use mdev ctl in the future we need to strart tacking mdevs like pci devices or pmem
11:25:05 noonedeadpunk I mean - nova jsut crash
11:25:16 sean-k-mooney[m] it should when you start the vm
11:26:04 sean-k-mooney[m] the current bug is just that a bug
11:26:07 noonedeadpunk hm... maybe it's result of resume_guests_state_on_host_boot then...
11:26:24 sean-k-mooney[m] maybe
11:26:30 noonedeadpunk As what I see when trying to start nova-compute - crash with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/
11:26:43 noonedeadpunk So to start nova compute I need mdev to be created
11:27:13 noonedeadpunk Need to try dropping resume_guests_state_on_host_boot indeed
11:31:01 sean-k-mooney[m] bauzas: wasnt there someone already working on a fix for that ^ by the way
11:31:41 sean-k-mooney[m] i remember talking to you about fixing it a few months ago but dont recall if you strated to impelent it but wasnt someone else looking at fixing the issue
11:32:05 sean-k-mooney[m] @noon
11:32:48 sean-k-mooney[m] noonedeadpunk: the issue i have with mdevctl is that nova expect that the mdevs do not exist in the normal code path
11:33:08 sean-k-mooney[m] so if you just precreate arbitary mdevs it will break the ablity to create vms
11:33:52 sean-k-mooney[m] and as a a tool it has little other utility if all you are doing is rectateing the mdevs used by the existing vms
11:34:21 sean-k-mooney[m] creating an mdev is just echoing a uuid into a file in /sys
11:37:25 sean-k-mooney[m] mdevctl also used to not be pakaged on anything other then fedora https://repology.org/project/mdevctl/versions its a little better now as tis actull in debian and ubuntu too but its not a tool that you could previously rely being in your distro package
11:37:34 noonedeadpunk well, I had other issue, but maybe because in the region we run V
11:38:00 dmitriis gibi: o/ Apologies for an extra ping, just wanted to ask if you're good with https://review.opendev.org/c/openstack/nova/+/829974 since you've reviewed it before.
11:38:04 dmitriis Not sure if it's appropriate to land it at this point or not but that's more of a fix + test change rather than a new feature.
11:38:38 noonedeadpunk So to start nova-compute we indeed had to echo uuid to /sys
11:39:06 sean-k-mooney[m] ya that is the workaround for now
11:39:15 noonedeadpunk but it's hard as you need to get resource from placement to understand mapping of uuid to mdev pci device
11:39:19 sean-k-mooney[m] i prefer recommendign that as its safer then mdevctl
11:39:43 sean-k-mooney[m] yes as i siad i tought someone had wirtten that up
11:40:02 noonedeadpunk yeah, I see
11:40:14 sean-k-mooney[m] i know bauzas started at one point
11:41:18 sean-k-mooney[m] i remember discussing the algrothim with them and how they would have to use placement to find the parent pci device to know whic device to create the mdev on since that is not in the xml
11:42:42 noonedeadpunk well another weird thing is that mdev uuid is just random thing, while tbh I'd expect it to be resource uuid. Which would make things easier...
11:44:03 noonedeadpunk As we have resource uuid and then mdev uuid is just another thing that is stored _only_ in xml
11:44:22 noonedeadpunk (in case I'm not missing anything
11:44:56 sean-k-mooney[m] its randome because we do not track them in the db
11:45:08 gibi dmitriis, sean-k-mooney[m], bauzas: I'm generally OK with https://review.opendev.org/c/openstack/nova/+/829974 I do noted that this patch is now wide the gap we need to close later as part of https://bugs.launchpad.net/nova/+bug/1961587 as it adds more sysfs calls to our neutron code.
11:45:10 sean-k-mooney[m] i have been trying to change that since before we actully added the mdev feature
11:46:41 sean-k-mooney[m] noonedeadpunk: what i would like use to evolve to is have operators precreat the mdevs and we would track them in the resouces table in the db
11:46:52 sean-k-mooney[m] then claim and allocate them to instances
11:47:04 sean-k-mooney[m] or tack them in the pcidevice table
11:47:33 sean-k-mooney[m] part of the reason i want to do that is i want to get rid of the persitnt libvirt domain xml
11:47:59 sean-k-mooney[m] right now vgpus are the only thing that need to the persitent domain xml
11:48:40 dmitriis gibi: ack, I am going to look at changing this to use extra info instead in a follow-up
11:48:41 noonedeadpunk well, I wasn't having much fun with plain qemu without libvirt, as if you don't need config you can jsut operate qemu directly?
11:50:46 noonedeadpunk tbh for me from operator prespective is preferable that nova manage mdev creation. as otherwise there will be tons of nasty hooks which everybody do. This can be handled by deployment tools ofc, but considering devices are not persistant and drop on reboot... dunno.
11:51:28 noonedeadpunk and if uuid for mdev will be taken as placement resource id - wouldn't it solve issue with persistant libvirt config?
11:52:08 noonedeadpunk as at time mdev is created, I guess allocation should be already claimed and resource provided where to create it?
11:52:20 gibi dmitriis: ack, thanks
11:57:50 sean-k-mooney[m] noonedeadpunk: you can have multiple mdevs attached to a vm so we cant use placemnt ids for the uuid
11:58:00 sean-k-mooney[m] also they have to be gloally unique
11:58:37 noonedeadpunk ah, indeed, if it's not Nvidia Ampere that would be the case...
11:58:52 noonedeadpunk (as there you have resource per vGPU)
11:58:58 sean-k-mooney[m] removing the persitent domain is not the same as raw qemu
11:59:14 sean-k-mooney[m] libvirt has 2 domains for each insthace
11:59:24 sean-k-mooney[m] the running one in memory and an xml file on disk
11:59:44 sean-k-mooney[m] on ocation they get out of sync and cause issues in managing the vm
12:00:08 sean-k-mooney[m] so i would like to get rid of the file on disk and just use the in memory one which is the acurrate one
12:00:27 noonedeadpunk yeah, I do agree herem that makes sense
12:01:10 noonedeadpunk just sugested that it could be first step to raw qemu, as had to deal with such solution on some cloud platform
12:01:22 sean-k-mooney[m] if you have a patch or code to recreate the mdevs on boot using placement we would be happy to include it
12:02:00 sean-k-mooney[m] are you in favor of raw qemu or against?
12:02:08 noonedeadpunk I'm not at the moment as just found that out during incident, but will try to allocate some resources
12:02:14 noonedeadpunk I'm personally against :)
12:02:33 sean-k-mooney[m] ack just checkign libvirt does some useful stuff for us
12:02:42 sean-k-mooney[m] so we are not currently plannig to remove it
12:02:47 noonedeadpunk s/just suggested/just assumed/
12:03:17 sean-k-mooney[m] we dont really want to have to manage cgroups directly for example
12:03:50 noonedeadpunk I guess it prings in selinux/apparmour rules as well
12:03:55 noonedeadpunk *brings
12:03:59 sean-k-mooney[m] yes
12:04:28 sean-k-mooney[m] it deals with selinux lables and a bunch of other things which we dont want to have to implement in nova
12:04:34 noonedeadpunk and some live migration features iirc
12:04:49 sean-k-mooney[m] yes libvirt does the cpu compatiblity check for us
12:05:13 sean-k-mooney[m] so removing the xml on disk simple. removign libvirt is basicaly a full rewrite of the dirver
12:05:26 sean-k-mooney[m] which is not something we have time for or really want to do
12:05:57 noonedeadpunk exeprience with raw qemu was quite painful
12:06:34 sean-k-mooney[m] the only real advantage to raw qemu would be if libvirt did not have support for feature x yet
12:06:55 sean-k-mooney[m] and at this point nova does not move fast enough for that to be an issue
12:07:10 noonedeadpunk just wanted to say the same:)
12:07:55 noonedeadpunk virtiofs great example of feature ppl love to get but nobody has capacity to help on with...
12:08:40 sean-k-mooney[m] actully that is on our radar
12:08:55 sean-k-mooney[m] it should hopefully get added in z
12:09:03 sean-k-mooney[m] for manilla shares
12:09:15 sean-k-mooney[m] but if you have other usecases for it let us know
12:10:03 sean-k-mooney[m] once we have the basic support in the libvirt driver we can extend it simply if cyborg or anything else wants to use it
12:10:30 sean-k-mooney[m] brb
12:12:41 gibi bauzas and others: I think we have a functional test instability https://paste.opendev.org/show/btHI7ErFfhKYFGdfoujl/
12:12:50 gibi e.g.: https://6779a19d629122a2ad74-09e4be48fe62aca6e4b03d954e19defe.ssl.cf1.rackcdn.com/828570/8/check/nova-tox-functional-py38/99a1e4f/testr_results.html
12:13:33 gibi recently I saw this pretty frequently but It happened in the past too (I saw matches from mid february)
12:14:25 gibi Im filing a bug...
12:20:19 gibi https://bugs.launchpad.net/nova/+bug/1964472

Earlier   Later