Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-03-10
08:21:39 opendevreview OpenStack Release Bot proposed openstack/placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/placement/+/832985
08:25:51 bauzas gibi: hey, maybe we can hold it until noon
08:25:59 kashyap Completely aside, TIL: with Py 3.7, "breakpoint()" is now an alternative to "import pdb ; pdb.set_trace()"
08:28:22 gibi bauzas: OK
08:28:41 gibi kashyap: ohh, nice. then I TIl that today too :)
08:29:08 bauzas gibi: do you know if someone already provided a nova change for using the new grenade-skip-level job ?
08:29:37 bauzas gibi: if no, I'll do it
08:29:46 kashyap gibi: https://peps.python.org/pep-0553/ :)
08:30:21 kashyap See the 4 points in the Rationale section
08:30:37 gibi bauzas: I only see this https://review.opendev.org/q/topic:test-grenade-skip-level
08:31:05 gibi that adds the job the check queue
08:31:20 EugenMayer gibi thank you. Which logs would you look for on the conductor?
08:33:13 EugenMayer gibi i think the candidate is https://gist.github.com/EugenMayer/db07bf3ecc8bad464afb0040700d644d
08:36:00 bauzas gibi: oh right, I forgot about it
08:36:11 bauzas https://review.opendev.org/c/openstack/tempest/+/830670/1/zuul.d/integrated-gate.yaml we test it for the whole integrated gate
08:36:11 gibi EugenMayer: yeah that means that service could not communicate with keystone
08:37:07 EugenMayer gibi are there somewhat rate limits or not? I guess it uses the LB to do that and it might be somewhat saturated. I could not understand why this suddently (but consistently) happens
08:38:24 gibi EugenMayer: [Errno 113] EHOSTUNREACH seems to be a network connection error to me
08:39:15 EugenMayer Seems so, but how could that happen out of a sudden, that's odd. Running xena with OVN here.
08:40:36 gibi kashyap: I could have been the author of this PEP :)
08:40:47 kashyap gibi: Haha, I'll believe you
08:40:58 gibi forgetting the semicolon
08:41:10 gibi that is typical :)
08:44:17 gibi EugenMayer: I have not furter ideas what can cause that. I suggest you to troubleshoot your network infra.
08:47:58 EugenMayer gibi well the entire infra is based on OVN / openstack .. that's my issue here. You assume asking over in neutron what could cause this, right?
08:48:08 EugenMayer (or how to track it down)
08:48:50 gibi EugenMayer: yeah you can try over there too
08:49:01 EugenMayer Thank you!
08:49:31 EugenMayer gibi isn't it odd alltogether that this happens, since the conductor is running on the controller itself?
08:50:11 EugenMayer gibi even the load-balancer that it is offered via is running on the controller, the host where actually the conductor uns on
08:50:31 gibi yeah it is definetly odd
09:56:10 bauzas damn, we're playing against Zuul by now
09:58:31 bauzas sean-k-mooney: we have 2 open bug reports for RC1 https://bugs.launchpad.net/nova/+bugs?field.tag=yoga-rc-potential that relate to https://review.opendev.org/c/openstack/nova/+/828570
09:59:08 bauzas sean-k-mooney: given I don't see your +2 for the main change, can we punt https://bugs.launchpad.net/nova/+bug/1949808 and https://bugs.launchpad.net/nova/+bug/1960412 off the RC1 ?
10:07:05 bauzas humpf, this looks sad we no longer have translations https://review.opendev.org/q/topic:zanata%252Ftranslations
10:07:58 bauzas last one we had was in ussuri https://review.opendev.org/c/openstack/nova/+/723160
10:08:33 bauzas my bad, xena
10:41:36 opendevreview kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156
10:41:37 opendevreview kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164
10:48:57 noonedeadpunk hey there! I have issue that is kind of related to https://bugs.launchpad.net/nova/+bug/1778563
10:49:43 noonedeadpunk so mdev get re-crearted and allocated during migration. But I guess nothing has been done when compute host got rebooted?
10:50:26 noonedeadpunk So with compute reboot mdev device are gone, so nova-compute jsut refuse to start
10:50:57 noonedeadpunk with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/
10:51:54 noonedeadpunk so basically on nova-compute start we need to ensure that devices exist same way we do during migration I guess?
10:52:29 noonedeadpunk it's on V just in case, so not sure maybe it's already fixed on later branches
11:08:19 gibi noonedeadpunk: I think it is an open bug https://bugs.launchpad.net/nova/+bug/1900800
11:10:38 noonedeadpunk oh, mdevctl define, nice, thanks!
11:11:38 gibi noonedeadpunk: happy to help :)
11:18:15 noonedeadpunk gibi: I wonder if it's worth to mention it on https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats ?
11:19:02 gibi noonedeadpunk: good point. I think it would be good to list it there. If you have time please push a small doc patch.
11:23:52 sean-k-mooney[m] we dont currenlty use mdevctl and if we wer too we would have to basically re write how we do mdev management
11:24:14 sean-k-mooney[m] today we expect nova to creat the medevs after a host reboot
11:24:20 sean-k-mooney[m] when it recreates the vms
11:24:52 sean-k-mooney[m] if we want to use mdev ctl in the future we need to strart tacking mdevs like pci devices or pmem
11:24:52 noonedeadpunk but it doesn't?
11:25:05 noonedeadpunk I mean - nova jsut crash
11:25:16 sean-k-mooney[m] it should when you start the vm
11:26:04 sean-k-mooney[m] the current bug is just that a bug
11:26:07 noonedeadpunk hm... maybe it's result of resume_guests_state_on_host_boot then...
11:26:24 sean-k-mooney[m] maybe
11:26:30 noonedeadpunk As what I see when trying to start nova-compute - crash with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/
11:26:43 noonedeadpunk So to start nova compute I need mdev to be created
11:27:13 noonedeadpunk Need to try dropping resume_guests_state_on_host_boot indeed
11:31:01 sean-k-mooney[m] bauzas: wasnt there someone already working on a fix for that ^ by the way
11:31:41 sean-k-mooney[m] i remember talking to you about fixing it a few months ago but dont recall if you strated to impelent it but wasnt someone else looking at fixing the issue
11:32:05 sean-k-mooney[m] @noon
11:32:48 sean-k-mooney[m] noonedeadpunk: the issue i have with mdevctl is that nova expect that the mdevs do not exist in the normal code path
11:33:08 sean-k-mooney[m] so if you just precreate arbitary mdevs it will break the ablity to create vms
11:33:52 sean-k-mooney[m] and as a a tool it has little other utility if all you are doing is rectateing the mdevs used by the existing vms
11:34:21 sean-k-mooney[m] creating an mdev is just echoing a uuid into a file in /sys
11:37:25 sean-k-mooney[m] mdevctl also used to not be pakaged on anything other then fedora https://repology.org/project/mdevctl/versions its a little better now as tis actull in debian and ubuntu too but its not a tool that you could previously rely being in your distro package
11:37:34 noonedeadpunk well, I had other issue, but maybe because in the region we run V
11:38:00 dmitriis gibi: o/ Apologies for an extra ping, just wanted to ask if you're good with https://review.opendev.org/c/openstack/nova/+/829974 since you've reviewed it before.
11:38:04 dmitriis Not sure if it's appropriate to land it at this point or not but that's more of a fix + test change rather than a new feature.
11:38:38 noonedeadpunk So to start nova-compute we indeed had to echo uuid to /sys
11:39:06 sean-k-mooney[m] ya that is the workaround for now
11:39:15 noonedeadpunk but it's hard as you need to get resource from placement to understand mapping of uuid to mdev pci device
11:39:19 sean-k-mooney[m] i prefer recommendign that as its safer then mdevctl
11:39:43 sean-k-mooney[m] yes as i siad i tought someone had wirtten that up
11:40:02 noonedeadpunk yeah, I see
11:40:14 sean-k-mooney[m] i know bauzas started at one point
11:41:18 sean-k-mooney[m] i remember discussing the algrothim with them and how they would have to use placement to find the parent pci device to know whic device to create the mdev on since that is not in the xml
11:42:42 noonedeadpunk well another weird thing is that mdev uuid is just random thing, while tbh I'd expect it to be resource uuid. Which would make things easier...
11:44:03 noonedeadpunk As we have resource uuid and then mdev uuid is just another thing that is stored _only_ in xml
11:44:22 noonedeadpunk (in case I'm not missing anything
11:44:56 sean-k-mooney[m] its randome because we do not track them in the db
11:45:08 gibi dmitriis, sean-k-mooney[m], bauzas: I'm generally OK with https://review.opendev.org/c/openstack/nova/+/829974 I do noted that this patch is now wide the gap we need to close later as part of https://bugs.launchpad.net/nova/+bug/1961587 as it adds more sysfs calls to our neutron code.
11:45:10 sean-k-mooney[m] i have been trying to change that since before we actully added the mdev feature
11:46:41 sean-k-mooney[m] noonedeadpunk: what i would like use to evolve to is have operators precreat the mdevs and we would track them in the resouces table in the db
11:46:52 sean-k-mooney[m] then claim and allocate them to instances
11:47:04 sean-k-mooney[m] or tack them in the pcidevice table
11:47:33 sean-k-mooney[m] part of the reason i want to do that is i want to get rid of the persitnt libvirt domain xml
11:47:59 sean-k-mooney[m] right now vgpus are the only thing that need to the persitent domain xml
11:48:40 dmitriis gibi: ack, I am going to look at changing this to use extra info instead in a follow-up
11:48:41 noonedeadpunk well, I wasn't having much fun with plain qemu without libvirt, as if you don't need config you can jsut operate qemu directly?
11:50:46 noonedeadpunk tbh for me from operator prespective is preferable that nova manage mdev creation. as otherwise there will be tons of nasty hooks which everybody do. This can be handled by deployment tools ofc, but considering devices are not persistant and drop on reboot... dunno.
11:51:28 noonedeadpunk and if uuid for mdev will be taken as placement resource id - wouldn't it solve issue with persistant libvirt config?
11:52:08 noonedeadpunk as at time mdev is created, I guess allocation should be already claimed and resource provided where to create it?
11:52:20 gibi dmitriis: ack, thanks
11:57:50 sean-k-mooney[m] noonedeadpunk: you can have multiple mdevs attached to a vm so we cant use placemnt ids for the uuid

Earlier   Later