| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-09 | |||
| 21:47:58 | opendevreview | Merged openstack/nova master: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/830966 | |
| 21:48:06 | opendevreview | Merged openstack/nova stable/queens: [stable-only] Fix sphinx doc job https://review.opendev.org/c/openstack/nova/+/830327 | |
| 22:48:17 | opendevreview | Artom Lifshitz proposed openstack/nova master: Update PCI requests in request spec on resize https://review.opendev.org/c/openstack/nova/+/806049 | |
| #openstack-nova - 2022-03-10 | |||
| 00:39:12 | opendevreview | Artom Lifshitz proposed openstack/nova master: Update PCI requests in request spec on resize https://review.opendev.org/c/openstack/nova/+/806049 | |
| 05:50:30 | elodilles | melwitt: thanks! :) | |
| 07:12:28 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:14:53 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:22:57 | EugenMayer | 'spawning' state and what hinders the VM to actually spawn? | |
| 07:22:57 | EugenMayer | Hello, out of a 'sudden' when deploying a series of VMs ussing terraform on openstack (spawning a k8s cluster of 4 nodes) some of them (usually 1) stucks in 'spawning' mode in openstack. We did this about 50 times already before, so it was working flawlessly - but suddenly it broke. What tools would i have to understand why they are in the | |
| 08:17:56 | gibi | bauzas: hi! prelude looks good to me. do we still wait for other +2s or should I just approve it | |
| 08:17:59 | gibi | ? | |
| 08:19:08 | gibi | EugenMayer: you have to track down what step that VM failed. I would first look at the conductor logs for error and the log of the nova-compute where the VM is scheduled to. | |
| 08:21:28 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/placement/+/832979 | |
| 08:21:31 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/placement/+/832981 | |
| 08:21:35 | opendevreview | OpenStack Release Bot proposed openstack/placement master: Update master for stable/yoga https://review.opendev.org/c/openstack/placement/+/832983 | |
| 08:21:39 | opendevreview | OpenStack Release Bot proposed openstack/placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/placement/+/832985 | |
| 08:25:51 | bauzas | gibi: hey, maybe we can hold it until noon | |
| 08:25:59 | kashyap | Completely aside, TIL: with Py 3.7, "breakpoint()" is now an alternative to "import pdb ; pdb.set_trace()" | |
| 08:28:22 | gibi | bauzas: OK | |
| 08:28:41 | gibi | kashyap: ohh, nice. then I TIl that today too :) | |
| 08:29:08 | bauzas | gibi: do you know if someone already provided a nova change for using the new grenade-skip-level job ? | |
| 08:29:37 | bauzas | gibi: if no, I'll do it | |
| 08:29:46 | kashyap | gibi: https://peps.python.org/pep-0553/ :) | |
| 08:30:21 | kashyap | See the 4 points in the Rationale section | |
| 08:30:37 | gibi | bauzas: I only see this https://review.opendev.org/q/topic:test-grenade-skip-level | |
| 08:31:05 | gibi | that adds the job the check queue | |
| 08:31:20 | EugenMayer | gibi thank you. Which logs would you look for on the conductor? | |
| 08:33:13 | EugenMayer | gibi i think the candidate is https://gist.github.com/EugenMayer/db07bf3ecc8bad464afb0040700d644d | |
| 08:36:00 | bauzas | gibi: oh right, I forgot about it | |
| 08:36:11 | gibi | EugenMayer: yeah that means that service could not communicate with keystone | |
| 08:36:11 | bauzas | https://review.opendev.org/c/openstack/tempest/+/830670/1/zuul.d/integrated-gate.yaml we test it for the whole integrated gate | |
| 08:37:07 | EugenMayer | gibi are there somewhat rate limits or not? I guess it uses the LB to do that and it might be somewhat saturated. I could not understand why this suddently (but consistently) happens | |
| 08:38:24 | gibi | EugenMayer: [Errno 113] EHOSTUNREACH seems to be a network connection error to me | |
| 08:39:15 | EugenMayer | Seems so, but how could that happen out of a sudden, that's odd. Running xena with OVN here. | |
| 08:40:36 | gibi | kashyap: I could have been the author of this PEP :) | |
| 08:40:47 | kashyap | gibi: Haha, I'll believe you | |
| 08:40:58 | gibi | forgetting the semicolon | |
| 08:41:10 | gibi | that is typical :) | |
| 08:44:17 | gibi | EugenMayer: I have not furter ideas what can cause that. I suggest you to troubleshoot your network infra. | |
| 08:47:58 | EugenMayer | gibi well the entire infra is based on OVN / openstack .. that's my issue here. You assume asking over in neutron what could cause this, right? | |
| 08:48:08 | EugenMayer | (or how to track it down) | |
| 08:48:50 | gibi | EugenMayer: yeah you can try over there too | |
| 08:49:01 | EugenMayer | Thank you! | |
| 08:49:31 | EugenMayer | gibi isn't it odd alltogether that this happens, since the conductor is running on the controller itself? | |
| 08:50:11 | EugenMayer | gibi even the load-balancer that it is offered via is running on the controller, the host where actually the conductor uns on | |
| 08:50:31 | gibi | yeah it is definetly odd | |
| 09:56:10 | bauzas | damn, we're playing against Zuul by now | |
| 09:58:31 | bauzas | sean-k-mooney: we have 2 open bug reports for RC1 https://bugs.launchpad.net/nova/+bugs?field.tag=yoga-rc-potential that relate to https://review.opendev.org/c/openstack/nova/+/828570 | |
| 09:59:08 | bauzas | sean-k-mooney: given I don't see your +2 for the main change, can we punt https://bugs.launchpad.net/nova/+bug/1949808 and https://bugs.launchpad.net/nova/+bug/1960412 off the RC1 ? | |
| 10:07:05 | bauzas | humpf, this looks sad we no longer have translations https://review.opendev.org/q/topic:zanata%252Ftranslations | |
| 10:07:58 | bauzas | last one we had was in ussuri https://review.opendev.org/c/openstack/nova/+/723160 | |
| 10:08:33 | bauzas | my bad, xena | |
| 10:41:36 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 10:41:37 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 10:48:57 | noonedeadpunk | hey there! I have issue that is kind of related to https://bugs.launchpad.net/nova/+bug/1778563 | |
| 10:49:43 | noonedeadpunk | so mdev get re-crearted and allocated during migration. But I guess nothing has been done when compute host got rebooted? | |
| 10:50:26 | noonedeadpunk | So with compute reboot mdev device are gone, so nova-compute jsut refuse to start | |
| 10:50:57 | noonedeadpunk | with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/ | |
| 10:51:54 | noonedeadpunk | so basically on nova-compute start we need to ensure that devices exist same way we do during migration I guess? | |
| 10:52:29 | noonedeadpunk | it's on V just in case, so not sure maybe it's already fixed on later branches | |
| 11:08:19 | gibi | noonedeadpunk: I think it is an open bug https://bugs.launchpad.net/nova/+bug/1900800 | |
| 11:10:38 | noonedeadpunk | oh, mdevctl define, nice, thanks! | |
| 11:11:38 | gibi | noonedeadpunk: happy to help :) | |
| 11:18:15 | noonedeadpunk | gibi: I wonder if it's worth to mention it on https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats ? | |
| 11:19:02 | gibi | noonedeadpunk: good point. I think it would be good to list it there. If you have time please push a small doc patch. | |
| 11:23:52 | sean-k-mooney[m] | we dont currenlty use mdevctl and if we wer too we would have to basically re write how we do mdev management | |
| 11:24:14 | sean-k-mooney[m] | today we expect nova to creat the medevs after a host reboot | |
| 11:24:20 | sean-k-mooney[m] | when it recreates the vms | |
| 11:24:52 | noonedeadpunk | but it doesn't? | |
| 11:24:52 | sean-k-mooney[m] | if we want to use mdev ctl in the future we need to strart tacking mdevs like pci devices or pmem | |
| 11:25:05 | noonedeadpunk | I mean - nova jsut crash | |
| 11:25:16 | sean-k-mooney[m] | it should when you start the vm | |
| 11:26:04 | sean-k-mooney[m] | the current bug is just that a bug | |
| 11:26:07 | noonedeadpunk | hm... maybe it's result of resume_guests_state_on_host_boot then... | |
| 11:26:24 | sean-k-mooney[m] | maybe | |
| 11:26:30 | noonedeadpunk | As what I see when trying to start nova-compute - crash with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/ | |
| 11:26:43 | noonedeadpunk | So to start nova compute I need mdev to be created | |
| 11:27:13 | noonedeadpunk | Need to try dropping resume_guests_state_on_host_boot indeed | |
| 11:31:01 | sean-k-mooney[m] | bauzas: wasnt there someone already working on a fix for that ^ by the way | |
| 11:31:41 | sean-k-mooney[m] | i remember talking to you about fixing it a few months ago but dont recall if you strated to impelent it but wasnt someone else looking at fixing the issue | |
| 11:32:05 | sean-k-mooney[m] | @noon | |
| 11:32:48 | sean-k-mooney[m] | noonedeadpunk: the issue i have with mdevctl is that nova expect that the mdevs do not exist in the normal code path | |
| 11:33:08 | sean-k-mooney[m] | so if you just precreate arbitary mdevs it will break the ablity to create vms | |
| 11:33:52 | sean-k-mooney[m] | and as a a tool it has little other utility if all you are doing is rectateing the mdevs used by the existing vms | |
| 11:34:21 | sean-k-mooney[m] | creating an mdev is just echoing a uuid into a file in /sys | |
| 11:37:25 | sean-k-mooney[m] | mdevctl also used to not be pakaged on anything other then fedora https://repology.org/project/mdevctl/versions its a little better now as tis actull in debian and ubuntu too but its not a tool that you could previously rely being in your distro package | |
| 11:37:34 | noonedeadpunk | well, I had other issue, but maybe because in the region we run V | |
| 11:38:00 | dmitriis | gibi: o/ Apologies for an extra ping, just wanted to ask if you're good with https://review.opendev.org/c/openstack/nova/+/829974 since you've reviewed it before. | |
| 11:38:04 | dmitriis | Not sure if it's appropriate to land it at this point or not but that's more of a fix + test change rather than a new feature. | |
| 11:38:38 | noonedeadpunk | So to start nova-compute we indeed had to echo uuid to /sys | |
| 11:39:06 | sean-k-mooney[m] | ya that is the workaround for now | |
| 11:39:15 | noonedeadpunk | but it's hard as you need to get resource from placement to understand mapping of uuid to mdev pci device | |
| 11:39:19 | sean-k-mooney[m] | i prefer recommendign that as its safer then mdevctl | |
| 11:39:43 | sean-k-mooney[m] | yes as i siad i tought someone had wirtten that up | |
| 11:40:02 | noonedeadpunk | yeah, I see | |
| 11:40:14 | sean-k-mooney[m] | i know bauzas started at one point | |
| 11:41:18 | sean-k-mooney[m] | i remember discussing the algrothim with them and how they would have to use placement to find the parent pci device to know whic device to create the mdev on since that is not in the xml | |
| 11:42:42 | noonedeadpunk | well another weird thing is that mdev uuid is just random thing, while tbh I'd expect it to be resource uuid. Which would make things easier... | |
| 11:44:03 | noonedeadpunk | As we have resource uuid and then mdev uuid is just another thing that is stored _only_ in xml | |
| 11:44:22 | noonedeadpunk | (in case I'm not missing anything | |