| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-09 | |||
| 17:50:59 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/822048 | |
| 17:51:00 | opendevreview | melanie witt proposed openstack/nova stable/ussuri: [rt] Apply migration context for incoming migrations https://review.opendev.org/c/openstack/nova/+/822050 | |
| 19:04:57 | elodilles | melwitt: could you please have a quick look at this stable/queens gate fix? periodic job is failing on it and it would be nice to make the error go away o:) https://review.opendev.org/c/openstack/nova/+/830327 | |
| 19:14:52 | melwitt | elodilles: sure thing | |
| 21:47:58 | opendevreview | Merged openstack/nova master: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/830966 | |
| 21:48:06 | opendevreview | Merged openstack/nova stable/queens: [stable-only] Fix sphinx doc job https://review.opendev.org/c/openstack/nova/+/830327 | |
| 22:48:17 | opendevreview | Artom Lifshitz proposed openstack/nova master: Update PCI requests in request spec on resize https://review.opendev.org/c/openstack/nova/+/806049 | |
| #openstack-nova - 2022-03-10 | |||
| 00:39:12 | opendevreview | Artom Lifshitz proposed openstack/nova master: Update PCI requests in request spec on resize https://review.opendev.org/c/openstack/nova/+/806049 | |
| 05:50:30 | elodilles | melwitt: thanks! :) | |
| 07:12:28 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:14:53 | opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | |
| 07:22:57 | EugenMayer | Hello, out of a 'sudden' when deploying a series of VMs ussing terraform on openstack (spawning a k8s cluster of 4 nodes) some of them (usually 1) stucks in 'spawning' mode in openstack. We did this about 50 times already before, so it was working flawlessly - but suddenly it broke. What tools would i have to understand why they are in the | |
| 07:22:57 | EugenMayer | 'spawning' state and what hinders the VM to actually spawn? | |
| 08:17:56 | gibi | bauzas: hi! prelude looks good to me. do we still wait for other +2s or should I just approve it | |
| 08:17:59 | gibi | ? | |
| 08:19:08 | gibi | EugenMayer: you have to track down what step that VM failed. I would first look at the conductor logs for error and the log of the nova-compute where the VM is scheduled to. | |
| 08:21:28 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/placement/+/832979 | |
| 08:21:31 | opendevreview | OpenStack Release Bot proposed openstack/placement stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/placement/+/832981 | |
| 08:21:35 | opendevreview | OpenStack Release Bot proposed openstack/placement master: Update master for stable/yoga https://review.opendev.org/c/openstack/placement/+/832983 | |
| 08:21:39 | opendevreview | OpenStack Release Bot proposed openstack/placement master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/placement/+/832985 | |
| 08:25:51 | bauzas | gibi: hey, maybe we can hold it until noon | |
| 08:25:59 | kashyap | Completely aside, TIL: with Py 3.7, "breakpoint()" is now an alternative to "import pdb ; pdb.set_trace()" | |
| 08:28:22 | gibi | bauzas: OK | |
| 08:28:41 | gibi | kashyap: ohh, nice. then I TIl that today too :) | |
| 08:29:08 | bauzas | gibi: do you know if someone already provided a nova change for using the new grenade-skip-level job ? | |
| 08:29:37 | bauzas | gibi: if no, I'll do it | |
| 08:29:46 | kashyap | gibi: https://peps.python.org/pep-0553/ :) | |
| 08:30:21 | kashyap | See the 4 points in the Rationale section | |
| 08:30:37 | gibi | bauzas: I only see this https://review.opendev.org/q/topic:test-grenade-skip-level | |
| 08:31:05 | gibi | that adds the job the check queue | |
| 08:31:20 | EugenMayer | gibi thank you. Which logs would you look for on the conductor? | |
| 08:33:13 | EugenMayer | gibi i think the candidate is https://gist.github.com/EugenMayer/db07bf3ecc8bad464afb0040700d644d | |
| 08:36:00 | bauzas | gibi: oh right, I forgot about it | |
| 08:36:11 | bauzas | https://review.opendev.org/c/openstack/tempest/+/830670/1/zuul.d/integrated-gate.yaml we test it for the whole integrated gate | |
| 08:36:11 | gibi | EugenMayer: yeah that means that service could not communicate with keystone | |
| 08:37:07 | EugenMayer | gibi are there somewhat rate limits or not? I guess it uses the LB to do that and it might be somewhat saturated. I could not understand why this suddently (but consistently) happens | |
| 08:38:24 | gibi | EugenMayer: [Errno 113] EHOSTUNREACH seems to be a network connection error to me | |
| 08:39:15 | EugenMayer | Seems so, but how could that happen out of a sudden, that's odd. Running xena with OVN here. | |
| 08:40:36 | gibi | kashyap: I could have been the author of this PEP :) | |
| 08:40:47 | kashyap | gibi: Haha, I'll believe you | |
| 08:40:58 | gibi | forgetting the semicolon | |
| 08:41:10 | gibi | that is typical :) | |
| 08:44:17 | gibi | EugenMayer: I have not furter ideas what can cause that. I suggest you to troubleshoot your network infra. | |
| 08:47:58 | EugenMayer | gibi well the entire infra is based on OVN / openstack .. that's my issue here. You assume asking over in neutron what could cause this, right? | |
| 08:48:08 | EugenMayer | (or how to track it down) | |
| 08:48:50 | gibi | EugenMayer: yeah you can try over there too | |
| 08:49:01 | EugenMayer | Thank you! | |
| 08:49:31 | EugenMayer | gibi isn't it odd alltogether that this happens, since the conductor is running on the controller itself? | |
| 08:50:11 | EugenMayer | gibi even the load-balancer that it is offered via is running on the controller, the host where actually the conductor uns on | |
| 08:50:31 | gibi | yeah it is definetly odd | |
| 09:56:10 | bauzas | damn, we're playing against Zuul by now | |
| 09:58:31 | bauzas | sean-k-mooney: we have 2 open bug reports for RC1 https://bugs.launchpad.net/nova/+bugs?field.tag=yoga-rc-potential that relate to https://review.opendev.org/c/openstack/nova/+/828570 | |
| 09:59:08 | bauzas | sean-k-mooney: given I don't see your +2 for the main change, can we punt https://bugs.launchpad.net/nova/+bug/1949808 and https://bugs.launchpad.net/nova/+bug/1960412 off the RC1 ? | |
| 10:07:05 | bauzas | humpf, this looks sad we no longer have translations https://review.opendev.org/q/topic:zanata%252Ftranslations | |
| 10:07:58 | bauzas | last one we had was in ussuri https://review.opendev.org/c/openstack/nova/+/723160 | |
| 10:08:33 | bauzas | my bad, xena | |
| 10:41:36 | opendevreview | kiran pawar proposed openstack/nova master: VMware: Split out VMwareAPISession https://review.opendev.org/c/openstack/nova/+/832156 | |
| 10:41:37 | opendevreview | kiran pawar proposed openstack/nova master: VMware: StableMoRefProxy for moref recovery https://review.opendev.org/c/openstack/nova/+/832164 | |
| 10:48:57 | noonedeadpunk | hey there! I have issue that is kind of related to https://bugs.launchpad.net/nova/+bug/1778563 | |
| 10:49:43 | noonedeadpunk | so mdev get re-crearted and allocated during migration. But I guess nothing has been done when compute host got rebooted? | |
| 10:50:26 | noonedeadpunk | So with compute reboot mdev device are gone, so nova-compute jsut refuse to start | |
| 10:50:57 | noonedeadpunk | with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/ | |
| 10:51:54 | noonedeadpunk | so basically on nova-compute start we need to ensure that devices exist same way we do during migration I guess? | |
| 10:52:29 | noonedeadpunk | it's on V just in case, so not sure maybe it's already fixed on later branches | |
| 11:08:19 | gibi | noonedeadpunk: I think it is an open bug https://bugs.launchpad.net/nova/+bug/1900800 | |
| 11:10:38 | noonedeadpunk | oh, mdevctl define, nice, thanks! | |
| 11:11:38 | gibi | noonedeadpunk: happy to help :) | |
| 11:18:15 | noonedeadpunk | gibi: I wonder if it's worth to mention it on https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#caveats ? | |
| 11:19:02 | gibi | noonedeadpunk: good point. I think it would be good to list it there. If you have time please push a small doc patch. | |
| 11:23:52 | sean-k-mooney[m] | we dont currenlty use mdevctl and if we wer too we would have to basically re write how we do mdev management | |
| 11:24:14 | sean-k-mooney[m] | today we expect nova to creat the medevs after a host reboot | |
| 11:24:20 | sean-k-mooney[m] | when it recreates the vms | |
| 11:24:52 | sean-k-mooney[m] | if we want to use mdev ctl in the future we need to strart tacking mdevs like pci devices or pmem | |
| 11:24:52 | noonedeadpunk | but it doesn't? | |
| 11:25:05 | noonedeadpunk | I mean - nova jsut crash | |
| 11:25:16 | sean-k-mooney[m] | it should when you start the vm | |
| 11:26:04 | sean-k-mooney[m] | the current bug is just that a bug | |
| 11:26:07 | noonedeadpunk | hm... maybe it's result of resume_guests_state_on_host_boot then... | |
| 11:26:24 | sean-k-mooney[m] | maybe | |
| 11:26:30 | noonedeadpunk | As what I see when trying to start nova-compute - crash with https://paste.openstack.org/show/b5m5sW2194PRjd2hLGtg/ | |
| 11:26:43 | noonedeadpunk | So to start nova compute I need mdev to be created | |
| 11:27:13 | noonedeadpunk | Need to try dropping resume_guests_state_on_host_boot indeed | |
| 11:31:01 | sean-k-mooney[m] | bauzas: wasnt there someone already working on a fix for that ^ by the way | |
| 11:31:41 | sean-k-mooney[m] | i remember talking to you about fixing it a few months ago but dont recall if you strated to impelent it but wasnt someone else looking at fixing the issue | |
| 11:32:05 | sean-k-mooney[m] | @noon | |
| 11:32:48 | sean-k-mooney[m] | noonedeadpunk: the issue i have with mdevctl is that nova expect that the mdevs do not exist in the normal code path | |
| 11:33:08 | sean-k-mooney[m] | so if you just precreate arbitary mdevs it will break the ablity to create vms | |
| 11:33:52 | sean-k-mooney[m] | and as a a tool it has little other utility if all you are doing is rectateing the mdevs used by the existing vms | |
| 11:34:21 | sean-k-mooney[m] | creating an mdev is just echoing a uuid into a file in /sys | |
| 11:37:25 | sean-k-mooney[m] | mdevctl also used to not be pakaged on anything other then fedora https://repology.org/project/mdevctl/versions its a little better now as tis actull in debian and ubuntu too but its not a tool that you could previously rely being in your distro package | |
| 11:37:34 | noonedeadpunk | well, I had other issue, but maybe because in the region we run V | |
| 11:38:00 | dmitriis | gibi: o/ Apologies for an extra ping, just wanted to ask if you're good with https://review.opendev.org/c/openstack/nova/+/829974 since you've reviewed it before. | |
| 11:38:04 | dmitriis | Not sure if it's appropriate to land it at this point or not but that's more of a fix + test change rather than a new feature. | |
| 11:38:38 | noonedeadpunk | So to start nova-compute we indeed had to echo uuid to /sys | |
| 11:39:06 | sean-k-mooney[m] | ya that is the workaround for now | |
| 11:39:15 | noonedeadpunk | but it's hard as you need to get resource from placement to understand mapping of uuid to mdev pci device | |
| 11:39:19 | sean-k-mooney[m] | i prefer recommendign that as its safer then mdevctl | |
| 11:39:43 | sean-k-mooney[m] | yes as i siad i tought someone had wirtten that up | |
| 11:40:02 | noonedeadpunk | yeah, I see | |
| 11:40:14 | sean-k-mooney[m] | i know bauzas started at one point | |