| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-11 | |||
| 09:54:44 | opendevreview | OpenStack Release Bot proposed openstack/nova stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/nova/+/833242 | |
| 09:54:50 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Update master for stable/yoga https://review.opendev.org/c/openstack/nova/+/833243 | |
| 09:54:56 | opendevreview | OpenStack Release Bot proposed openstack/nova master: Add Python3 zed unit tests https://review.opendev.org/c/openstack/nova/+/833244 | |
| 10:02:02 | opendevreview | ribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs(manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 10:02:03 | opendevreview | ribaudr proposed openstack/nova master: [WIP] Enable and use COMPUTE_STORAGE_VIRTIO_FS and COMPUTE_MEM_BACKING_FILE traits. https://review.opendev.org/c/openstack/nova/+/833090 | |
| 10:03:40 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: [stable-only] Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/nova/+/833241 | |
| 10:03:41 | opendevreview | Elod Illes proposed openstack/nova stable/yoga: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/nova/+/833242 | |
| 10:07:40 | bauzas | gibi_pto: enjoy your time off | |
| 10:09:49 | bauzas | elodilles: so not around too on Monday and Tuesday ? | |
| 10:13:54 | elodilles | bauzas: yepp, those are holidays here | |
| 10:14:06 | bauzas | cool, enjoy then ! | |
| 10:15:30 | elodilles | bauzas: thanks :) | |
| 10:16:01 | elodilles | bauzas: though i also might be around, will look at IRC and participate on the weekly meeting :) | |
| 10:16:16 | bauzas | elodilles: this is holidays | |
| 10:16:17 | elodilles | but i don't promise anything o:) | |
| 10:16:32 | elodilles | bauzas: yepp :D | |
| 10:16:33 | bauzas | elodilles: don't be around and rest | |
| 10:16:54 | elodilles | bauzas: will do that as well ;) | |
| 10:17:10 | bauzas | in August, I'll be off for 3.5 weeks and I'd be back 2 days before FF... | |
| 10:17:38 | elodilles | that sounds like a brave thing :D | |
| 10:18:32 | elodilles | but the team is here, so there will be surely someone who can help in :) | |
| 10:19:05 | elodilles | will it be a big family - tesla tour around France? :) | |
| 10:24:27 | opendevreview | changxin xiao proposed openstack/nova master: Fix openstack/nova git repo https://bugs.launchpad.net/nova/+bug/1964548 https://review.opendev.org/c/openstack/nova/+/833248 | |
| 10:28:18 | bauzas | elodilles: just the usual annual family holidays with 2 weeks in Corsica, indeed | |
| 10:28:45 | elodilles | that's also sounds relaxing :) | |
| 10:42:05 | bauzas | with 2 kids ? not exactly relaxing | |
| 10:42:16 | bauzas | but at least, it's called "holidays" | |
| 10:42:38 | bauzas | the only change is I won't dad taxi every day | |
| 10:44:11 | opendevreview | Maxim Korezkij proposed openstack/nova master: fixup! Handle volume attachments https://review.opendev.org/c/openstack/nova/+/833257 | |
| 10:45:39 | opendevreview | Maxim Korezkij proposed openstack/nova master: Handle volume attachments https://review.opendev.org/c/openstack/nova/+/833234 | |
| 11:57:53 | elodilles | bauzas: let's say, different kind of relaxing :D | |
| 12:46:50 | opendevreview | Merged openstack/nova stable/yoga: [stable-only] Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/nova/+/833241 | |
| 12:51:51 | opendevreview | Merged openstack/nova stable/yoga: [stable-only] Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/nova/+/833242 | |
| 13:55:23 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Adds regression test for bug LP#1944619 https://review.opendev.org/c/openstack/nova/+/833166 | |
| 13:55:24 | opendevreview | Erlon R. Cruz proposed openstack/nova master: Fix pre_live_migration rollback https://review.opendev.org/c/openstack/nova/+/815324 | |
| 14:01:32 | bauzas | huzzah, we now have master be Zed :) | |
| 14:01:46 | bauzas | thanks elodilles | |
| 14:02:07 | chateaulav | all hail Zed | |
| 14:29:39 | dansmith | kashyap: I wonder if you might be interested in chasing a CI failure I've seen a couple times on centos jobs, where we ask libvirt to detach a volume, and it just never happens | |
| 14:30:14 | dansmith | it's just, AFAICT, a simple file-based volume so I dunno if it's a guest refusing to let it go or what | |
| 14:36:22 | kashyap | dansmith: Got a bug or a link? (Detach volume sucks the marrow out of my life ... but got used to it :D) | |
| 14:36:35 | dansmith | yeah, lemme get one | |
| 14:39:00 | kashyap | Is it this one? - https://bugs.launchpad.net/nova/+bug/1960346 | |
| 14:41:57 | dansmith | kashyap: https://zuul.opendev.org/t/openstack/build/87df2018e335440f830b08fe1a05bfb7/logs | |
| 14:42:29 | dansmith | kashyap: ah, looks like it! | |
| 14:43:15 | kashyap | Okay, I was hoping: "Oh, not again, not a new one" -- me and Gibi recently spent a few days chasing it down | |
| 14:43:18 | dansmith | so what's the outcome there? it looks like just making a centos job non-voting :) | |
| 14:45:04 | kashyap | There was a direct thread about it w/ libvirt folks. We had two hypotheses: | |
| 14:45:30 | kashyap | (Actually I noted both in the bug. Lemme look what's the current status) | |
| 14:45:56 | kashyap | 1) the guest OS didn't confirm the detach | |
| 14:45:56 | kashyap | dansmith: Okay, to summarize, our two hypotheses were these: | |
| 14:45:57 | kashyap | 2) there was a recent bug in QEMU triggered by using JSON syntax for `-device` | |
| 14:46:05 | kashyap | It turns out to be #1 | |
| 14:46:22 | dansmith | but why does this seem to only happen on the centos jobs? it's all the same cirros in the guest right? | |
| 14:46:25 | kashyap | For now, we've hacked around it by adding extra delay :-( | |
| 14:47:19 | dansmith | are you talking about the wait_until=SSHABLE patches in terms of the delay? | |
| 14:47:42 | kashyap | dansmith: Good question! Damned if I know, why it's happening only on CentOS jobs. (I'm tempted to say "something to do w/ virt-package versions") | |
| 14:48:16 | kashyap | dansmith: No, that's the bigger thing that's not merged yet (IIRC). There was a 120-sec delay patch from Gibi ... lemme look | |
| 14:48:18 | dansmith | because I think it's still happening, AFAICT | |
| 14:48:26 | dansmith | oh jeez, 120s? | |
| 14:48:34 | sean-k-mooney | kashyap: the centos 9 stream jobs are much much newer version fo libvirt and qemu vs ubuntu | |
| 14:49:06 | sean-k-mooney | i think sshable has merged or mostly merged in the last day or two | |
| 14:49:12 | dansmith | right, which is why I'm not sure how we could ever really run on something so bleeding edge where we need stability | |
| 14:49:22 | sean-k-mooney | im not sure if all of the patches are landed but there was proggess on them | |
| 14:49:44 | dansmith | sean-k-mooney: that landed on 3/3 I think | |
| 14:49:45 | sean-k-mooney | well its the same version we will be releaseing stable wallaby on downstream | |
| 14:50:06 | sean-k-mooney | also ubuntu 22.04 will have similar version when it releases | |
| 14:50:24 | sean-k-mooney | so its kind of good that centos 9 stream is catching this | |
| 14:51:29 | dansmith | sean-k-mooney: doesn't stream9's version track upstream closer, like almost constantly moving? | |
| 14:51:49 | kashyap | Not quite; it is the "upstream of RHEL" | |
| 14:52:00 | kashyap | So not as bleeding as Fedora, but not as "stable" as RHEL either | |
| 14:52:07 | sean-k-mooney | dansmith: the pacakges are older the fedora but newer then rhel but not by much | |
| 14:52:13 | kashyap | So you might get the worst of everything w/ Stream :D | |
| 14:52:20 | dansmith | heh | |
| 14:53:00 | sean-k-mooney | effectivly stream is what would be in the next point release of rhel | |
| 14:53:23 | dansmith | well, I'd be concerned if we're adding sleeps to paper over something that would be a real problem with the host triggering the guest to release the block device | |
| 14:53:36 | dansmith | seems like something must have changed if the guest is identical | |
| 14:53:46 | sean-k-mooney | dansmith: the sleep was to see if it was related to the kernel booting | |
| 14:53:50 | sean-k-mooney | not an actual fix | |
| 14:53:56 | sean-k-mooney | before the sshable serires landed | |
| 14:54:05 | dansmith | sean-k-mooney: ah okay I thought kashyap was suggesting that was the workaround | |
| 14:54:07 | sean-k-mooney | at least that was my understanding | |
| 14:54:11 | dansmith | [06:46:25] <kashyap> For now, we've hacked around it by adding extra delay :-( | |
| 14:54:19 | dansmith | this ^ but maybe that's not what he meant | |
| 14:54:40 | sean-k-mooney | the sleep i belive was in tempest not nova if its the patch im thinking of | |
| 14:54:51 | kashyap | dansmith: Sorry, I should've been clearer; I don't see the 120sec patch in tree; but I swear Gibi mentioned it in a thread | |
| 14:55:45 | dansmith | sean-k-mooney: yeah I assumed tempest | |
| 14:55:58 | kashyap | sean-k-mooney: For the SSHable series to land, "someone" (a body) needs to shepherd it...Not sure who has the will for it | |
| 14:56:15 | sean-k-mooney | kashyap: gmann took it over | |
| 14:56:19 | sean-k-mooney | i think its landed already | |
| 14:56:27 | dansmith | yeah it's already merged | |
| 14:56:46 | dansmith | and I still see fails, like this one I think from five days later: | |
| 14:56:57 | dansmith | https://zuul.opendev.org/t/openstack/build/ee63e247893c42a69e096e14f4305850 | |
| 14:57:01 | dansmith | I haven't dug into that one yet but looks the same | |
| 14:58:00 | sean-k-mooney | ya we did not know if the sshable woudl fix it we just hoped it woudl reduce the issue. we saw some cases where test wer doing attach, detach and live migrate all before the kernel was at thet login prompt form the inital boot | |
| 14:58:26 | sean-k-mooney | and the kernel then crashed during the migration | |
| 14:58:55 | dansmith | okay, but nova retries things like the volume detach like ten times, | |
| 14:59:10 | dansmith | so I would expect that would resolve that race for detach right/ | |
| 14:59:12 | sean-k-mooney | ya so we need to remove that retry | |
| 14:59:36 | sean-k-mooney | gibi and i spoke about this a while ago | |