| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-11 | |||
| 14:02:07 | chateaulav | all hail Zed | |
| 14:29:39 | dansmith | kashyap: I wonder if you might be interested in chasing a CI failure I've seen a couple times on centos jobs, where we ask libvirt to detach a volume, and it just never happens | |
| 14:30:14 | dansmith | it's just, AFAICT, a simple file-based volume so I dunno if it's a guest refusing to let it go or what | |
| 14:36:22 | kashyap | dansmith: Got a bug or a link? (Detach volume sucks the marrow out of my life ... but got used to it :D) | |
| 14:36:35 | dansmith | yeah, lemme get one | |
| 14:39:00 | kashyap | Is it this one? - https://bugs.launchpad.net/nova/+bug/1960346 | |
| 14:41:57 | dansmith | kashyap: https://zuul.opendev.org/t/openstack/build/87df2018e335440f830b08fe1a05bfb7/logs | |
| 14:42:29 | dansmith | kashyap: ah, looks like it! | |
| 14:43:15 | kashyap | Okay, I was hoping: "Oh, not again, not a new one" -- me and Gibi recently spent a few days chasing it down | |
| 14:43:18 | dansmith | so what's the outcome there? it looks like just making a centos job non-voting :) | |
| 14:45:04 | kashyap | There was a direct thread about it w/ libvirt folks. We had two hypotheses: | |
| 14:45:30 | kashyap | (Actually I noted both in the bug. Lemme look what's the current status) | |
| 14:45:56 | kashyap | dansmith: Okay, to summarize, our two hypotheses were these: | |
| 14:45:56 | kashyap | 1) the guest OS didn't confirm the detach | |
| 14:45:57 | kashyap | 2) there was a recent bug in QEMU triggered by using JSON syntax for `-device` | |
| 14:46:05 | kashyap | It turns out to be #1 | |
| 14:46:22 | dansmith | but why does this seem to only happen on the centos jobs? it's all the same cirros in the guest right? | |
| 14:46:25 | kashyap | For now, we've hacked around it by adding extra delay :-( | |
| 14:47:19 | dansmith | are you talking about the wait_until=SSHABLE patches in terms of the delay? | |
| 14:47:42 | kashyap | dansmith: Good question! Damned if I know, why it's happening only on CentOS jobs. (I'm tempted to say "something to do w/ virt-package versions") | |
| 14:48:16 | kashyap | dansmith: No, that's the bigger thing that's not merged yet (IIRC). There was a 120-sec delay patch from Gibi ... lemme look | |
| 14:48:18 | dansmith | because I think it's still happening, AFAICT | |
| 14:48:26 | dansmith | oh jeez, 120s? | |
| 14:48:34 | sean-k-mooney | kashyap: the centos 9 stream jobs are much much newer version fo libvirt and qemu vs ubuntu | |
| 14:49:06 | sean-k-mooney | i think sshable has merged or mostly merged in the last day or two | |
| 14:49:12 | dansmith | right, which is why I'm not sure how we could ever really run on something so bleeding edge where we need stability | |
| 14:49:22 | sean-k-mooney | im not sure if all of the patches are landed but there was proggess on them | |
| 14:49:44 | dansmith | sean-k-mooney: that landed on 3/3 I think | |
| 14:49:45 | sean-k-mooney | well its the same version we will be releaseing stable wallaby on downstream | |
| 14:50:06 | sean-k-mooney | also ubuntu 22.04 will have similar version when it releases | |
| 14:50:24 | sean-k-mooney | so its kind of good that centos 9 stream is catching this | |
| 14:51:29 | dansmith | sean-k-mooney: doesn't stream9's version track upstream closer, like almost constantly moving? | |
| 14:51:49 | kashyap | Not quite; it is the "upstream of RHEL" | |
| 14:52:00 | kashyap | So not as bleeding as Fedora, but not as "stable" as RHEL either | |
| 14:52:07 | sean-k-mooney | dansmith: the pacakges are older the fedora but newer then rhel but not by much | |
| 14:52:13 | kashyap | So you might get the worst of everything w/ Stream :D | |
| 14:52:20 | dansmith | heh | |
| 14:53:00 | sean-k-mooney | effectivly stream is what would be in the next point release of rhel | |
| 14:53:23 | dansmith | well, I'd be concerned if we're adding sleeps to paper over something that would be a real problem with the host triggering the guest to release the block device | |
| 14:53:36 | dansmith | seems like something must have changed if the guest is identical | |
| 14:53:46 | sean-k-mooney | dansmith: the sleep was to see if it was related to the kernel booting | |
| 14:53:50 | sean-k-mooney | not an actual fix | |
| 14:53:56 | sean-k-mooney | before the sshable serires landed | |
| 14:54:05 | dansmith | sean-k-mooney: ah okay I thought kashyap was suggesting that was the workaround | |
| 14:54:07 | sean-k-mooney | at least that was my understanding | |
| 14:54:11 | dansmith | [06:46:25] <kashyap> For now, we've hacked around it by adding extra delay :-( | |
| 14:54:19 | dansmith | this ^ but maybe that's not what he meant | |
| 14:54:40 | sean-k-mooney | the sleep i belive was in tempest not nova if its the patch im thinking of | |
| 14:54:51 | kashyap | dansmith: Sorry, I should've been clearer; I don't see the 120sec patch in tree; but I swear Gibi mentioned it in a thread | |
| 14:55:45 | dansmith | sean-k-mooney: yeah I assumed tempest | |
| 14:55:58 | kashyap | sean-k-mooney: For the SSHable series to land, "someone" (a body) needs to shepherd it...Not sure who has the will for it | |
| 14:56:15 | sean-k-mooney | kashyap: gmann took it over | |
| 14:56:19 | sean-k-mooney | i think its landed already | |
| 14:56:27 | dansmith | yeah it's already merged | |
| 14:56:46 | dansmith | and I still see fails, like this one I think from five days later: | |
| 14:56:57 | dansmith | https://zuul.opendev.org/t/openstack/build/ee63e247893c42a69e096e14f4305850 | |
| 14:57:01 | dansmith | I haven't dug into that one yet but looks the same | |
| 14:58:00 | sean-k-mooney | ya we did not know if the sshable woudl fix it we just hoped it woudl reduce the issue. we saw some cases where test wer doing attach, detach and live migrate all before the kernel was at thet login prompt form the inital boot | |
| 14:58:26 | sean-k-mooney | and the kernel then crashed during the migration | |
| 14:58:55 | dansmith | okay, but nova retries things like the volume detach like ten times, | |
| 14:59:10 | dansmith | so I would expect that would resolve that race for detach right/ | |
| 14:59:12 | sean-k-mooney | ya so we need to remove that retry | |
| 14:59:36 | sean-k-mooney | gibi and i spoke about this a while ago | |
| 14:59:58 | sean-k-mooney | kashyap: can proably verify but i think qemu now considers it an error to detach an already detaching vloume | |
| 15:00:00 | dansmith | is libvirt/qemu only requesting the guest drop it once? because subsequent ones seem to be a libvirt refusal to try again | |
| 15:00:16 | dansmith | yeah I see that error in the logs after the first attempt | |
| 15:00:50 | sean-k-mooney | ya so it used to be undefiend behavior that on some release aborted the detach then qemu started rejecting it and | |
| 15:00:55 | kashyap | sean-k-mooney: Yes, that's right - a device that is already being unplugged, QEMU will consider another attempt at it an error | |
| 15:01:03 | sean-k-mooney | im not really sure what the intended behavior is now | |
| 15:01:26 | dansmith | so .. what to do? if the guest isn't ready to handle the signal we're just screwed until reboot or something? | |
| 15:01:29 | dansmith | that seems pretty broken | |
| 15:02:21 | sean-k-mooney | i dont know honestly. i dont think we want to do what we did in the past for snapshot which was stop and start the vm | |
| 15:02:28 | sean-k-mooney | when live snapshots were not possible | |
| 15:02:36 | dansmith | yeah :/ | |
| 15:02:43 | sean-k-mooney | other then that i dont know of a way to force this form nova | |
| 15:02:52 | dansmith | but again, I'm still curious about why this seems to only happen on centos hosts | |
| 15:03:03 | dansmith | unless you think that behavior changed between 20.04's libvirt and now? | |
| 15:03:12 | sean-k-mooney | we were seeign it on ubuntu too with q35 | |
| 15:03:19 | kashyap | dansmith: So, this was the DNM patch of 120sec delay: https://review.opendev.org/c/openstack/devstack/+/828705/8/.zuul.yaml | |
| 15:03:39 | sean-k-mooney | i dont think we have seen this with pc and ubuntu | |
| 15:03:52 | dansmith | sean-k-mooney: oh, are the centos jobs all q35 by default? | |
| 15:04:03 | sean-k-mooney | no i dont think so | |
| 15:04:04 | kashyap | The thing is, CirrOS needs 10 sec to boot in our CI, but Nova returns "ACTIVE" when the guest spawns. | |
| 15:04:23 | dansmith | kashyap: ah that's the timeout not a delay, and just runs more futile retries I guess? | |
| 15:04:28 | sean-k-mooney | but it could be a combination of the differnet way attach is done with q35 and the new versions treatment of pc | |
| 15:04:57 | kashyap | Right, I read the timeout as "delay before the detach" | |
| 15:04:59 | sean-k-mooney | dansmith: we coudl try using a debian or maybe ubuntu latest job to verify | |
| 15:05:12 | sean-k-mooney | ill check nodepool quickly | |
| 15:05:22 | sean-k-mooney | but i think we have a debian 11 image avaiable | |
| 15:05:37 | sean-k-mooney | and i think it will have a similar libvirt as c9s | |
| 15:05:43 | kashyap | dansmith: Also, yes, Q35 does have some additional hidden special bugs with hot unplug. I was told <cough> CentOS/RHEL 8.6 has better fixes in that area | |
| 15:06:02 | kashyap | (Ouch, I should not use the c-word in these times) | |
| 15:06:18 | dansmith | kashyap: may I suggest <ahem> | |
| 15:07:04 | kashyap | Heh, sure | |
| 15:07:57 | sean-k-mooney | debian-bullseye is there so we could try recreating it on that | |
| 15:08:35 | kashyap | dansmith: Incidentally, I was supposed to work with Red Hat QE today/Monday to test those bits | |
| 15:08:53 | dansmith | sean-k-mooney: so what's the point of doing that? to see if it seems to be characteristic of new libvirt/qemu and not something else about stream9 itself? | |
| 15:09:00 | kashyap | ("those bits" == supposed fixes in QEMU from 8.6) | |
| 15:09:15 | dansmith | kashyap: does that make it to stream9 at some point I hope? | |
| 15:09:33 | kashyap | Yes, definitely. They should. | |