| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-10 | |||
| 08:48:17 | gibi | bauzas: we hit by the missing paramiko typedef after 0.9 | |
| 08:48:37 | bauzas | it was a minor version upgrade from mypy ? woah | |
| 08:49:01 | bauzas | the fix is still -1 from Zuul, not related ? | |
| 08:49:55 | gibi | bauzas: the failing nova-next and nova-grenade-multinode in that patch fails with thing that I saw before on master so I rule them unrelated | |
| 08:49:56 | bauzas | gibi: can we just remove https://github.com/openstack/nova/blob/master/tox.ini#L57 for the moment ? | |
| 08:50:23 | gibi | bauzas: landing that removal need to pass the same test as the patch that is in the check queue | |
| 08:50:37 | gibi | if your tests are unstable then both equally hard to land | |
| 08:51:28 | gibi | s/your/our/ | |
| 08:52:08 | bauzas | gibi: if we would remove https://github.com/openstack/nova/blob/master/tox.ini#L57 then the jobs wouldn't be running | |
| 08:52:32 | gibi | bauzas: removing something from tox ini still triggers nova-next and nova-grenade-multinode isn't it? | |
| 08:52:34 | bauzas | gibi: so, we could fix the gate issue *and then* trying to adding this change | |
| 08:52:53 | bauzas | gibi: good question, AFAIK, I wasn't knowing | |
| 08:53:11 | stephenfin | lyarwood: artom: The reason 'mypy --install-types' wasn't enough is that that requires an existing mypy cache (.mypy_cache), which will only be created if you run mypy. So you'd have to run mypy, wait for it to potentially fail, run '--install-types', then run mypy again | |
| 08:53:20 | bauzas | gibi: lemme try to see this | |
| 08:53:32 | stephenfin | It worked locally because I had the cache already, but failed in the gate because it's a new env | |
| 08:53:51 | bauzas | gibi: stephenfin: so we could just remove mypy first, then trying https://review.opendev.org/c/openstack/nova/+/795533 to be merged, and then adding again mypy | |
| 08:54:18 | gibi | bauzas: go ahead | |
| 08:54:27 | bauzas | doing it now | |
| 08:54:32 | stephenfin | why can't https://review.opendev.org/c/openstack/nova/+/795533 merge? | |
| 08:54:37 | stephenfin | the requirements change has merged | |
| 08:54:51 | stephenfin | and that's the correct fix | |
| 08:55:25 | gibi | I don't want to block bauzas to try another angle. If nova-next and nova-grenade-mutlinode does not trigger a tox.ini change then we might faster land the tox.ini change than the requirement change. | |
| 08:55:40 | gibi | s/trigger a/ trigger on a/ | |
| 08:55:55 | stephenfin | oh, I'm possibly missing context. Is there another issue now? | |
| 08:56:17 | stephenfin | i.e. with nova-next and nova-grenade-multinode ? | |
| 08:56:24 | gibi | we needed to recheck the nova requirement patch couple of times already as it always hit someting either in those jobvs | |
| 08:56:42 | bauzas | stephenfin: the problem is that we run lots of jobs | |
| 08:56:45 | stephenfin | do we have a fix for those jobs? | |
| 08:56:45 | gibi | totally unrealted problems in unstable tests | |
| 08:56:51 | gibi | I don't think so | |
| 08:57:05 | bauzas | stephenfin: so I'm trying to see whether we could just remove the issue without running all of them | |
| 08:58:03 | kashyap | lyarwood: Oh, BTW -- when you're back: the "cancelled" in the QEMU logs has a special meaning for NBD. I forgot that I documented this myself upstream :D | |
| 08:58:20 | kashyap | lyarwood: See step (4) here: https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#qmp-invocation-for-live-storage-migration-with-drive-mirror-nbd | |
| 08:58:28 | bauzas | gibi: stephenfin: I guess we have a bug ? | |
| 08:58:49 | gibi | stephenfin: we have abou 10% failing rate on master in those two jobs | |
| 08:59:05 | opendevreview | Victor Coutellier proposed openstack/nova master: Allow configuration of direct-snapshot feature https://review.opendev.org/c/openstack/nova/+/794837 | |
| 08:59:37 | kashyap | lyarwood: In short, once the mirroring from src --> dest completes, and the _READY event is emitted, source libvirtd issues QMP `block-job-cancel` to gracefully end the mirroring. | |
| 09:00:11 | gibi | so I guess we are just unlucky with the req patch | |
| 09:00:38 | stephenfin | I think so | |
| 09:01:24 | stephenfin | fwiw though I wouldn't be in favour of simply dropping the requirements fix so we can merge other stuff, if the reason the requirements patch is failing is unrelated | |
| 09:01:56 | opendevreview | Sylvain Bauza proposed openstack/nova master: Removing mypy to fix the nova CI https://review.opendev.org/c/openstack/nova/+/795744 | |
| 09:01:57 | gibi | stephenfin: not dropping the req fix | |
| 09:02:08 | gibi | stephenfin: if it lands then we are done | |
| 09:02:21 | gibi | if bauzas's patch lands first, then we revert that when yours land | |
| 09:02:33 | bauzas | gibi: stephenfin: patch is up for just removing mypy run until we fix the requirements | |
| 09:02:39 | stephenfin | that makes no sense to me though | |
| 09:02:46 | stephenfin | the mypy run is causing the other failures | |
| 09:02:48 | stephenfin | *isn't | |
| 09:03:01 | bauzas | stephenfin: sure | |
| 09:03:08 | stephenfin | so they have an equal chance of failing randomly | |
| 09:03:23 | bauzas | I'm just proposing this one because I guess nova-next WONT be running on my patch | |
| 09:03:35 | bauzas | it's just a tox.ini change | |
| 09:03:39 | stephenfin | but what does this achieve? | |
| 09:03:39 | gibi | stephenfin: not equal chance iff the tox.ini change does not trigger the unstable jobs | |
| 09:03:56 | gibi | stephenfin: if it trigger the same job, then I agree that it mypy removal patch is pointless | |
| 09:04:08 | stephenfin | even if it doesn't, so what? | |
| 09:04:23 | bauzas | gibi: stephenfin: yup, indeed, if this runs the same jobs, then nevermind my one | |
| 09:04:53 | bauzas | stephenfin: I was off yesterday but from what I've seen our gate is blocked | |
| 09:05:03 | bauzas | I just want to unblock it asap | |
| 09:05:10 | stephenfin | right, but it's blocked because of the flaky tests | |
| 09:05:16 | stephenfin | not because of the mypy thing | |
| 09:05:16 | bauzas | the clean fix requires a requirement bump | |
| 09:05:22 | stephenfin | we have a fix for the mypy thing | |
| 09:05:29 | bauzas | stephenfin: yup, I saw it | |
| 09:05:43 | bauzas | stephenfin: I'm just trying to see whether we can land things easier | |
| 09:06:00 | bauzas | again, unblocking the gate seems to me the most important | |
| 09:06:11 | bauzas | we could revert stuff later | |
| 09:06:16 | stephenfin | but you still won't be able to land anything that causes nova-next or nova-grenade-mutlinode to run? | |
| 09:06:20 | stephenfin | so it's not unblocked | |
| 09:06:38 | bauzas | stephenfin: that's a classic chicken-and-egg issue | |
| 09:06:47 | bauzas | you have 2 unrelated gate issues | |
| 09:06:48 | stephenfin | I mean, if they're flaky then they're flaky for everything, surely? | |
| 09:07:07 | bauzas | stephenfin: I don't disagree | |
| 09:07:16 | bauzas | 20% is a high rate of flakiness | |
| 09:07:18 | stephenfin | then we achieve nothing with this | |
| 09:07:45 | gibi | for me moving from a full red due to pep8 (mypy) failure to a falky red due to unstable jobs is still progress | |
| 09:07:47 | bauzas | except we go from 100% of failures to a random 20% from what I understando | |
| 09:07:52 | bauzas | this | |
| 09:08:40 | bauzas | either way, I proposed but I don't have opinions | |
| 09:08:59 | bauzas | the last call, tho, is that I need to get my kids in a min | |
| 09:09:59 | bauzas | https://zuul.opendev.org/t/openstack/status#795744 | |
| 09:10:14 | bauzas | we don't run the flaky jobs | |
| 09:10:52 | bauzas | 30 mins roughly sorry | |
| 09:11:00 | stephenfin | idk, if the failure rate is that high that we can't land a simple reqs patch, that would suggest we need to be working on the other jobs. I see lyarwood has already been looking. I can start now too | |
| 09:11:39 | bauzas | stephenfin: yup, we need to parallelize efforts, I don't disagree | |
| 09:12:01 | bauzas | my patch is just a hack | |
| 09:12:17 | bauzas | and we need to consider themigrate and next jobs as the top prio | |
| 09:12:22 | bauzas | I absolutely don't disagree | |
| 09:12:43 | bauzas | and I also absolutely agree we need to land the types-paramiko changes | |
| 09:13:18 | bauzas | it's just, again, a way to unblock the gate even if flakey with the other jobs | |
| 09:13:21 | opendevreview | Victor Coutellier proposed openstack/nova master: Allow configuration of direct-snapshot feature https://review.opendev.org/c/openstack/nova/+/794837 | |
| 09:50:24 | gibi | the requirement patch failed again. I'm looking at the grenade failure... | |
| 09:50:36 | gibi | https://f141fb01d9c1d07df646-94aaf0771088c81abb9a09d47e91a608.ssl.cf1.rackcdn.com/795533/5/check/nova-grenade-multinode/24485b8/testr_results.html | |
| 09:55:04 | elodilles | gibi: sorry, meanwhile i've commented 'recheck' on it :X | |
| 09:55:11 | gibi | elodilles: no worries | |
| 09:55:15 | gibi | the failure is unrelated | |
| 09:55:32 | gibi | but based on this morining discussion above I feel we need to stabilize our gate | |
| 09:55:54 | bauzas | I'm back | |
| 09:56:52 | bauzas | gibi: stephenfin disagrees with the quick fix approach of removing mypy so I don't want to opiniate here | |
| 09:57:25 | bauzas | what i agree tho is that the gate failures are a PITA that need other pair of eyes | |