| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-17 | |||
| 16:31:34 | elodilles | nothing more to add for now :X | |
| 16:31:55 | bauzas | elodilles: at one time, we'll need to cut the rope on the older branches | |
| 16:32:38 | sean-k-mooney | older starting form ? | |
| 16:32:38 | elodilles | bauzas: indeed. now ussuri is in bad shape for weeks now... | |
| 16:32:50 | sean-k-mooney | hum | |
| 16:32:52 | bauzas | sean-k-mooney: ussuri | |
| 16:33:00 | sean-k-mooney | so we woudl be droping train and below too | |
| 16:33:23 | elodilles | actually it fails due to multiple intermittent failures | |
| 16:33:27 | bauzas | we need help or some action in order to mitigate the blocker | |
| 16:33:42 | bauzas | elodilles: yup, constant rechecks | |
| 16:33:45 | sean-k-mooney | bauzas: what is the blocker exactly | |
| 16:34:01 | bauzas | the failure ratio is so high it becomes unpractical to merge | |
| 16:34:15 | sean-k-mooney | right but on what issue | |
| 16:34:22 | bauzas | a couple of them | |
| 16:34:29 | bauzas | elodilles: we tracked them, right | |
| 16:34:46 | sean-k-mooney | ok i was wonderign if there was one we shoudl look at specificly | |
| 16:34:57 | elodilles | sean-k-mooney: guest kernel panics, volume timeouts, | |
| 16:35:12 | sean-k-mooney | as in volume detach | |
| 16:35:19 | sean-k-mooney | is it resize related? | |
| 16:35:44 | sean-k-mooney | becasue we noticed today that reize is not suing hte sshable change yet | |
| 16:35:56 | sean-k-mooney | gibi proposed a patch to cover that | |
| 16:36:08 | elodilles | also this bug comes up often: https://bugs.launchpad.net/nova/+bug/1901739 | |
| 16:36:40 | sean-k-mooney | im not familar with that | |
| 16:37:22 | sean-k-mooney | looks like lee fixed that in V but it has not been backported | |
| 16:37:52 | gibi | it is a mix of fixes already due to zuul migration and ubuntu version cahgne | |
| 16:37:55 | gibi | chnage | |
| 16:37:56 | sean-k-mooney | ok we can proably move on but if you have a list elodilles please share | |
| 16:38:48 | elodilles | sean-k-mooney: well it's on the recheck list on this patch: https://review.opendev.org/c/openstack/nova/+/838033 | |
| 16:38:51 | elodilles | :/ | |
| 16:39:28 | sean-k-mooney | can we ask infra to force merge that | |
| 16:39:57 | elodilles | sean-k-mooney: it won't help to other patches, they will need the same rechecks | |
| 16:40:17 | sean-k-mooney | yes i know | |
| 16:40:35 | bauzas | in general, we can try to release the jobs | |
| 16:40:50 | bauzas | in order to merge one specific patch we want | |
| 16:41:05 | elodilles | yes, one way is to decrease the test coverage | |
| 16:41:05 | gmann | +1, better than force merge | |
| 16:41:06 | bauzas | we did that in some past | |
| 16:41:19 | sean-k-mooney | ok | |
| 16:41:26 | gibi | here that would mean to cut out devstack based jobs | |
| 16:41:29 | sean-k-mooney | we can make some non voting for now | |
| 16:41:35 | gibi | an rely on unit and functional | |
| 16:41:43 | gibi | *and | |
| 16:41:45 | bauzas | this was ages ago | |
| 16:41:47 | gmann | yeah just for temporary time | |
| 16:41:53 | elodilles | gibi: or some volume related tests. though i don't know which is worse :S | |
| 16:41:53 | bauzas | but I still remember the magic formula | |
| 16:42:22 | gibi | gmann: it would not be temporary if we disable jobs we will never fix tem | |
| 16:42:25 | gibi | them | |
| 16:42:34 | sean-k-mooney | well i asusme we have a limited set of patches that we want to merge and we woudl turn these abck on like droping lc jobs | |
| 16:42:54 | sean-k-mooney | so i was epecting disable patch then revert | |
| 16:43:02 | bauzas | gibi: the idea is to relax the jobs before merging patches we want and then enabling again the jobs | |
| 16:43:19 | gibi | bauzas: and will we do it for every patch we want to merge? | |
| 16:43:21 | bauzas | and see whether the failure ratio drops again | |
| 16:43:23 | gmann | yeah that is what I thought, enabling them again | |
| 16:43:40 | gibi | bauzas: it would work iff we would have patches in queue that fixes those breaks but we dont | |
| 16:43:48 | bauzas | gibi: no, we need to identify a set of patches we'd like to merge in order to help with CI stability | |
| 16:44:01 | gibi | we don't have the fixes :/ | |
| 16:44:33 | gibi | it is not like merge these 5 patches to stabilize ussuri, we don't have those 5 patches ready | |
| 16:44:50 | gibi | we have random patches backported and waiting | |
| 16:44:51 | gmann | humm | |
| 16:45:25 | bauzas | ok, we won't obviously solve this problem by now | |
| 16:45:33 | gmann | one thing to note if we are removing integration tests coverage then it is better to make branch EOL like we did for ocata | |
| 16:45:38 | gibi | so something like: 1) collect the issue to be fixed in ussurit to stabilize CI 2) create fixes for them 3) force merge them 4) profit | |
| 16:45:39 | bauzas | let's state we all know about this problem and we'll figure out the solutions later | |
| 16:45:58 | bauzas | gibi: yeah, sounds 1/ and 2/ are not done yet | |
| 16:45:59 | elodilles | maybe if that "SSHABLE" fix helps things in ussuri we have one, but still we probably have other failures | |
| 16:46:30 | gibi | elodilles: a buncs of SSHABLE already landed, so we should see them help | |
| 16:46:38 | gmann | elodilles: that is again difficult as we are not supposed to use tempest master for ussuri so not all master change will go for ussuri testing | |
| 16:46:43 | bauzas | elodilles: ideally an etherpad of doom may help gathering ideas and feedback | |
| 16:46:48 | gibi | gmann: ahh | |
| 16:46:50 | gibi | that explains | |
| 16:46:55 | gmann | I have patch up to cap tempest for ussuri but that is not merged yet | |
| 16:46:58 | sean-k-mooney | gmann: wait we are not? | |
| 16:47:08 | sean-k-mooney | gmann: temest is ment to be branchless upstream | |
| 16:47:13 | gmann | but we have stopped ussuri support in tempest master | |
| 16:47:31 | sean-k-mooney | hum | |
| 16:47:37 | sean-k-mooney | because of what reason? | |
| 16:47:41 | gmann | sean-k-mooney: yes and master tempest is for SUpported branch and we cannot support EM branches | |
| 16:47:42 | sean-k-mooney | python version ? | |
| 16:48:04 | sean-k-mooney | shoudl that not be a project choice | |
| 16:48:16 | gmann | EM branches - #link https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html | |
| 16:48:17 | sean-k-mooney | tempest should not have to support it | |
| 16:48:42 | sean-k-mooney | but project that support em branches shoudl still be able to choose which tempest to run | |
| 16:48:46 | gmann | sean-k-mooney: we have to pin tempest for EM branch so that using master there can break them anytime | |
| 16:49:06 | gmann | sean-k-mooney: yeah, you can but no guarantee if master tempest will work | |
| 16:49:19 | gmann | as long as it is passing it is ok to use | |
| 16:49:58 | sean-k-mooney | ack | |
| 16:50:12 | sean-k-mooney | so i think i would prefer to keep tempest uncapped until ti breaks | |
| 16:50:14 | elodilles | (well, i guess changes on tempest master can affect not only EM branches, but they are affected most likely) | |
| 16:50:30 | sean-k-mooney | elodilles: well nova uses microversion | |
| 16:50:40 | sean-k-mooney | so it should work form our point of view | |
| 16:50:53 | gmann | elodilles: for supported branches we make sure tempest does not break them but for EM it is not the case | |
| 16:50:57 | sean-k-mooney | chagnes to tempest should not break our stable branches expcti if there ar package dep issues | |
| 16:51:14 | gmann | plan is : by default devstack will cap the tempest in ussuri and project can override it in jobs | |
| 16:51:15 | bauzas | I need to timestop the conversation | |
| 16:51:15 | sean-k-mooney | which i guess is why this was done | |
| 16:51:41 | gmann | yeah, we can discuss it in qa or nova channel after meeting | |
| 16:51:49 | sean-k-mooney | +1 | |
| 16:51:51 | elodilles | +1 | |
| 16:51:52 | bauzas | gmann: sean-k-mooney: gibi: you're all free to continue the conversation after the meeting | |