| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-28 | |||
| 16:01:57 | elodilles | o/ | |
| 16:02:24 | gibi | \o | |
| 16:02:44 | bauzas | let's start | |
| 16:02:52 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:03 | bauzas | No Critical bug | |
| 16:03:16 | bauzas | #link 13 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | |
| 16:03:25 | bauzas | thanks for people who triaged them :) | |
| 16:03:41 | bauzas | one open bug marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential | |
| 16:03:42 | gibi | bauzas: was it you ?:) | |
| 16:03:51 | bauzas | gibi: shhhhhhhhtttt :p | |
| 16:04:16 | bauzas | so, https://bugs.launchpad.net/nova/+bug/1945323 is proposed against RC* | |
| 16:04:18 | gibi | :x | |
| 16:04:40 | bauzas | tl;dr: last osbrick release is creating problems for NVMe volumes | |
| 16:05:01 | bauzas | the fix is provided https://review.opendev.org/c/openstack/nova/+/811413 but not yet merged | |
| 16:05:44 | bauzas | we'll discuss about RC releases in the next topic, but here the question is, should we try to merge ^ before Thursday as it's the deadline for RC | |
| 16:05:49 | bauzas | ? | |
| 16:06:01 | bauzas | lyarwood had concerns and I have an opinion | |
| 16:06:14 | gibi | I think if the fix is valid then we should aim to land it | |
| 16:06:21 | gibi | but the gate state might not allow that | |
| 16:06:37 | bauzas | given the gate is bad at the moment (again, we will discuss this later), we could try to land it but at least we should provide a releasenotes documentation | |
| 16:06:52 | bauzas | telling about the bug | |
| 16:07:25 | bauzas | so, in case we can't land the fix, operators would still see the documentation and for example packagers wouldn't use the latest osbrick if they want to support NVMe | |
| 16:07:30 | bauzas | thoughts ? | |
| 16:07:46 | bauzas | remember that we have a "bugs" section in nova relnotes | |
| 16:07:55 | bauzas | (this gives a "known bugs" section) | |
| 16:08:09 | gibi | I agree to land a reno, regardless of the fix | |
| 16:08:39 | bauzas | given I proposed, I need to write it | |
| 16:08:48 | bauzas | but someone could write it if they want :) | |
| 16:09:01 | bauzas | e0ne: around ? | |
| 16:09:20 | bauzas | we're discussing about your open bug https://bugs.launchpad.net/nova/+bug/1945323 | |
| 16:09:27 | e0ne | bauzas: hi | |
| 16:09:36 | bauzas | and we're proposing you to write a reno change documenting the known bug | |
| 16:10:12 | bauzas | so in case we can't merge the fix before thursday's deadline for RCs, operators would have a documentation explaining how to workaround | |
| 16:10:24 | e0ne | ok | |
| 16:10:29 | e0ne | fair enough | |
| 16:10:32 | bauzas | e0ne: can you write it ? | |
| 16:10:43 | e0ne | I understand that we found it too late | |
| 16:10:49 | bauzas | I wish you would be better than me explaining the issue and how to workaround for an ops | |
| 16:10:54 | e0ne | bauzas: ok, will do it a bit later today | |
| 16:11:14 | bauzas | e0ne: the problem is not really about the time, this is more about the gate block we have atm | |
| 16:11:22 | lyarwood | sorry back | |
| 16:11:33 | gmann | on gate url issue - I think we need to go with no trailing '/' in both place but we need to push changes on stable branches first to make grenade job work. basically in reverse order. I will try after an hr or so, currently in another meeting. | |
| 16:11:39 | bauzas | lyarwood: you had concerns on this bug, btw. take the mic | |
| 16:12:06 | bauzas | gmann: we'll discuss the gate problems litterally after this one :) | |
| 16:12:11 | gmann | k | |
| 16:12:21 | lyarwood | bauzas: I don't even think it's a valid nova bug at the moment, it doesn't look like os-brick has actually changed it's return value with that fix | |
| 16:12:36 | lyarwood | bauzas: and even if it had that's on os-brick not nova | |
| 16:12:54 | bauzas | do we have a reproduction test ? | |
| 16:12:58 | bauzas | e0ne: ^ | |
| 16:12:58 | lyarwood | tbh this just smells like a poorly tested codepath | |
| 16:13:17 | e0ne | lyarwood: there was an issue with CI. it's fixed now | |
| 16:13:50 | bauzas | e0ne: you mean that CI wasn't using the latest osbrick hence not seeing the issue ? | |
| 16:13:54 | e0ne | mellanox CI should fail with this issue on os-brick patches now | |
| 16:14:00 | bauzas | ack | |
| 16:14:51 | bauzas | fwiw, gibi also has another bugfix he wants to merge that's not a regression, so I guess we will provide a Xena stable release like 2 weeks after GA | |
| 16:15:05 | e0ne | bauzas: when I merged that patch I was confused by mellanox CI Cinder-tgtadm | |
| 16:15:14 | lyarwood | yup either way I don't think this needs to land in rc | |
| 16:15:29 | gibi | bauzas: yupp this is a 1/3 xena regression and 2/3 existing bug in wallaby https://bugs.launchpad.net/nova/+bug/1945310 | |
| 16:15:59 | gibi | bauzas: and I'm fine to include it in 24.0.1 later | |
| 16:16:09 | bauzas | ok, then let's do this | |
| 16:16:12 | e0ne | lyarwood: ok, I totally understand you. thanks for letting me discuss it | |
| 16:16:24 | bauzas | e0ne: ping me when you're done with the reno patch | |
| 16:16:34 | e0ne | bauzas: ok, will do | |
| 16:16:35 | bauzas | e0ne: use the 'bugs' section for it please | |
| 16:16:50 | bauzas | cool | |
| 16:17:06 | bauzas | and for the fix itself, let's review it sooner than later in master | |
| 16:17:30 | bauzas | so we could backport it to stable/xena as quickly as we can | |
| 16:17:46 | bauzas | lyarwood: do you had time to review the fix itself ? | |
| 16:18:02 | lyarwood | I already have several times | |
| 16:18:12 | bauzas | nice | |
| 16:18:25 | bauzas | so we need a second core volunteering for reviewing it :) | |
| 16:18:27 | lyarwood | https://review.opendev.org/c/openstack/os-brick/+/800014/7/os_brick/initiator/connectors/nvmeof.py#199 and FWIW I think this is a regression in os-briclk | |
| 16:18:42 | bauzas | hah | |
| 16:18:52 | bauzas | so, no nova impact ? | |
| 16:19:00 | e0ne | lyarwood: I'll discuss it in cinder meeting tomorrow too | |
| 16:19:11 | bauzas | ok, then let's clarify the solution later | |
| 16:19:29 | bauzas | the reno doc is the priority, the fix itself needs a bit more care I guess | |
| 16:19:31 | bauzas | moving on | |
| 16:19:41 | bauzas | #topic Gate status | |
| 16:19:48 | bauzas | Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | |
| 16:19:55 | bauzas | we triaged a few other ones | |
| 16:20:00 | bauzas | but, | |
| 16:20:05 | bauzas | CI is blocked due to nova-grenade-multinode issue and apache placement issue | |
| 16:20:08 | bauzas | gmann: your turn | |
| 16:20:38 | gmann | frickler: ianw_pto figured it out on apache2 security bugs, you might have seen in email | |
| 16:21:03 | gmann | we have two option 1. no trailing '/' 2. having trailing '/' in both url and target | |
| 16:21:26 | gmann | later one seems causing keystone error so we might need to go with first one | |
| 16:21:30 | bauzas | grabbing the link for the sake of understanding | |
| 16:21:45 | gibi | that is the current fix proposal #link https://review.opendev.org/c/openstack/devstack/+/811399 | |
| 16:21:54 | bauzas | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html | |
| 16:22:12 | gmann | but we need to do it on stable branches first due to grenade jobs , I will try to push changes in stable in reverse order and see if it all work till master. | |
| 16:22:25 | bauzas | tl:dr: apache config is swallowing the placement endpoint URL | |
| 16:22:46 | gmann | so in summary it may take time, hope today we get fixes in | |
| 16:23:08 | gibi | gmann: the stable/victoria fix failing with ceph issue now https://review.opendev.org/c/openstack/devstack/+/811399 | |
| 16:23:08 | gmann | gibi: yeah and ceph things on victoria is another thing | |
| 16:23:17 | bauzas | gmann: any pointer for the nova-grenade-multinode issue ? | |
| 16:23:20 | gibi | gmann: yepp, it is not relataed but it is blocking | |
| 16:23:26 | gmann | true | |
| 16:23:39 | bauzas | gmann: can't see any bug for either the apache issue or the grenade thing | |
| 16:23:51 | gmann | bauzas: nova-grenade-multinode you mean for neutron-truck extension right? | |