Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-09-28
16:03:41 bauzas one open bug marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential
16:03:42 gibi bauzas: was it you ?:)
16:03:51 bauzas gibi: shhhhhhhhtttt :p
16:04:16 bauzas so, https://bugs.launchpad.net/nova/+bug/1945323 is proposed against RC*
16:04:18 gibi :x
16:04:40 bauzas tl;dr: last osbrick release is creating problems for NVMe volumes
16:05:01 bauzas the fix is provided https://review.opendev.org/c/openstack/nova/+/811413 but not yet merged
16:05:44 bauzas we'll discuss about RC releases in the next topic, but here the question is, should we try to merge ^ before Thursday as it's the deadline for RC
16:05:49 bauzas ?
16:06:01 bauzas lyarwood had concerns and I have an opinion
16:06:14 gibi I think if the fix is valid then we should aim to land it
16:06:21 gibi but the gate state might not allow that
16:06:37 bauzas given the gate is bad at the moment (again, we will discuss this later), we could try to land it but at least we should provide a releasenotes documentation
16:06:52 bauzas telling about the bug
16:07:25 bauzas so, in case we can't land the fix, operators would still see the documentation and for example packagers wouldn't use the latest osbrick if they want to support NVMe
16:07:30 bauzas thoughts ?
16:07:46 bauzas remember that we have a "bugs" section in nova relnotes
16:07:55 bauzas (this gives a "known bugs" section)
16:08:09 gibi I agree to land a reno, regardless of the fix
16:08:39 bauzas given I proposed, I need to write it
16:08:48 bauzas but someone could write it if they want :)
16:09:01 bauzas e0ne: around ?
16:09:20 bauzas we're discussing about your open bug https://bugs.launchpad.net/nova/+bug/1945323
16:09:27 e0ne bauzas: hi
16:09:36 bauzas and we're proposing you to write a reno change documenting the known bug
16:10:12 bauzas so in case we can't merge the fix before thursday's deadline for RCs, operators would have a documentation explaining how to workaround
16:10:24 e0ne ok
16:10:29 e0ne fair enough
16:10:32 bauzas e0ne: can you write it ?
16:10:43 e0ne I understand that we found it too late
16:10:49 bauzas I wish you would be better than me explaining the issue and how to workaround for an ops
16:10:54 e0ne bauzas: ok, will do it a bit later today
16:11:14 bauzas e0ne: the problem is not really about the time, this is more about the gate block we have atm
16:11:22 lyarwood sorry back
16:11:33 gmann on gate url issue - I think we need to go with no trailing '/' in both place but we need to push changes on stable branches first to make grenade job work. basically in reverse order. I will try after an hr or so, currently in another meeting.
16:11:39 bauzas lyarwood: you had concerns on this bug, btw. take the mic
16:12:06 bauzas gmann: we'll discuss the gate problems litterally after this one :)
16:12:11 gmann k
16:12:21 lyarwood bauzas: I don't even think it's a valid nova bug at the moment, it doesn't look like os-brick has actually changed it's return value with that fix
16:12:36 lyarwood bauzas: and even if it had that's on os-brick not nova
16:12:54 bauzas do we have a reproduction test ?
16:12:58 lyarwood tbh this just smells like a poorly tested codepath
16:12:58 bauzas e0ne: ^
16:13:17 e0ne lyarwood: there was an issue with CI. it's fixed now
16:13:50 bauzas e0ne: you mean that CI wasn't using the latest osbrick hence not seeing the issue ?
16:13:54 e0ne mellanox CI should fail with this issue on os-brick patches now
16:14:00 bauzas ack
16:14:51 bauzas fwiw, gibi also has another bugfix he wants to merge that's not a regression, so I guess we will provide a Xena stable release like 2 weeks after GA
16:15:05 e0ne bauzas: when I merged that patch I was confused by mellanox CI Cinder-tgtadm
16:15:14 lyarwood yup either way I don't think this needs to land in rc
16:15:29 gibi bauzas: yupp this is a 1/3 xena regression and 2/3 existing bug in wallaby https://bugs.launchpad.net/nova/+bug/1945310
16:15:59 gibi bauzas: and I'm fine to include it in 24.0.1 later
16:16:09 bauzas ok, then let's do this
16:16:12 e0ne lyarwood: ok, I totally understand you. thanks for letting me discuss it
16:16:24 bauzas e0ne: ping me when you're done with the reno patch
16:16:34 e0ne bauzas: ok, will do
16:16:35 bauzas e0ne: use the 'bugs' section for it please
16:16:50 bauzas cool
16:17:06 bauzas and for the fix itself, let's review it sooner than later in master
16:17:30 bauzas so we could backport it to stable/xena as quickly as we can
16:17:46 bauzas lyarwood: do you had time to review the fix itself ?
16:18:02 lyarwood I already have several times
16:18:12 bauzas nice
16:18:25 bauzas so we need a second core volunteering for reviewing it :)
16:18:27 lyarwood https://review.opendev.org/c/openstack/os-brick/+/800014/7/os_brick/initiator/connectors/nvmeof.py#199 and FWIW I think this is a regression in os-briclk
16:18:42 bauzas hah
16:18:52 bauzas so, no nova impact ?
16:19:00 e0ne lyarwood: I'll discuss it in cinder meeting tomorrow too
16:19:11 bauzas ok, then let's clarify the solution later
16:19:29 bauzas the reno doc is the priority, the fix itself needs a bit more care I guess
16:19:31 bauzas moving on
16:19:41 bauzas #topic Gate status
16:19:48 bauzas Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure
16:19:55 bauzas we triaged a few other ones
16:20:00 bauzas but,
16:20:05 bauzas CI is blocked due to nova-grenade-multinode issue and apache placement issue
16:20:08 bauzas gmann: your turn
16:20:38 gmann frickler: ianw_pto figured it out on apache2 security bugs, you might have seen in email
16:21:03 gmann we have two option 1. no trailing '/' 2. having trailing '/' in both url and target
16:21:26 gmann later one seems causing keystone error so we might need to go with first one
16:21:30 bauzas grabbing the link for the sake of understanding
16:21:45 gibi that is the current fix proposal #link https://review.opendev.org/c/openstack/devstack/+/811399
16:21:54 bauzas #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html
16:22:12 gmann but we need to do it on stable branches first due to grenade jobs , I will try to push changes in stable in reverse order and see if it all work till master.
16:22:25 bauzas tl:dr: apache config is swallowing the placement endpoint URL
16:22:46 gmann so in summary it may take time, hope today we get fixes in
16:23:08 gmann gibi: yeah and ceph things on victoria is another thing
16:23:08 gibi gmann: the stable/victoria fix failing with ceph issue now https://review.opendev.org/c/openstack/devstack/+/811399
16:23:17 bauzas gmann: any pointer for the nova-grenade-multinode issue ?
16:23:20 gibi gmann: yepp, it is not relataed but it is blocking
16:23:26 gmann true
16:23:39 bauzas gmann: can't see any bug for either the apache issue or the grenade thing
16:23:51 gmann bauzas: nova-grenade-multinode you mean for neutron-truck extension right?
16:23:56 bauzas this is correct
16:23:57 gibi gmann: right
16:24:20 gmann yeah, I have lyarwood patch in list. let's get this apache things in then I will take a look into that
16:24:28 lyarwood https://review.opendev.org/c/openstack/grenade/+/811117 is the fix for that
16:24:31 lyarwood I can create a bug
16:24:32 gibi yepp we need to fix the apache first anyhow
16:24:34 gmann yeah ^^

Earlier   Later