Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-05-11
15:08:38 opendevreview Elod Illes proposed openstack/nova stable/xena: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882967
15:09:49 opendevreview Elod Illes proposed openstack/nova stable/wallaby: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882968
15:11:00 opendevreview Elod Illes proposed openstack/nova stable/victoria: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882969
15:12:10 opendevreview Elod Illes proposed openstack/nova stable/ussuri: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882970
15:13:22 opendevreview Elod Illes proposed openstack/nova stable/train: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882971
15:29:22 opendevreview Sylvain Bauza proposed openstack/nova stable/zed: Revert "Debug Nova APIs call failures" https://review.opendev.org/c/openstack/nova/+/882786
16:17:57 bauzas folks, I'm also saying it loudly, tomorrow I'll be working and looking at specs
16:52:21 dansmith melwitt: the backport checker fix is going to fail on functional, that db table race thing
16:54:43 melwitt argh
16:54:58 dansmith is there a bug open for that/
16:55:56 melwitt yes, sec
16:56:10 dansmith I dunno why zuul hasn't kicked it out so I can recheck it yet
16:56:30 melwitt I believe it's this one https://bugs.launchpad.net/nova/+bug/1946339
16:57:14 dansmith hmm, similar at least
16:57:18 melwitt gibi has done a lot of work to improve the situation but it's a gnarly issue
16:58:16 dansmith okay yeah that's the same
16:58:20 JayF Hey; I re-proposed the Ironic sharding spec for this cycle about 2 weeks ago. It's not gotten any reviews. If anyone can take a look I'd appreciate it: https://review.opendev.org/c/openstack/nova-specs/+/881643
16:58:50 melwitt there is a change I think we could do that might help but I haven't proposed it yet bc it hadn't been happening very often for a long time there
16:59:14 dansmith melwitt: okay gibi just commented on the bug that we've seen an. uptick recently
16:59:37 melwitt yeah, I just saw that too
17:12:49 dansmith fungi: there's something that has completed all our jobs in gate that we need to recheck, but it's just sitting there in the queue and I'm not sure why
17:13:24 dansmith it's already identified as failed and out of the queue, but not.. uh, reporting or whatever
17:13:25 clarkb dansmith: because the changes ahead of it haven't finished
17:13:37 clarkb only the first thing in the queue can report
17:13:48 clarkb doing so removes it from the queue and then the next item can be processed
17:13:54 dansmith I thought if nothing in front could have caused the failure on that job it would come out immediately
17:14:04 clarkb zuul doesn't have that information so can't do that
17:14:14 dansmith hmm, okay
17:14:25 clarkb by putting things in the same queue you are asserting a failure in one may be caused by the other
17:14:31 clarkb and zuul is operating on that knowledge
17:14:37 dansmith okay
17:15:03 dansmith but it already shows it as failed out (meaning the fork in the line) so I thought that was it saying it knows the things behind it no longer depend
17:15:41 clarkb correct the things behind it no longer depend on it. But the things ahead of it may be where the actual bug is
17:16:02 clarkb in that case you want to evict the broken stuff ahead and restart the things ehind
17:16:18 dansmith so the green checks behind this are based on skipping it or with it applied?
17:16:38 clarkb the green checks behind are based on skipping the one that has failed
17:16:41 dansmith I assume with it applied and they'll restart if it decides it was legit to kick it out?
17:16:47 dansmith hmm okay
17:16:50 clarkb the unknown is the not yet completed jobs ahead of it
17:16:55 dansmith so if it doesn't get kicked out they restart?
17:16:57 fungi but they'll all be tested again from scratch if something else ahead of all of those fails a job
17:17:49 clarkb you have 5 changes, 6th is a failure, then X behind. Zuul does not know if the failure was caused by the 5 changes at the front so it does not completely evict the 6th until it processes the 5 ahead of it
17:17:50 fungi until all changes ahead of the failing change merge successfully, zuul can't be sure that there's something wrong with that change
17:18:18 dansmith it's too bad we can't mark a job as isolated or something, because this is only running nova unit tests, but it's held up as if it has the same dependencies as something with a tempest (which is why the queue needs to be shared)
17:18:28 dansmith obviously not a very common case
17:19:12 dansmith fungi: ack, the fork in the graph makes it look to me like it's already "out" but yeah okay
17:19:36 dansmith "out of consideration" I should say
17:20:37 dansmith but yeah I guess I thought there was job affinity and not just place-in-the-queue
17:24:31 fungi right, if the failure were due to a change ahead of it in an oslo lib, the bug in that oslo change might fail on some other job which exposed the same bug through some other tests which aren't the nova unit test job
17:24:39 fungi it's all fairly abstract from zuul's perspective
17:25:41 dansmith yeah, probably safer that way I guess, it's just not how I thought it worked
17:29:59 sean-k-mooney JayF: i think we have just been a bit busy and missed it
17:30:49 sean-k-mooney JayF: one of the things we agree at the ptg however as not to auto reappove previosly approves specs if there was no code proposed in the previos cycle
17:31:19 sean-k-mooney JayF: i know you were working on the iroinc side fo that last cyle
17:31:25 JayF sean-k-mooney: that's an interesting case; there was lots of code landed last cycle related to that spec. None in nova though (we had to get the Ironic API released, which we have)
17:31:30 sean-k-mooney JayF: how is that going
17:31:38 JayF Ironic shards API exists, was shipped in Antelope
17:31:47 sean-k-mooney ack
17:31:49 JayF openstacksdk support for it is landed, unsure if released but it can be if eneded
17:32:00 JayF I'm working on Ironic CLI support for that, which is only really needed once the Nova stuff is released
17:32:12 JayF right now, if that spec doesn't hit a speed bump, we've hit every milestone on time
17:32:21 sean-k-mooney cool
17:32:33 sean-k-mooney are you planning to work on the nova part this cycle
17:32:51 sean-k-mooney assuimg its the same as the spec form last cycle
17:33:01 sean-k-mooney i dont really see any issues with it
17:33:11 sean-k-mooney as long as there is someone to work on it we can review
17:33:18 JayF I believe John Garbutt is going to be doing most of the heavy lifting, with Julia and I as backup / docs writing
17:33:52 sean-k-mooney ok have they confirmed that since john has been out of active nova dev for a while
17:34:08 JayF I have confirmed that downstream
17:34:27 JayF He helped us with the design, and wrote the spec last cycle which was approved.
17:35:06 JayF Either way, regardless of which human writes the code, it's our intent to implement the spec as listed. I sure hope John does it; his familiarity will save a lot of time but even if not, this is too important to let it live/die on one persons' shoulders.
17:35:32 sean-k-mooney ack
17:36:25 sean-k-mooney ill try an review it proably monday at this point but if they want to +2 it i can proably +w it assuming its basically the same as last cycle.
17:36:40 sean-k-mooney i was happy with the desgin previously
17:37:02 sean-k-mooney and i dont think anythin has maritarly change on the nova side that woudl affect it
17:37:07 JayF I appreciate it. My only urgency in getting the spec merged is I believe there's a deadline in the nova process for things we want to land this cycle, yeah?
17:37:29 sean-k-mooney there technially is but its milestone too
17:37:35 sean-k-mooney *two
17:37:56 sean-k-mooney so July 6th
17:37:59 JayF aha, I was worried it was -1
17:38:11 JayF sounds good :) thanks Sean!
17:38:22 sean-k-mooney no we encurage peopel to submit the first draft before m1
17:38:27 sean-k-mooney you have time
17:38:46 JayF I'm going to use some of that time now to land the ironic cli for shards o/ ty again
17:41:38 sean-k-mooney since i have it open im going to do a quick pass on it and compre to last release but then i need to swap to somethign else.
17:42:02 sean-k-mooney JayF: the ironic cli is now a osc plugin yes
17:42:13 sean-k-mooney or does ironic still have a standalone cli too
17:42:56 JayF sean-k-mooney: yes-ish. We have a plugin for OSC which can also operate independently (e.g. with just Ironic client plugin installed, you can still run `baremetal whatever`)
17:43:17 JayF but if the primary openstack cli client is installed, `openstack baremetal whatever` works
17:43:37 sean-k-mooney oh neat
17:43:44 JayF single codebase, same command structure, just prefix for when it's integrated vs no prefix when it's not
17:44:04 JayF that's also why all the Ironic docs use `baremetal X` instead of `openstack baremetal X` (the non-openstack-namespaced version works universally)
17:44:17 sean-k-mooney well without i assume the "prefix" is the binary name
17:45:13 sean-k-mooney so ironic baremetal X ? vs openstack baremental X
17:45:22 JayF Gonna be honest; I've done very little work in the clients. Part of why I'm speaking inexactly is my knowledge is inexact.
17:45:34 JayF No, it's `openstack baremetal X` or `baremetal X` (no Ironic at any point)
17:45:34 sean-k-mooney no worries
17:46:03 sean-k-mooney ok so then teh console script entryp oint and the binary on the path is called "baremental" then
17:46:53 JayF https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L25 we have both a binary and the entrypoints setup
17:46:53 sean-k-mooney thhat woudl be yes https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L27

Earlier   Later