Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2023-05-11
17:16:50 clarkb the unknown is the not yet completed jobs ahead of it
17:16:55 dansmith so if it doesn't get kicked out they restart?
17:16:57 fungi but they'll all be tested again from scratch if something else ahead of all of those fails a job
17:17:49 clarkb you have 5 changes, 6th is a failure, then X behind. Zuul does not know if the failure was caused by the 5 changes at the front so it does not completely evict the 6th until it processes the 5 ahead of it
17:17:50 fungi until all changes ahead of the failing change merge successfully, zuul can't be sure that there's something wrong with that change
17:18:18 dansmith it's too bad we can't mark a job as isolated or something, because this is only running nova unit tests, but it's held up as if it has the same dependencies as something with a tempest (which is why the queue needs to be shared)
17:18:28 dansmith obviously not a very common case
17:19:12 dansmith fungi: ack, the fork in the graph makes it look to me like it's already "out" but yeah okay
17:19:36 dansmith "out of consideration" I should say
17:20:37 dansmith but yeah I guess I thought there was job affinity and not just place-in-the-queue
17:24:31 fungi right, if the failure were due to a change ahead of it in an oslo lib, the bug in that oslo change might fail on some other job which exposed the same bug through some other tests which aren't the nova unit test job
17:24:39 fungi it's all fairly abstract from zuul's perspective
17:25:41 dansmith yeah, probably safer that way I guess, it's just not how I thought it worked
17:29:59 sean-k-mooney JayF: i think we have just been a bit busy and missed it
17:30:49 sean-k-mooney JayF: one of the things we agree at the ptg however as not to auto reappove previosly approves specs if there was no code proposed in the previos cycle
17:31:19 sean-k-mooney JayF: i know you were working on the iroinc side fo that last cyle
17:31:25 JayF sean-k-mooney: that's an interesting case; there was lots of code landed last cycle related to that spec. None in nova though (we had to get the Ironic API released, which we have)
17:31:30 sean-k-mooney JayF: how is that going
17:31:38 JayF Ironic shards API exists, was shipped in Antelope
17:31:47 sean-k-mooney ack
17:31:49 JayF openstacksdk support for it is landed, unsure if released but it can be if eneded
17:32:00 JayF I'm working on Ironic CLI support for that, which is only really needed once the Nova stuff is released
17:32:12 JayF right now, if that spec doesn't hit a speed bump, we've hit every milestone on time
17:32:21 sean-k-mooney cool
17:32:33 sean-k-mooney are you planning to work on the nova part this cycle
17:32:51 sean-k-mooney assuimg its the same as the spec form last cycle
17:33:01 sean-k-mooney i dont really see any issues with it
17:33:11 sean-k-mooney as long as there is someone to work on it we can review
17:33:18 JayF I believe John Garbutt is going to be doing most of the heavy lifting, with Julia and I as backup / docs writing
17:33:52 sean-k-mooney ok have they confirmed that since john has been out of active nova dev for a while
17:34:08 JayF I have confirmed that downstream
17:34:27 JayF He helped us with the design, and wrote the spec last cycle which was approved.
17:35:06 JayF Either way, regardless of which human writes the code, it's our intent to implement the spec as listed. I sure hope John does it; his familiarity will save a lot of time but even if not, this is too important to let it live/die on one persons' shoulders.
17:35:32 sean-k-mooney ack
17:36:25 sean-k-mooney ill try an review it proably monday at this point but if they want to +2 it i can proably +w it assuming its basically the same as last cycle.
17:36:40 sean-k-mooney i was happy with the desgin previously
17:37:02 sean-k-mooney and i dont think anythin has maritarly change on the nova side that woudl affect it
17:37:07 JayF I appreciate it. My only urgency in getting the spec merged is I believe there's a deadline in the nova process for things we want to land this cycle, yeah?
17:37:29 sean-k-mooney there technially is but its milestone too
17:37:35 sean-k-mooney *two
17:37:56 sean-k-mooney so July 6th
17:37:59 JayF aha, I was worried it was -1
17:38:11 JayF sounds good :) thanks Sean!
17:38:22 sean-k-mooney no we encurage peopel to submit the first draft before m1
17:38:27 sean-k-mooney you have time
17:38:46 JayF I'm going to use some of that time now to land the ironic cli for shards o/ ty again
17:41:38 sean-k-mooney since i have it open im going to do a quick pass on it and compre to last release but then i need to swap to somethign else.
17:42:02 sean-k-mooney JayF: the ironic cli is now a osc plugin yes
17:42:13 sean-k-mooney or does ironic still have a standalone cli too
17:42:56 JayF sean-k-mooney: yes-ish. We have a plugin for OSC which can also operate independently (e.g. with just Ironic client plugin installed, you can still run `baremetal whatever`)
17:43:17 JayF but if the primary openstack cli client is installed, `openstack baremetal whatever` works
17:43:37 sean-k-mooney oh neat
17:43:44 JayF single codebase, same command structure, just prefix for when it's integrated vs no prefix when it's not
17:44:04 JayF that's also why all the Ironic docs use `baremetal X` instead of `openstack baremetal X` (the non-openstack-namespaced version works universally)
17:44:17 sean-k-mooney well without i assume the "prefix" is the binary name
17:45:13 sean-k-mooney so ironic baremetal X ? vs openstack baremental X
17:45:22 JayF Gonna be honest; I've done very little work in the clients. Part of why I'm speaking inexactly is my knowledge is inexact.
17:45:34 JayF No, it's `openstack baremetal X` or `baremetal X` (no Ironic at any point)
17:45:34 sean-k-mooney no worries
17:46:03 sean-k-mooney ok so then teh console script entryp oint and the binary on the path is called "baremental" then
17:46:53 JayF https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L25 we have both a binary and the entrypoints setup
17:46:53 sean-k-mooney thhat woudl be yes https://github.com/openstack/python-ironicclient/blob/master/setup.cfg#L27
17:46:57 JayF heh jinx
17:47:08 fungi dansmith: melwitt: (or anybody else plugged into ossa-2023-003), do you happen to know if the vulnerability affects iscsi based deployments that don't rely on multipathd? i asked just now in https://launchpad.net/bugs/2004555 because an operator reached out to me directly with the question
17:47:20 dansmith fungi: I just replied and pinged gorka
17:47:28 fungi oh, perfect. thanks!
17:47:34 sean-k-mooney JayF: no worreis just had not seen that done before but that was what i was expecting
20:25:39 opendevreview Merged openstack/nova master: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882956
20:25:47 opendevreview Merged openstack/nova stable/2023.1: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882964
20:26:28 dansmith woot
22:40:34 opendevreview Merged openstack/nova stable/yoga: Remove deleted projects from flavor access list https://review.opendev.org/c/openstack/nova/+/881314
22:40:41 opendevreview Merged openstack/nova stable/zed: Ironic: retry when node not available https://review.opendev.org/c/openstack/nova/+/867924
22:40:48 opendevreview Merged openstack/nova stable/zed: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882965
23:34:28 opendevreview Merged openstack/nova master: doc: Update version info https://review.opendev.org/c/openstack/nova/+/880614
#openstack-nova - 2023-05-12
06:32:58 opendevreview Amit Uniyal proposed openstack/nova master: WIP: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284
07:14:17 opendevreview Amit Uniyal proposed openstack/nova master: WIP: Reproducer for dangling volumes https://review.opendev.org/c/openstack/nova/+/881457
07:14:17 opendevreview Amit Uniyal proposed openstack/nova master: WIP: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284
08:29:57 opendevreview Merged openstack/nova stable/yoga: CI: fix backport validator for new branch naming https://review.opendev.org/c/openstack/nova/+/882966
08:32:38 opendevreview Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075
09:48:56 opendevreview Alexey Stupnikov proposed openstack/nova stable/xena: Remove deleted projects from flavor access list https://review.opendev.org/c/openstack/nova/+/883014
09:52:29 SvenKieske hey there, we're currently implementing changes in kolla-ansible so nova uses service-tokens to talk to cinder, to address the vuln released 2 days ago. we hit an 500 Server Error during volume attachment. Should I report a bug, or might this just be a spurious failure?
09:53:05 SvenKieske our change: https://review.opendev.org/c/openstack/kolla-ansible/+/882893 the logged error: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11b/882893/3/check/kolla-ansible-rocky9-cephadm/11b1baf/primary/logs/ansible/test-core-openstack
09:54:16 SvenKieske Command is: "openstack server add volume kolla_boot_test test_volume --device /dev/vdb"
09:54:50 SvenKieske Error: HttpException: 500: Server Error for url: http://192.0.2.10:8774/v2.1/servers/2177019c-a915-4bda-a898-4fa806901144/os-volume_attachments, Unexpected API Error.
09:55:33 opendevreview Amit Uniyal proposed openstack/nova master: WIP: Reproducer for dangling volumes https://review.opendev.org/c/openstack/nova/+/881457
09:55:33 opendevreview Amit Uniyal proposed openstack/nova master: WIP: Delete dangling bdms https://review.opendev.org/c/openstack/nova/+/882284
10:53:04 bauzas SvenKieske: what's saying the nova-api log ?
10:54:04 SvenKieske bauzas: I'll take a look
10:54:26 bauzas 2023-05-11 16:15:22.782 25 DEBUG nova.compute.api [None req-92ee72e6-a5e8-47b1-8428-47a71835f010 6ffc4ae1295442999283b4c88ba497e9 a6ca4c77593c46a2b5e6c3c651e7e436 - - default default] Failed to fetch instance by id kolla_boot_test get /var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/api.py:2983 2023-05-11 16:15:22.782 25 INFO nova.api.openstack.wsgi [None req-92ee72e6-a5e8-47b1-8428-47a71835f010 6ffc4ae129544299928
10:54:26 bauzas 3b4c88ba497e9 a6ca4c77593c46a2b5e6c3c651e7e436 - - default default] HTTP exception thrown: Instance kolla_boot_test could not be found.
10:59:23 bauzas I don't see any instance creation in the nova-api.txt log
11:02:23 bauzas yeah, as well here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11b/882893/3/check/kolla-ansible-rocky9-cephadm/11b1baf/primary/logs/kolla/nova/nova-api-access.txt
11:04:09 bauzas and I don't see in the nova-api logs where os-volume_attachments is called :(
12:15:06 SvenKieske bauzas: I think I got an error/stacktrace, preparing a pastebin..
12:22:07 SvenKieske bauzas: https://paste.opendev.org/show/819994/
12:25:32 SvenKieske what's interesting is that it throws a 401 in the first few lines but what I get at the high level cli command is a 500 server error..mhm let me double check if this is really correlated
12:27:13 bauzas SvenKieske: yeah ok, so that's indeed due to https://review.opendev.org/c/openstack/kolla-ansible/+/882893
12:29:29 SvenKieske yeah, but I see there are even more errors in keystone... it can't find the service project.. I'll circle back with my fellow kolla contributors who know more than me about our test setup, thanks so far!
12:33:28 bauzas ack
12:54:53 opendevreview Danylo Vodopianov proposed openstack/nova master: Packed virtqueue support was added. https://review.opendev.org/c/openstack/nova/+/876075

Earlier   Later