Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2026-08-19
12:36:27 sean-k-mooney ya i can see that being a problem
12:36:50 sean-k-mooney also ya addign task state to the decoreato rmakes seesne
12:36:58 gibi and it might happen more often in the grenade job as it uses a single conductor setup instead of a super and a cell conductor
12:37:19 gibi so the two instance.save happens in the same conductor
12:37:37 sean-k-mooney maybe although i dont think that would make it that much slower with onloy 1-2 vms
12:37:43 sean-k-mooney *1-2 computes
12:37:48 gibi yeah, I'm just guessing here
12:37:58 sean-k-mooney i think the race hunch is a good one
12:38:09 sean-k-mooney jus tnot entirly sure why its more common in grenade
12:38:10 gibi we will see. if the precondition makes the tempest test fail early that will prove it
12:38:47 gibi it can be something with threading + single conductor as well. I think only grenade is doing that now
12:39:00 sean-k-mooney is the tempest tst properly waiting for it to be shleve offloaded or just unshilving straight away
12:39:05 gibi threading allows more random task switching
12:39:27 gibi sean-k-mooney: even if the tempest waits for the shelve_offload state we might set that early with a task state
12:39:30 gibi let me check
12:39:49 sean-k-mooney gibi: well im thinking that tempest shoudl retry the unshleve
12:40:03 sean-k-mooney at least on a 409
12:40:10 sean-k-mooney which is what that decoreator shoudl be returning
12:40:58 sean-k-mooney since the transtion form shelve to shelve offloaded is configurabel and depend on if its BFV or BFI
12:41:14 sean-k-mooney or using ceph
12:41:28 gibi https://github.com/openstack/tempest/blob/fa340b1f5fff8f57a018be206644c81b6d36b3d4/tempest/common/compute.py#L388-L400
12:41:30 sean-k-mooney so i dont think we can write the test to know which to expect
12:41:37 gibi so it depends on the tempest config what we are waiting for
12:41:42 sean-k-mooney but it can retyr the unshelve on 409 conflict and still be correct
12:42:10 gibi I can imagine we end up only waiting for SHELVED state and issue an unshelve
12:42:19 sean-k-mooney yep
12:42:37 sean-k-mooney but as i said for images_type=rbd and bfv we go drictly to shelve_offloaded i think
12:42:41 gibi OK we will see and if it is the case then we need to fix nova and tempest as well
12:42:43 sean-k-mooney regradless fo the offload time
12:43:33 sean-k-mooney ya we should have ci reulst in a while and we can see
12:45:53 gibi yepp
14:15:09 dansmith gibi: I just +2'd this if you want to re-add yours after the test coverage fix: https://review.opendev.org/c/openstack/nova/+/998571
14:23:28 gibi dansmith: ohh sure. Thanks for the reviews there
14:30:00 gibi Uggla: I left some feedback in the repro and fix in https://review.opendev.org/c/openstack/nova/+/999978
14:30:33 gibi I agree this is a valid repro but for the fix we might need to make a step back and see if we can make a wider fix
14:30:56 gibi having an instance without task state and without instance.host during a period of evac sounds a source of different issuees
14:45:05 gibi Uggla: on the same note I will need to drop at 17:45 latest from the upstream triage call today
14:47:37 opendevreview Balazs Gibizer proposed openstack/nova master: [DNM]: Troubleshoot race condition in unshelve vif plug https://review.opendev.org/c/openstack/nova/+/1001294
14:47:38 opendevreview Balazs Gibizer proposed openstack/nova master: [WIP]Add task_state = None as pre-condition of unshelve https://review.opendev.org/c/openstack/nova/+/1001485
14:47:56 gibi grr ^^ that was in merge conflict while I thought I just waiting for the CI result
15:29:51 Uggla Upstream triage meeting
15:29:52 Uggla meet.google.com/zjr-rxus-hzj
15:48:42 gmaan gibi: thanks, I was thinking to do some check on conductor side but this is better
15:49:56 gmaan gibi: sambork sorry i missed eventlet meeting today. if anything for me to review, please let me know.
15:52:05 sambork np it was mostly about perf test
15:55:45 gmaan ack
16:50:57 opendevreview Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211
17:11:10 opendevreview Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211
18:01:58 melwitt gmaan: can you remind me if there is a fix idea for https://bugs.launchpad.net/nova/+bug/2160254 ? anecdotally I'm seeing a lot of failures on patches I'm following
18:02:50 gmaan melwitt: not yet, I did not debug it yet. most probably I will be able to get on this after FF
18:03:00 gmaan is it happening too frequently ?
18:03:45 melwitt seems so but that is just my personal observation on patches I'm following. I guess I should check opensearch
18:05:18 gmaan k, let me see if I can get some time today. it may get worst during FF week :)
18:06:21 melwitt gmaan: looks like it's just my bad luck, I see only one hit per day at most
18:07:43 gmaan ok, I used to see it mulitple times in the week when i reported bug but agfter that not much occurance, but thanks for reminder, i will see if there is any obious mistake in test
19:17:32 gmaan gibi: i saw notification failure in functional threading job, it seems 'start' migration event happened instead of 'end' https://zuul.opendev.org/t/openstack/build/07c0f0aecebb42fc99ec209ad6a9df3e/log/job-output.txt#13762
20:56:47 opendevreview Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572
21:01:04 gmaan dansmith: gibi: ^^ this is the further refactoring of thread pool executors as discussed in original change
21:20:16 opendevreview Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211
#openstack-nova - 2026-08-20
03:03:02 opendevreview Ghanshyam Maan proposed openstack/nova master: Shutdown thread pool executors during graceful shutdown https://review.opendev.org/c/openstack/nova/+/1000211
03:57:00 opendevreview Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572
04:00:55 gmaan gibi: i saw the same notification test failing again, reported the bug with trackback and link https://bugs.launchpad.net/nova/+bug/2164606
04:01:29 opendevreview Ghanshyam Maan proposed openstack/nova master: Create services required thread pool executors during service initialization https://review.opendev.org/c/openstack/nova/+/1001572
13:38:36 dansmith gouthamr: I've restarted m-shr and no reconcile has happened. I've captured the logs for the service during the restart, a dump of the manila database, and a capture of what osc shows me
13:38:51 dansmith gouthamr: anything else you want me to capture (other service logs) before I force fix the share and move on?
14:21:38 gouthamr dansmith: o/ ty, looking
14:57:04 opendevreview Brian Haley proposed openstack/nova master: Change neutron API calls to use project_id https://review.opendev.org/c/openstack/nova/+/994321
15:22:17 gouthamr dansmith: in meetings and got distracted; did you have the log posted somewhere? i can take a look at it and i'm thinking we have an eventlet bug.. I don't need anyhing else though, so yes.. force-fix the share with "access delete --unrestrict" as admin
15:22:50 dansmith gouthamr: no, I was waiting to see if you wanted me to capture anything else.. I'll post what I have in a sec then
15:23:01 gouthamr dansmith: i reported this https://bugs.launchpad.net/manila/+bug/2164609 and started investigating.. my hunch is this has to do with the eventlet backend (default today)
15:23:31 gouthamr dansmith: tyty
15:23:51 dansmith gouthamr: the logs and things could go in the bug.. do you want the DB dump there as well (or not at all?)
15:26:17 dansmith oh, is that bug for something more specific and thus not quite appropriate?
15:26:29 dansmith FWIW, it does seem like restarting the service takes a long time, like it's waiting for something
15:28:18 gmaan dansmith: btw, this is ready. CI is flaky over other unrelated failure so zuul -1 https://review.opendev.org/c/openstack/nova/+/1000211/14
15:28:46 dansmith ack
15:28:49 gmaan thanks
15:44:28 dansmith gouthamr: the log is too big to paste, as is the sql dump of course, so I emailed them to you and you can decide what to do witht hem
15:48:10 gouthamr ah! thank you dansmith, got that email - this is super helpful
15:49:39 dansmith gouthamr: using --unrestrict let me delete the stuck access rule. attempting to start the instance after that re-created the rule and the instance started (for the first time ever)
15:50:26 dansmith so that's good, I was able to get it working again with just admin/api stuff and no surgery.. IIRC, this would all have required surgery to resolve before you added the idempotent give-myself-access-always-on-start behavior
15:50:32 dansmith so a definite win for robustness, IMHO
15:50:33 gouthamr wow 😅
16:06:25 gouthamr dansmith: ty for pushing for that robustness.. this looks a lot better than i initially intended it
16:06:37 gouthamr will get back to you about the m-shr issues
17:40:44 opendevreview Brian Haley proposed openstack/nova master: Change neutron API calls to use project_id https://review.opendev.org/c/openstack/nova/+/994321
18:06:57 gmaan another bug (most prob tempest schema) happening in gate, reported https://bugs.launchpad.net/nova/+bug/2164677
18:08:01 gmaan currently I am seeing these many failure happening in CI, not sure we have etherpad to track them ( Uggla may knows ) but listing here just in case anyone else want it to check it before recheck
18:08:09 gmaan https://bugs.launchpad.net/nova/+bug/2164606
18:08:10 gmaan https://bugs.launchpad.net/nova/+bug/2163552
18:08:16 gmaan https://bugs.launchpad.net/nova/+bug/2163448
18:08:28 gmaan https://bugs.launchpad.net/nova/+bug/2160254
18:08:35 gmaan https://bugs.launchpad.net/nova/+bug/2164677
18:20:11 melwitt gmaan: I thought a schema validation failure would be 100% fail rate, so I'm a bit confused for this. the example linked in https://bugs.launchpad.net/nova/+bug/2164677 patch is also not touching any API area so it's not that either
18:22:44 gmaan melwitt: indeed. it is a clear error in tempest schema but not sure why it was/is not failing 100% there is 'instance_uuid' in required field instead of 'server_uuid' field https://github.com/openstack/tempest/blob/master/tempest/lib/api_schema/response/compute/v2_1/servers.py#L555
18:26:09 melwitt gmaan: hm, it looks like it's /servers/{server_id}/migrations API has the 'server_uuid'
18:27:15 melwitt https://github.com/openstack/nova/blob/ec47550235877e82f529f6a8c99f32dd9ea9c1a8/nova/api/openstack/compute/schemas/server_migrations.py#L85
18:27:33 gmaan melwitt: yeah, i think i got it. this schema is for list server in-progress live migration and most of time it may be empty so passing but this time it catch the in-progress live migration and hit the schema bug
18:27:45 melwitt ahhh
18:28:09 melwitt makes sense

Earlier   Later