| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-08-31 | |||
| 16:58:36 | gibi | :) | |
| 16:58:39 | gmaan | thanks | |
| 16:59:51 | gmaan | monday morning, need more coffee | |
| 17:43:20 | gouthamr | dansmith: welcome back | |
| 17:47:34 | gouthamr | sean-k-mooney: dansmith : ack on the review comments; I’ll get to them as soon as I get to my laptop | |
| 18:12:02 | gmaan | sean-k-mooney: as you pinged me on this, I think returning 400 is more suitable here, returning 409 need microversion https://review.opendev.org/c/openstack/nova/+/946223 | |
| 18:12:07 | gmaan | i commented on it | |
| 18:27:22 | sean-k-mooney | i would not really agree that all action apis are indivgual interfaces | |
| 18:27:35 | sean-k-mooney | but i dont have a stong perfence | |
| 18:28:18 | sean-k-mooney | its a single endpoing form my persecitie and the return code shoudl not really vary based on the request body once the request is valid | |
| 18:28:37 | sean-k-mooney | but ya either is fine if you prefer the 400 that works for me | |
| 19:08:35 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron networks https://review.opendev.org/c/openstack/nova/+/928022 | |
| 19:08:35 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 19:08:36 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 19:59:36 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | |
| 21:25:38 | gmaan | historically, nova action APIs has not been best interface but they are individual operations and interface which is why we have different set of return code per action APIs | |
| 21:26:04 | opendevreview | Merged openstack/nova master: ovo: Drop deprecated item_cls argument https://review.opendev.org/c/openstack/nova/+/1002937 | |
| 22:25:37 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Fix config option registration https://review.opendev.org/c/openstack/os-vif/+/1003173 | |
| 22:25:58 | opendevreview | Stephen Finucane proposed openstack/os-vif master: Fix config option registration https://review.opendev.org/c/openstack/os-vif/+/1003173 | |
| 22:29:40 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (2/3) https://review.opendev.org/c/openstack/nova/+/1002657 | |
| 22:29:40 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (3/3) https://review.opendev.org/c/openstack/nova/+/1002359 | |
| 22:29:41 | opendevreview | Stephen Finucane proposed openstack/nova master: neutron: Simplify client creation https://review.opendev.org/c/openstack/nova/+/1002658 | |
| 22:29:41 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Tweak endpoint_override logic https://review.opendev.org/c/openstack/nova/+/1003174 | |
| 22:29:42 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Use discover_version for microversion negotiation https://review.opendev.org/c/openstack/nova/+/1003175 | |
| 22:29:42 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (1/3) https://review.opendev.org/c/openstack/nova/+/1003176 | |
| 22:29:43 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop support for v2 password auth https://review.opendev.org/c/openstack/nova/+/1003177 | |
| 22:29:56 | stephenfin | melwitt: I went down a rabbit hole 🤦 | |
| 23:03:27 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Use discover_version for microversion negotiation https://review.opendev.org/c/openstack/nova/+/1003175 | |
| 23:03:27 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (1/3) https://review.opendev.org/c/openstack/nova/+/1003176 | |
| 23:03:28 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (2/3) https://review.opendev.org/c/openstack/nova/+/1002657 | |
| 23:03:28 | opendevreview | Stephen Finucane proposed openstack/nova master: cinder: Replace legacy auth options (3/3) https://review.opendev.org/c/openstack/nova/+/1002359 | |
| 23:03:29 | opendevreview | Stephen Finucane proposed openstack/nova master: neutron: Simplify client creation https://review.opendev.org/c/openstack/nova/+/1002658 | |
| 23:03:29 | opendevreview | Stephen Finucane proposed openstack/nova master: Drop support for v2 password auth https://review.opendev.org/c/openstack/nova/+/1003177 | |
| 23:03:30 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: cinder: Simplify client creation https://review.opendev.org/c/openstack/nova/+/1003188 | |
| #openstack-nova - 2026-09-01 | |||
| 00:51:26 | opendevreview | Merged openstack/os-vif master: Fix config option registration https://review.opendev.org/c/openstack/os-vif/+/1003173 | |
| 04:07:56 | opendevreview | Adam McArthur proposed openstack/nova master: Use safe allocation deletion on instance delete https://review.opendev.org/c/openstack/nova/+/1003208 | |
| 07:11:09 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 07:11:10 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 07:38:10 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 07:38:10 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 07:48:10 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron subnets https://review.opendev.org/c/openstack/nova/+/962190 | |
| 07:48:11 | opendevreview | Lajos Katona proposed openstack/nova master: Use SDK for Neutron extensions https://review.opendev.org/c/openstack/nova/+/962270 | |
| 08:07:04 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 08:07:04 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 08:28:57 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Support cold migration with virtiofs shares https://review.opendev.org/c/openstack/nova/+/989633 | |
| 08:29:10 | opendevreview | Goutham Pacha Ravi proposed openstack/nova master: Reconcile stale Manila share access rules https://review.opendev.org/c/openstack/nova/+/1001916 | |
| 08:33:55 | gibi | gmaan: https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/service.py#L435 this combined with the _get_manager_shutdown_timeout returning 0 is a big trap in the functional tests. It means test cases leaving a thread behind that finishes after the test case finished and a new one starts and they are calling things on the manager that calling things on the global | |
| 08:34:01 | gibi | executor state. | |
| 08:40:29 | gibi | I filed https://bugs.launchpad.net/nova/+bug/2165991 | |
| 08:54:51 | gibi | btw new hit on TIMED_OUT and no new hit on DB locked. But I still checking out the rest of the functional-threading failures happend yesterday (zuul is non cooperative): | |
| 08:55:04 | gibi | I mean no new hit | |
| 08:55:41 | gibi | I belive the DB locked issue is real as I saw it locally once or twice in extreme scenarios | |
| 08:55:54 | gibi | (extrem means by introducing lot of additional DB churn) | |
| 08:56:05 | gibi | but still no stable reproducer | |
| 09:03:06 | gibi | I have one suspicious functional-threading fauilure from yesterday https://5125be418d93521dc669-217c5d59ee9178b06ecc76d8426e75cb.ssl.cf1.rackcdn.com/openstack/1ff94c6e70154943aae8d53d58fbcdd5/testr_results.html I will look deeper into it | |
| 09:33:03 | opendevreview | Ashish Gupta proposed openstack/nova master: Fix init_host crash migration test under native threading https://review.opendev.org/c/openstack/nova/+/1001369 | |
| 10:36:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: [test]Ensure shutdown thread is waited on https://review.opendev.org/c/openstack/nova/+/1003245 | |
| 10:36:54 | gibi | gmaan: ^^ fix for the above reported bug | |
| 10:37:13 | gibi | gmaan: I will put a follow up on top with 3 functional test fixed that haning the shutdown | |
| 10:52:44 | gibi | gmaan: also if I see that graceful shutdown logs a task as live_migration_at_dest which RPC handler it is? post...at_dest? rollback...at_dest? | |
| 11:14:57 | gibi | I started logging stack trace at _record_task_start and I see something strange | |
| 11:15:30 | gibi | the _record_task_start live_migration_at_dest actually happens from File "/home/gibi/upstream/git/openstack/nova/nova/compute/manager.py", line 9431, in pre_live_migration | |
| 11:15:59 | gibi | is this correct that we name the task live_migration_at_dest when it is actually a pre_live_migration RPC? | |
| 11:18:27 | gibi | we need at least some help for devs to know what hangs when we log that | |
| 11:19:13 | gibi | as it was a pretty long goose chase for me to find out that pre_live_migration records a task named live_migration_at_dest but that task only finished when the whole live migration terminated. | |
| 11:19:37 | gibi | which make sense after I discovered all this but was pretty strange when I did not know about that logic | |
| 11:23:17 | gibi | btw I think we have a path that rolls back the live migration but never finishes the live_migration_at_dest task | |
| 11:23:39 | gibi | filing a bug and then we can discuss if it is a realistic scenario | |
| 11:23:52 | gibi | (it is hit by a functional test so it is at least triggerable) | |
| 11:34:59 | gibi | (I cannot open a bug as launchpad is dead) | |
| 11:35:18 | gibi | so https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L10553-L10562 here if the code takes the else branch as do_cleanup is false then the task will never finish | |
| 12:00:23 | gibi | https://bugs.launchpad.net/nova/+bug/2166009 | |
| 12:00:57 | sean-k-mooney | _rollback_live_migration can be called in _cleanup_pre_live_migration and 2 places in _do_live_migration before and after the migrtion is called | |
| 12:01:14 | sean-k-mooney | i dont really recall off the top of my head what the critia for do_cleanup is | |
| 12:01:32 | sean-k-mooney | looks like its determeidn by https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9968 _live_migration_cleanup_flags | |
| 12:01:57 | sean-k-mooney | i guess its tryign to figure out if its on shared storage? | |
| 12:02:10 | sean-k-mooney | but we defintly dont want to alwasy do that | |
| 12:02:52 | sean-k-mooney | do you think its only a testing issue or a production one | |
| 12:04:25 | gibi | I feel like this is a real code path we can take | |
| 12:04:47 | gibi | and it result in a task kept pending forever in the graceful shutdown task tracker | |
| 12:04:48 | sean-k-mooney | yes if we are on shared storage i think we will take it (nfs) | |
| 12:06:01 | sean-k-mooney | or perhasp if we fail in pre-live migrtion | |
| 12:06:08 | sean-k-mooney | as there is nothign to clean up on the dest | |
| 12:07:26 | gibi | the complication here as far as I see is that this task is in the dest compute's tracker and the source computes rolls the migration back and never calls the dest compute so the dest compute cannot finish the task | |
| 12:07:30 | sean-k-mooney | it looks like do_cleanup, destroy_disks are only ever used inside that if | |
| 12:08:11 | sean-k-mooney | well should it be in the dest tracker at that point | |
| 12:08:52 | sean-k-mooney | i havent fully follosed you but if it roling back before the call to pre live migrate on dest it wont be there | |
| 12:09:45 | gibi | if you want to trace it there are at least two function test cases that hitting the codepath (noted in the bugreport) | |
| 12:09:51 | sean-k-mooney | so the edge case would have to haappend form pre live migratoin at soruce or the call to migrateToURI3 at the libivrt level | |
| 12:11:25 | sean-k-mooney | if we are seing live_migration_at_dest i think that impleise we are rooling back at the libvirt migrate call | |
| 12:11:29 | gibi | test nova.tests.functional.regressions.test_bug_1888395.TestLiveMigrationRollbackWithoutMultiplePortBindings this is the cleaner repro and it trigger dom.fail_job() | |
| 12:12:24 | sean-k-mooney | as in this is happen right at the point of no return since we do not supprot rollback if that call complete without error | |
| 12:13:36 | sean-k-mooney | thats still a little impresices we do not supprot calling rollabck if that libvirt job that was started by that call complete without failing | |
| 12:16:18 | gibi | this also triggers the path and it uses abort not rollback nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSampleWithMultipleCompute._test_live_migration_abort | |
| 12:17:55 | sean-k-mooney | gibi: so we are recordign the task start here https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9426-L9432 | |
| 12:18:09 | sean-k-mooney | but never callign https://github.com/openstack/nova/blob/d3a1c95d06c2d97368207c2728fc262796683ad8/nova/compute/manager.py#L9441-L9444 | |
| 12:18:29 | gibi | yeah | |
| 12:18:43 | gibi | I think rollback at dest and post at dest call that helper that finishes the task | |
| 12:18:55 | gibi | but in this scenario there is no rpc to the dest that can call that helper | |
| 12:19:19 | sean-k-mooney | if we just passed destory disks false when we woudl have had do_cleanup=false | |