| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 19:43:23 | Zhan[m] | so let's just rollback the port binding changes in _rollback_live_migration? | |
| 19:43:56 | Zhan[m] | for failure cases like this. | |
| 19:44:13 | sean-k-mooney | if the migration failure does nto kill the vm and it resulmes activating the port binding on the dest will deactivate tehm on the souce and some network backend will stop delivering packet to the soruce | |
| 19:44:38 | sean-k-mooney | so we woudl be addigng nework downtiem in the failure case that did nto exist before | |
| 19:44:52 | sean-k-mooney | however that may or may not be accpatable | |
| 19:46:44 | sean-k-mooney | im going to quickly try hacking somethign to see how it behviaves one sec | |
| 19:49:31 | Zhan[m] | I tested this (rollback when failure) in my environment, it works properly, like I see port binding gets updated on dest, then I wrote a function to roll back (basically delete, create, and re-activate bindings on source), then the VM regained network connectivity and is still on source. | |
| 19:50:58 | Zhan[m] | yes we would add more downtime, but I would think that failure doesn't really happen that often (correct me if I'm wrong), so we would be trading benefits for something happens less frequent for benefits for something happens more frequent (migration success case). | |
| 19:51:12 | Zhan[m] | *downtime for failure cases | |
| 19:52:31 | opendevreview | sean mooney proposed openstack/nova master: [DNM] testing early activation https://review.opendev.org/c/openstack/nova/+/966106 | |
| 19:53:39 | sean-k-mooney | Zhan[m]: in the happy path i think that is effectivly what you want to do | |
| 19:53:53 | Zhan[m] | yes | |
| 19:54:15 | Zhan[m] | the only concern I think is the unhappy path :) | |
| 19:56:08 | sean-k-mooney | ya, so the reason im debating this in my head is this is already complciated and im not sure we want to make it more complicted by adding aother event handeler | |
| 19:57:47 | Zhan[m] | To ease the process a little bit, so basically: 1. activate port bindings when paused 2. If migration ok later, all is well 3. If migration not ok later, delete, recreate, reactivate the port binding on the source side | |
| 19:58:22 | sean-k-mooney | well you just activate them again and delete the dest ones | |
| 19:58:24 | Zhan[m] | It is complicated :( I spent quite some time digging all these too | |
| 19:58:49 | sean-k-mooney | but you also have to cater for neturon backend that dont supprot multipel port bidnign | |
| 19:58:59 | sean-k-mooney | because we have not made those unsupproted yet | |
| 19:59:42 | Zhan[m] | I'm not sure on that, I got exceptions when I try to re-enable the source bindings and I saw some fields are missing in DB, so that's why I recreated the bindings, but this is not important, the idea is the same. | |
| 19:59:42 | sean-k-mooney | there are some network backend that dont supprot the multipe prot bidnging exteion and in those case we only have one and we jsut update the bindings:host field | |
| 20:00:09 | sean-k-mooney | i gues it depend on if they are delete or not | |
| 20:00:22 | Zhan[m] | right, if backend doesn't support I think the port binding extension will not be used right? | |
| 20:00:33 | sean-k-mooney | but ya we have to recrate them if they are i though activating them only deactived teh souce one but didnt delete tehm | |
| 20:00:37 | Zhan[m] | yeah in that case migrate_instance_start won't do anything, so | |
| 20:01:38 | Zhan[m] | they are not deleted, but as far as what I'm seeing in the DB, some metadata are deleted so hmmm not very sure on that | |
| 20:01:46 | sean-k-mooney | so what i woudl like is if we do have a dedicated port biding activation phase we use it for both pre and post copy | |
| 20:01:55 | sean-k-mooney | that is how it works today | |
| 20:02:36 | sean-k-mooney | what i dont wnat to have is 1 way to activate them for pre-copy, 1 way for post copy adn 1 way for backend that dont suprot it | |
| 20:02:38 | Zhan[m] | ok so like extracting it out of post live migration and event handler function? | |
| 20:02:52 | sean-k-mooney | right | |
| 20:03:30 | sean-k-mooney | we woudl pull out the networkign bits and do them alwsy in a new funciton that woudl be before psot-copy and move post copy to happen alwasy on the migration complete event | |
| 20:03:46 | Zhan[m] | I see, let's put that maybe as a stage 2 thing? I'll work on this first and see how we refactor this process later | |
| 20:04:04 | sean-k-mooney | well we woudl want ot have both before merging any code | |
| 20:04:06 | Zhan[m] | obviously the spec will come first | |
| 20:04:23 | sean-k-mooney | but ya spec first to fiutre out exaclty how it shoudl look | |
| 20:04:44 | sean-k-mooney | is this somehtin you think you would have time to work on this cycle | |
| 20:04:46 | Zhan[m] | ok let's start with the spec first, thanks for looking into this too!! | |
| 20:05:01 | Zhan[m] | I'll see and get back to you on this. | |
| 20:05:08 | sean-k-mooney | ack no worries | |
| 20:21:06 | Zhan[m] | Ok so I'll be working on the spec now and submit it soonish, but regarding the code (the refining part) may come later in Q1, consider we may have some back and forth in the spec. | |
| 20:21:56 | Zhan[m] | I'll put the goal of spec to be 1. this change 2. put networking bits in the right place. | |
| 20:26:36 | sean-k-mooney | Zhan[m]: we have selected december 4 or 8th for spec feeze | |
| 20:26:51 | sean-k-mooney | with feature freeze aroudn the middle fo febuary | |
| 20:27:02 | sean-k-mooney | just for context so you are not surpised | |
| 20:27:14 | sean-k-mooney | normally sepc freeze would be around january 8th | |
| 20:34:08 | Zhan[m] | understood, thanks for the heads up. In this case I think I'll submit the spec before Dec 4 and I can open a PR in Q1 but it can definitely wait until the freeze is over. | |
| 20:44:46 | sean-k-mooney | well its better to submit it early not late as if it comes in mid to late q1 it will be unlikely to land this cycle | |
| 20:46:09 | sean-k-mooney | the offical release is Wednesday, April 1, 2026. freautre freeze is February 26, 2026 | |
| 20:46:45 | sean-k-mooney | so fi you only puth the patch up for review in late january or early febuary we likely wont have time to review it | |
| 20:50:49 | Zhan[m] | Ok, I'll see if I can make it there, if not, what will be the next cycle? | |
| #openstack-nova - 2025-11-05 | |||
| 05:41:31 | opendevreview | Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304 | |
| 06:32:38 | opendevreview | Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304 | |
| 09:01:50 | mkuroha | bauzas: gibi: sean-k-mooney: I’ve updated the feature that copies the applied provider config files. I’d appreciate it if you could review it when you have time. https://review.opendev.org/c/openstack/nova/+/948304 | |
| 09:23:06 | zigo | gmaan: Since you wrote these tests, any idea what could have gone wrong? https://bugs.launchpad.net/nova/+bug/2130703 | |
| 10:00:44 | bauzas | mkuroha: as said in the change, pease reprovide the spec for 2026.1 so we could reapprove it | |
| 10:09:49 | mkuroha | bauzas: I see, understood. I'll recreate it for the 2026.1 spec. | |
| 10:10:40 | bauzas | mkuroha: thanks, ping me again once you're done, just copy the file and add in the history section something like "2025.2: Approved; 2026.1: Reproposed" | |
| 10:10:52 | bauzas | should be quick | |
| 10:14:00 | mkuroha | ok! thanks. | |
| 10:18:00 | opendevreview | Masanori Kuroha proposed openstack/nova-specs master: Copy the spec approved in 2025.2 to 2026.1 https://review.opendev.org/c/openstack/nova-specs/+/966151 | |
| 10:20:20 | mkuroha | bauzas: Thank you for the comment, I created: https://review.opendev.org/c/openstack/nova-specs/+/966151 | |
| 11:14:46 | opendevreview | Merged openstack/nova master: api: Pre-query not deleted members in server groups https://review.opendev.org/c/openstack/nova/+/959696 | |
| 11:15:54 | opendevreview | Merged openstack/nova master: setup: Remove pbr's wsgi_scripts https://review.opendev.org/c/openstack/nova/+/902688 | |
| 11:40:03 | opendevreview | Merged openstack/nova stable/2025.1: Update start_service() function in test https://review.opendev.org/c/openstack/nova/+/963845 | |
| 11:40:14 | opendevreview | Merged openstack/nova stable/2025.1: Adds regression test for bug LP#2085135 https://review.opendev.org/c/openstack/nova/+/963846 | |
| 12:25:07 | opendevreview | Merged openstack/nova stable/2025.1: Reset the mapped field of nodes at service deletion https://review.opendev.org/c/openstack/nova/+/963847 | |
| 12:26:24 | sean-k-mooney | Zhan[m]: if you have input on https://review.opendev.org/c/openstack/nova/+/966106/comment/80b6a2a0_591c5cab/ feel free to add to it | |
| 12:27:22 | sean-k-mooney | for context of others ^ is related to https://bugs.launchpad.net/nova/+bug/2128665 which is a very specific edgecase that results in live migration downtime when using calico | |
| 12:27:44 | sean-k-mooney | it could impact other backend but shoudl not impact ml2/ovs or recent version fo ml2/ovn | |
| 12:28:44 | sean-k-mooney | im conflcited on if this shoudl be fixed, if it should be fixed as a bug or as a feature to split post-live-migration into networking and the rest | |
| 12:52:33 | opendevreview | Masanori Kuroha proposed openstack/nova-specs master: Copy the spec approved in 2025.2 to 2026.1 https://review.opendev.org/c/openstack/nova-specs/+/966151 | |
| 14:00:01 | Zhan[m] | sean-k-mooney: Thanks, I will put comments there too. | |
| 14:02:06 | Zhan[m] | Just for my own understanding, because we are not using ovn: from what you describe, the ovn driver in some sense is not really depending on the port binding activation but via the RARP packet (which I assume will be sent as soon as the destination QEMU process is resumed)? | |
| 14:03:30 | Zhan[m] | calico is more of using the port binding activation as the trigger for switching the network to the dest QEMU so yes that's why we're hitting | |
| 14:11:06 | sean-k-mooney | Zhan[m]: so the ml2/ovs ml2/linuxbrige (before they deleted it) and as of about a year ago ml2/ovn driver | |
| 14:11:24 | sean-k-mooney | all wired up the port is there was a portbinidng for the host | |
| 14:11:30 | sean-k-mooney | not just an active one | |
| 14:11:42 | sean-k-mooney | for calico this si tricky because the tap does not exsit yet | |
| 14:12:21 | sean-k-mooney | Zhan[m]: hoever we coudl modify the supprot for vif_type=tap to create the tap in pre-live-migrtion | |
| 14:12:30 | sean-k-mooney | we are condiering doing that for ovn | |
| 14:12:39 | sean-k-mooney | becuse ovn while it strcitly sepaking does not need it | |
| 14:12:49 | sean-k-mooney | can take a long time to install the flow in larger clouds | |
| 14:13:20 | sean-k-mooney | basically whith many vm and port the ovn db perfromcen cna resulting in reconsitaion taking over 30s in some cases | |
| 14:14:07 | sean-k-mooney | Zhan[m]: we modifed ovn to allow the trafic swithc to be triggred by the RARP packet sent by qemu when the vm unpauses | |
| 14:14:40 | sean-k-mooney | Zhan[m]: for ml2/ovs and ml2/linux bridge because they relyied on mac learning the RARP is the only thing that cause the traffic to be rerouted | |
| 14:15:17 | sean-k-mooney | for calico that a bit harder becuase its an l3 backend so the mac learning frames are likely entirly ignored | |
| 14:15:49 | sean-k-mooney | Zhan[m]: if you wanted to improve downtime mroe in calico then using the RARP to triger the update would be a consitent approch | |
| 14:26:24 | Zhan[m] | Indeed it's tricky I think for l3 in general, thanks for the explanation. | |
| 14:27:46 | Zhan[m] | but wondering if ml2/ovs can switch traffic with the RARP, what is the role that the port binding is playing here? | |
| 14:28:15 | Zhan[m] | the way we use calico that i do see port bindings to the destination are created in pre live migraiton | |
| 14:29:26 | opendevreview | Julien LE JEUNE proposed openstack/nova stable/2024.2: Adds regression test for bug LP#2044235 https://review.opendev.org/c/openstack/nova/+/961602 | |
| 14:29:57 | opendevreview | Julien LE JEUNE proposed openstack/nova stable/2024.2: nova-conductor puts instance in error state https://review.opendev.org/c/openstack/nova/+/961775 | |
| 14:34:57 | opendevreview | Julien LE JEUNE proposed openstack/nova stable/2024.2: nova-conductor puts instance in error state https://review.opendev.org/c/openstack/nova/+/961775 | |
| 15:18:33 | opendevreview | Henry Richter proposed openstack/nova master: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/955657 | |
| 15:29:34 | opendevreview | Nicolai Ruckel proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/959682 | |
| 15:40:59 | nicolairuckel | sean-k-mooney, thanks for your feedback. That fixed almost everything. | |
| 15:41:24 | nicolairuckel | I was so sure I already tried that but maybe that was before you told me that I need to recreate the VM. | |
| 15:41:30 | sean-k-mooney | cool ill pull in your updated code later in the week | |
| 15:41:36 | nicolairuckel | thanks | |