Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-04
19:26:09 sean-k-mooney ah ok
19:26:29 sean-k-mooney THP is awsome if it works and terible otherwise
19:26:36 sean-k-mooney espically when combind iwht ksm
19:26:44 sean-k-mooney it causes a lot of non determinism
19:27:03 Zhan[m] exactly lol, but yeah, that's why we also wanna tweak nova to just skip the cleanup
19:28:55 sean-k-mooney hum i was not aware of that bug but reading it that definly going to slow things donw
19:29:12 sean-k-mooney is that fixed upstream?
19:29:19 Zhan[m] yes, what QEMU does in migration is that it will read the page first and then write
19:29:24 Zhan[m] it's fixed in 6.13 IIRC
19:29:46 Zhan[m] so all of the following writes done by the VM will be regular pages
19:29:57 Zhan[m] which is painful....
19:30:31 sean-k-mooney ah ok
19:30:34 Zhan[m] https://github.com/torvalds/linux/commit/5c00ff742bf5caf85f60e1c73999f99376fb865d yeah 6.13
19:30:57 sean-k-mooney 6.13 is resonably new so that proably not in a lot of disto kernels
19:31:39 sean-k-mooney ubutnu has 6.14 in the HWE kernsl but it default to 6.11 for 24.04
19:31:59 sean-k-mooney rhel 10 is 6.12 + backports
19:31:59 Zhan[m] yeah but anyways, nova doesn't really need to wait for the cleanup to be done.
19:32:09 Zhan[m] if it's doing cleanup, then it means the VM has been resumed on dest
19:32:19 sean-k-mooney ya
19:32:29 Zhan[m] which we can start at least setting up the network
19:32:37 sean-k-mooney if we can trigger form that point we likely cna just go direclty ot post-live-migrate
19:33:08 sean-k-mooney but we cant relibly tirgger on the cleanup starting then we can expore just doing the network activation
19:34:08 Zhan[m] yeah just the network I feel is fine, everything else can be pushed back
19:34:26 Zhan[m] what VM users ultimately care about is just network connectivity and if the VM is running or not
19:34:49 Zhan[m] which in that case the VM is running so we should setup the network
19:36:01 sean-k-mooney what i woudl suggest is push a patch to hack https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L274-L321
19:36:21 sean-k-mooney oh
19:36:23 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L295-L298
19:36:56 Zhan[m] right, I'm thinking if the migration fails, we rollback the port bindings back to source
19:37:11 sean-k-mooney Zhan[m]: so based on that comment we cant realy on that only beign sent on sucssess
19:37:20 sean-k-mooney whgich mean we have to call guest.get_job_info()
19:37:42 Zhan[m] right but that get_job_info will hang until cleanup is done
19:37:48 sean-k-mooney right
19:38:11 sean-k-mooney but we cant just assume its ok to activate the netork on the dest based on VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED
19:38:49 Zhan[m] right, but if migration fails at last, then we can just rollback the port binding change back to source
19:39:11 sean-k-mooney no becaue if the migtion failed that does not mean the guest crashed
19:39:20 sean-k-mooney it can unpase on the souce
19:39:35 sean-k-mooney so we dont want to have to rool back the prot bidning in that case
19:40:38 Zhan[m] hmm why we can't?
19:41:23 sean-k-mooney i was goign tot setted modfiyin tghis to just tirger post-live-migration and see what happesn in ci
19:41:48 sean-k-mooney https://github.com/openstack/nova/commit/aa87b9c288d316b85079e681e0df24354ec1912c
19:42:09 sean-k-mooney well because of https://bugs.launchpad.net/nova/+bug/1788014
19:42:31 Zhan[m] right that is what I put in that bug report
19:43:06 Zhan[m] the question is "As a result, failed live migrations will inadvertantly trigger activation of the port bindings on the destination host, which deactivates the source host port bindings, and then _rollback_live_migration will delete those activated dest host port bindings and leave the source host port bindings deactivated."
19:43:23 Zhan[m] so let's just rollback the port binding changes in _rollback_live_migration?
19:43:56 Zhan[m] for failure cases like this.
19:44:13 sean-k-mooney if the migration failure does nto kill the vm and it resulmes activating the port binding on the dest will deactivate tehm on the souce and some network backend will stop delivering packet to the soruce
19:44:38 sean-k-mooney so we woudl be addigng nework downtiem in the failure case that did nto exist before
19:44:52 sean-k-mooney however that may or may not be accpatable
19:46:44 sean-k-mooney im going to quickly try hacking somethign to see how it behviaves one sec
19:49:31 Zhan[m] I tested this (rollback when failure) in my environment, it works properly, like I see port binding gets updated on dest, then I wrote a function to roll back (basically delete, create, and re-activate bindings on source), then the VM regained network connectivity and is still on source.
19:50:58 Zhan[m] yes we would add more downtime, but I would think that failure doesn't really happen that often (correct me if I'm wrong), so we would be trading benefits for something happens less frequent for benefits for something happens more frequent (migration success case).
19:51:12 Zhan[m] *downtime for failure cases
19:52:31 opendevreview sean mooney proposed openstack/nova master: [DNM] testing early activation https://review.opendev.org/c/openstack/nova/+/966106
19:53:39 sean-k-mooney Zhan[m]: in the happy path i think that is effectivly what you want to do
19:53:53 Zhan[m] yes
19:54:15 Zhan[m] the only concern I think is the unhappy path :)
19:56:08 sean-k-mooney ya, so the reason im debating this in my head is this is already complciated and im not sure we want to make it more complicted by adding aother event handeler
19:57:47 Zhan[m] To ease the process a little bit, so basically: 1. activate port bindings when paused 2. If migration ok later, all is well 3. If migration not ok later, delete, recreate, reactivate the port binding on the source side
19:58:22 sean-k-mooney well you just activate them again and delete the dest ones
19:58:24 Zhan[m] It is complicated :( I spent quite some time digging all these too
19:58:49 sean-k-mooney but you also have to cater for neturon backend that dont supprot multipel port bidnign
19:58:59 sean-k-mooney because we have not made those unsupproted yet
19:59:42 Zhan[m] I'm not sure on that, I got exceptions when I try to re-enable the source bindings and I saw some fields are missing in DB, so that's why I recreated the bindings, but this is not important, the idea is the same.
19:59:42 sean-k-mooney there are some network backend that dont supprot the multipe prot bidnging exteion and in those case we only have one and we jsut update the bindings:host field
20:00:09 sean-k-mooney i gues it depend on if they are delete or not
20:00:22 Zhan[m] right, if backend doesn't support I think the port binding extension will not be used right?
20:00:33 sean-k-mooney but ya we have to recrate them if they are i though activating them only deactived teh souce one but didnt delete tehm
20:00:37 Zhan[m] yeah in that case migrate_instance_start won't do anything, so
20:01:38 Zhan[m] they are not deleted, but as far as what I'm seeing in the DB, some metadata are deleted so hmmm not very sure on that
20:01:46 sean-k-mooney so what i woudl like is if we do have a dedicated port biding activation phase we use it for both pre and post copy
20:01:55 sean-k-mooney that is how it works today
20:02:36 sean-k-mooney what i dont wnat to have is 1 way to activate them for pre-copy, 1 way for post copy adn 1 way for backend that dont suprot it
20:02:38 Zhan[m] ok so like extracting it out of post live migration and event handler function?
20:02:52 sean-k-mooney right
20:03:30 sean-k-mooney we woudl pull out the networkign bits and do them alwsy in a new funciton that woudl be before psot-copy and move post copy to happen alwasy on the migration complete event
20:03:46 Zhan[m] I see, let's put that maybe as a stage 2 thing? I'll work on this first and see how we refactor this process later
20:04:04 sean-k-mooney well we woudl want ot have both before merging any code
20:04:06 Zhan[m] obviously the spec will come first
20:04:23 sean-k-mooney but ya spec first to fiutre out exaclty how it shoudl look
20:04:44 sean-k-mooney is this somehtin you think you would have time to work on this cycle
20:04:46 Zhan[m] ok let's start with the spec first, thanks for looking into this too!!
20:05:01 Zhan[m] I'll see and get back to you on this.
20:05:08 sean-k-mooney ack no worries
20:21:06 Zhan[m] Ok so I'll be working on the spec now and submit it soonish, but regarding the code (the refining part) may come later in Q1, consider we may have some back and forth in the spec.
20:21:56 Zhan[m] I'll put the goal of spec to be 1. this change 2. put networking bits in the right place.
20:26:36 sean-k-mooney Zhan[m]: we have selected december 4 or 8th for spec feeze
20:26:51 sean-k-mooney with feature freeze aroudn the middle fo febuary
20:27:02 sean-k-mooney just for context so you are not surpised
20:27:14 sean-k-mooney normally sepc freeze would be around january 8th
20:34:08 Zhan[m] understood, thanks for the heads up. In this case I think I'll submit the spec before Dec 4 and I can open a PR in Q1 but it can definitely wait until the freeze is over.
20:44:46 sean-k-mooney well its better to submit it early not late as if it comes in mid to late q1 it will be unlikely to land this cycle
20:46:09 sean-k-mooney the offical release is Wednesday, April 1, 2026. freautre freeze is February 26, 2026
20:46:45 sean-k-mooney so fi you only puth the patch up for review in late january or early febuary we likely wont have time to review it
20:50:49 Zhan[m] Ok, I'll see if I can make it there, if not, what will be the next cycle?
#openstack-nova - 2025-11-05
05:41:31 opendevreview Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304
06:32:38 opendevreview Masanori Kuroha proposed openstack/nova master: Copy applied provider config https://review.opendev.org/c/openstack/nova/+/948304
09:01:50 mkuroha bauzas: gibi: sean-k-mooney: I’ve updated the feature that copies the applied provider config files. I’d appreciate it if you could review it when you have time. https://review.opendev.org/c/openstack/nova/+/948304
09:23:06 zigo gmaan: Since you wrote these tests, any idea what could have gone wrong? https://bugs.launchpad.net/nova/+bug/2130703
10:00:44 bauzas mkuroha: as said in the change, pease reprovide the spec for 2026.1 so we could reapprove it

Earlier   Later