| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-11-04 | |||
| 15:15:55 | sean-k-mooney | across openstack we are ment ot have <package>.wsgi.<api endpoint) | |
| 15:16:23 | sean-k-mooney | so you could monkey patch here https://github.com/openstack/nova/blob/master/nova/wsgi/__init__.py but im not sure it is required | |
| 15:16:25 | gibi | hm but our setup.cfg ponts to nova-api-wsgi = nova.api.openstack.compute.wsgi:init_application | |
| 15:16:49 | sean-k-mooney | that in wsgi script | |
| 15:16:52 | sean-k-mooney | but that is being deleted | |
| 15:17:05 | sean-k-mooney | it will generate the same script effectivly | |
| 15:17:42 | sean-k-mooney | we agreed at the ptg to finally delete https://github.com/openstack/nova/blob/master/setup.cfg#L87-L89 | |
| 15:17:51 | gibi | but until that setup.cfg is a thing I need to make sure that works. So today I cannot move the monkey patch to nova/wsgi because that will not be executed if the setup.cfg defined entry point is used | |
| 15:18:35 | gibi | I agree that once the setup.cfg does not have such entry point defined and only the nova/wsgi is used then I can move the monkey_patch | |
| 15:18:37 | sean-k-mooney | right but lest just find and merge stephesn patch | |
| 15:18:44 | sean-k-mooney | then you can move it if needed | |
| 15:18:50 | gibi | OK | |
| 15:19:14 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/902688 | |
| 15:19:40 | sean-k-mooney | he updated the docs too | |
| 15:20:22 | sean-k-mooney | so its a slightly longer review then i was plannign but ill re review that today/tomorrow | |
| 15:20:53 | gibi | so that patch suggest the deployers do the right thing and use nova/wsgi | |
| 15:21:17 | sean-k-mooney | yep which is how devstack has worked for 2 releases now | |
| 15:25:33 | gibi | I need to respin my defaulting series anyhow so I can rebase on top of stephenfin and move the monkey patch accordingly | |
| 15:37:29 | stephenfin | sean-k-mooney: if you have time, would you be able to continue and grab the other 3 patches on top of that one? https://review.opendev.org/c/openstack/nova/+/953703 | |
| 16:00:05 | sean-k-mooney | my 1:1 just finished so sure. | |
| 16:10:09 | gibi | stephenfin: could you quickly fix the versioning issue in https://review.opendev.org/c/openstack/nova/+/902688/9#message-72481479904c69ce6f3adbcdfb26143c6f85756a then we can land that | |
| 16:13:33 | sean-k-mooney | gibi: ah the versionchanged line | |
| 16:13:39 | sean-k-mooney | ya that shoudl eb 33.0.0 | |
| 16:14:14 | opendevreview | Stephen Finucane proposed openstack/nova master: setup: Remove pbr's wsgi_scripts https://review.opendev.org/c/openstack/nova/+/902688 | |
| 16:14:15 | opendevreview | Stephen Finucane proposed openstack/nova master: Migrate mypy configuration to pyproject.toml https://review.opendev.org/c/openstack/nova/+/953703 | |
| 16:14:15 | opendevreview | Stephen Finucane proposed openstack/nova master: Migrate codespell configuration to pyproject.toml https://review.opendev.org/c/openstack/nova/+/953704 | |
| 16:14:16 | opendevreview | Stephen Finucane proposed openstack/nova master: Migrate setup configuration to pyproject.toml https://review.opendev.org/c/openstack/nova/+/953705 | |
| 16:14:16 | opendevreview | Stephen Finucane proposed openstack/nova master: pre-commit: Bump versions https://review.opendev.org/c/openstack/nova/+/966089 | |
| 16:14:22 | stephenfin | gibi: sean-k-mooney: done | |
| 16:14:53 | sean-k-mooney | stephenfin: did you see https://review.opendev.org/c/openstack/nova/+/953704/4/pyproject.toml | |
| 16:15:03 | sean-k-mooney | im fien with not changing it just noded it was false before | |
| 16:15:06 | gibi | stephenfin: thanks | |
| 16:15:24 | sean-k-mooney | as in im fine with the patch as is just making sure you saw it | |
| 16:18:20 | stephenfin | sean-k-mooney: replied on both | |
| 16:20:06 | sean-k-mooney | thanks | |
| 16:29:37 | Zhan[m] | hi friends, I'm trying to work on some improvements for pre-copy live migrations - to reduce VMs' network connectivity loss time. I put my ideas and research in https://bugs.launchpad.net/nova/+bug/2128665 and would love some feedback before I start coding and making changes. does anyone mind taking a look? I can create a blueprint if it's needed. thanks!! | |
| 16:45:04 | sean-k-mooney | Zhan[m]: you are effectivly askign to trigger post-live-migration very slightly earlier | |
| 16:45:22 | sean-k-mooney | when pre-copy triggers a pause of the guest. | |
| 16:45:39 | sean-k-mooney | that only happens if you have set force-complete as the timeout action or if you manually do that | |
| 16:46:06 | sean-k-mooney | we do this for post-copy migration already when it enter the post copy phase | |
| 16:46:32 | sean-k-mooney | im not sure how safe ti woudl be to do that in this case vs the current migration completion event | |
| 16:46:54 | sean-k-mooney | Zhan[m]: this likely need more then a bug ie. perhaps a spec | |
| 16:47:26 | sean-k-mooney | or atleast more condierdation the a quick review of your propal to think though the implciiton for upgrades and the diffenent failure modes | |
| 16:49:15 | Zhan[m] | understood, I will go ahead and create a spec. wanna get some early feedback to see if this idea make sense or not. | |
| 16:49:48 | Zhan[m] | we are not using post-copy (I know that we'll avoid this with post-copy) so hence this is just for pre-copy | |
| 16:49:50 | sean-k-mooney | we might be able to use a slightly earlier event | |
| 16:50:40 | Zhan[m] | from my understanding, the paused event should be the earliest? (basically when the process is paused for switch-over) | |
| 16:51:24 | sean-k-mooney | yes but we need to think about the recover cases if there si a failure after pause | |
| 16:51:39 | sean-k-mooney | we currently do it in complete because we know we cant rollback at that point | |
| 16:52:20 | sean-k-mooney | Zhan[m]: just to confirm you are currenlty using pre-copy not post-copy | |
| 16:52:27 | sean-k-mooney | is there a reason you are not using post-copy | |
| 16:55:56 | Zhan[m] | > if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AQcXzaU3yYgbieAMs5LdQxhfkMQjWLQ8n98jM_lIB8gwWH9ApwZ_vdawDM9LNz0brOl_UevcPCkv3f-67_yYcUZCeal-T3mAAG1hdHJpeC5vcmcvRFZtSG9LWGxZaFNCZ0hrYnpXbVh0WnNi>) | |
| 16:56:15 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AYPX5olphEwNM-oVQEAeW1n4bjBMOCL7JcBa2--xwCdXhT0nA_sPuR_jnKT0C0oDEmSyzPWDZpPKsZ1wc-FdO-dCeal-VAbgAG1hdHJpeC5vcmcvRUtYVUxDZVBiekRLck1wZ0hpSW1ZVWNn>) | |
| 16:56:51 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AfGmfWXp6q289_-pAg_3r2CNmHk8GLA3AAnfqOAWmBViUGQMwTDwe8p12QyJHYlvI7HoFg0OieVLTgc2FNFdS1dCeal-XNrgAG1hdHJpeC5vcmcvZkprTUVOeWlRR0htWmlYc2hLZVhYZ0ha>) | |
| 16:58:00 | Zhan[m] | * - if there si a failure after pause... (full message at <https://matrix.org/oftc/media/v1/media/download/AQbdtBpPhkIEZJLqRgxshCfnuFsmdpJyW0XBuScJnchhklfJJ0L11j5bwq4dTpkoh88wJMokcThxBRTCLw1ECLJCeal-bZagAG1hdHJpeC5vcmcvQXNCTUJqVWtqUWF0cmJKRG1XRURxbmF1>) | |
| 16:58:37 | Zhan[m] | I think this change can help regardless, maybe in some environments, pre-copy migrations are preferred. | |
| 17:05:50 | Zhan[m] | We've also seen some crazy wait time (20+ seconds) between when the VM is paused and when libvirt says the migration is done (the completion event), so that's where our motivation comes from. | |
| 17:48:08 | gibi | I saw working live migration with native threaded compute \o/ | |
| 17:48:11 | gibi | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_071/openstack/07142d96394445fc9d4d5ad58b963223/testr_results.html | |
| 18:12:30 | sean-k-mooney | gibi: nice | |
| 18:14:10 | sean-k-mooney | gibi: do you know what is causing the volume issues | |
| 18:14:17 | gibi | not yet | |
| 18:14:41 | sean-k-mooney | well its less broken then it could have been | |
| 18:14:54 | gibi | I'm still not convinced that the lifecycle events are actually working between libvirt and nova in the gate. Locally it is working for me so I need to look more tomorrow | |
| 18:14:57 | sean-k-mooney | i assum this is lvm? | |
| 18:15:57 | sean-k-mooney | ok ya it is cidner lvm | |
| 18:16:57 | gibi | yeah it is not ceph I'm sure | |
| 18:17:08 | gibi | but I drop now. Continue from here tomorrow | |
| 18:17:26 | sean-k-mooney | gibi: looksl ilke there are are a buch of multipahd error bug i cant tell at a glance if that brick being brick or actully imporant | |
| 18:19:18 | sean-k-mooney | gibi: when we stop monkeypatching nova-comptue | |
| 18:19:28 | sean-k-mooney | we ar ealso unmonkeypatchign os-brick | |
| 18:19:40 | sean-k-mooney | i hope that is not related | |
| 18:20:16 | sean-k-mooney | i do see that some of the iscsiadm commadns failed os hopefuly it a unrealted ci faliure and a recheck will jsut work | |
| 18:20:42 | sean-k-mooney | most of the time the iscsiadm dicovery is workign so it coudl be load/timing related | |
| 18:49:28 | sean-k-mooney | Zhan[m]: multi line matix message are hard to read on the irc side as we only get the first line and link to the rest of the message | |
| 18:49:52 | sean-k-mooney | Zhan[m]: but no i htink the issue you are hitting is sepcific to pre-copy migration | |
| 18:50:17 | Zhan[m] | ah apologies, first time using this | |
| 18:50:42 | sean-k-mooney | thats ok | |
| 18:50:51 | sean-k-mooney | we also get a new message everytime you edit a message | |
| 18:51:28 | Zhan[m] | yes this is specific to pre-copy. When I was reading https://docs.openstack.org/nova/2024.1/configuration/config.html#libvirt.live_migration_permit_post_copy, I think it's saying that post-copy won't be used until we hit timeout right? | |
| 18:52:35 | sean-k-mooney | no post-copy will be enabeld by libvirt without nova's intervention or contol | |
| 18:52:57 | sean-k-mooney | the way post-copy works is qemu does an inital precoppy pahse whiel the instnace is running | |
| 18:53:17 | sean-k-mooney | it then internally decied if it is making enough forward progress to proceed with a standar pre-copy migration | |
| 18:54:14 | sean-k-mooney | if not after the 2nd pre-copy compeltes it will swap to post copy mode where the vm will be resumed on teh dest | |
| 18:54:34 | sean-k-mooney | we trigger off the suspend call in post-copy case | |
| 18:56:54 | sean-k-mooney | Zhan[m]: effectivly what nova does is tell qemu it is allwoed to use post-copy and it will make the decion of when to do it | |
| 18:56:56 | Zhan[m] | thanks, that's interesting to know, I was thinking we are controlling this with Nova when reading the config | |
| 18:57:29 | sean-k-mooney | the config option just tell nova to pass the flag to librit to give it permision to use post-copy | |
| 18:58:12 | Zhan[m] | but talking back, do you think this is something that can be improved on for pre-copy? | |
| 18:58:24 | Zhan[m] | I'll definitely play around with post-copy too. | |
| 18:59:51 | sean-k-mooney | im unsure https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html is the spec where we defien dthe current sematicn for prot activation during live migration | |
| 19:00:57 | sean-k-mooney | if we change when we do the activateion (it may be ok) we need ot review the libvirt docuemtion and lifecycle fo migrton evnet | |
| 19:02:07 | Zhan[m] | yes I will list relevant events, job stats, and related stuff in the spec, as well as any edge cases I can think of. | |
| 19:02:16 | sean-k-mooney | Zhan[m]: today the only way to do the activation sooner is to do post_live_migration sooner which may or may not be safe | |
| 19:02:39 | sean-k-mooney | failing that we need to register a specific addtional event handeler just for the port activation case | |
| 19:03:19 | sean-k-mooney | that shoudl be possible but post_live_migration woudl have to handel the fact that the bining is actived earlier and they clean up code for error/abort woudl too | |
| 19:03:27 | Zhan[m] | Currently, I'm thinking we just do the networking part - activate port bindings earlier like what's done previously | |
| 19:03:58 | sean-k-mooney | well when we activate it on the test network backend are ment to start routing traffic to the dest as well | |
| 19:04:23 | sean-k-mooney | in the past with ovn that would have resulting in packet beign rescived on the source host being drop | |
| 19:04:56 | sean-k-mooney | today we replciat the packt to the inactive port bidnign sepcified in the migrating_to filed in the port bidning profile | |