| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-16 | |||
| 12:09:14 | opendevreview | Merged openstack/nova master: Move ComputeManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948186 | |
| 13:45:03 | noonedeadpunk | hey folks! am I right that there is no way so far to supply any arbitrary arguments to as migration_flags ? | |
| 13:48:37 | noonedeadpunk | specifically thinking about VIR_MIGRATE_PARALLEL and VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS now | |
| 14:08:02 | sean-k-mooney | noonedeadpunk: be design no not really. that an internal interface of nova's libvirt driver | |
| 14:08:26 | sean-k-mooney | we add flag when we supprot new features in our config. | |
| 14:08:50 | sean-k-mooney | unofficlly you very ocationlly can enable some thing via the migration uri via query args | |
| 14:10:30 | sean-k-mooney | we woudl need to actully add supprot for that in nova properly if we were to supprot it | |
| 14:15:59 | noonedeadpunk | yeah, I was planning to look into adding this thing to nova, but also wondering about how to workaround current failures of migration | |
| 14:16:17 | noonedeadpunk | As it seems to ways so far is either to disable tls, or increase throughput | |
| 14:24:01 | noonedeadpunk | do you think I'd be needing to submit a full-fledged spec for that? | |
| 14:26:56 | dansmith | for which? always-on migration flags, config knob to enable one, or something like generic flag merging support? | |
| 14:27:23 | dansmith | in either of those cases we probably want to document the need, risk, config behavior, etc | |
| 14:30:54 | gibi | eventlet sync https://meet.google.com/bcy-uqoz-hje | |
| 14:32:40 | noonedeadpunk | dansmith: config knob I believe. to enable/disable parallel and amount of forks | |
| 14:32:56 | noonedeadpunk | ok, I guess I'll come up with smth relatively simple | |
| 14:46:09 | dansmith | noonedeadpunk: I think a spec for that yeah | |
| 15:19:12 | noonedeadpunk | sean-k-mooney: do you recall the issue I was coming with for live migration failures a month ago or so? | |
| 15:19:46 | noonedeadpunk | as I was able to reproduce it on plain libvirt+qemu on ubuntu | |
| 15:20:02 | noonedeadpunk | and pretty much it's kinda a single usecase which is failing.... | |
| 15:20:27 | noonedeadpunk | virsh migrate --live --auto-converge --persistent --copy-storage-all --tls instance-00016a41 qemu+tls://compute10/system | |
| 15:20:56 | noonedeadpunk | if I: drop tls, add parallel or use compress - migration passes. | |
| 15:21:37 | noonedeadpunk | but this also is really confusing... how in the world uwing qemu+tls without --tls does not actually use tls... | |
| 15:22:07 | sean-k-mooney | noonedeadpunk: reading back and yes vagly | |
| 15:23:24 | sean-k-mooney | spec vs specless blueprint in this context partly comes down to the upgrade impact, the other factor being what version fo libvirt/qemu supprot it ectra | |
| 15:23:28 | noonedeadpunk | but anyway, wanted to confirm it has nothing to do with nova. But I'm very surprised nobody else was reporting or coming with that | |
| 15:23:39 | sean-k-mooney | we do allow new config option that default to the old bevhioar without a spec | |
| 15:23:50 | noonedeadpunk | intention was to keep the behavior for sure | |
| 15:24:00 | sean-k-mooney | btu if it impact upgrades in a mixed comptue node version case it almost alwasy needs a spec | |
| 15:25:00 | sean-k-mooney | noonedeadpunk: but if i recall correctly you were seeing qemu crashes right | |
| 15:25:46 | noonedeadpunk | I was thinking to add an option for instance live_migration_parallel_connections with default of 1, and when it's >1 add https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS and libvirt.VIR_MIGRATE_PARALLEL | |
| 15:25:53 | noonedeadpunk | sean-k-mooney: I do | |
| 15:26:19 | noonedeadpunk | also thinking to report a bug to qemu, and gather some debug logs now | |
| 15:26:34 | sean-k-mooney | so i dont actlly have a problem with a config option that set hte number of parralel connection and use that to enable this | |
| 15:27:10 | sean-k-mooney | but beyond that we woudl need to make sure the version of libvirt supprot that | |
| 15:27:19 | sean-k-mooney | and qemu | |
| 15:27:24 | sean-k-mooney | both on the source and dest host | |
| 15:27:39 | sean-k-mooney | and we may or may not want ot schdluer on it. | |
| 15:27:48 | noonedeadpunk | any gotchas on how to trace that back? | |
| 15:28:31 | sean-k-mooney | you have replicated this with that virsh migrate command yes | |
| 15:28:54 | sean-k-mooney | if you have ideally without openstack at all i think we shoudl file a libvirt but for that | |
| 15:29:08 | noonedeadpunk | yeah, it's without openstack at all | |
| 15:29:10 | noonedeadpunk | or well | |
| 15:29:28 | dansmith | I think a spec to explain how this works and what the considerations are is important.. like, does this have a dependency on the remote version? do we want to schedule for the feature? if not, why not? | |
| 15:29:37 | noonedeadpunk | I copied big chunk of domain xml, minus nova metadata | |
| 15:30:05 | sean-k-mooney | ok but you actully created the vm wiht libvirt driectly right | |
| 15:30:11 | noonedeadpunk | yep | |
| 15:30:19 | noonedeadpunk | just 2 independent hosts | |
| 15:30:20 | sean-k-mooney | if you can do that its eaiser for the libvirt maintainer ot recreate this | |
| 15:30:26 | noonedeadpunk | right | |
| 15:31:25 | sean-k-mooney | so i think there are 3 things that can/should happne 1 file a libvirt bug, 2 we can dicuss a spec for the parallel connection supprot, 3 we can discss a workaroudn for your production cloud | |
| 15:31:56 | noonedeadpunk | so parallel migration was added in libvirt 5.2: https://github.com/libvirt/libvirt/commit/d3ea986af24fdb320a54854b6d6668a51ecb0cd0 | |
| 15:32:20 | sean-k-mooney | thats a good data point as its below our min version, what about qemu? | |
| 15:32:40 | noonedeadpunk | I'm not that fast, sorry | |
| 15:32:46 | sean-k-mooney | its fine | |
| 15:33:47 | sean-k-mooney | gemini things it was in qemu 2.1 but im not sure i turst it | |
| 15:34:00 | sean-k-mooney | https://wiki.qemu.org/Features/Migration-Multiple-fds | |
| 15:35:16 | sean-k-mooney | by the way i know there ws also interest in enabling this for vgpu live migration | |
| 15:35:18 | noonedeadpunk | like hard to find so far, as plenty of related work to multifd was done last year | |
| 15:35:44 | sean-k-mooney | Multifd introduced in QEMU 2.11 in late 2017 by Juan Quintela | |
| 15:36:02 | noonedeadpunk | yeah, ok, so then min versions are passing for sure | |
| 15:36:10 | sean-k-mooney | https://kvm-forum.qemu.org/2024/kvm-forum-2024-multifd-device-state-transfer_3K5EQIG.pdf#page=7 | |
| 15:36:40 | noonedeadpunk | yeah, excatly what I see is 10gbit | |
| 15:36:50 | noonedeadpunk | which is annoying on 100gbit cards... | |
| 15:37:18 | sean-k-mooney | and when you have gpus with large amount of vram that they are afctivly updating | |
| 15:37:20 | noonedeadpunk | I was thinking also about rdma option, but we don't have hugepages everywhere... | |
| 15:38:03 | noonedeadpunk | (or well, we have only on *very* selected hosts as added them after you helped last time) | |
| 15:38:17 | sean-k-mooney | enabling multiple connections is a generically useful feature so i woudl keep that proposal simple | |
| 15:38:24 | noonedeadpunk | ok, cool, so I'll do 1 and 2 for now then | |
| 15:38:59 | sean-k-mooney | i woudl proably do somthing like a min comptue service version check to know if this si supproted | |
| 15:39:17 | sean-k-mooney | which avoid the need to schdler on it but if you file a spec we can disucss that there | |
| 15:39:45 | sean-k-mooney | for 3 your production case. | |
| 15:40:01 | sean-k-mooney | i assume you need tls and cant take the perfromance overhead fo using ssh instead | |
| 15:40:32 | noonedeadpunk | I'm not sure it's working either... | |
| 15:40:46 | noonedeadpunk | so far I'm thinking about doing a local backport of proposal | |
| 15:41:07 | sean-k-mooney | ack | |
| 15:41:32 | sean-k-mooney | you have been around openstack long enough to know the pros and cons of that | |
| 15:42:02 | noonedeadpunk | I can't think of better option either :( | |
| 15:42:37 | noonedeadpunk | and then hope it would be merged for 2026.1... | |
| 15:43:12 | sean-k-mooney | if using ssh worked instead of qemu+tls it would act as a mitigation although the better option woudl be for the qemu/libvirt bug to be fixed | |
| 15:44:00 | noonedeadpunk | I kinda still don't get how qemu+tls without --tls works tbh | |
| 15:45:29 | sean-k-mooney | the protocol is ment to imply --tls i think although there are 2 qemu connection to consier. you have teh conenction to your local libvirt and the connection to the remote one | |
| 15:45:43 | noonedeadpunk | yes, right, qemu/libvirt fix would be really nice, though I'd guess the lead time might be even higher then waiting for spec to merge | |
| 15:45:43 | sean-k-mooney | it hink --tls is specifying that tls shoudl be used for the remote one | |
| 15:46:37 | noonedeadpunk | sean-k-mooney: well. it seems so, but qemu+tls://compute10/system - compute10 is a remote host... | |
| 15:46:55 | sean-k-mooney | also to be fair i never actully use virsh to do migration manullly so im not famiarly with the specific here | |
| 15:47:11 | noonedeadpunk | and also I have listen_tls = 1 and listen_tcp = 0 for /etc/libvirt/libvirtd.conf | |
| 15:47:14 | noonedeadpunk | so wtrf... | |
| 15:48:09 | noonedeadpunk | probably I should try going to their IRC with this though | |
| 15:50:47 | sean-k-mooney | https://libvirt.org/migration.html implie that the uri si enough but it does nto say one way or another | |
| 15:51:25 | sean-k-mooney | there is a way to force the use fo tls via migrate_tls_force in /etc/libvirt/qemu.conf but that shoudl nto be required espically with nova | |
| 15:53:00 | sean-k-mooney | oh right you use -c to specify the locall connection to virsh | |
| 15:54:08 | noonedeadpunk | sean-k-mooney: https://gitlab.com/qemu-project/qemu/-/issues/1937 | |
| 15:58:42 | sean-k-mooney | oh you found a 2023 issue in qemu | |
| 16:01:13 | noonedeadpunk | yeah | |
| 16:01:20 | sean-k-mooney | looks like dan bareeange is activlly engaged | |
| 16:01:35 | sean-k-mooney | i can mentio nto him that this affect nova as well today | |
| 16:01:39 | noonedeadpunk | and they in #virt suggesting using parallel as general rule | |
| 16:01:51 | sean-k-mooney | they also recommended multi-fd migration | |
| 16:02:18 | sean-k-mooney | so thats another reason to consider your proposal and add supprot in nova | |
| 16:04:18 | noonedeadpunk | yeah, I will work on it tomorrow then :) | |