| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-16 | |||
| 10:42:07 | sean-k-mooney | well i guess i can propose a revert and depend on and test that as well | |
| 10:57:52 | opendevreview | sean mooney proposed openstack/nova master: move ceph jobs to debian to avoid bug 2116852 https://review.opendev.org/c/openstack/nova/+/955179 | |
| 10:58:23 | sean-k-mooney | so something like ^ | |
| 10:59:27 | sean-k-mooney | we can see if that passes and decied if we want to keep it disabled or just do that. we could also consdier doign the debian swap in the parent devstack jobs so it fixes it for everyone | |
| 10:59:51 | sean-k-mooney | so devstack-plugin-ceph-tempest-py3 and devstack-plugin-ceph-multinode-tempest-py3 | |
| 11:00:47 | sean-k-mooney | which are defiend in https://github.com/openstack/devstack-plugin-ceph/blob/master/.zuul.yaml | |
| 11:29:05 | gibi | I've added the Ceph project on https://bugs.launchpad.net/nova/+bug/2116852 to let ubuntu see the issue | |
| 12:09:14 | opendevreview | Merged openstack/nova master: Move ComputeManager to use spawn_on https://review.opendev.org/c/openstack/nova/+/948186 | |
| 13:45:03 | noonedeadpunk | hey folks! am I right that there is no way so far to supply any arbitrary arguments to as migration_flags ? | |
| 13:48:37 | noonedeadpunk | specifically thinking about VIR_MIGRATE_PARALLEL and VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS now | |
| 14:08:02 | sean-k-mooney | noonedeadpunk: be design no not really. that an internal interface of nova's libvirt driver | |
| 14:08:26 | sean-k-mooney | we add flag when we supprot new features in our config. | |
| 14:08:50 | sean-k-mooney | unofficlly you very ocationlly can enable some thing via the migration uri via query args | |
| 14:10:30 | sean-k-mooney | we woudl need to actully add supprot for that in nova properly if we were to supprot it | |
| 14:15:59 | noonedeadpunk | yeah, I was planning to look into adding this thing to nova, but also wondering about how to workaround current failures of migration | |
| 14:16:17 | noonedeadpunk | As it seems to ways so far is either to disable tls, or increase throughput | |
| 14:24:01 | noonedeadpunk | do you think I'd be needing to submit a full-fledged spec for that? | |
| 14:26:56 | dansmith | for which? always-on migration flags, config knob to enable one, or something like generic flag merging support? | |
| 14:27:23 | dansmith | in either of those cases we probably want to document the need, risk, config behavior, etc | |
| 14:30:54 | gibi | eventlet sync https://meet.google.com/bcy-uqoz-hje | |
| 14:32:40 | noonedeadpunk | dansmith: config knob I believe. to enable/disable parallel and amount of forks | |
| 14:32:56 | noonedeadpunk | ok, I guess I'll come up with smth relatively simple | |
| 14:46:09 | dansmith | noonedeadpunk: I think a spec for that yeah | |
| 15:19:12 | noonedeadpunk | sean-k-mooney: do you recall the issue I was coming with for live migration failures a month ago or so? | |
| 15:19:46 | noonedeadpunk | as I was able to reproduce it on plain libvirt+qemu on ubuntu | |
| 15:20:02 | noonedeadpunk | and pretty much it's kinda a single usecase which is failing.... | |
| 15:20:27 | noonedeadpunk | virsh migrate --live --auto-converge --persistent --copy-storage-all --tls instance-00016a41 qemu+tls://compute10/system | |
| 15:20:56 | noonedeadpunk | if I: drop tls, add parallel or use compress - migration passes. | |
| 15:21:37 | noonedeadpunk | but this also is really confusing... how in the world uwing qemu+tls without --tls does not actually use tls... | |
| 15:22:07 | sean-k-mooney | noonedeadpunk: reading back and yes vagly | |
| 15:23:24 | sean-k-mooney | spec vs specless blueprint in this context partly comes down to the upgrade impact, the other factor being what version fo libvirt/qemu supprot it ectra | |
| 15:23:28 | noonedeadpunk | but anyway, wanted to confirm it has nothing to do with nova. But I'm very surprised nobody else was reporting or coming with that | |
| 15:23:39 | sean-k-mooney | we do allow new config option that default to the old bevhioar without a spec | |
| 15:23:50 | noonedeadpunk | intention was to keep the behavior for sure | |
| 15:24:00 | sean-k-mooney | btu if it impact upgrades in a mixed comptue node version case it almost alwasy needs a spec | |
| 15:25:00 | sean-k-mooney | noonedeadpunk: but if i recall correctly you were seeing qemu crashes right | |
| 15:25:46 | noonedeadpunk | I was thinking to add an option for instance live_migration_parallel_connections with default of 1, and when it's >1 add https://libvirt.org/html/libvirt-libvirt-domain.html#VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS and libvirt.VIR_MIGRATE_PARALLEL | |
| 15:25:53 | noonedeadpunk | sean-k-mooney: I do | |
| 15:26:19 | noonedeadpunk | also thinking to report a bug to qemu, and gather some debug logs now | |
| 15:26:34 | sean-k-mooney | so i dont actlly have a problem with a config option that set hte number of parralel connection and use that to enable this | |
| 15:27:10 | sean-k-mooney | but beyond that we woudl need to make sure the version of libvirt supprot that | |
| 15:27:19 | sean-k-mooney | and qemu | |
| 15:27:24 | sean-k-mooney | both on the source and dest host | |
| 15:27:39 | sean-k-mooney | and we may or may not want ot schdluer on it. | |
| 15:27:48 | noonedeadpunk | any gotchas on how to trace that back? | |
| 15:28:31 | sean-k-mooney | you have replicated this with that virsh migrate command yes | |
| 15:28:54 | sean-k-mooney | if you have ideally without openstack at all i think we shoudl file a libvirt but for that | |
| 15:29:08 | noonedeadpunk | yeah, it's without openstack at all | |
| 15:29:10 | noonedeadpunk | or well | |
| 15:29:28 | dansmith | I think a spec to explain how this works and what the considerations are is important.. like, does this have a dependency on the remote version? do we want to schedule for the feature? if not, why not? | |
| 15:29:37 | noonedeadpunk | I copied big chunk of domain xml, minus nova metadata | |
| 15:30:05 | sean-k-mooney | ok but you actully created the vm wiht libvirt driectly right | |
| 15:30:11 | noonedeadpunk | yep | |
| 15:30:19 | noonedeadpunk | just 2 independent hosts | |
| 15:30:20 | sean-k-mooney | if you can do that its eaiser for the libvirt maintainer ot recreate this | |
| 15:30:26 | noonedeadpunk | right | |
| 15:31:25 | sean-k-mooney | so i think there are 3 things that can/should happne 1 file a libvirt bug, 2 we can dicuss a spec for the parallel connection supprot, 3 we can discss a workaroudn for your production cloud | |
| 15:31:56 | noonedeadpunk | so parallel migration was added in libvirt 5.2: https://github.com/libvirt/libvirt/commit/d3ea986af24fdb320a54854b6d6668a51ecb0cd0 | |
| 15:32:20 | sean-k-mooney | thats a good data point as its below our min version, what about qemu? | |
| 15:32:40 | noonedeadpunk | I'm not that fast, sorry | |
| 15:32:46 | sean-k-mooney | its fine | |
| 15:33:47 | sean-k-mooney | gemini things it was in qemu 2.1 but im not sure i turst it | |
| 15:34:00 | sean-k-mooney | https://wiki.qemu.org/Features/Migration-Multiple-fds | |
| 15:35:16 | sean-k-mooney | by the way i know there ws also interest in enabling this for vgpu live migration | |
| 15:35:18 | noonedeadpunk | like hard to find so far, as plenty of related work to multifd was done last year | |
| 15:35:44 | sean-k-mooney | Multifd introduced in QEMU 2.11 in late 2017 by Juan Quintela | |
| 15:36:02 | noonedeadpunk | yeah, ok, so then min versions are passing for sure | |
| 15:36:10 | sean-k-mooney | https://kvm-forum.qemu.org/2024/kvm-forum-2024-multifd-device-state-transfer_3K5EQIG.pdf#page=7 | |
| 15:36:40 | noonedeadpunk | yeah, excatly what I see is 10gbit | |
| 15:36:50 | noonedeadpunk | which is annoying on 100gbit cards... | |
| 15:37:18 | sean-k-mooney | and when you have gpus with large amount of vram that they are afctivly updating | |
| 15:37:20 | noonedeadpunk | I was thinking also about rdma option, but we don't have hugepages everywhere... | |
| 15:38:03 | noonedeadpunk | (or well, we have only on *very* selected hosts as added them after you helped last time) | |
| 15:38:17 | sean-k-mooney | enabling multiple connections is a generically useful feature so i woudl keep that proposal simple | |
| 15:38:24 | noonedeadpunk | ok, cool, so I'll do 1 and 2 for now then | |
| 15:38:59 | sean-k-mooney | i woudl proably do somthing like a min comptue service version check to know if this si supproted | |
| 15:39:17 | sean-k-mooney | which avoid the need to schdler on it but if you file a spec we can disucss that there | |
| 15:39:45 | sean-k-mooney | for 3 your production case. | |
| 15:40:01 | sean-k-mooney | i assume you need tls and cant take the perfromance overhead fo using ssh instead | |
| 15:40:32 | noonedeadpunk | I'm not sure it's working either... | |
| 15:40:46 | noonedeadpunk | so far I'm thinking about doing a local backport of proposal | |
| 15:41:07 | sean-k-mooney | ack | |
| 15:41:32 | sean-k-mooney | you have been around openstack long enough to know the pros and cons of that | |
| 15:42:02 | noonedeadpunk | I can't think of better option either :( | |
| 15:42:37 | noonedeadpunk | and then hope it would be merged for 2026.1... | |
| 15:43:12 | sean-k-mooney | if using ssh worked instead of qemu+tls it would act as a mitigation although the better option woudl be for the qemu/libvirt bug to be fixed | |
| 15:44:00 | noonedeadpunk | I kinda still don't get how qemu+tls without --tls works tbh | |
| 15:45:29 | sean-k-mooney | the protocol is ment to imply --tls i think although there are 2 qemu connection to consier. you have teh conenction to your local libvirt and the connection to the remote one | |
| 15:45:43 | noonedeadpunk | yes, right, qemu/libvirt fix would be really nice, though I'd guess the lead time might be even higher then waiting for spec to merge | |
| 15:45:43 | sean-k-mooney | it hink --tls is specifying that tls shoudl be used for the remote one | |
| 15:46:37 | noonedeadpunk | sean-k-mooney: well. it seems so, but qemu+tls://compute10/system - compute10 is a remote host... | |
| 15:46:55 | sean-k-mooney | also to be fair i never actully use virsh to do migration manullly so im not famiarly with the specific here | |
| 15:47:11 | noonedeadpunk | and also I have listen_tls = 1 and listen_tcp = 0 for /etc/libvirt/libvirtd.conf | |
| 15:47:14 | noonedeadpunk | so wtrf... | |
| 15:48:09 | noonedeadpunk | probably I should try going to their IRC with this though | |
| 15:50:47 | sean-k-mooney | https://libvirt.org/migration.html implie that the uri si enough but it does nto say one way or another | |
| 15:51:25 | sean-k-mooney | there is a way to force the use fo tls via migrate_tls_force in /etc/libvirt/qemu.conf but that shoudl nto be required espically with nova | |
| 15:53:00 | sean-k-mooney | oh right you use -c to specify the locall connection to virsh | |
| 15:54:08 | noonedeadpunk | sean-k-mooney: https://gitlab.com/qemu-project/qemu/-/issues/1937 | |
| 15:58:42 | sean-k-mooney | oh you found a 2023 issue in qemu | |