| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-07-16 | |||
| 15:25:53 | noonedeadpunk | sean-k-mooney: I do | |
| 15:26:19 | noonedeadpunk | also thinking to report a bug to qemu, and gather some debug logs now | |
| 15:26:34 | sean-k-mooney | so i dont actlly have a problem with a config option that set hte number of parralel connection and use that to enable this | |
| 15:27:10 | sean-k-mooney | but beyond that we woudl need to make sure the version of libvirt supprot that | |
| 15:27:19 | sean-k-mooney | and qemu | |
| 15:27:24 | sean-k-mooney | both on the source and dest host | |
| 15:27:39 | sean-k-mooney | and we may or may not want ot schdluer on it. | |
| 15:27:48 | noonedeadpunk | any gotchas on how to trace that back? | |
| 15:28:31 | sean-k-mooney | you have replicated this with that virsh migrate command yes | |
| 15:28:54 | sean-k-mooney | if you have ideally without openstack at all i think we shoudl file a libvirt but for that | |
| 15:29:08 | noonedeadpunk | yeah, it's without openstack at all | |
| 15:29:10 | noonedeadpunk | or well | |
| 15:29:28 | dansmith | I think a spec to explain how this works and what the considerations are is important.. like, does this have a dependency on the remote version? do we want to schedule for the feature? if not, why not? | |
| 15:29:37 | noonedeadpunk | I copied big chunk of domain xml, minus nova metadata | |
| 15:30:05 | sean-k-mooney | ok but you actully created the vm wiht libvirt driectly right | |
| 15:30:11 | noonedeadpunk | yep | |
| 15:30:19 | noonedeadpunk | just 2 independent hosts | |
| 15:30:20 | sean-k-mooney | if you can do that its eaiser for the libvirt maintainer ot recreate this | |
| 15:30:26 | noonedeadpunk | right | |
| 15:31:25 | sean-k-mooney | so i think there are 3 things that can/should happne 1 file a libvirt bug, 2 we can dicuss a spec for the parallel connection supprot, 3 we can discss a workaroudn for your production cloud | |
| 15:31:56 | noonedeadpunk | so parallel migration was added in libvirt 5.2: https://github.com/libvirt/libvirt/commit/d3ea986af24fdb320a54854b6d6668a51ecb0cd0 | |
| 15:32:20 | sean-k-mooney | thats a good data point as its below our min version, what about qemu? | |
| 15:32:40 | noonedeadpunk | I'm not that fast, sorry | |
| 15:32:46 | sean-k-mooney | its fine | |
| 15:33:47 | sean-k-mooney | gemini things it was in qemu 2.1 but im not sure i turst it | |
| 15:34:00 | sean-k-mooney | https://wiki.qemu.org/Features/Migration-Multiple-fds | |
| 15:35:16 | sean-k-mooney | by the way i know there ws also interest in enabling this for vgpu live migration | |
| 15:35:18 | noonedeadpunk | like hard to find so far, as plenty of related work to multifd was done last year | |
| 15:35:44 | sean-k-mooney | Multifd introduced in QEMU 2.11 in late 2017 by Juan Quintela | |
| 15:36:02 | noonedeadpunk | yeah, ok, so then min versions are passing for sure | |
| 15:36:10 | sean-k-mooney | https://kvm-forum.qemu.org/2024/kvm-forum-2024-multifd-device-state-transfer_3K5EQIG.pdf#page=7 | |
| 15:36:40 | noonedeadpunk | yeah, excatly what I see is 10gbit | |
| 15:36:50 | noonedeadpunk | which is annoying on 100gbit cards... | |
| 15:37:18 | sean-k-mooney | and when you have gpus with large amount of vram that they are afctivly updating | |
| 15:37:20 | noonedeadpunk | I was thinking also about rdma option, but we don't have hugepages everywhere... | |
| 15:38:03 | noonedeadpunk | (or well, we have only on *very* selected hosts as added them after you helped last time) | |
| 15:38:17 | sean-k-mooney | enabling multiple connections is a generically useful feature so i woudl keep that proposal simple | |
| 15:38:24 | noonedeadpunk | ok, cool, so I'll do 1 and 2 for now then | |
| 15:38:59 | sean-k-mooney | i woudl proably do somthing like a min comptue service version check to know if this si supproted | |
| 15:39:17 | sean-k-mooney | which avoid the need to schdler on it but if you file a spec we can disucss that there | |
| 15:39:45 | sean-k-mooney | for 3 your production case. | |
| 15:40:01 | sean-k-mooney | i assume you need tls and cant take the perfromance overhead fo using ssh instead | |
| 15:40:32 | noonedeadpunk | I'm not sure it's working either... | |
| 15:40:46 | noonedeadpunk | so far I'm thinking about doing a local backport of proposal | |
| 15:41:07 | sean-k-mooney | ack | |
| 15:41:32 | sean-k-mooney | you have been around openstack long enough to know the pros and cons of that | |
| 15:42:02 | noonedeadpunk | I can't think of better option either :( | |
| 15:42:37 | noonedeadpunk | and then hope it would be merged for 2026.1... | |
| 15:43:12 | sean-k-mooney | if using ssh worked instead of qemu+tls it would act as a mitigation although the better option woudl be for the qemu/libvirt bug to be fixed | |
| 15:44:00 | noonedeadpunk | I kinda still don't get how qemu+tls without --tls works tbh | |
| 15:45:29 | sean-k-mooney | the protocol is ment to imply --tls i think although there are 2 qemu connection to consier. you have teh conenction to your local libvirt and the connection to the remote one | |
| 15:45:43 | noonedeadpunk | yes, right, qemu/libvirt fix would be really nice, though I'd guess the lead time might be even higher then waiting for spec to merge | |
| 15:45:43 | sean-k-mooney | it hink --tls is specifying that tls shoudl be used for the remote one | |
| 15:46:37 | noonedeadpunk | sean-k-mooney: well. it seems so, but qemu+tls://compute10/system - compute10 is a remote host... | |
| 15:46:55 | sean-k-mooney | also to be fair i never actully use virsh to do migration manullly so im not famiarly with the specific here | |
| 15:47:11 | noonedeadpunk | and also I have listen_tls = 1 and listen_tcp = 0 for /etc/libvirt/libvirtd.conf | |
| 15:47:14 | noonedeadpunk | so wtrf... | |
| 15:48:09 | noonedeadpunk | probably I should try going to their IRC with this though | |
| 15:50:47 | sean-k-mooney | https://libvirt.org/migration.html implie that the uri si enough but it does nto say one way or another | |
| 15:51:25 | sean-k-mooney | there is a way to force the use fo tls via migrate_tls_force in /etc/libvirt/qemu.conf but that shoudl nto be required espically with nova | |
| 15:53:00 | sean-k-mooney | oh right you use -c to specify the locall connection to virsh | |
| 15:54:08 | noonedeadpunk | sean-k-mooney: https://gitlab.com/qemu-project/qemu/-/issues/1937 | |
| 15:58:42 | sean-k-mooney | oh you found a 2023 issue in qemu | |
| 16:01:13 | noonedeadpunk | yeah | |
| 16:01:20 | sean-k-mooney | looks like dan bareeange is activlly engaged | |
| 16:01:35 | sean-k-mooney | i can mentio nto him that this affect nova as well today | |
| 16:01:39 | noonedeadpunk | and they in #virt suggesting using parallel as general rule | |
| 16:01:51 | sean-k-mooney | they also recommended multi-fd migration | |
| 16:02:18 | sean-k-mooney | so thats another reason to consider your proposal and add supprot in nova | |
| 16:04:18 | noonedeadpunk | yeah, I will work on it tomorrow then :) | |
| 16:18:43 | sean-k-mooney | noonedeadpunk: so there is a possibel workaround via cofnig files assuming you do not need fips | |
| 16:19:33 | sean-k-mooney | noonedeadpunk: i was talking to danpb downstream about this and https://issues.redhat.com/browse/RHEL-103240?focusedId=27595864&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-27595864 is a workaround | |
| 16:20:04 | sean-k-mooney | noonedeadpunk: you can disabel AES as one fo the allowed encyption methods | |
| 16:20:26 | noonedeadpunk | ah, he mentioned that in IRC actually | |
| 16:21:02 | noonedeadpunk | but thanks for pointing to that report | |
| 16:21:15 | sean-k-mooney | that breaks fips compliance but it might eb a better option then a downstream code change if its somethign that was accpable form a secuirty perspective until the issue is fixed properly | |
| 16:21:21 | noonedeadpunk | I think I will try it, as all other options are kinda not perfect | |
| 16:21:50 | noonedeadpunk | I don't want tunneled migrations as we are have local storages here and there | |
| 16:22:09 | noonedeadpunk | and parallel and compress both not solving issue but making it more distant | |
| 16:22:25 | noonedeadpunk | so it's a question of how big and loaded VM should be | |
| 16:23:38 | sean-k-mooney | well as someone who prefers the simple life of migratin unlooed cirros iamges in my devstack small an light :) but customer want to run real workloads, how unreasonabel :) | |
| 16:23:42 | noonedeadpunk | It's also a question if there could be more side-effects from disabling aes | |
| 16:24:22 | sean-k-mooney | so there are some, if you dont use aes and use one of the other algroatims you may not be able to leverage aes-ni or similer exteions to offload the encyprtion to hardware | |
| 16:24:49 | noonedeadpunk | yeah, and we start talking if customer moved from VMWare with their 1Tb MSSQL with 200gb of ram... | |
| 16:24:52 | sean-k-mooney | the side effect of that would be a bottelneck on througput | |
| 16:25:21 | noonedeadpunk | yeah, but there's also like OVN on host likely using AES as well | |
| 16:25:32 | sean-k-mooney | i remember jay pipes talking about one of the customers mirantas had wehre they litrally ran 1 VM per compute node that used all the cpus/ram | |
| 16:25:38 | noonedeadpunk | or well, ovs | |
| 16:26:22 | sean-k-mooney | the litrally just wanted to have an api to create/delete vms effectivly and provide some levle of security via a vm absrction | |
| 16:26:29 | noonedeadpunk | I wonder how scheduling work for them if they need to evacuate the VM on hypervisor failure | |
| 16:26:47 | sean-k-mooney | hopefully they had hot spares | |
| 16:27:04 | sean-k-mooney | as in free hosts | |
| 16:27:18 | sean-k-mooney | so i beliv ethis dropin file is specific to qemu | |
| 16:27:34 | sean-k-mooney | i.e. its createing /etc/crypto-policies/local.d/gnutls-qemu.config | |
| 16:27:48 | sean-k-mooney | but it woudl apply to all tls coonection use by qemu fro anything it doing | |
| 16:28:07 | sean-k-mooney | but again im not an experit on this so that may be incorrect | |
| 16:28:14 | noonedeadpunk | well, this would be fine then | |
| 16:28:42 | noonedeadpunk | I was a bit more, if crypto-policies just do fileglob of /etc/crypto-policies/local.d/*.config | |
| 16:28:50 | noonedeadpunk | but will check on that | |
| 16:29:04 | sean-k-mooney | i think that is what the QEMU= at the start of it is doing not the file name | |