| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-15 | |||
| 08:28:01 | tafkamax | thats a good docs | |
| 09:49:14 | stephenfin | sean-k-mooney: Great, thanks | |
| 09:51:25 | sean-k-mooney | so in the job that failing most often i think i can actully trun off the neutron-periocs but im goign t chat about that wiht the neturon folk after it have coffee | |
| 10:30:34 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 10:47:43 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix PEP-765 syntax warning https://review.opendev.org/c/openstack/nova/+/988636 | |
| 10:48:57 | tafkamax | I have a question regarding the live migrations. they take forever to sync the last part of ram. And looking at the logs I don't know if I am looking at the correct thing. | |
| 10:49:09 | tafkamax | nova.virt.libvirt.migration [req-f5188e51-02b9-4338-b3e7-53e162a3f3c2 req-f369ecbf-a79d-4ddf-b065-eea729d54da1 2337b7a2b5e34752a86ab5d61c525327 0530bca34f3f4d2c92a3495d56a1e065 - - default default] [instance: d5cf4b01-d97d-4395-8dd2-c4930b57e0c9] Increasing downtime to 50 ms after 0 sec elapsed time during the start of the migration | |
| 10:49:44 | tafkamax | what downtime is it? because https://docs.openstack.org/nova/2025.1/configuration/config.html#libvirt.live_migration_downtime <- this downtime starts at 500ms | |
| 10:50:46 | tafkamax | And it is constantly hovering at 1% and 0% memory remaining at takes like ~20minutes to finish | |
| 10:55:07 | tafkamax | Aha it completed faster after changing live_migration_downtime_delay from default 75 to 25 | |
| 10:55:31 | stephenfin | tafkamax: That happens if you have a a busy VM. If there's lot of writes to memory in the VM, the writes can happen faster than libvirt can sync things on the destination | |
| 10:56:06 | stephenfin | You want to look at live_migration_downtime_delay, live_migration_permit_post_copy and live_migration_permit_auto_converge (this one particularly) | |
| 10:56:30 | tafkamax | I have already setup the auto_converge and post_copy | |
| 10:56:52 | tafkamax | ^ the logs, the first downtime ~11:43 to 12:03 had the default 75seconds | |
| 10:57:15 | tafkamax | changing to 25seconds in ~13:42 made the downtime change faster and then it completed faster | |
| 10:58:07 | tafkamax | whatabout live_migration_downtime and downtime_steps though? | |
| 10:58:18 | tafkamax | or 500 is the max and it starts from 50ms? | |
| 11:06:15 | tafkamax | I will now move steps from 10 to 6 | |
| 11:18:43 | tafkamax | ok seems I got the hang of it now how it caluculates it | |
| 11:53:54 | gibi | fyi cores, IOThreads in Gazpacho breaks the live migration of pre-existing VMs with pinned CPUs https://bugs.launchpad.net/nova/+bug/2152697 | |
| 11:54:16 | gibi | it is due to an wrong assumption that iothreadpin always exists in the XML | |
| 11:54:19 | gibi | https://github.com/openstack/nova/commit/53a613d9948826ec9a4cd4a502f7a5d1b2dc87d7#diff-1f1f3f935853a67b1239220cd9f8c28278734732c5c28e8d238f62860391b1ecR273 | |
| 12:49:53 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 13:15:33 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 13:23:47 | sean-k-mooney | gibi: didnt we fix that | |
| 13:23:52 | sean-k-mooney | i guess we never backported it? | |
| 13:26:54 | gibi | sean-k-mooney: we fixed the pinned cpu case and fixed the live migration case but not the live migration of a pre-existing VM wwith pinned cpu case :D | |
| 13:27:35 | gibi | the live migration fix introduced the unconditional check for the iothreadpin xml tag | |
| 13:27:56 | gibi | which only exists with VMs that are created (or rebooted) since the IOThread feature | |
| 13:37:08 | sean-k-mooney | im surpised our multinode grenade job didnt pick that up | |
| 13:37:16 | sean-k-mooney | but ok that shoudl not be hard to fixx | |
| 13:37:29 | sean-k-mooney | we just need ot gate that code by the xml having it | |
| 13:43:03 | gibi | I guess our grenade does not have pinned VMs | |
| 13:43:19 | sean-k-mooney | oh is this only for pinned vms | |
| 13:43:29 | sean-k-mooney | oh no | |
| 13:43:35 | sean-k-mooney | its not pined exaclytu | |
| 13:43:46 | sean-k-mooney | its that we are not using cpu_share_set or cpu_dedicated_set | |
| 13:43:59 | sean-k-mooney | so since we are not using either of them we never generate teh element | |
| 13:44:06 | gibi | nope | |
| 13:44:13 | sean-k-mooney | cpu_shared_set woudl be enough to catch this i think | |
| 13:44:29 | sean-k-mooney | gibi: are you plannign to work on a patch to make it conditional? | |
| 13:44:44 | gibi | I'm working on a functional reproducer | |
| 13:45:07 | sean-k-mooney | ack cool im happy to review both so add me or ping me when its up | |
| 13:45:22 | gibi | during live migration we update the src XML, for a pre-existing VM, the src XML does not have any iothreads or iothreadpin field | |
| 13:45:52 | sean-k-mooney | yep we have had this tyep of issue before | |
| 13:46:09 | sean-k-mooney | i think we have exsing functional tests that simulate somethign similar | |
| 13:51:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769 | |
| 13:52:01 | gibi | sean-k-mooney: this is the reproducer ^^ | |
| 13:53:48 | sean-k-mooney | am `del list(conn._vms.values())[0]._def["iothreads"]` | |
| 13:53:58 | sean-k-mooney | ok i see why that works | |
| 13:54:43 | sean-k-mooney | the list is convertign the the iterator int a lis that containes refence to the actul objects in the _vms dict | |
| 13:54:55 | sean-k-mooney | so we are deleting trhough the list | |
| 13:55:14 | sean-k-mooney | coudl we do this a littel iffenrtly | |
| 13:55:44 | sean-k-mooney | vm_domain = next(conn._vms.values()) | |
| 13:55:54 | gibi | we can use next yes | |
| 13:56:06 | sean-k-mooney | del vm_domain._def["..."] | |
| 13:56:38 | sean-k-mooney | i can see why what you have works on a second or third reading but its a littel non obvious | |
| 13:57:09 | sean-k-mooney | gibi: over all the repoducer is more or less doign what i expect | |
| 13:57:27 | sean-k-mooney | as noted however this shoudl also manifest if you have jsut set cpu_shared_set and are not using dedicated | |
| 13:57:47 | sean-k-mooney | so it migh be nice to add a second test for floating vms | |
| 13:58:30 | sean-k-mooney | otherwise this looks like a reasonabel repoducer. it was more or less what i was expecting you to write | |
| 13:59:45 | gibi | I tried an instance with shared CPUs before and it did not hit the same issue | |
| 13:59:53 | gibi | the problematic code is under a condition | |
| 13:59:54 | gibi | https://github.com/openstack/nova/commit/53a613d9948826ec9a4cd4a502f7a5d1b2dc87d7#diff-1f1f3f935853a67b1239220cd9f8c28278734732c5c28e8d238f62860391b1ecR260-R262 | |
| 13:59:57 | gibi | about pinning | |
| 14:00:10 | sean-k-mooney | we do pin shared cpus | |
| 14:00:21 | sean-k-mooney | if an only if cpu_share_set is defiend | |
| 14:01:40 | sean-k-mooney | oh | |
| 14:01:56 | sean-k-mooney | actully in that case we only set vcpu cpuset=... | |
| 14:02:07 | sean-k-mooney | so we pin it but we dont generate the element | |
| 14:02:22 | sean-k-mooney | ok then ya i see this would only impacte pinned vms that were created before iothread | |
| 14:05:10 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774 | |
| 14:08:21 | sean-k-mooney | gibi: +2 on the repoducer +1 on the fix. ill see if i can try and trest this on monday and loop back to the review after the ci has ran | |
| 14:08:40 | sean-k-mooney | gibi: we might want to add a release note but that my only feedback really right now | |
| 14:08:52 | sean-k-mooney | maybe also a unit test for the branchign bevhior in the migration module | |
| 14:09:20 | sean-k-mooney | just to test _update_numa_xml | |
| 14:09:27 | gibi | yeah I can do that | |
| 14:17:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769 | |
| 14:17:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774 | |
| 14:20:43 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix live migration with pinned VM and iothreads https://review.opendev.org/c/openstack/nova/+/988774 | |
| 14:20:49 | gibi | now with reno and unit test :) | |
| 14:21:15 | gibi | ^^ cc dansmith as we chatted about it downstream | |
| 14:24:30 | sean-k-mooney | gibi: so one nuance about your fix | |
| 14:24:49 | sean-k-mooney | it will work but you intentally choose not to add an iothread to the running instnace | |
| 14:24:58 | sean-k-mooney | on live migrate | |
| 14:25:30 | sean-k-mooney | we could also do that perhasp in a followup patch | |
| 14:26:34 | sean-k-mooney | im personally fine with not doing that and just saying you need to reboot to get the iothread jsut notign that qemu/libvirt techniall supprot adding and removign them on a runing instance | |
| 14:27:36 | gibi | I think it is safer not to try to add it during live migration | |
| 14:27:47 | sean-k-mooney | ya that was my feelign too | |
| 14:27:52 | sean-k-mooney | just wanted to raise the possiblity | |
| 14:28:01 | gibi | so if somebody explicity want this then that is a new small feature to me | |
| 14:29:56 | sean-k-mooney | cool well still +1 lets see what ci says and ill loop back | |
| 14:31:11 | sean-k-mooney | gibi: nice find by the way idd you just come across it or was it somethign you noticed in a ci failreu or something | |
| 14:31:26 | sean-k-mooney | oh you were backporting this downstream right | |
| 14:31:38 | gibi | yepp I tested my backport downstream | |
| 14:31:53 | gibi | there it is simpler to thest the upgrade | |
| 14:32:16 | gibi | by just changing the nova-compute container image | |
| 14:32:42 | gibi | as we had bugs before with live migration and pinned CPUs, I tested the combination of those :) | |
| 14:32:56 | sean-k-mooney | ack i need to revive and rewrite my ci change | |
| 14:33:01 | opendevreview | Masanori Ueno proposed openstack/nova master: DNM: Add functional test for NUMA live migration overcommit bug https://review.opendev.org/c/openstack/nova/+/988777 | |