| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-14 | |||
| 16:27:11 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Filter out (more) eventlet deprecation warnings https://review.opendev.org/c/openstack/nova/+/974686 | |
| 16:27:12 | opendevreview | Stephen Finucane proposed openstack/nova master: utils: Pass string to lock https://review.opendev.org/c/openstack/nova/+/978433 | |
| 16:27:12 | opendevreview | Stephen Finucane proposed openstack/nova master: libvirt: Fix logging type error https://review.opendev.org/c/openstack/nova/+/978432 | |
| 16:27:13 | opendevreview | Stephen Finucane proposed openstack/nova master: privsep: Correct arg order https://review.opendev.org/c/openstack/nova/+/978435 | |
| 16:27:13 | opendevreview | Stephen Finucane proposed openstack/nova master: privsep: Pass strings, not ints https://review.opendev.org/c/openstack/nova/+/978434 | |
| 16:27:14 | opendevreview | Stephen Finucane proposed openstack/nova master: Remove more dead code https://review.opendev.org/c/openstack/nova/+/978436 | |
| 16:27:14 | opendevreview | Stephen Finucane proposed openstack/nova master: privsep: Remove dead code https://review.opendev.org/c/openstack/nova/+/977111 | |
| 16:27:15 | opendevreview | Stephen Finucane proposed openstack/nova master: Add ruff-check https://review.opendev.org/c/openstack/nova/+/974441 | |
| 16:27:16 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.conf https://review.opendev.org/c/openstack/nova/+/974688 | |
| 16:27:16 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.cmd https://review.opendev.org/c/openstack/nova/+/705657 | |
| 16:27:18 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to remaining top-level modules https://review.opendev.org/c/openstack/nova/+/705658 | |
| 16:27:18 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.console https://review.opendev.org/c/openstack/nova/+/974689 | |
| 16:27:20 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Correct import issues https://review.opendev.org/c/openstack/nova/+/974221 | |
| 16:27:20 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.virt, nova.virt.libvirt https://review.opendev.org/c/openstack/nova/+/974220 | |
| 16:27:22 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Disallow incomplete defs https://review.opendev.org/c/openstack/nova/+/974690 | |
| 16:27:22 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Enable incremental checks https://review.opendev.org/c/openstack/nova/+/974222 | |
| 16:27:24 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.privsep https://review.opendev.org/c/openstack/nova/+/977112 | |
| 16:27:24 | opendevreview | Stephen Finucane proposed openstack/nova master: mypy: Disallow untyped defs (where possible) https://review.opendev.org/c/openstack/nova/+/974725 | |
| 16:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: Add beartype for runtime type validation https://review.opendev.org/c/openstack/nova/+/988651 | |
| 16:27:26 | opendevreview | Stephen Finucane proposed openstack/nova master: typing: Add hints to nova.policy https://review.opendev.org/c/openstack/nova/+/977153 | |
| 17:03:17 | opendevreview | Stephen Finucane proposed openstack/nova master: WIP: Add beartype for runtime type validation https://review.opendev.org/c/openstack/nova/+/988651 | |
| 19:01:37 | opendevreview | Merged openstack/nova master: Extend the tempest coverage in nova-alt-config https://review.opendev.org/c/openstack/nova/+/988599 | |
| 19:17:08 | opendevreview | Merged openstack/os-vif master: typing: Return Self from PluginBase.load https://review.opendev.org/c/openstack/os-vif/+/986890 | |
| 19:17:11 | opendevreview | Merged openstack/os-vif master: trivial: Convert add_keepalives to function https://review.opendev.org/c/openstack/os-vif/+/986891 | |
| 19:17:39 | opendevreview | Merged openstack/os-vif master: trivial: Specify args to OvsPlugin._create_vif_port https://review.opendev.org/c/openstack/os-vif/+/986892 | |
| 19:17:41 | opendevreview | Merged openstack/os-vif master: typing: Add typing to vif_plug_ovs module https://review.opendev.org/c/openstack/os-vif/+/976363 | |
| 19:48:42 | sean-k-mooney | stephenfin: ya so ill push a patch to stablise the os-vif job | |
| 19:48:44 | sean-k-mooney | https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/#?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(discover:(columns:!(hosts_region,project,build_change,build_name),interval:d,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:kuery,query:'message | |
| 19:48:46 | sean-k-mooney | :%22invoked%20oom-killer%22')) several fo the failure today were OOM events, at least 1 was a timeout and another was cidner | |
| 19:48:59 | sean-k-mooney | oh https://tinyurl.com/m3uv2cf6 | |
| 19:49:04 | sean-k-mooney | is the short url | |
| 19:51:57 | sean-k-mooney | ill remove cinder entirly form those jobs add 15 mins and make sure they have 8G of swap i think they already do but thing like the number of workers used by neutron-perodic and might aslo need to be tweaked | |
| 19:58:22 | sean-k-mooney | neutron-periodic lools liek it defaultign to 1 worker per core | |
| 20:12:51 | sean-k-mooney | https://paste.opendev.org/show/bT5BWqSArjzeicYc010d/ | |
| 20:15:26 | sean-k-mooney | that derived form the oom data | |
| 20:15:42 | sean-k-mooney | i think it might we worth turning this in devstack rather then the job | |
| 20:17:09 | sean-k-mooney | neuton on that deploymetn is using about 2GB of ram | |
| 23:56:50 | opendevreview | Merged openstack/os-vif master: Drop support for Python 3.10 https://review.opendev.org/c/openstack/os-vif/+/987500 | |
| #openstack-nova - 2026-05-15 | |||
| 08:18:23 | tafkamax | Hi I have a question about live-migration. I am getting this error: Operation not permitted: libvirt.libvirtError: internal error: unable to execute QEMU command 'migrate-set-capabilities': Postcopy is not supported: Userfaultfd not available: Operation not permitted | |
| 08:20:39 | tafkamax | I am using kolla-ansible and set the options under [libvirt]... (full message at <https://matrix.org/oftc/media/v1/media/download/ARBKYWZG4vFezBf8AdfbKO35RwPbQknzRNnZhzshrYdIgP66r5d8eL5kRgH6dYMm6kvXhVs1S3zU0FLXLsvUWH1CeectI0ogAG1hdHJpeC5vcmcvT1NZR0RscVlSZk9kZkhkR2ZOc3VHeHF5>) | |
| 08:20:46 | tafkamax | running 2025.1 release | |
| 08:20:54 | tafkamax | are these even supported? | |
| 08:20:59 | tafkamax | or how can I find out then? | |
| 08:21:08 | tafkamax | or do i need to enable a kernel module? | |
| 08:22:38 | tafkamax | * or do i need to enable the sysctl option "vm.unprivileged_userfaultfd=1" | |
| 08:25:26 | Mike-- | I checked nova changelog and version 14.0.0 introduced some of those options and should work I'd say | |
| 08:25:49 | tafkamax | I think I need to fool around with the sysctl options | |
| 08:26:06 | Mike-- | https://github.com/openstack/nova/blob/master/releasenotes/source/newton.rst | |
| 08:27:23 | Mike-- | also found: https://github.com/openstack/nova/blob/master/doc/source/admin/configuring-migrations.rst | |
| 08:27:58 | tafkamax | ok thx! | |
| 08:28:01 | tafkamax | thats a good docs | |
| 09:49:14 | stephenfin | sean-k-mooney: Great, thanks | |
| 09:51:25 | sean-k-mooney | so in the job that failing most often i think i can actully trun off the neutron-periocs but im goign t chat about that wiht the neturon folk after it have coffee | |
| 10:30:34 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 10:47:43 | opendevreview | Pavlo Shchelokovskyy proposed openstack/nova master: Fix PEP-765 syntax warning https://review.opendev.org/c/openstack/nova/+/988636 | |
| 10:48:57 | tafkamax | I have a question regarding the live migrations. they take forever to sync the last part of ram. And looking at the logs I don't know if I am looking at the correct thing. | |
| 10:49:09 | tafkamax | nova.virt.libvirt.migration [req-f5188e51-02b9-4338-b3e7-53e162a3f3c2 req-f369ecbf-a79d-4ddf-b065-eea729d54da1 2337b7a2b5e34752a86ab5d61c525327 0530bca34f3f4d2c92a3495d56a1e065 - - default default] [instance: d5cf4b01-d97d-4395-8dd2-c4930b57e0c9] Increasing downtime to 50 ms after 0 sec elapsed time during the start of the migration | |
| 10:49:44 | tafkamax | what downtime is it? because https://docs.openstack.org/nova/2025.1/configuration/config.html#libvirt.live_migration_downtime <- this downtime starts at 500ms | |
| 10:50:46 | tafkamax | And it is constantly hovering at 1% and 0% memory remaining at takes like ~20minutes to finish | |
| 10:55:07 | tafkamax | Aha it completed faster after changing live_migration_downtime_delay from default 75 to 25 | |
| 10:55:31 | stephenfin | tafkamax: That happens if you have a a busy VM. If there's lot of writes to memory in the VM, the writes can happen faster than libvirt can sync things on the destination | |
| 10:56:06 | stephenfin | You want to look at live_migration_downtime_delay, live_migration_permit_post_copy and live_migration_permit_auto_converge (this one particularly) | |
| 10:56:30 | tafkamax | I have already setup the auto_converge and post_copy | |
| 10:56:52 | tafkamax | ^ the logs, the first downtime ~11:43 to 12:03 had the default 75seconds | |
| 10:57:15 | tafkamax | changing to 25seconds in ~13:42 made the downtime change faster and then it completed faster | |
| 10:58:07 | tafkamax | whatabout live_migration_downtime and downtime_steps though? | |
| 10:58:18 | tafkamax | or 500 is the max and it starts from 50ms? | |
| 11:06:15 | tafkamax | I will now move steps from 10 to 6 | |
| 11:18:43 | tafkamax | ok seems I got the hang of it now how it caluculates it | |
| 11:53:54 | gibi | fyi cores, IOThreads in Gazpacho breaks the live migration of pre-existing VMs with pinned CPUs https://bugs.launchpad.net/nova/+bug/2152697 | |
| 11:54:16 | gibi | it is due to an wrong assumption that iothreadpin always exists in the XML | |
| 11:54:19 | gibi | https://github.com/openstack/nova/commit/53a613d9948826ec9a4cd4a502f7a5d1b2dc87d7#diff-1f1f3f935853a67b1239220cd9f8c28278734732c5c28e8d238f62860391b1ecR273 | |
| 12:49:53 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 13:15:33 | opendevreview | Thibaut Démaret proposed openstack/nova master: libvirt: add disk rotation_rate support for local disks https://review.opendev.org/c/openstack/nova/+/979693 | |
| 13:23:47 | sean-k-mooney | gibi: didnt we fix that | |
| 13:23:52 | sean-k-mooney | i guess we never backported it? | |
| 13:26:54 | gibi | sean-k-mooney: we fixed the pinned cpu case and fixed the live migration case but not the live migration of a pre-existing VM wwith pinned cpu case :D | |
| 13:27:35 | gibi | the live migration fix introduced the unconditional check for the iothreadpin xml tag | |
| 13:27:56 | gibi | which only exists with VMs that are created (or rebooted) since the IOThread feature | |
| 13:37:08 | sean-k-mooney | im surpised our multinode grenade job didnt pick that up | |
| 13:37:16 | sean-k-mooney | but ok that shoudl not be hard to fixx | |
| 13:37:29 | sean-k-mooney | we just need ot gate that code by the xml having it | |
| 13:43:03 | gibi | I guess our grenade does not have pinned VMs | |
| 13:43:19 | sean-k-mooney | oh is this only for pinned vms | |
| 13:43:29 | sean-k-mooney | oh no | |
| 13:43:35 | sean-k-mooney | its not pined exaclytu | |
| 13:43:46 | sean-k-mooney | its that we are not using cpu_share_set or cpu_dedicated_set | |
| 13:43:59 | sean-k-mooney | so since we are not using either of them we never generate teh element | |
| 13:44:06 | gibi | nope | |
| 13:44:13 | sean-k-mooney | cpu_shared_set woudl be enough to catch this i think | |
| 13:44:29 | sean-k-mooney | gibi: are you plannign to work on a patch to make it conditional? | |
| 13:44:44 | gibi | I'm working on a functional reproducer | |
| 13:45:07 | sean-k-mooney | ack cool im happy to review both so add me or ping me when its up | |
| 13:45:22 | gibi | during live migration we update the src XML, for a pre-existing VM, the src XML does not have any iothreads or iothreadpin field | |
| 13:45:52 | sean-k-mooney | yep we have had this tyep of issue before | |
| 13:46:09 | sean-k-mooney | i think we have exsing functional tests that simulate somethign similar | |
| 13:51:51 | opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2152697 https://review.opendev.org/c/openstack/nova/+/988769 | |
| 13:52:01 | gibi | sean-k-mooney: this is the reproducer ^^ | |
| 13:53:48 | sean-k-mooney | am `del list(conn._vms.values())[0]._def["iothreads"]` | |
| 13:53:58 | sean-k-mooney | ok i see why that works | |