| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-06-02 | |||
| 16:03:38 | elodilles | sean-k-mooney: so i'd say let's focus now on fixing it on master o:) but it just my opinion o:) | |
| 16:03:46 | melwitt | yeah I'd have to double check but it _might_ be ok bc of the change we made to make only the essential resources (vcpu, ram, disk) required for enforcement and the rest treated as unlimited | |
| 16:04:38 | melwitt | (i.e. the required for enforcement resource list is configurable) | |
| 16:05:21 | melwitt | yeah I think we all agree for now let it bake on master and backport is not an immediate thought | |
| 16:08:59 | elodilles | melwitt: +1 | |
| 16:17:51 | melwitt | CI has seemed so unhappy lately 😩 | |
| 16:38:05 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 18:35:53 | opendevreview | Merged openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 18:36:07 | opendevreview | Merged openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | |
| 18:36:47 | opendevreview | Merged openstack/nova stable/2026.1: Add reproducer test for bug 2105896 https://review.opendev.org/c/openstack/nova/+/989519 | |
| 18:43:55 | opendevreview | Merged openstack/nova stable/2026.1: Fix error when multiple security group in a project with same name https://review.opendev.org/c/openstack/nova/+/989520 | |
| 18:44:25 | opendevreview | Merged openstack/nova master: Add agentic coding guidance and docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:21:25 | frickler | sean-k-mooney: looks like you have uncovered a new failure mode on https://zuul.opendev.org/t/openstack/build/79d870c283264c169171f3a7a332ba3c already (ssh auth fail), maybe that's the next thing that can happen when the first boot is interrupted and the cloud-init process doesn't properly finish? | |
| 19:25:35 | melwitt | I think I have seen that failure mode before but couldn't figure out whether or how it doesn't have the right authorized_keys | |
| 19:28:15 | melwitt | seems like it's there, unless that doesn't mean what I thought it means | |
| 19:28:15 | melwitt | === sshd host keys === | |
| 19:28:15 | melwitt | -----BEGIN SSH HOST KEY KEYS----- | |
| 19:28:15 | melwitt | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk9qvrsW4suQOSW7BD+7S6v5Fakm6C6e6bxhoWpAi5nH7LRsFbzYj7M+bSvE0mQ64/qwoZaT6lmXFexmdfIkUSevz18TWecvg45bi4nx+sCEK25jtBXaqdGo1Bz7TU44/M4Bsj8rE1yiSHasoZ/qMrEJ4maMFZkSaub0CvqIe1NaRXRR1T/T9JLmA4IXGLN6gOKkw8NXfIRHbVwnQuNpcvCjifJlIKxWJXQq+Ttwq94apHZ0vqyg1/KMygHz6DUCU7kdRGv3/51ROKvJuwmLStjkwB8SyfcvJzQcjwkazX6DYWY7C/VEteP4ng2RfGouIOJRpjQxscsNdTG4aWgs0n root@tempest-serveractionstestothera-server-190417456 | |
| 19:28:15 | melwitt | ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOxWzqADqCihEY+t+o3XDe0hEIagEoU8YZYIjgdXuuJhvdz3NMiyaWJ4iYnFlL1ewByK/CzXuSJCQLzLOQJjtM= root@tempest-serveractionstestothera-server-190417456 | |
| 19:28:16 | melwitt | -----END SSH HOST KEY KEYS----- | |
| 19:30:35 | sean-k-mooney | frickler: that migh tnae sesne if we dont inject ths ssh key or ornly partly confirue dhcp or similar | |
| 19:30:36 | frickler | that's host key, not authorized_keys. there's "Printing cirros user authorized keys" before with no actual key shown, but I'll need to compare against a log for a successful boot | |
| 19:31:18 | sean-k-mooney | frickler: i also found one of the ohter kernel panics the other day but its a very very rare one | |
| 19:31:33 | melwitt | oh. mah bad | |
| 19:32:09 | sean-k-mooney | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_17d/openstack/17d6defdcff648cbbe950cf0137c7906/testr_results.html | |
| 19:32:23 | frickler | "cirros-apply-local already run per instance" might be a hint supporting my theory | |
| 19:32:27 | sean-k-mooney | https://paste.opendev.org/show/bd4gViIvUmb0CjIUcvh4/ | |
| 19:32:59 | sean-k-mooney | frickler: right so if we got far enough to touch the file sayign we are runing or have run our first init script | |
| 19:33:12 | sean-k-mooney | but we have not sync'd the file system or completeded it | |
| 19:33:24 | sean-k-mooney | it possibel the dhcp or other config wont survive the reboot | |
| 19:34:40 | sean-k-mooney | frickler: so the patch to use my custom image is currently in the gate queu | |
| 19:34:59 | sean-k-mooney | so we shoudl start geting data soon on if prepoulating the root fs helps | |
| 19:35:31 | sean-k-mooney | oh no | |
| 19:35:33 | sean-k-mooney | it failed | |
| 19:35:36 | frickler | sean-k-mooney: no, the failure above occured in gate, so you'd need to recheck | |
| 19:35:52 | sean-k-mooney | ah i tought htis was form a diffent patch | |
| 19:36:13 | sean-k-mooney | ok os ya that either the fact that that sometime just fails anyway | |
| 19:36:36 | sean-k-mooney | or that reason athat sometimes fails is also interupted first boot | |
| 19:37:21 | sean-k-mooney | its kind of undortuet we dont caure the console before and after the reboot | |
| 19:40:19 | sean-k-mooney | /dev/root resized successfully [took 0.34s] | |
| 19:40:21 | sean-k-mooney | Printing cirros user authorized keys | |
| 19:40:23 | sean-k-mooney | === system information === | |
| 19:40:33 | sean-k-mooney | so it does not appare to have pritned any authorised keys | |
| 19:40:40 | frickler | well ... we might be able to save the console data before we issue the resize/reboot and only dump it in case of a failure? | |
| 19:40:58 | sean-k-mooney | and since we are not passign a fallback password | |
| 19:41:00 | sean-k-mooney | tempest.lib.exceptions.SSHTimeout: Connection to the 172.24.5.241 via SSH timed out. | |
| 19:41:02 | sean-k-mooney | User: cirros, Password: None | |
| 19:41:07 | sean-k-mooney | its failing | |
| 19:41:17 | frickler | anyway, I'm out for today, will check back tomorrow | |
| 19:41:18 | sean-k-mooney | frickler: i dont see why we could | |
| 19:41:20 | sean-k-mooney | not | |
| 19:42:22 | sean-k-mooney | frickler: no worres ill recheck this for now and ill try and take a look at the cirros-init path so wee where we are markign the first boot as done vs wehre we have actuly sotre the autorisze dkeys ectra | |
| 19:43:33 | sean-k-mooney | we could mask the issue by providing the defautl cirros users pass word in the tempest config but i woudl prefer not too if we can avoid that | |
| 20:43:33 | opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | |
| 20:50:39 | opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | |
| #openstack-nova - 2026-06-03 | |||
| 07:12:22 | ralonsoh | sean-k-mooney, gibi hello folks. Yesterday I was looking at the issues with pyroute2 and asyncio in the nova-compute agent | |
| 07:12:48 | ralonsoh | I know this is not a solution but it could help, as a workaround, to wrap all pyroute2 operations with a privsep context | |
| 07:13:17 | ralonsoh | these operations will be executed in the daemon process instead of running in the main process | |
| 07:13:55 | ralonsoh | that will avoid the asyncio problem experienced when executing non-privileged commands | |
| 07:14:06 | ralonsoh | that will allow to bump the pyroute2 version | |
| 07:24:25 | gibi | ralonsoh: as I'm not sure about the root cause of the issue I cannot judge if moving the pyroute2 calls to another process will help or not. Have you tried to see if it helps? | |
| 08:20:51 | ralonsoh | gibi no, but I can propose a os-vif patch, at least for the OVS related commands | |
| 08:20:54 | ralonsoh | let me try it | |
| 08:21:35 | gibi | cool | |
| 08:34:08 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyPyroute2._lookup_interface`` private https://review.opendev.org/c/openstack/os-vif/+/991292 | |
| 08:47:21 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyRoute2.exists`` privileged in the OVS library https://review.opendev.org/c/openstack/os-vif/+/991293 | |
| 08:54:51 | ralonsoh | gibi, I've proposed https://review.opendev.org/c/openstack/requirements/+/973210, depending on ^^ | |
| 08:55:17 | ralonsoh | nova-alt-configurations-os-vif should install this os-vif patch | |
| 08:55:40 | opendevreview | dalekseev proposed openstack/nova master: Restrict machine type check to QEMU instances https://review.opendev.org/c/openstack/nova/+/991137 | |
| 09:12:44 | opendevreview | Joan Gilabert proposed openstack/nova master: move compile earlier https://review.opendev.org/c/openstack/nova/+/950516 | |
| 09:12:44 | opendevreview | Joan Gilabert proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 09:17:03 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug #2120927 https://review.opendev.org/c/openstack/nova/+/991294 | |
| 09:17:03 | opendevreview | ribaudr proposed openstack/nova master: Fix shelve-offload/unshelve race wiping instance host https://review.opendev.org/c/openstack/nova/+/991295 | |
| 10:17:49 | sean-k-mooney | ralonsoh: im not really a fan of that idea by the way | |
| 10:18:32 | sean-k-mooney | it woudl perfer to just run the command in a futureist process pool | |
| 10:18:43 | sean-k-mooney | then to pretened the call neeed privlage escalation | |
| 10:19:01 | sean-k-mooney | or provide a cli driver that we use when eventlet is enabled | |
| 10:20:00 | sean-k-mooney | nova-compute already defaults to threaded mode. in septempeber we can start removing eventlet support for the 2027.1 cycle if we choose too | |
| 10:20:10 | sean-k-mooney | at which point we wont need the privsep hack | |
| 10:26:35 | sean-k-mooney | frickler: so the next issue after you fix the first-boot being interupted is apparently gettign shceulded to a rax xen host where the cirrors image that 11 seconds for the intram to start 19 cectd for /etc/init.d/rc.sysinit: and 58 second to get to the longin prome when we only wait 15 seconds for the system to boot... | |
| 10:27:09 | sean-k-mooney | so i dont think we can fix randaom slow nodes but thats progress at least | |
| 10:30:14 | ralonsoh | sean-k-mooney, but if you are running in threaded mode, why pyroute has this issue? | |
| 10:30:39 | sean-k-mooney | im not sure it does have it in threaed mode | |
| 10:30:56 | sean-k-mooney | but we supprot both modes and we only enabeld threadign by defautl this cycle | |
| 10:31:21 | sean-k-mooney | as in its only been that way for like 2 months | |
| 10:32:02 | sean-k-mooney | ralonsoh: i have not had time to look at the actual issue in quite a whiel | |
| 10:32:15 | sean-k-mooney | but perhaps there are other approch we coudl take | |
| 10:32:23 | sean-k-mooney | is there a way to repoduce this issue locally? | |
| 10:32:39 | sean-k-mooney | my understanding was it was only seen in ci intermitently | |
| 10:34:18 | ralonsoh | no, that happens always in the CI when bumping pyroute to 0.9.6 | |
| 10:34:35 | ralonsoh | I'm not sure exactly in what command | |
| 10:36:31 | sean-k-mooney | oh ok if its reptable can we trigger ti with tempest locally in devstack | |
| 10:36:56 | sean-k-mooney | im askign because if we can i can maybe take a look at it in a vm and see if i cna debug it a bit | |
| 12:04:16 | opendevreview | Joan Gilabert proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 12:45:09 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 13:31:29 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyPyroute2._lookup_interface`` private https://review.opendev.org/c/openstack/os-vif/+/991292 | |
| 13:32:23 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyRoute2.exists`` privileged in the OVS library https://review.opendev.org/c/openstack/os-vif/+/991293 | |
| 13:34:58 | opendevreview | ribaudr proposed openstack/nova master: Add reproducer for bug #2117544 https://review.opendev.org/c/openstack/nova/+/991350 | |
| 13:34:58 | opendevreview | ribaudr proposed openstack/nova master: Deserialize JSON properties from volume_image_metadata https://review.opendev.org/c/openstack/nova/+/991351 | |
| 13:36:43 | opendevreview | ribaudr proposed openstack/nova master: Deserialize JSON properties from volume_image_metadata https://review.opendev.org/c/openstack/nova/+/991351 | |