| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-06-02 | |||
| 18:35:53 | opendevreview | Merged openstack/nova master: Fix unified limits to include all resource types https://review.opendev.org/c/openstack/nova/+/975872 | |
| 18:36:07 | opendevreview | Merged openstack/nova master: Poison eventlet import in native threading mode https://review.opendev.org/c/openstack/nova/+/986282 | |
| 18:36:47 | opendevreview | Merged openstack/nova stable/2026.1: Add reproducer test for bug 2105896 https://review.opendev.org/c/openstack/nova/+/989519 | |
| 18:43:55 | opendevreview | Merged openstack/nova stable/2026.1: Fix error when multiple security group in a project with same name https://review.opendev.org/c/openstack/nova/+/989520 | |
| 18:44:25 | opendevreview | Merged openstack/nova master: Add agentic coding guidance and docs https://review.opendev.org/c/openstack/nova/+/984540 | |
| 19:21:25 | frickler | sean-k-mooney: looks like you have uncovered a new failure mode on https://zuul.opendev.org/t/openstack/build/79d870c283264c169171f3a7a332ba3c already (ssh auth fail), maybe that's the next thing that can happen when the first boot is interrupted and the cloud-init process doesn't properly finish? | |
| 19:25:35 | melwitt | I think I have seen that failure mode before but couldn't figure out whether or how it doesn't have the right authorized_keys | |
| 19:28:15 | melwitt | ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCOxWzqADqCihEY+t+o3XDe0hEIagEoU8YZYIjgdXuuJhvdz3NMiyaWJ4iYnFlL1ewByK/CzXuSJCQLzLOQJjtM= root@tempest-serveractionstestothera-server-190417456 | |
| 19:28:15 | melwitt | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk9qvrsW4suQOSW7BD+7S6v5Fakm6C6e6bxhoWpAi5nH7LRsFbzYj7M+bSvE0mQ64/qwoZaT6lmXFexmdfIkUSevz18TWecvg45bi4nx+sCEK25jtBXaqdGo1Bz7TU44/M4Bsj8rE1yiSHasoZ/qMrEJ4maMFZkSaub0CvqIe1NaRXRR1T/T9JLmA4IXGLN6gOKkw8NXfIRHbVwnQuNpcvCjifJlIKxWJXQq+Ttwq94apHZ0vqyg1/KMygHz6DUCU7kdRGv3/51ROKvJuwmLStjkwB8SyfcvJzQcjwkazX6DYWY7C/VEteP4ng2RfGouIOJRpjQxscsNdTG4aWgs0n root@tempest-serveractionstestothera-server-190417456 | |
| 19:28:15 | melwitt | -----BEGIN SSH HOST KEY KEYS----- | |
| 19:28:15 | melwitt | === sshd host keys === | |
| 19:28:15 | melwitt | seems like it's there, unless that doesn't mean what I thought it means | |
| 19:28:16 | melwitt | -----END SSH HOST KEY KEYS----- | |
| 19:30:35 | sean-k-mooney | frickler: that migh tnae sesne if we dont inject ths ssh key or ornly partly confirue dhcp or similar | |
| 19:30:36 | frickler | that's host key, not authorized_keys. there's "Printing cirros user authorized keys" before with no actual key shown, but I'll need to compare against a log for a successful boot | |
| 19:31:18 | sean-k-mooney | frickler: i also found one of the ohter kernel panics the other day but its a very very rare one | |
| 19:31:33 | melwitt | oh. mah bad | |
| 19:32:09 | sean-k-mooney | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_17d/openstack/17d6defdcff648cbbe950cf0137c7906/testr_results.html | |
| 19:32:23 | frickler | "cirros-apply-local already run per instance" might be a hint supporting my theory | |
| 19:32:27 | sean-k-mooney | https://paste.opendev.org/show/bd4gViIvUmb0CjIUcvh4/ | |
| 19:32:59 | sean-k-mooney | frickler: right so if we got far enough to touch the file sayign we are runing or have run our first init script | |
| 19:33:12 | sean-k-mooney | but we have not sync'd the file system or completeded it | |
| 19:33:24 | sean-k-mooney | it possibel the dhcp or other config wont survive the reboot | |
| 19:34:40 | sean-k-mooney | frickler: so the patch to use my custom image is currently in the gate queu | |
| 19:34:59 | sean-k-mooney | so we shoudl start geting data soon on if prepoulating the root fs helps | |
| 19:35:31 | sean-k-mooney | oh no | |
| 19:35:33 | sean-k-mooney | it failed | |
| 19:35:36 | frickler | sean-k-mooney: no, the failure above occured in gate, so you'd need to recheck | |
| 19:35:52 | sean-k-mooney | ah i tought htis was form a diffent patch | |
| 19:36:13 | sean-k-mooney | ok os ya that either the fact that that sometime just fails anyway | |
| 19:36:36 | sean-k-mooney | or that reason athat sometimes fails is also interupted first boot | |
| 19:37:21 | sean-k-mooney | its kind of undortuet we dont caure the console before and after the reboot | |
| 19:40:19 | sean-k-mooney | /dev/root resized successfully [took 0.34s] | |
| 19:40:21 | sean-k-mooney | Printing cirros user authorized keys | |
| 19:40:23 | sean-k-mooney | === system information === | |
| 19:40:33 | sean-k-mooney | so it does not appare to have pritned any authorised keys | |
| 19:40:40 | frickler | well ... we might be able to save the console data before we issue the resize/reboot and only dump it in case of a failure? | |
| 19:40:58 | sean-k-mooney | and since we are not passign a fallback password | |
| 19:41:00 | sean-k-mooney | tempest.lib.exceptions.SSHTimeout: Connection to the 172.24.5.241 via SSH timed out. | |
| 19:41:02 | sean-k-mooney | User: cirros, Password: None | |
| 19:41:07 | sean-k-mooney | its failing | |
| 19:41:17 | frickler | anyway, I'm out for today, will check back tomorrow | |
| 19:41:18 | sean-k-mooney | frickler: i dont see why we could | |
| 19:41:20 | sean-k-mooney | not | |
| 19:42:22 | sean-k-mooney | frickler: no worres ill recheck this for now and ill try and take a look at the cirros-init path so wee where we are markign the first boot as done vs wehre we have actuly sotre the autorisze dkeys ectra | |
| 19:43:33 | sean-k-mooney | we could mask the issue by providing the defautl cirros users pass word in the tempest config but i woudl prefer not too if we can avoid that | |
| 20:43:33 | opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | |
| 20:50:39 | opendevreview | Dan Smith proposed openstack/nova master: Avoid reading 4GiB of reason text from RFB https://review.opendev.org/c/openstack/nova/+/991220 | |
| #openstack-nova - 2026-06-03 | |||
| 07:12:22 | ralonsoh | sean-k-mooney, gibi hello folks. Yesterday I was looking at the issues with pyroute2 and asyncio in the nova-compute agent | |
| 07:12:48 | ralonsoh | I know this is not a solution but it could help, as a workaround, to wrap all pyroute2 operations with a privsep context | |
| 07:13:17 | ralonsoh | these operations will be executed in the daemon process instead of running in the main process | |
| 07:13:55 | ralonsoh | that will avoid the asyncio problem experienced when executing non-privileged commands | |
| 07:14:06 | ralonsoh | that will allow to bump the pyroute2 version | |
| 07:24:25 | gibi | ralonsoh: as I'm not sure about the root cause of the issue I cannot judge if moving the pyroute2 calls to another process will help or not. Have you tried to see if it helps? | |
| 08:20:51 | ralonsoh | gibi no, but I can propose a os-vif patch, at least for the OVS related commands | |
| 08:20:54 | ralonsoh | let me try it | |
| 08:21:35 | gibi | cool | |
| 08:34:08 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyPyroute2._lookup_interface`` private https://review.opendev.org/c/openstack/os-vif/+/991292 | |
| 08:47:21 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyRoute2.exists`` privileged in the OVS library https://review.opendev.org/c/openstack/os-vif/+/991293 | |
| 08:54:51 | ralonsoh | gibi, I've proposed https://review.opendev.org/c/openstack/requirements/+/973210, depending on ^^ | |
| 08:55:17 | ralonsoh | nova-alt-configurations-os-vif should install this os-vif patch | |
| 08:55:40 | opendevreview | dalekseev proposed openstack/nova master: Restrict machine type check to QEMU instances https://review.opendev.org/c/openstack/nova/+/991137 | |
| 09:12:44 | opendevreview | Joan Gilabert proposed openstack/nova master: Add mtty/mdpy support for testing fake mdevs https://review.opendev.org/c/openstack/nova/+/898100 | |
| 09:12:44 | opendevreview | Joan Gilabert proposed openstack/nova master: move compile earlier https://review.opendev.org/c/openstack/nova/+/950516 | |
| 09:17:03 | opendevreview | ribaudr proposed openstack/nova master: Fix shelve-offload/unshelve race wiping instance host https://review.opendev.org/c/openstack/nova/+/991295 | |
| 09:17:03 | opendevreview | ribaudr proposed openstack/nova master: Add regression test for bug #2120927 https://review.opendev.org/c/openstack/nova/+/991294 | |
| 10:17:49 | sean-k-mooney | ralonsoh: im not really a fan of that idea by the way | |
| 10:18:32 | sean-k-mooney | it woudl perfer to just run the command in a futureist process pool | |
| 10:18:43 | sean-k-mooney | then to pretened the call neeed privlage escalation | |
| 10:19:01 | sean-k-mooney | or provide a cli driver that we use when eventlet is enabled | |
| 10:20:00 | sean-k-mooney | nova-compute already defaults to threaded mode. in septempeber we can start removing eventlet support for the 2027.1 cycle if we choose too | |
| 10:20:10 | sean-k-mooney | at which point we wont need the privsep hack | |
| 10:26:35 | sean-k-mooney | frickler: so the next issue after you fix the first-boot being interupted is apparently gettign shceulded to a rax xen host where the cirrors image that 11 seconds for the intram to start 19 cectd for /etc/init.d/rc.sysinit: and 58 second to get to the longin prome when we only wait 15 seconds for the system to boot... | |
| 10:27:09 | sean-k-mooney | so i dont think we can fix randaom slow nodes but thats progress at least | |
| 10:30:14 | ralonsoh | sean-k-mooney, but if you are running in threaded mode, why pyroute has this issue? | |
| 10:30:39 | sean-k-mooney | im not sure it does have it in threaed mode | |
| 10:30:56 | sean-k-mooney | but we supprot both modes and we only enabeld threadign by defautl this cycle | |
| 10:31:21 | sean-k-mooney | as in its only been that way for like 2 months | |
| 10:32:02 | sean-k-mooney | ralonsoh: i have not had time to look at the actual issue in quite a whiel | |
| 10:32:15 | sean-k-mooney | but perhaps there are other approch we coudl take | |
| 10:32:23 | sean-k-mooney | is there a way to repoduce this issue locally? | |
| 10:32:39 | sean-k-mooney | my understanding was it was only seen in ci intermitently | |
| 10:34:18 | ralonsoh | no, that happens always in the CI when bumping pyroute to 0.9.6 | |
| 10:34:35 | ralonsoh | I'm not sure exactly in what command | |
| 10:36:31 | sean-k-mooney | oh ok if its reptable can we trigger ti with tempest locally in devstack | |
| 10:36:56 | sean-k-mooney | im askign because if we can i can maybe take a look at it in a vm and see if i cna debug it a bit | |
| 12:04:16 | opendevreview | Joan Gilabert proposed openstack/nova master: WIP : Add mtty support to nova-next https://review.opendev.org/c/openstack/nova/+/922140 | |
| 12:45:09 | opendevreview | Ashish Gupta proposed openstack/nova master: tests: file-backed SQLite with WAL in threading mode for Database and CellDatabases Fixtures https://review.opendev.org/c/openstack/nova/+/988583 | |
| 13:31:29 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyPyroute2._lookup_interface`` private https://review.opendev.org/c/openstack/os-vif/+/991292 | |
| 13:32:23 | opendevreview | Rodolfo Alonso proposed openstack/os-vif master: Make ``PyRoute2.exists`` privileged in the OVS library https://review.opendev.org/c/openstack/os-vif/+/991293 | |
| 13:34:58 | opendevreview | ribaudr proposed openstack/nova master: Deserialize JSON properties from volume_image_metadata https://review.opendev.org/c/openstack/nova/+/991351 | |
| 13:34:58 | opendevreview | ribaudr proposed openstack/nova master: Add reproducer for bug #2117544 https://review.opendev.org/c/openstack/nova/+/991350 | |
| 13:36:43 | opendevreview | ribaudr proposed openstack/nova master: Deserialize JSON properties from volume_image_metadata https://review.opendev.org/c/openstack/nova/+/991351 | |
| 13:50:25 | opendevreview | Takashi Kajinami proposed openstack/nova-specs master: libvirt: AMD SEV-SNP support https://review.opendev.org/c/openstack/nova-specs/+/983376 | |
| 14:40:00 | Uggla | I guess there is a dependency resolution issue with grenade and stable/2025.1 https://zuul.opendev.org/t/openstack/build/23ff7cd6c0e649a9b83bb7df6f9dfc7b | |
| 14:40:19 | Uggla | elodilles are you aware of ^ | |
| 14:50:20 | elodilles | Uggla: yes, rpds-py does not support python 3.10 anymore. on QA channel it was discussed the other day and the plan is to release tempest and pin it for jobs that still uses Ubuntu Jammy 22.04 that has python3.10 by default | |
| 14:52:05 | Uggla | elodilles thx, so I will wait the fix. | |
| 14:56:44 | elodilles | +1 | |
| 14:59:52 | gmaan | elodilles: Uggla which job failing, I can fix that as i sent in email | |