| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-cyborg - 2026-06-10 | |||
| 10:47:35 | jgilaber | ack, will remove the duplication after the current ci run finishes | |
| 15:56:32 | sean-k-mooney | jgilaber: so it booted correctly https://paste.opendev.org/show/b9TDtQ29jhj8jJfAy5Ge/ | |
| 15:57:03 | sean-k-mooney | but but ssh to the vm failed | |
| 15:57:19 | sean-k-mooney | 2026-06-10 14:15:22.306 107224 ERROR tempest.lib.common.ssh paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.24.5.245 | |
| 15:57:54 | sean-k-mooney | the fallback password is also incorrect | |
| 15:57:56 | sean-k-mooney | User: cirros, Password: password | |
| 15:58:43 | sean-k-mooney | it shoudl be gocubsgo but its trying to use the ssh key anyway | |
| 16:01:13 | sean-k-mooney | there are a buch of unrelated libvirt issues as well again related to libvie migration that dont obviously looks to be related to yoru change but its odd | |
| 16:02:07 | jgilaber | yes, I saw that I was missing some pre playbook from the whitebox plugin | |
| 16:02:28 | jgilaber | https://github.com/openstack/whitebox-tempest-plugin/blob/master/playbooks/whitebox/pre.yaml | |
| 16:02:42 | sean-k-mooney | oh | |
| 16:02:45 | jgilaber | I suspect this task might be needed for ssh https://github.com/openstack/whitebox-tempest-plugin/blob/b63d8bbda7af90f3ac2f463d34439d638d0f16df/playbooks/whitebox/pre.yaml#L24 | |
| 16:02:58 | jgilaber | and the compute_nodes template was missing for some test as well | |
| 16:03:01 | sean-k-mooney | am that not stictly required | |
| 16:03:13 | sean-k-mooney | although thte tempest key part might be | |
| 16:03:56 | sean-k-mooney | the hugepage setup is not needed for what we need but it does not hurg to have this and have it disabeld which i think is the default | |
| 16:04:13 | jgilaber | the non-vgpu test that are failing are because of the setting 'live_migrate_back_and_forth' which the whitebox plugin defaults to true | |
| 16:04:22 | jgilaber | I think | |
| 16:04:28 | sean-k-mooney | jgilaber: ya ok well lets see but non of that shoudl impact the libvirt connection issues | |
| 16:04:39 | sean-k-mooney | well... | |
| 16:04:57 | sean-k-mooney | so some of the whitebox tests restart nova-comptue | |
| 16:05:26 | sean-k-mooney | live_migrate_back_and_forth should work in general | |
| 16:05:55 | sean-k-mooney | but ok it might be triggering an old qemu bug | |
| 16:06:12 | sean-k-mooney | there was a case where qemu woudl crash when that was eneable when usign block migration | |
| 16:06:23 | sean-k-mooney | and that is waht is failing | |
| 16:06:36 | sean-k-mooney | so maybe we are hitting that | |
| 16:08:30 | jgilaber | hmm maybe, I disabled because of this comment https://github.com/openstack/tempest/blob/531167f42631b9b79fdd73455772dc166d047ae6/tempest/api/compute/admin/test_live_migration.py#L172 and also it was the biggest diff between the tempest conf in this job and run from a different change | |
| 16:53:48 | sean-k-mooney | im wodneirng if there is an interaction betwen that and live_migration_parallel_connections: 4 | |
| 16:54:08 | sean-k-mooney | in anycase it does not hurt to turn it off for now | |
| 16:54:29 | sean-k-mooney | as you nted this is mainly for grenade but not solely | |
| 16:55:29 | opendevreview | sean mooney proposed openstack/cyborg master: Add SRBAC base rules and enforce_new_defaults override https://review.opendev.org/c/openstack/cyborg/+/992292 | |
| 16:55:30 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate ARQ policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992293 | |
| 16:55:31 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate deployable policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992295 | |
| 16:55:31 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate device policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992294 | |
| 16:55:32 | opendevreview | sean mooney proposed openstack/cyborg master: Add SRBAC policy concepts documentation https://review.opendev.org/c/openstack/cyborg/+/992297 | |
| 16:55:32 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate attribute policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992296 | |
| 16:55:33 | opendevreview | sean mooney proposed openstack/cyborg master: Use request context as policy target https://review.opendev.org/c/openstack/cyborg/+/992722 | |
| 16:55:33 | opendevreview | sean mooney proposed openstack/cyborg master: Add cyborg-tempest-srbac Zuul job https://review.opendev.org/c/openstack/cyborg/+/992332 | |
| 16:57:39 | opendevreview | sean mooney proposed openstack/cyborg master: Use request context as policy target https://review.opendev.org/c/openstack/cyborg/+/992722 | |
| 16:57:40 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate ARQ policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992293 | |
| 16:57:40 | opendevreview | sean mooney proposed openstack/cyborg master: Add SRBAC base rules and enforce_new_defaults override https://review.opendev.org/c/openstack/cyborg/+/992292 | |
| 16:57:41 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate deployable policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992295 | |
| 16:57:41 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate device policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992294 | |
| 16:57:42 | opendevreview | sean mooney proposed openstack/cyborg master: Add SRBAC policy concepts documentation https://review.opendev.org/c/openstack/cyborg/+/992297 | |
| 16:57:42 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate attribute policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992296 | |
| 16:57:43 | opendevreview | sean mooney proposed openstack/cyborg master: Add cyborg-tempest-srbac Zuul job https://review.opendev.org/c/openstack/cyborg/+/992332 | |
| 17:06:14 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate device policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992294 | |
| 17:06:14 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate ARQ policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992293 | |
| 17:06:15 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate attribute policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992296 | |
| 17:06:15 | opendevreview | sean mooney proposed openstack/cyborg master: Migrate deployable policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992295 | |
| 17:06:16 | opendevreview | sean mooney proposed openstack/cyborg master: Add cyborg-tempest-srbac Zuul job https://review.opendev.org/c/openstack/cyborg/+/992332 | |
| 17:06:16 | opendevreview | sean mooney proposed openstack/cyborg master: Add SRBAC policy concepts documentation https://review.opendev.org/c/openstack/cyborg/+/992297 | |
| #openstack-cyborg - 2026-06-12 | |||
| 08:51:14 | jgilaber | Hi sean-k-mooney, I was wrong the last time we talked about the mtty whitebox vpgu tests. I thought the pre playbook was needed to configure the ssh key to the instances | |
| 08:51:40 | jgilaber | but the tests actually create a ssh keypair and pass it when creating the instance | |
| 08:52:21 | jgilaber | after some more digging and asking claude I saw some warnings in the qemu logs for the instance https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_583/openstack/58333f7196aa482c9d2b7c737045b906/controller/logs/libvirt/libvirt/qemu/instance-000000f6_log.txt | |
| 08:53:13 | jgilaber | do you think those could be preventing the instance os from booting? From the tempest logs the networking ssh keys and security groups seems correct | |
| 08:55:10 | sean-k-mooney | hum | |
| 08:55:42 | sean-k-mooney | so maybe | |
| 08:55:55 | sean-k-mooney | have you treid doing this locally in devstack yet? | |
| 08:56:18 | sean-k-mooney | we may need to twak the kernel parmaters slightly like im doing in my pci sim job | |
| 08:56:30 | jgilaber | no, I tried before without nova | |
| 08:56:50 | jgilaber | I can update my env and try quickly | |
| 08:57:25 | sean-k-mooney | so pci_sim_sysfs_write Y /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts | |
| 08:57:49 | sean-k-mooney | that just echo Y | sudo tee /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts | |
| 08:59:27 | sean-k-mooney | failing that we can try a newer qemu/kernel by movign the job to debian 13 if its currently on ubutnu 24.04 | |
| 08:59:49 | sean-k-mooney | i will hopefully have 26.04 working in the ci in the next week or two as well | |
| 09:00:11 | sean-k-mooney | tehre are some kernel parmater related to eventfd that we might also ned to tweak | |
| 09:01:24 | sean-k-mooney | sepcificly vm.unprivileged_userfaultfd should be set to 1 | |
| 09:04:41 | sean-k-mooney | asking gemini it say the issue si cosmetic | |
| 09:05:02 | sean-k-mooney | jgilaber: for what its worth i did see the vm boot probely in at least one of the test runs | |
| 09:05:15 | sean-k-mooney | it just didnt get networking properly | |
| 09:05:30 | jgilaber | ack thanks, I'll try in my devstack and see what happens | |
| 09:06:06 | sean-k-mooney | oh | |
| 09:06:12 | sean-k-mooney | gemina has a good point | |
| 09:06:27 | sean-k-mooney | so we used to use 64mb cirros images in the past | |
| 09:06:43 | sean-k-mooney | but we stoped becasuse newer cirros iamge need a little more ram to boot properly | |
| 09:06:55 | sean-k-mooney | we now use 128 or 192 mb flavors | |
| 09:07:03 | sean-k-mooney | but this vm was using a 64mb flavor | |
| 09:07:17 | sean-k-mooney | with 64mb it was prone ot ooming or very slow boots | |
| 09:08:00 | sean-k-mooney | that is proably worth looking ito and or fixing to match the new defautls we use in devstack | |
| 09:08:11 | jgilaber | hmm that's worth a shot while I test things locally | |
| 09:08:21 | jgilaber | I need to update the patch to revert some changes anyway | |
| 13:41:48 | jgilaber | sean-k-mooney, it's taken me a while but I got a vm with a mtty attached in my devstack | |
| 13:42:04 | jgilaber | lscpi -nn -k gives: "00:05.0 Class 0700: 4348:3253 serial" | |
| 13:43:32 | jgilaber | and in the ci job now it seems to ssh into the vm just fine | |
| 13:43:36 | jgilaber | https://fad10fc5f2c070b32033-beb494a5db5cda67da23a1381224245f.ssl.cf5.rackcdn.com/openstack/5f8a4405ba0f46c78746e4583a18d6e7/testr_results.html | |
| 13:44:17 | sean-k-mooney | cool | |
| 13:44:46 | sean-k-mooney | ok so now its the pre playbook | |
| 13:44:48 | sean-k-mooney | FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/compute_nodes.yaml' | |
| 13:45:13 | jgilaber | yes, I'm thinking about creating a new base job in the whitebox plugin | |
| 13:45:16 | sean-k-mooney | so your running thest that are tryign to ssh into the vms or connect to the db so it need that file | |
| 13:45:31 | sean-k-mooney | you could | |
| 13:45:43 | sean-k-mooney | but we dont wnat nova-next to intherit form whitebox | |
| 13:46:09 | sean-k-mooney | we could do this testignin the vtpm job instead | |
| 13:46:15 | sean-k-mooney | that inherits form whitebox alrady | |
| 13:46:35 | sean-k-mooney | that might be a better base | |
| 13:47:06 | jgilaber | ack, if there no particular reason to use nova-next i can switch | |
| 13:47:25 | jgilaber | from nova pov what would be better, add the tests to the vptm job or create a new job just for this? | |
| 13:47:26 | sean-k-mooney | we jus tchose it because it test nonstandard things | |
| 13:47:34 | sean-k-mooney | btu nova-next is getting quite big | |