Earlier  
Later
New Search
Posted Nick Remark
#openstack-cyborg - 2026-06-09
15:11:34 jgilaber I was testing the zuul config and see if I had the configuration right and then I push a patch to make that check accept mdev devices
15:11:37 sean-k-mooney so ill looint why ocne we have actul logs
15:11:47 jgilaber ack thanks!
15:13:02 sean-k-mooney my guess is the whitebox plugin may have confirued something tha tbrok live migrtion or some other unrelated issues but ill need to look at the nova-comptue logs to figure that out rather then just the tempest log stream
15:14:13 sean-k-mooney i know you didnt set the cpu config options in the josb def for whitebox but i dont recall what the devstack plugisn defaults are
15:15:08 jgilaber yes I saw your comment I probably missed a few settings, I tried to add the minimal set to get the test to run and then iteratively add the rest
15:15:19 jgilaber to make sure that I understood what each does
15:15:35 sean-k-mooney ya once we have a full run we can look at the rendered config files and then see what we need to tweak
15:15:50 sean-k-mooney i dont thin that will be hard
15:16:22 sean-k-mooney the ohter option we have is to write new tests in tempest iteslf
15:16:34 sean-k-mooney that done need whitebox
15:30:48 opendevreview Joan Gilabert proposed openstack/cyborg-specs master: Add generic mdev driver spec for 2026.2 https://review.opendev.org/c/openstack/cyborg-specs/+/982276
17:06:07 sean-k-mooney jgilaber: so the live migtion fialed because the qemu process crashed
18:01:54 jgilaber sean-k-mooney, do you think it is related to the patch?
18:02:35 sean-k-mooney no
18:02:40 sean-k-mooney i think you jsut got unlucky
18:02:49 jgilaber ack, thanks
18:03:16 jgilaber I pushed a new version because the tempest regex was wrong
18:03:30 jgilaber and I saw no reference to the whitebox
18:06:53 sean-k-mooney that also failing i think but ill take a look when its done
18:07:38 sean-k-mooney we may want to disabel the other jobs while we are debuging this
18:08:07 sean-k-mooney just so it does not us as much gate resouces and or take as long
18:08:15 sean-k-mooney but we can do that tomorrow
18:14:46 jgilaber good point, I'll do that tomorrow
19:48:25 blanson[m] \\\=-0987654321KL;'
#openstack-cyborg - 2026-06-10
01:54:26 opendevreview chandan kumar proposed openstack/cyborg master: Use GenericDriver.update() as the FPGA programming interface https://review.opendev.org/c/openstack/cyborg/+/991598
02:00:53 opendevreview chandan kumar proposed openstack/cyborg master: Remove broken image signature verification https://review.opendev.org/c/openstack/cyborg/+/991027
02:00:53 opendevreview chandan kumar proposed openstack/cyborg master: Use GenericDriver.update() as the FPGA programming interface https://review.opendev.org/c/openstack/cyborg/+/991598
10:04:09 sean-k-mooney jgilaber: progress
10:04:12 sean-k-mooney Details: [{'status': 400, 'title': 'Bad Request', 'detail': "The server could not comply with the request since it is either malformed or otherwise incorrect.\n\n Invalid query string parameters: Additional properties are not allowed ('in_tree' was unexpected) Failed validating 'additionalProperties' in schema: {'type': 'object', 'properties': {'name': {'type':
10:04:14 sean-k-mooney 'string'}, 'uuid': {'type': 'string', 'format': 'uuid'}}, 'additionalProperties': False} On instance: {'in_tree': '58f9b863-0a2f-4e9b-a2e9-84d92707beab'} ", 'request_id': 'req-6792f268-11b3-4a14-ada4-f87e733e5058'}]
10:04:23 sean-k-mooney so the whitebox test ran
10:04:39 sean-k-mooney but apprently we are calling placment incorrectly
10:04:56 sean-k-mooney and or tempst has incorrect schma validation
10:05:31 sean-k-mooney 400 GET https://213.32.77.206/placement/resource_providers?in_tree=58f9b863-0a2f-4e9b-a2e9-84d92707beab 0.028s
10:05:47 sean-k-mooney that looks like placement rejecting it but my guess is this is a microversion issue
10:07:27 sean-k-mooney New in version 1.14
10:07:40 jgilaber yes, I'm working on a patch for that
10:08:12 sean-k-mooney looking a tthe hearder we are not passign any header or are getting defaulted back to 'openstack-api-version': 'placement 1.0'
10:08:26 jgilaber I'm waiting to get the vendor id of the device in the guest vm to configure the test properly
10:08:42 sean-k-mooney ack
10:09:22 sean-k-mooney the vendor id and prouct ida are 4348:3253
10:10:26 sean-k-mooney os i think you neeed to set that as hex so you want 4348 i think
10:11:05 sean-k-mooney its set here https://github.com/torvalds/linux/blob/master/samples/vfio-mdev/mtty.c#L252 in reviert byte order
10:11:34 sean-k-mooney well or here https://github.com/torvalds/linux/blob/master/samples/vfio-mdev/README.rst?plain=1#L59
10:13:22 sean-k-mooney for mdpy which is actully predentign to be a gpu its using redhats vendor id
10:13:30 sean-k-mooney https://github.com/torvalds/linux/blob/master/samples/vfio-mdev/mdpy-defs.h#L12
10:13:53 sean-k-mooney because it s using the allcoation redhat provide too qemu
10:27:55 jgilaber that seems right, I've pushed a patch to set the placement min microversion, let's see what happens next
10:29:32 sean-k-mooney it is set in some test but not in the smoke test
10:30:07 sean-k-mooney https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_vgpu.py#L316
10:32:17 sean-k-mooney jgilaber: ah you setin it in the base class ya i was debating propsing you do that cool
10:32:35 sean-k-mooney we could clean up some of the other palces where its now beign set to the same value later
10:33:10 sean-k-mooney jgilaber: can you update the nova patch to depend on that one to test it end to end?
10:33:45 sean-k-mooney if it passes i cna fast appove that but if you need to respin it woudl be nice to remove the duplication
10:47:00 jgilaber already set the depends-on
10:47:20 sean-k-mooney cool
10:47:35 jgilaber ack, will remove the duplication after the current ci run finishes
15:56:32 sean-k-mooney jgilaber: so it booted correctly https://paste.opendev.org/show/b9TDtQ29jhj8jJfAy5Ge/
15:57:03 sean-k-mooney but but ssh to the vm failed
15:57:19 sean-k-mooney 2026-06-10 14:15:22.306 107224 ERROR tempest.lib.common.ssh paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 172.24.5.245
15:57:54 sean-k-mooney the fallback password is also incorrect
15:57:56 sean-k-mooney User: cirros, Password: password
15:58:43 sean-k-mooney it shoudl be gocubsgo but its trying to use the ssh key anyway
16:01:13 sean-k-mooney there are a buch of unrelated libvirt issues as well again related to libvie migration that dont obviously looks to be related to yoru change but its odd
16:02:07 jgilaber yes, I saw that I was missing some pre playbook from the whitebox plugin
16:02:28 jgilaber https://github.com/openstack/whitebox-tempest-plugin/blob/master/playbooks/whitebox/pre.yaml
16:02:42 sean-k-mooney oh
16:02:45 jgilaber I suspect this task might be needed for ssh https://github.com/openstack/whitebox-tempest-plugin/blob/b63d8bbda7af90f3ac2f463d34439d638d0f16df/playbooks/whitebox/pre.yaml#L24
16:02:58 jgilaber and the compute_nodes template was missing for some test as well
16:03:01 sean-k-mooney am that not stictly required
16:03:13 sean-k-mooney although thte tempest key part might be
16:03:56 sean-k-mooney the hugepage setup is not needed for what we need but it does not hurg to have this and have it disabeld which i think is the default
16:04:13 jgilaber the non-vgpu test that are failing are because of the setting 'live_migrate_back_and_forth' which the whitebox plugin defaults to true
16:04:22 jgilaber I think
16:04:28 sean-k-mooney jgilaber: ya ok well lets see but non of that shoudl impact the libvirt connection issues
16:04:39 sean-k-mooney well...
16:04:57 sean-k-mooney so some of the whitebox tests restart nova-comptue
16:05:26 sean-k-mooney live_migrate_back_and_forth should work in general
16:05:55 sean-k-mooney but ok it might be triggering an old qemu bug
16:06:12 sean-k-mooney there was a case where qemu woudl crash when that was eneable when usign block migration
16:06:23 sean-k-mooney and that is waht is failing
16:06:36 sean-k-mooney so maybe we are hitting that
16:08:30 jgilaber hmm maybe, I disabled because of this comment https://github.com/openstack/tempest/blob/531167f42631b9b79fdd73455772dc166d047ae6/tempest/api/compute/admin/test_live_migration.py#L172 and also it was the biggest diff between the tempest conf in this job and run from a different change
16:53:48 sean-k-mooney im wodneirng if there is an interaction betwen that and live_migration_parallel_connections: 4
16:54:08 sean-k-mooney in anycase it does not hurt to turn it off for now
16:54:29 sean-k-mooney as you nted this is mainly for grenade but not solely
16:55:29 opendevreview sean mooney proposed openstack/cyborg master: Add SRBAC base rules and enforce_new_defaults override https://review.opendev.org/c/openstack/cyborg/+/992292
16:55:30 opendevreview sean mooney proposed openstack/cyborg master: Migrate ARQ policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992293
16:55:31 opendevreview sean mooney proposed openstack/cyborg master: Migrate device policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992294
16:55:31 opendevreview sean mooney proposed openstack/cyborg master: Migrate deployable policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992295
16:55:32 opendevreview sean mooney proposed openstack/cyborg master: Migrate attribute policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992296
16:55:32 opendevreview sean mooney proposed openstack/cyborg master: Add SRBAC policy concepts documentation https://review.opendev.org/c/openstack/cyborg/+/992297
16:55:33 opendevreview sean mooney proposed openstack/cyborg master: Add cyborg-tempest-srbac Zuul job https://review.opendev.org/c/openstack/cyborg/+/992332
16:55:33 opendevreview sean mooney proposed openstack/cyborg master: Use request context as policy target https://review.opendev.org/c/openstack/cyborg/+/992722
16:57:39 opendevreview sean mooney proposed openstack/cyborg master: Use request context as policy target https://review.opendev.org/c/openstack/cyborg/+/992722
16:57:40 opendevreview sean mooney proposed openstack/cyborg master: Add SRBAC base rules and enforce_new_defaults override https://review.opendev.org/c/openstack/cyborg/+/992292
16:57:40 opendevreview sean mooney proposed openstack/cyborg master: Migrate ARQ policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992293
16:57:41 opendevreview sean mooney proposed openstack/cyborg master: Migrate device policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992294
16:57:41 opendevreview sean mooney proposed openstack/cyborg master: Migrate deployable policies to DocumentedRuleDefault https://review.opendev.org/c/openstack/cyborg/+/992295

Earlier   Later