| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2026-05-20 | |||
| 17:56:58 | sean-k-mooney | and then start loadinga lot of module and doign some netwrok config not long after | |
| 17:57:05 | sean-k-mooney | tha tkind of olooks like it restarted | |
| 17:59:32 | sean-k-mooney | melwitt: ok ya it was stoped and started | |
| 17:59:46 | sean-k-mooney | │May 12 21:08:38 npe61172270ee04 sudo[44964]: stack : PWD=/opt/stack ; USER=root ; COMMAND=/usr/bin/systemctl stop libvirtd │ | |
| 17:59:48 | sean-k-mooney | │May 12 21:08:38 npe61172270ee04 sudo[44964]: pam_unix(sudo:session): session opened for user root(uid=0) by stack(uid=1001) │ | |
| 17:59:50 | sean-k-mooney | │May 12 21:08:38 npe61172270ee04 sudo[44964]: pam_unix(sudo:session): session closed for user root │ | |
| 17:59:52 | sean-k-mooney | │May 12 21:08:39 npe61172270ee04 sudo[45008]: stack : PWD=/opt/stack ; USER=root ; COMMAND=/usr/bin/systemctl start libvirtd │ | |
| 17:59:54 | sean-k-mooney | │May 12 21:08:39 npe61172270ee04 sudo[45008]: pam_unix(sudo:session): session opened for user root(uid=0) by stack(uid=1001) | |
| 18:01:15 | sean-k-mooney | melwitt: so this is likely oen of 2 thing | |
| 18:01:45 | sean-k-mooney | the whitebox tempest plugin need to run in concrance 1 becasue it has test that do thing like restart libivirt | |
| 18:01:56 | sean-k-mooney | so it could be a whitbox test | |
| 18:02:14 | sean-k-mooney | or this could be beacuse fo a post playbook | |
| 18:02:19 | melwitt | sean-k-mooney: oh huh ok. let me check if there is that | |
| 18:02:38 | melwitt | thanks | |
| 18:03:08 | sean-k-mooney | i used `curl https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1ef/openstack/1efc16477b414adbaf673b00520aa5cf/compute-host/logs/syslog.txt | zcat | lnav` | |
| 18:03:17 | sean-k-mooney | to look that the syslog locally | |
| 18:03:31 | sean-k-mooney | but tis clearlly shoing that sudo was use to invoke sysclt | |
| 18:03:40 | sean-k-mooney | *systemctl | |
| 18:04:28 | sean-k-mooney | test_vtpm_creation_after_virtqemud_restart | |
| 18:04:37 | sean-k-mooney | that is proably the test i would look at first | |
| 18:05:12 | sean-k-mooney | yep | |
| 18:05:14 | sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute/test_vtpm.py#L168 | |
| 18:05:39 | melwitt | ugh ok | |
| 18:05:43 | sean-k-mooney | melwitt: gmaan so we have never gotten aroudn to makeign the serial decorator from tempet proper | |
| 18:05:46 | sean-k-mooney | work in whitebox | |
| 18:06:07 | melwitt | that makes it all make sense. thanks | |
| 18:06:18 | sean-k-mooney | so you either need to splity the job into 2 jobs or run it serially. | |
| 18:07:00 | sean-k-mooney | we have done both at diffent time but we currently set concrrency 1 | |
| 18:07:02 | sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L176 | |
| 18:08:00 | sean-k-mooney | ah https://github.com/openstack/nova/blob/master/.zuul.yaml#L484 | |
| 18:08:05 | melwitt | yeah, I had seen that but did not know the root reason why /facepalm. I'm gonna include this specifically in my code comment | |
| 18:08:32 | sean-k-mooney | we shoudl jsut make whitebox use the serial decorator properly eventully | |
| 18:08:40 | sean-k-mooney | maybe let ai spin on that for a while | |
| 18:08:50 | melwitt | yeah... I know. that was my bad. but I did not see why these would have to be serial, I completely missed the restart thing | |
| 18:09:10 | sean-k-mooney | but ya its one of those thing that you only know about because you have the scars | |
| 18:09:28 | sean-k-mooney | melwitt: not at all this is very easy to miss | |
| 18:10:15 | melwitt | "who wouldn't want concurrency!" "oh." | |
| 18:10:29 | gmaan | sean-k-mooney: yeah but is there any known issue for not using serial decorator in whitebox? | |
| 18:10:47 | melwitt | well, mystery solved haha | |
| 18:10:48 | sean-k-mooney | gmaan: jparker tried, hit an isssue and we never got around to it | |
| 18:10:54 | sean-k-mooney | so gmaan no | |
| 18:11:09 | sean-k-mooney | we tried it once there was a bug we reverted back to not using it | |
| 18:11:15 | gmaan | ohk, I can try but sometime start of next month or when get time | |
| 18:11:42 | sean-k-mooney | https://opendev.org/openstack/whitebox-tempest-plugin/commit/a8986a86c03e32daeebcf4e0fb65c91aede2248a | |
| 18:11:46 | melwitt | that means it would let one test be serial test without having to make all the others? | |
| 18:12:02 | sean-k-mooney | im not conviece it was enable properly in te first palce | |
| 18:12:19 | sean-k-mooney | # Decorator support for serial does not land into tempest until 34.0.0. | |
| 18:12:26 | sean-k-mooney | lol ok well that not a proplem any more | |
| 18:12:56 | sean-k-mooney | melwitt: yes | |
| 18:13:07 | melwitt | sounds like a good decorator | |
| 18:13:12 | gmaan | sean-k-mooney: ack, i was searching the link | |
| 18:13:16 | sean-k-mooney | melwitt: it uses a file based reder writere lock | |
| 18:13:58 | sean-k-mooney | melwitt: basiclly when you use this all the test that dont need to run serially acriore the reasder lock and hte ones that do aquire the writer lock | |
| 18:14:18 | sean-k-mooney | and hten the filesysem lock syncoised across the workers | |
| 18:14:29 | melwitt | makes sense | |
| 18:14:31 | sean-k-mooney | but in addtion ot that we run the serial tests last | |
| 18:14:40 | sean-k-mooney | becuase of how we order testes base don name | |
| 18:14:49 | sean-k-mooney | at least in upstream tempest | |
| 18:15:22 | sean-k-mooney | melwitt: https://github.com/openstack/tempest/commit/dfb304355b46882696ef26386637836577be8db7 | |
| 18:15:42 | sean-k-mooney | it was an optimisation that gibi help imeplemnta few years ago | |
| 18:16:39 | gmaan | yeah, that resolved the aggregates tests issue | |
| 18:16:40 | melwitt | very cool | |
| 18:16:45 | sean-k-mooney | hehe https://github.com/openstack/tempest/commit/73ba33773daf1df1be792b616842dd389fd325bc | |
| 18:16:50 | sean-k-mooney | looks like you have used it before | |
| 18:17:04 | melwitt | uhhh lol | |
| 18:17:06 | gmaan | so for now, concurrency=1 will work for VTPM case as they are from same test class | |
| 18:17:37 | sean-k-mooney | yep also we run like 8 tests in that job | |
| 18:17:44 | sean-k-mooney | so concrance 1 is not a big issue | |
| 18:17:50 | melwitt | well, I'm not gonna forget it again now haha | |
| 18:18:16 | melwitt | I put it at its own job bc it needs nested virt and was wary to tie up those machines too much with other tests that don't need it | |
| 18:18:26 | sean-k-mooney | for whiht box in general iw touls be nicer to not need that obvioulsy as many of the test can run in parralel | |
| 18:18:35 | gmaan | yeah | |
| 18:19:02 | sean-k-mooney | well mystery solved | |
| 18:19:13 | sean-k-mooney | feel free to pin and i can review the zuul change quickly | |
| 18:19:51 | melwitt | yeah. this is good. I was struggling to write the code comment for needing concurrency=1 so I was looking at details again. bc I was not seeing why. ugh. | |
| 18:29:29 | opendevreview | melanie witt proposed openstack/nova master: Use tempest_concurrency=1 for nova-vtpm job https://review.opendev.org/c/openstack/nova/+/984864 | |
| 18:30:41 | melwitt | gmaan, sean-k-mooney: ^ | |
| 18:39:15 | gmaan | melwitt: ack, thanks | |
| 18:46:53 | sean-k-mooney | looks good to me | |
| 19:41:33 | melwitt | I'm sure someone already said this but https://bugs.launchpad.net/tempest/+bug/2153382 is affecting nova-next too Details: {'code': 401, 'title': 'Unauthorized', 'message': 'The request you have made requires authentication.'} | |
| 19:53:48 | melwitt | (just replied on the ML) | |
| 19:54:23 | gmaan | ohk, I did not realize nova-next also run that test | |
| 19:55:07 | gmaan | anyways fix is in gate, I will merge and updaet ML once test results are out https://review.opendev.org/c/openstack/tempest/+/938766 | |
| 19:56:24 | melwitt | thanks gmaan | |
| 19:57:18 | melwitt | nova-multi-cell too. sent my reply too hastily | |
| 20:01:00 | sean-k-mooney | you know our gate would pass more frequetly if we just turned off cidner :P | |
| 20:01:39 | sean-k-mooney | gmaan: was the change form a 403 to a 401 intentioanly by the way | |
| 20:02:40 | gmaan | it was due to change in default value of service_token_roles_required in keystonemiddlwware | |
| 20:02:57 | sean-k-mooney | its not really a 401 Unauthorized Error as i understnad it. the service token is a valid token but it does not have the requrie roles right? | |
| 20:02:58 | gmaan | and tempest test did not handle it beacuse it was disable it in CI. | |
| 20:03:07 | gmaan | yes | |
| 20:03:08 | sean-k-mooney | so it really shoudl be a s 403 Forbidden Error | |
| 20:03:29 | sean-k-mooney | becuase it authrization issue not authentication | |
| 20:04:12 | sean-k-mooney | so the real fix woudl be in cidner to make ti return a 403 right? | |
| 20:05:14 | gmaan | I am not sure, if service token does not have a required token then it does not come to cinder itself that if operation is forbidden or not. | |
| 20:05:45 | gmaan | so that is why 401 seems valid one as passed token and required roles are not valid | |
| 20:06:05 | sean-k-mooney | but 401 is not about roles or permeison | |
| 20:06:13 | sean-k-mooney | its about is the token exprie/valid | |
| 20:06:37 | melwitt | I thought that would include roles too though, no? | |
| 20:06:52 | sean-k-mooney | a 401 is an indtication to the client that it should reauthicate | |