| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-01-13 | |||
| 14:37:09 | dansmith | if we let it get out of control we'll never recover | |
| 14:37:35 | bauzas | sean-k-mooney: gmann: ok, so by now we only check functional tests using legacy RBAC | |
| 14:37:37 | bauzas | https://review.opendev.org/c/openstack/placement/+/869525/3/placement/tests/functional/fixtures/placement.py | |
| 14:37:43 | bauzas | I'm OK with this | |
| 14:38:08 | bauzas | but will we have a FUP modifying our tests for using the new defaults ? | |
| 14:38:37 | dansmith | bauzas: no, the main devstack job runs with old | |
| 14:38:42 | dansmith | and a new devstack job with new | |
| 14:39:03 | dansmith | that's what I had him add, so we could command them still to be old while we transition | |
| 14:40:38 | sean-k-mooney | well bauzas is askign about functional tests | |
| 14:40:44 | bauzas | dansmith: ok, I understand it so placement won't yet support new defaults, only nova/neutron/cinder blah, right? | |
| 14:40:59 | sean-k-mooney | at some point we shoudl test with new placemnt defaults too | |
| 14:41:05 | bauzas | that's my point | |
| 14:41:16 | sean-k-mooney | althogh i dont know if we want to just swap or add new tests | |
| 14:41:17 | bauzas | I just wanna make sure what we will do | |
| 14:41:37 | sean-k-mooney | largly for placment it wont matter much | |
| 14:41:39 | dansmith | oh sorry I thought you meant you thought we weren't getting any old coverage | |
| 14:41:48 | sean-k-mooney | as we always talk to it as admin more or less now anyway | |
| 14:41:49 | dansmith | right doesn't matter much for placement | |
| 14:42:56 | dansmith | glance does have a functional job running on both, but mostly because they have specific functional tests for it and their functional tests are a lot more realistic than ours | |
| 14:43:14 | dansmith | do we even do policy checking in our functionals? maybe some of them? | |
| 14:45:26 | bauzas | we don't check policy in our tests AFAIK | |
| 14:45:29 | bauzas | butn, | |
| 14:45:59 | bauzas | we sometimes ask for the admin role | |
| 14:46:39 | bauzas | so I guess that while nova will work with new defaults, we'll then call placement using old defaults, right? | |
| 14:47:49 | opendevreview | Sofia Enriquez proposed openstack/nova master: WIP: Implement encryption on backingStore https://review.opendev.org/c/openstack/nova/+/870012 | |
| 14:48:52 | dansmith | bauzas: I don't know that there's really much difference for placement | |
| 14:49:05 | dansmith | admin is still admin, and we don't use the user's token like we do when we call neutron, cinder, etc | |
| 14:56:17 | dansmith | I've also seen this grenade failure where we fail on the old side waiting for compute to be registered in the catalog | |
| 15:12:48 | dansmith | bauzas: gibi sean-k-mooney: it would be really good to get this in as soon as we can, it's seriously annoying debugging the other gate fails without it: https://review.opendev.org/c/openstack/nova/+/869900 | |
| 15:14:22 | bauzas | dansmith: gmann: sent new RBAC defaults patch to the gate | |
| 15:14:29 | bauzas | looking now at your CI fix | |
| 15:14:33 | dansmith | bauzas: thanks | |
| 15:16:19 | dansmith | bauzas: that 9900 patch avoids us spewing thousands of warnings on each run about that deprecated function, | |
| 15:16:27 | dansmith | which makes reading other log dumps pretty hard | |
| 15:17:23 | bauzas | dansmith: I don't see what you mean :p https://review.opendev.org/c/openstack/oslo.messaging/+/862419/4/oslo_messaging/rpc/client.py#389 | |
| 15:17:29 | bauzas | (joking) | |
| 15:17:41 | dansmith | mmhmm :) | |
| 15:17:42 | bauzas | every single instanciation raises a warning | |
| 15:17:47 | bauzas | lovely | |
| 15:17:57 | dansmith | yeah, which we apparently do A LOT | |
| 15:18:28 | bauzas | I wasn't expecting it | |
| 15:18:45 | bauzas | but I guess those are for tests | |
| 15:18:48 | dansmith | 15,664 times in just one n-api log run | |
| 15:18:54 | bauzas | https://review.opendev.org/c/openstack/nova/+/869900/6/nova/tests/fixtures/nova.py | |
| 15:19:00 | dansmith | search RPC in here: https://391a5777f99d615e9bdd-6109476be9a7a65d8252c7a651ade8fd.ssl.cf1.rackcdn.com/863919/6/check/tempest-integrated-compute/6af4a31/controller/logs/screen-n-api.txt | |
| 15:19:08 | dansmith | we log it even in production, tens of thousands of times | |
| 15:19:08 | bauzas | woah | |
| 15:19:24 | dansmith | and by "we" I mean "we on behalf of o.msg" | |
| 15:19:34 | sean-k-mooney | so any reason fro me to not +2w this | |
| 15:19:35 | bauzas | oh wait | |
| 15:19:38 | sean-k-mooney | it looks ok to me | |
| 15:19:42 | bauzas | this isn't a singleton | |
| 15:19:44 | sean-k-mooney | it has the requiremtn bump | |
| 15:20:02 | sean-k-mooney | ci will kick it back if it fails again on the recheck | |
| 15:20:02 | dansmith | https://imgur.com/a/0Vvw1ss | |
| 15:20:04 | bauzas | sean-k-mooney: sent to the gate too, for the gosh sake we could merge it eventually | |
| 15:20:22 | dansmith | note the scroll bar showing all the matches | |
| 15:20:28 | bauzas | dansmith: yeah, look at https://review.opendev.org/c/openstack/nova/+/869900/6/nova/rpc.py | |
| 15:20:38 | sean-k-mooney | what was the nova-next failure | |
| 15:20:39 | dansmith | yup | |
| 15:20:42 | bauzas | everytime we call get_client, we instantiate a RPCService | |
| 15:20:50 | dansmith | yup | |
| 15:21:22 | bauzas | I would have preferred this being a singleton, but meh noxw | |
| 15:21:23 | bauzas | now | |
| 15:21:28 | dansmith | well, | |
| 15:21:35 | gmann | dansmith: bauzas dansmith : for functional test testing placement new defaults. I will switch it once placement new defaults are merged, otherwise we need to add system scope token to make placement exiting defaults - https://review.opendev.org/c/openstack/placement/+/865618 | |
| 15:21:39 | dansmith | it can't always be because of cell stuff so we need to pool | |
| 15:21:51 | bauzas | dansmith: ah true | |
| 15:22:01 | bauzas | gmann: wfm | |
| 15:23:11 | gmann | dansmith: bauzas: sean-k-mooney: as you were on rbac things, can you check this which was missed in original change. keeping legacy admin for 'os-tenant-networks' policy https://review.opendev.org/c/openstack/nova/+/865071 | |
| 15:25:04 | sean-k-mooney | oh thats projec treader of gloabl admin | |
| 15:25:15 | sean-k-mooney | where as the curren tbehavior woudl requrie you to have admin on the current project | |
| 15:26:01 | gmann | yeah keeping legacy admin unimpacted | |
| 15:26:15 | sean-k-mooney | i.e. admin implies member implies reader but the project reader will also check the project in your token matches | |
| 15:27:09 | sean-k-mooney | i guess we dont have any tempest tests covering that | |
| 15:27:37 | sean-k-mooney | or it would have blocked eitehr enablign the new default or where we broke it | |
| 15:31:54 | gmann | sean-k-mooney: this ADMIN is just role:admin not just project admin. so rule is "role:admin or project-reader" where we do not check project_id for admin role token | |
| 15:45:48 | sean-k-mooney | yep that is what i was expecting | |
| 16:47:32 | opendevreview | Sofia Enriquez proposed openstack/nova master: WIP: Implement encryption on backingStore https://review.opendev.org/c/openstack/nova/+/870012 | |
| 17:03:34 | sean-k-mooney | i guess sofia has got time to start on ^ again but that spec is not appvoed for this cycle | |
| 17:15:34 | sean-k-mooney | on a related note to the other warning it looks like we are spaming the functionl logs with an sqlachemy deprecation warning too | |
| 17:15:43 | sean-k-mooney | https://paste.opendev.org/show/bWlWeI8pWyiwWyvJ7NtC/ ^ dansmith | |
| 17:16:18 | sean-k-mooney | im not seeign a nova line there | |
| 17:16:28 | sean-k-mooney | so i guess this needs to be fixed in oslo_db | |
| 17:25:27 | dansmith | sean-k-mooney: yeah I've seen that too | |
| 17:25:40 | dansmith | it's annoying, just less annoying :) | |
| 17:26:09 | sean-k-mooney | ya os i just mentioned it on the oslo channel and look at code search | |
| 17:26:25 | sean-k-mooney | other then in some puppet code this is off everywhere | |
| 17:26:38 | sean-k-mooney | so i think oslo.db just need to drop the kwarg and do a release | |
| 17:27:05 | kashyap | Is this Oslo DB error know issue in the "nova-tox-functional-py38" job? | |
| 17:27:22 | sean-k-mooney | its just a log message | |
| 17:27:26 | sean-k-mooney | its not breakign anything | |
| 17:27:35 | kashyap | {0} nova.tests.functional.libvirt.test_vgpu.VGPUTests.test_resize_servers_with_vgpu [6.373304s] ... FAILED() | |
| 17:27:36 | sean-k-mooney | but it makes runnign the test more annrying | |
| 17:27:46 | kashyap | (From here: https://zuul.opendev.org/t/openstack/build/a229b41daba64b6f8dfdeca8c839e9f7) | |
| 17:28:35 | sean-k-mooney | thats a diffent oslo.db thing then i was talkign about | |
| 17:29:19 | sean-k-mooney | kashyap: that actully looks like a real bug | |
| 17:29:32 | kashyap | Again: it was not hit in the previous 3 runs :-( | |
| 17:29:36 | sean-k-mooney | im not sure why we woudl get a db conflict like that in a fucntional test | |
| 17:30:16 | kashyap | sean-k-mooney: But I agree - it "looks" on the surface like a real bug, but I'm not confident if it's _really_ a DB conflict, or a PEBKAC in the test or ... env snafu | |