| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-03-25 | |||
| 14:31:58 | dansmith | I removed all those fixture setups in gabbits.py and no change | |
| 14:32:25 | dansmith | right, I'm sure it's the attempt to make it pretty, but it has made it completely un-debuggable, AFAICT | |
| 14:32:28 | sean-k-mooney | i proably shoudl be doing somethign else bug you peaked my interest | |
| 14:32:44 | sean-k-mooney | do you have a patch i can pully or aplly and see if i can replicate it | |
| 14:33:18 | dansmith | I can push it up | |
| 14:34:44 | opendevreview | Dan Smith proposed openstack/placement master: Reproduce bug 2104040: allocate in over-capacity https://review.opendev.org/c/openstack/placement/+/945464 | |
| 14:34:45 | opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | |
| 14:34:54 | dansmith | sean-k-mooney: ^ | |
| 14:35:09 | sean-k-mooney | the secodn one is causign the 500 ya | |
| 14:35:16 | dansmith | maybe the actual thing failing will get hit by a tempest test and I can see what the effing problem is :) | |
| 14:35:17 | dansmith | yeah | |
| 14:36:49 | sean-k-mooney | well placmeent does not use eventlet so in theory i can also run this under a debugger in vscoded and set a breakpoint | |
| 14:37:14 | sean-k-mooney | i have never tried that with placment but i have got that to work wiht some of watchers unit tests | |
| 14:37:19 | dansmith | if I knew where to set a breakpoint I could do the same :) | |
| 14:37:44 | sean-k-mooney | you can make the debugger tirgger on uncaught excetpions | |
| 14:38:18 | sean-k-mooney | im going ot make sure i can repoduce the fialure first and then ill try something more fancy | |
| 14:39:48 | dansmith | thanks, maybe I've got tunnel vision on something | |
| 14:44:23 | sean-k-mooney | ... we still have not merget the patch to move form psycopg2 to psycopg2-binary... | |
| 14:44:32 | sean-k-mooney | oh we didnt propose that ot placment | |
| 14:44:36 | sean-k-mooney | ill fix that later | |
| 14:45:10 | sean-k-mooney | if we use psycopg2-binary instead we dont need the postgress development headers to be able to compile it | |
| 14:47:41 | sean-k-mooney | dansmith: so i ran the functional tests and everything passed | |
| 14:47:52 | sean-k-mooney | oh nevermind | |
| 14:47:59 | sean-k-mooney | that with master i have not applied you change yet | |
| 14:48:39 | sean-k-mooney | i did see "TransactionFactory already started, not reconfiguring." pirnted once or twice which is a bit suss but it does nto seam to be breaking anything | |
| 14:49:07 | dansmith | master passes for me too :) | |
| 14:49:40 | dansmith | I think I've just guessed the thing it's failing on by trial and error, which is good, but it's like debugging in the dark | |
| 14:50:35 | sean-k-mooney | so i have output | |
| 14:50:41 | sean-k-mooney | and two failed testcases | |
| 14:51:01 | sean-k-mooney | https://paste.opendev.org/show/baZsKMeiO9mOxyGz2OlY/ | |
| 14:51:11 | dansmith | ...right | |
| 14:51:15 | dansmith | that's just gabbit output | |
| 14:51:30 | dansmith | not the logging from the wsgi app to show you where the 500 was | |
| 14:51:38 | sean-k-mooney | ah you want the api logs form the service right | |
| 14:51:51 | dansmith | yeah, so I can see, you know, the *actual* problem | |
| 14:53:15 | sean-k-mooney | right well now that i know which tests are failing i can see if i can get more output sicn ei an just target those specific tests | |
| 15:05:29 | Uggla | sean-k-mooney, do you think we should have a cross team nova-watcher session at the PTG ? | |
| 15:05:45 | sean-k-mooney | sigh... the way we dynmically load teh gabit tst means that they are not discoverd wehn we use pytest and the way we alias the global fixtures module with our own breaks unittests discoveray | |
| 15:06:59 | sean-k-mooney | Uggla: am we could the topics i wanted to raise are not only relvent to watcher but it does not need to be super formal | |
| 15:07:05 | dansmith | sean-k-mooney: yep, already tried that, but you can use testtools directly, just not for a specific test. you have to run all the gabbit tests | |
| 15:07:46 | sean-k-mooney | did you try the stestr pdb supprot? | |
| 15:10:15 | sean-k-mooney | ... AttributeError: module 'placement.tests.functional.test_api' has no attribute 'allocations-bug-1778743_post_multiple_allocations' | |
| 15:10:22 | sean-k-mooney | right | |
| 15:11:09 | Uggla | sean-k-mooney, ok so I will not create a specific time slot only for watcher and will "distribute" the topic in the agenda where it fits best. Is that ok ? | |
| 15:11:26 | sean-k-mooney | sounds good | |
| 15:12:21 | sean-k-mooney | dansmith: so passing --pdb to stestr didnt help either how fun | |
| 15:13:32 | dansmith | lovely | |
| 15:16:14 | opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | |
| 15:16:37 | sean-k-mooney | dansmith: https://github.com/cdent/gabbi/commit/2469b1617fde2ee62f42ef48fc38a7d075c1dd62#diff-af026a8f8ce5b1d6bf2c2b9a33c71ba0566feb828ca0fbc8ec4f3970c2835c75 | |
| 15:16:46 | sean-k-mooney | appreently you can mark a test as verbose | |
| 15:17:14 | dansmith | and that lets logs go? | |
| 15:19:20 | sean-k-mooney | not exactly https://paste.opendev.org/show/bI2SFFiDmfx51VXu7CrD/ | |
| 15:19:36 | sean-k-mooney | you do get more output | |
| 15:19:54 | dansmith | unhelpful output | |
| 15:20:19 | sean-k-mooney | its jsut increasing the verbosity of the client | |
| 15:26:56 | dansmith | yeah | |
| 15:33:59 | sean-k-mooney | ok i have logs but i need to change code | |
| 15:34:08 | sean-k-mooney | ill make this configuable in a env var | |
| 15:34:46 | sean-k-mooney | tldr set this to false https://github.com/openstack/placement/blob/master/placement/tests/functional/fixtures/capture.py#L55 | |
| 15:35:12 | dansmith | WAT | |
| 15:35:16 | dansmith | set to false? | |
| 15:35:35 | sean-k-mooney | yep to turn off the log capture and allow it to print to stderr | |
| 15:35:36 | dansmith | I have commented out the usage of that thing entirely and I still don't get logs | |
| 15:36:09 | dansmith | do you also have to disable the stdio capture? | |
| 15:36:10 | sean-k-mooney | i also invoked it liek this wiht a few other changes | |
| 15:36:12 | sean-k-mooney | OS_DEBUG=1 OS_LOG_CAPTURE=1 tox -e functional -- --failing | |
| 15:36:43 | dansmith | okay are you going to push up a change so I can see what all? | |
| 15:36:49 | sean-k-mooney | yep | |
| 15:37:03 | dansmith | aight | |
| 15:37:22 | sean-k-mooney | i mean its just this https://termbin.com/gj5q | |
| 15:37:39 | sean-k-mooney | but not all of that is relevent | |
| 15:38:11 | sean-k-mooney | ill past you the output so you can review | |
| 15:38:19 | sean-k-mooney | then ill figure out what bits actully matter | |
| 15:39:44 | sean-k-mooney | https://termbin.com/zacx | |
| 15:40:40 | sean-k-mooney | File "/home/smooney/repos/placement/placement/objects/allocation.py", line 262, in _check_capacity_exceeded | |
| 15:40:41 | sean-k-mooney | prev_used = prev_usage[alloc.resource_provider.id][rc_id] | |
| 15:40:43 | opendevreview | Dan Smith proposed openstack/placement master: Reproduce bug 2104040: allocate in over-capacity https://review.opendev.org/c/openstack/placement/+/945464 | |
| 15:40:43 | opendevreview | Dan Smith proposed openstack/placement master: Fix placement allocate while over-capacity https://review.opendev.org/c/openstack/placement/+/945465 | |
| 15:40:43 | sean-k-mooney | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ | |
| 15:40:45 | sean-k-mooney | KeyError: 0 | |
| 15:41:34 | dansmith | OS_LOG_CAPTURE=1 sets capture_logs=True on oslo right? | |
| 15:41:55 | sean-k-mooney | i need to see if thats actully needed but proably | |
| 15:42:17 | dansmith | very confusing if we need to set that on the parent *and* =False on the child.. I suspect one of those and not the other is to blame | |
| 15:42:36 | sean-k-mooney | ok so that is not relevnet | |
| 15:42:42 | sean-k-mooney | i can set it to 0 and it still work | |
| 15:42:48 | sean-k-mooney | i was gong to use that to contole it however | |
| 15:43:27 | dansmith | okay, isn't that capture_logs flag telling oslo to capture the logs in memory so they can be reported with the test failure? | |
| 15:43:29 | sean-k-mooney | so instead of hardcoding this to true defualt that to 0 or 1 i dont know what makes the most sense and allow you to override it | |
| 15:43:43 | dansmith | I'm not sure why we'd ever want that to be false unless the reporting is wrong | |
| 15:45:27 | sean-k-mooney | i need to read the related code to answer properly but this is the relevnet code https://github.com/openstack/oslotest/blob/1fe445863e124f8a10c1496fb8244c745e546b21/oslotest/log.py#L35-L91 | |
| 15:46:13 | sean-k-mooney | this is why i have not pushed a patch yet im trying to load context and then figurout what the correct behavior should be before commiting and pushing | |
| 15:47:36 | Uggla | nova meeting in aound 15mn. | |
| 15:48:03 | sean-k-mooney | dansmith: by hardcodign it to ture we were alsways addign the fake loonger | |
| 15:48:25 | sean-k-mooney | which make shte logs aviable in the tests | |
| 15:48:37 | dansmith | right, but that's the intent so it gets reported to the test runner so the runner can report the test fail with logs right? | |
| 15:48:39 | sean-k-mooney | btu im not sure if that will attach them to the tests to print on output | |
| 15:48:47 | dansmith | yeah, idk | |
| 15:49:03 | dansmith | so many layers and years of crust, I'm not even sure how it's supposed to work | |
| 15:49:20 | sean-k-mooney | since we know what to flip we can easly expose that bool as an env var in any case | |
| 15:51:43 | sean-k-mooney | dansmith: so its wrapping a logging fixture form fixtures and nuke_handeler is set to true | |