Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-10-06
12:04:36 gibi this helps If True, when limiting allocation candidate results, the results will be a random sampling of the full result set.
12:04:39 gibi sorry
12:04:41 gibi wrong buffer
12:04:47 gibi this helps https://review.opendev.org/c/openstack/placement/+/963052/1/placement/tests/functional/test_allocation_candidates.py#265
12:05:08 gibi this does not really help https://review.opendev.org/c/openstack/placement/+/963053/2/placement/tests/functional/test_allocation_candidates.py
12:05:10 sean-k-mooney gibi: so inlining _consolidate_allocation_requests helps but not exceeds_capacity
12:05:14 gibi yep
12:06:30 sean-k-mooney i see form the comment you have been trying to use some profiles to help with this, have you found that useful
12:06:41 gibi I will not trying to inline more, the profiler shows that the time is spent on hashing and eqality checking the building blocks but I don't see easy moves there
12:07:02 gibi I used just the built in cProfiler
12:07:09 gibi as you can see in the commented out code
12:07:22 sean-k-mooney ya i was just wondering if the output was useful
12:07:32 sean-k-mooney i have never used that before hence the question
12:08:04 gibi it tells you what called how much and what time it took so it shows you where your spending your execution time
12:08:43 gibi it showed that consolidate is more expensive than exeeced_capacity
12:08:57 sean-k-mooney ack, i have used flamegraph profilers in other lanaguges in the past just never really used those types of tools much with openstack because they hate eventlets
12:09:13 sean-k-mooney placement fortunetly does not use eventlet so that is not an issue
12:09:22 gibi yeah, I just enabled it in athe _merge_candidate call so no parallism was involved
12:09:33 sean-k-mooney but i tried to use them on nova once on the schduler and they were not really that useful when used as external tools
12:10:19 gibi I'm now providing test coverage for the prune patch and we can decide together when dansmith is up that which inlining patch if any we want to take in
12:10:41 sean-k-mooney how do you feel about this overall approch for the current bug. are you happy that it gives us enough breathing room for the current reproted edgecase to move forward with for now?
12:12:23 gibi I think the prune patch already give us time to re-thing the modeling. I think the inlining of the _consoliodate call is still have a good ROI on code complexity. I would at least merge these two optimization. I will then go and update our backlog space about the problem with the new learning and we can discuss on the PTG. My attached customer case probably closable with only the prune patch in
12:12:43 gibi s/space/spec/
12:12:45 sean-k-mooney im kind of suprised that manual function inlineing has such an effect in a dynmic language like python. what python version were you testing with out of interest?
12:13:15 gibi Python 3.12.10 (main, Apr 9 2025, 04:44:59) [GCC 14.2.0] on linux
12:13:36 sean-k-mooney ack i wonder if the impact is large on say 3.10 or 3.9
12:13:43 gibi note that it wasn't pure inlining, it was actually using the commulative nature of the call to save actual steps
12:14:02 gibi cumulative
12:14:24 sean-k-mooney ah ok i was actully trying to pull up the orginal code to comare whiel we were chatting but i have not fond that yet
12:14:47 sean-k-mooney wel i have not been trying very hard :)
12:15:42 sean-k-mooney oh its in your previous patch
12:15:45 gibi consolidate merges the allocations of the different request groups, with the new product algo this was done on each prefix of those groups. but merging G1 + G2 and G1 + G2 + G3 can be optimized if G1 + G2 is only calculated once not twice
12:15:52 sean-k-mooney that explains why it was not in github https://review.opendev.org/c/openstack/placement/+/962776/5/placement/objects/allocation_candidate.py#850
12:16:34 sean-k-mooney so really your folding the exceeds_capacity logic inter the generation
12:16:45 gibi yes
12:17:09 sean-k-mooney ok ya that make more sense as to why it would have a benifit
12:17:14 gibi because of the result of that check informs what part of the product space can be ignored
12:18:02 gibi i.e. if G1+G2 fails the check then G1+G2+Gx+...Gy will awas fail the check so they can be ignored
12:18:29 sean-k-mooney right so we might be able to not inline the fucntion if we return more info beyond True or False like the atribute that exceed capsity
12:19:51 sean-k-mooney i dont really object to the inlineing DRY is nice but im ok wiht specific code where we have a perfroamce need for it
12:20:24 sean-k-mooney im just wonderign if we can get the benift of a testibale function while also achiving the speed up if we adjust the return value to provdie more context
12:21:52 sean-k-mooney looking at https://github.com/openstack/placement/blob/master/placement/objects/research_context.py#L345
12:22:30 sean-k-mooney the behvior is differnt enough that i dont think it makes sense to reuse exceeds_capacity and modify it as i was suggesting
12:24:41 gibi I will think about making the code cleaner once I have pinned down the behavior with some unit test
12:25:47 sean-k-mooney ack i think we coudl factor https://review.opendev.org/c/openstack/placement/+/963052/1/placement/objects/allocation_candidate.py#746 out into its own helper if we wanted but having its own impleaiton i belive is resonable
13:26:13 opendevreview Julien Le Jeune proposed openstack/nova master: Update start_service() function in test https://review.opendev.org/c/openstack/nova/+/963156
13:34:17 bauzas dansmith: remember when you asked me why my comments were usually having two spaces between the hashtag and the first word ? looks to me this is due to autopep8 which automatically 'fixes' my comments 🤦
13:34:43 dansmith bauzas: I don't, but.. good to know
13:38:37 bauzas still have to confirm, but unless I'm fool, I don't understand why those extra spaces suddently appear on my code while I doublechecked before committing I wasn't having them
13:39:02 bauzas so this is either autopep8 or vscode (was having the same problem before I started using Cursor)
13:39:23 bauzas I was becoming crazy tbh
13:40:12 bauzas I'll do further tests with --no-verify
13:40:21 bauzas so it would skip autopep8
14:03:15 sean-k-mooney bauzas: its from pep8 https://peps.python.org/pep-0008/#inline-comments
14:03:38 sean-k-mooney bauzas: "An inline comment is a comment on the same line as a statement. Inline comments should be separated by at least two spaces from the statement. They should start with a # and a single space."
14:03:55 bauzas I'm not talking of inline statements sorry
14:04:19 sean-k-mooney oh ok. never mind then
14:04:26 bauzas just a whole comment line like # TODO(x) : This is a comment
14:04:57 sean-k-mooney ah well ther eshoudl be 1 space between the # and TODO
14:04:59 bauzas I'm writing "# TODO" and eventually it commits "#  TODO" with two spaces
14:05:23 sean-k-mooney hum that is odd indeed
14:05:44 sean-k-mooney i have not seen that but that could be the agressive option. although im not sure which guidelien would be influcancing that
14:06:16 sean-k-mooney the change to drop agressive=3 is still open and mergable so if ye want to proceed with that do
14:06:39 sean-k-mooney bauzas: do you happen to have a linke to the code that was formated?
14:07:04 bauzas that's always when I write a new comment line
14:07:11 sean-k-mooney i woudl be interested in seign the example. given we did not have to reformat any code when i enabeld that i was really not expectign it to have such an effect
14:07:15 bauzas maybe this is just a codium issue
14:07:24 sean-k-mooney ya maybe
14:07:40 bauzas but everytime I need to doublecheck what I commit, that's super cumbersome
14:07:45 sean-k-mooney you did have an issue with codium adding a nobreaking space at one point
14:08:31 bauzas honestly, I need time to investigate what's wrong but I'm rushing into delivering a fix for https://bugs.launchpad.net/nova/+bug/2125935
14:09:15 bauzas so I'll check that later
14:09:26 sean-k-mooney ack
14:41:07 opendevreview Balazs Gibizer proposed openstack/placement master: Prune a_c search space by invalid prefixes https://review.opendev.org/c/openstack/placement/+/962776
14:41:07 opendevreview Balazs Gibizer proposed openstack/placement master: Inline _consolidate_allocation_requests https://review.opendev.org/c/openstack/placement/+/963052
14:41:08 opendevreview Balazs Gibizer proposed openstack/placement master: Inline exceeds_capacity https://review.opendev.org/c/openstack/placement/+/963053
14:42:02 gibi dansmith: this is how unit test for the new product algo will look like https://review.opendev.org/c/openstack/placement/+/962776/5
14:48:25 gibi https://review.opendev.org/c/openstack/placement/+/962776/6/placement/tests/unit/objects/test_allocation_candidate.py
14:48:27 dansmith gibi: okay.. that will require some study as well
14:49:34 gibi no worries. I have plenty of quirks to debug now that the unit test show some strange steps the algo makes
14:56:08 dansmith gibi: so all those "bug duplicate" ones are things it should have filtered?
15:03:47 gibi those are steps in the algo that should not be made. (I guess we have a off by one index error somewhere)
15:04:16 gibi somehow we check the same product 3 times instead of one
15:04:54 gibi I will figure it out why...
15:05:07 gibi don't hang up on that now.
15:05:28 gibi but the test already shows where we skip products so that is what is interesting
15:05:31 gibi for review
15:05:41 opendevreview Julien Le Jeune proposed openstack/nova master: Update start_service() function in test https://review.opendev.org/c/openstack/nova/+/963156
15:08:04 dansmith gibi: yeah, when I first skimmed it I thought you were identifying the things that would be skipped by using yours over the standard product, so just wanted to confirm that it was actually further skipping that needed to happen
15:08:10 dansmith I'm deeper into it now so it's obvious
15:20:25 dansmith gibi: can you respond here: https://review.opendev.org/c/openstack/placement/+/962776/comment/05b0ce6d_48710288/ ?
15:20:48 dansmith I'm trying to grok the test first, so maybe I'm being stupid
15:26:42 bauzas dansmith: fwiw, fill_metadata() seems non deterministic when you use a wrong context
15:27:29 bauzas ie. in my reproducer, some host with two instances has one with metadata set while the other not
15:28:05 dansmith bauzas: I dunno why you think that's possible :D
15:28:14 bauzas I know my patch fixes the whole problem but I'm puzzled why I'm still getting metadata only for one (and that could explain why your UT didn't catch the failure)
15:28:22 dansmith bauzas: I think it's more likely that you're using a shared context and it's being mutated in a way that seems like it's changing
15:28:33 bauzas yeah, that's what I thought
15:29:01 bauzas I'm passing the same context object for both instances so this probably was mutate
15:29:04 bauzas mutated

Earlier   Later