Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-05
09:58:19 sean-k-mooney[m] there are 13 chose 8 combinations
09:58:39 sean-k-mooney[m] that 1287
09:58:46 gibi that is not that much
09:58:55 gibi but if there is 1000 computes
09:58:59 sean-k-mooney[m] 1287*1000
09:58:59 gibi or just 100
09:59:07 gibi then that is sizeable
09:59:16 sean-k-mooney[m] ya it will grow quickly
10:00:03 sean-k-mooney[m] sorry no
10:00:18 songwenping_ there are extra 8 computes without gpu.
10:00:22 sean-k-mooney[m] its 13 hosts each with 8 gpus and the vm is asking for 8
10:00:53 sean-k-mooney[m] so it wont explode like that there is only 1 allocation pooible per host
10:01:01 sean-k-mooney[m] *possible
10:01:01 gibi nope
10:01:08 gibi if you have 8 groups and 8 gpus
10:01:19 gibi then each group can be satisfied by each gpu
10:01:25 sean-k-mooney[m] you think it would b n squared
10:01:41 sean-k-mooney[m] oh hum maybe
10:01:48 gibi 8!
10:02:48 gibi 40320
10:03:00 sean-k-mooney[m] ya… that would be bad
10:03:33 sean-k-mooney[m] i hope thats not whats happening or that will cause issue for pci devices in placemnt too
10:03:43 gibi this is coming from the fact that placement treats groups and RPs individually. even if we have very similar RPs and very similar groups
10:04:02 gibi I can make a test case for 8 PCI devs :)
10:04:09 gibi in nova func test
10:04:12 gibi so we can prove it
10:04:32 sean-k-mooney[m] perhaps start with 4
10:05:08 sean-k-mooney[m] i mean if its a gabbit test then sure 8
10:05:13 gibi ack. first 1 will go and try to stabilize the fair lock unit tests then I will look at the 4-8 PCI issue
10:05:45 sean-k-mooney[m] we really need to not do all possible permuations on the placement side
10:06:11 gibi if this is real factorial then we need to introduce an per host limit for a_c query
10:06:28 sean-k-mooney[m] ya i was debating if that was the only option
10:06:33 gibi if the groups are different by trait then we need all permutations
10:06:51 gibi otherwise we might not found the right one
10:07:16 sean-k-mooney[m] well an allocation candiate by defieniton meets the requirements
10:07:16 gibi the trick here is that we have identical groups and identicaly PRs
10:07:23 sean-k-mooney[m] so it depend on whyer this is happening
10:07:48 gibi to find a_c placement needs to iterate permutations I think in the general case
10:07:51 sean-k-mooney[m] anyway something to look into i guess
10:07:55 gibi yepp
10:08:36 sean-k-mooney[m] im going to grab a coffee and take my blood pressre meds and then check on freya be back in about 10 mins
10:09:02 gibi ack
10:09:22 gibi for me coffee is the blood pressure med
10:32:32 sean-k-mooney gibi: should fastener reintoudce the workaround they had
10:35:20 gibi that is a way too but I have less authority over that project
10:35:51 sean-k-mooney isnit it an oslo deliverable too
10:36:09 sean-k-mooney or has it move out of openstack
10:36:56 sean-k-mooney oh its not an openstack project
10:37:02 sean-k-mooney i tough it used to be at one point
10:38:21 sean-k-mooney i guess we can fix it in oslo but we should leth the know that under eventlet the rentrant guarentee is broken
10:38:25 sean-k-mooney https://github.com/harlowja/fasteners#-overview
10:38:35 sean-k-mooney then note that it should be reentrant
10:44:17 sean-k-mooney gibi: https://github.com/harlowja/fasteners/issues/86
10:44:17 sean-k-mooney that also intereisng
10:44:17 sean-k-mooney https://github.com/harlowja/fasteners/pull/87/files
10:44:17 sean-k-mooney elif not self.has_pending_writers:
10:44:17 sean-k-mooney elif (self._writer == me) or not self.has_pending_writers:
10:49:45 gibi I'm not sure I follow how this connects to your current problem.
10:49:52 gibi our
10:50:16 sean-k-mooney its relying on threading.current_thread
10:50:35 sean-k-mooney an now it allows you to reaquire the lock if tha tis the same
10:50:47 sean-k-mooney but with spwan_n
10:51:00 sean-k-mooney that means two greenthread coudl get the same lock if they run on the same os thread
10:51:17 gibi yes, ReaderWriterLock rely on current_thread for reentrancy
10:52:01 sean-k-mooney this was changed in januaary
10:52:16 sean-k-mooney could you try downgrading fastener to 0.17.2
10:52:30 gibi but the fact that ReaderWriterLock depends on current_thread was not introduced there, it was there before https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L169
10:52:32 sean-k-mooney gibi: it did not used to in the reader writer case
10:53:04 sean-k-mooney right but it used to prevent geting the reader lock if there were any writers
10:53:46 gibi olso depends on the writer lock it seems https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L288
10:53:56 gibi and the writer part had reentrancy before 0.17.2
10:54:34 sean-k-mooney im oging to try your oslo repoducer and downgrade it just to see
10:54:49 gibi and writer lock is affected independently from the 0.17.2 https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L208
10:56:36 sean-k-mooney synconise is takign a write_lock ya?
10:56:44 sean-k-mooney if so then its not related to that change
10:57:00 sean-k-mooney but the is_writer code is not eventlet safe
10:57:09 sean-k-mooney likely becuase of the workaround you mentioned they remvoed
10:57:51 sean-k-mooney https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913
10:58:31 sean-k-mooney so 0.15 dropped that
11:00:12 gibi yes it is broken since 0.15
11:01:10 gibi it is effecting master and yoga
11:01:15 gibi in xena we have < 0.15
11:12:53 sean-k-mooney https://github.com/harlowja/fasteners/issues/96
11:13:20 sean-k-mooney at leasst they can triage ^ and decied if its somethign they want to fix
11:20:23 gibi thanks
11:33:01 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
11:33:24 gibi added the fasteners issue link and hopefully stabilized the unit test ^^
11:34:00 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
11:34:20 sean-k-mooney im going to propose two reverts to fasteneres and assocaite it with the issue link too incase they deciced that that is approicate
11:35:30 gibi ack
11:40:33 sean-k-mooney https://github.com/harlowja/fasteners/pull/97
11:47:10 gibi thanks
13:00:34 opendevreview Balazs Gibizer proposed openstack/nova master: Show candidate combinatorial explosion by dev number https://review.opendev.org/c/openstack/nova/+/855885
13:00:46 gibi sean-k-mooney: here are the numbers of the combinatorial explosion
13:00:48 gibi ^^
13:01:16 gibi in case of single device per RP (ie PCI or PF) we have worst case factorial amount of candidates
13:01:48 gibi in case a single RP provides more than one devices (n VFs for a PF RP) then worst case we have exponential candidates
13:04:09 gibi and placement first generate all of them then limit the result based on the limit queryparam https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/research_context.py#L289-L292
13:13:19 gibi so this probably needs a bit of (probably massive) refactoring if we want to avoid placement to blow up on 8 devices
13:13:34 gibi we need to inline the limit somehow

Earlier   Later