Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-05
09:59:16 sean-k-mooney[m] ya it will grow quickly
10:00:03 sean-k-mooney[m] sorry no
10:00:18 songwenping_ there are extra 8 computes without gpu.
10:00:22 sean-k-mooney[m] its 13 hosts each with 8 gpus and the vm is asking for 8
10:00:53 sean-k-mooney[m] so it wont explode like that there is only 1 allocation pooible per host
10:01:01 sean-k-mooney[m] *possible
10:01:01 gibi nope
10:01:08 gibi if you have 8 groups and 8 gpus
10:01:19 gibi then each group can be satisfied by each gpu
10:01:25 sean-k-mooney[m] you think it would b n squared
10:01:41 sean-k-mooney[m] oh hum maybe
10:01:48 gibi 8!
10:02:48 gibi 40320
10:03:00 sean-k-mooney[m] ya… that would be bad
10:03:33 sean-k-mooney[m] i hope thats not whats happening or that will cause issue for pci devices in placemnt too
10:03:43 gibi this is coming from the fact that placement treats groups and RPs individually. even if we have very similar RPs and very similar groups
10:04:02 gibi I can make a test case for 8 PCI devs :)
10:04:09 gibi in nova func test
10:04:12 gibi so we can prove it
10:04:32 sean-k-mooney[m] perhaps start with 4
10:05:08 sean-k-mooney[m] i mean if its a gabbit test then sure 8
10:05:13 gibi ack. first 1 will go and try to stabilize the fair lock unit tests then I will look at the 4-8 PCI issue
10:05:45 sean-k-mooney[m] we really need to not do all possible permuations on the placement side
10:06:11 gibi if this is real factorial then we need to introduce an per host limit for a_c query
10:06:28 sean-k-mooney[m] ya i was debating if that was the only option
10:06:33 gibi if the groups are different by trait then we need all permutations
10:06:51 gibi otherwise we might not found the right one
10:07:16 sean-k-mooney[m] well an allocation candiate by defieniton meets the requirements
10:07:16 gibi the trick here is that we have identical groups and identicaly PRs
10:07:23 sean-k-mooney[m] so it depend on whyer this is happening
10:07:48 gibi to find a_c placement needs to iterate permutations I think in the general case
10:07:51 sean-k-mooney[m] anyway something to look into i guess
10:07:55 gibi yepp
10:08:36 sean-k-mooney[m] im going to grab a coffee and take my blood pressre meds and then check on freya be back in about 10 mins
10:09:02 gibi ack
10:09:22 gibi for me coffee is the blood pressure med
10:32:32 sean-k-mooney gibi: should fastener reintoudce the workaround they had
10:35:20 gibi that is a way too but I have less authority over that project
10:35:51 sean-k-mooney isnit it an oslo deliverable too
10:36:09 sean-k-mooney or has it move out of openstack
10:36:56 sean-k-mooney oh its not an openstack project
10:37:02 sean-k-mooney i tough it used to be at one point
10:38:21 sean-k-mooney i guess we can fix it in oslo but we should leth the know that under eventlet the rentrant guarentee is broken
10:38:25 sean-k-mooney https://github.com/harlowja/fasteners#-overview
10:38:35 sean-k-mooney then note that it should be reentrant
10:44:17 sean-k-mooney gibi: https://github.com/harlowja/fasteners/issues/86
10:44:17 sean-k-mooney that also intereisng
10:44:17 sean-k-mooney https://github.com/harlowja/fasteners/pull/87/files
10:44:17 sean-k-mooney elif not self.has_pending_writers:
10:44:17 sean-k-mooney elif (self._writer == me) or not self.has_pending_writers:
10:49:45 gibi I'm not sure I follow how this connects to your current problem.
10:49:52 gibi our
10:50:16 sean-k-mooney its relying on threading.current_thread
10:50:35 sean-k-mooney an now it allows you to reaquire the lock if tha tis the same
10:50:47 sean-k-mooney but with spwan_n
10:51:00 sean-k-mooney that means two greenthread coudl get the same lock if they run on the same os thread
10:51:17 gibi yes, ReaderWriterLock rely on current_thread for reentrancy
10:52:01 sean-k-mooney this was changed in januaary
10:52:16 sean-k-mooney could you try downgrading fastener to 0.17.2
10:52:30 gibi but the fact that ReaderWriterLock depends on current_thread was not introduced there, it was there before https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L169
10:52:32 sean-k-mooney gibi: it did not used to in the reader writer case
10:53:04 sean-k-mooney right but it used to prevent geting the reader lock if there were any writers
10:53:46 gibi olso depends on the writer lock it seems https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L288
10:53:56 gibi and the writer part had reentrancy before 0.17.2
10:54:34 sean-k-mooney im oging to try your oslo repoducer and downgrade it just to see
10:54:49 gibi and writer lock is affected independently from the 0.17.2 https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L208
10:56:36 sean-k-mooney synconise is takign a write_lock ya?
10:56:44 sean-k-mooney if so then its not related to that change
10:57:00 sean-k-mooney but the is_writer code is not eventlet safe
10:57:09 sean-k-mooney likely becuase of the workaround you mentioned they remvoed
10:57:51 sean-k-mooney https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913
10:58:31 sean-k-mooney so 0.15 dropped that
11:00:12 gibi yes it is broken since 0.15
11:01:10 gibi it is effecting master and yoga
11:01:15 gibi in xena we have < 0.15
11:12:53 sean-k-mooney https://github.com/harlowja/fasteners/issues/96
11:13:20 sean-k-mooney at leasst they can triage ^ and decied if its somethign they want to fix
11:20:23 gibi thanks
11:33:01 opendevreview Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717
11:33:24 gibi added the fasteners issue link and hopefully stabilized the unit test ^^
11:34:00 opendevreview Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718
11:34:20 sean-k-mooney im going to propose two reverts to fasteneres and assocaite it with the issue link too incase they deciced that that is approicate
11:35:30 gibi ack
11:40:33 sean-k-mooney https://github.com/harlowja/fasteners/pull/97
11:47:10 gibi thanks
13:00:34 opendevreview Balazs Gibizer proposed openstack/nova master: Show candidate combinatorial explosion by dev number https://review.opendev.org/c/openstack/nova/+/855885
13:00:46 gibi sean-k-mooney: here are the numbers of the combinatorial explosion
13:00:48 gibi ^^
13:01:16 gibi in case of single device per RP (ie PCI or PF) we have worst case factorial amount of candidates
13:01:48 gibi in case a single RP provides more than one devices (n VFs for a PF RP) then worst case we have exponential candidates
13:04:09 gibi and placement first generate all of them then limit the result based on the limit queryparam https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/research_context.py#L289-L292
13:13:19 gibi so this probably needs a bit of (probably massive) refactoring if we want to avoid placement to blow up on 8 devices
13:13:34 gibi we need to inline the limit somehow
13:15:45 gibi but by that we would potentially loose viable candidates
13:16:12 gibi so placement alone cannot decide where to limiot
13:23:49 gibi So modeling similar PFs of PCI devs does not help as it would lead to the VF scenario.
13:24:39 gibi Also we cannot model count=n as a single group as placement never splits a suffixed group to fit it into multiple RP
13:26:31 gibi while for nova it would be enough to have a small number of candidates per compute host, while we have nova side PCI filtering we need all candidates from placement as we don't know which will fulfill the nova side filtering
13:44:29 sean-k-mooney gibi: ya so this is exactly why we did not want each VF to ba an RP
13:44:40 sean-k-mooney we were very concerned it would explode like this

Earlier   Later