Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-09-05
09:55:05 sean-k-mooney[m] those are not tracked in placment then
09:55:18 auniyal_ Hi
09:55:23 auniyal_ please review these
09:55:25 auniyal_ https://review.opendev.org/c/openstack/nova/+/854980
09:55:25 auniyal_ https://review.opendev.org/c/openstack/nova/+/854979
09:55:25 auniyal_ backporting
09:55:25 auniyal_ https://review.opendev.org/c/openstack/nova/+/854499
09:55:25 auniyal_ https://review.opendev.org/c/openstack/nova/+/852171
09:56:17 sean-k-mooney[m] songwenping_: on master we now can track pci devics in placment but in any other release pci devices are not tracked in placment
09:56:34 songwenping_ _ID_1DB6&required6=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&resources=MEMORY_MB%3A32%2CVCPU%3A1&resources1=PGPU%3A1&resources2=PGPU%3A1&resources3=PGPU%3A1&resources4=PGPU%3A1&resources5=PGPU%3A1&resources6=PGPU%3A1" -H "Accept: application/json" -H "OpenStack-API-Version: placement 1.29" -H "User-Agent: openstacksdk/0.99.0 keystoneauth1/4.6.0 python-requests/2.27.1 CPython/3.8.10" -H "X-Auth-Token: gAAAAABjFbkj8nz24Q7A6J0qmjpdZHfWM
09:56:34 songwenping_ sean-k-mooney[m]: we use cyborg to manage pgpu, the request url is :curl -g -i -X GET "http://10.7.20.73/placement/allocation_candidates?limit=1000&group_policy=none&required1=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required2=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required3=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required4=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required5=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT
09:56:35 songwenping_ vZidJOT9iYhV2MQCngcYQHhSQmjsGJofkYoT087tAISpf3IniDGwPTHXz_-8x-1nF60WavSYFgEd-5l3_ENrGumaHuU1yfhMJqZu06IR4SXacjA1g6ImSSEfLbfQ9zrPouB0roFokHPmPy3-UpnFZE"
09:56:46 gibi sean-k-mooney[m], sean-k-mooney[m]: is it via cyborg? because then it might be tracked in placement
09:56:53 sean-k-mooney[m] ok
09:58:05 sean-k-mooney[m] in that case perhaps they are hitting the compintorial explosion we were worried about with tracking VFs directly
09:58:11 gibi probably there too many possible candidates
09:58:19 sean-k-mooney[m] there are 13 chose 8 combinations
09:58:39 sean-k-mooney[m] that 1287
09:58:46 gibi that is not that much
09:58:55 gibi but if there is 1000 computes
09:58:59 gibi or just 100
09:58:59 sean-k-mooney[m] 1287*1000
09:59:07 gibi then that is sizeable
09:59:16 sean-k-mooney[m] ya it will grow quickly
10:00:03 sean-k-mooney[m] sorry no
10:00:18 songwenping_ there are extra 8 computes without gpu.
10:00:22 sean-k-mooney[m] its 13 hosts each with 8 gpus and the vm is asking for 8
10:00:53 sean-k-mooney[m] so it wont explode like that there is only 1 allocation pooible per host
10:01:01 gibi nope
10:01:01 sean-k-mooney[m] *possible
10:01:08 gibi if you have 8 groups and 8 gpus
10:01:19 gibi then each group can be satisfied by each gpu
10:01:25 sean-k-mooney[m] you think it would b n squared
10:01:41 sean-k-mooney[m] oh hum maybe
10:01:48 gibi 8!
10:02:48 gibi 40320
10:03:00 sean-k-mooney[m] ya… that would be bad
10:03:33 sean-k-mooney[m] i hope thats not whats happening or that will cause issue for pci devices in placemnt too
10:03:43 gibi this is coming from the fact that placement treats groups and RPs individually. even if we have very similar RPs and very similar groups
10:04:02 gibi I can make a test case for 8 PCI devs :)
10:04:09 gibi in nova func test
10:04:12 gibi so we can prove it
10:04:32 sean-k-mooney[m] perhaps start with 4
10:05:08 sean-k-mooney[m] i mean if its a gabbit test then sure 8
10:05:13 gibi ack. first 1 will go and try to stabilize the fair lock unit tests then I will look at the 4-8 PCI issue
10:05:45 sean-k-mooney[m] we really need to not do all possible permuations on the placement side
10:06:11 gibi if this is real factorial then we need to introduce an per host limit for a_c query
10:06:28 sean-k-mooney[m] ya i was debating if that was the only option
10:06:33 gibi if the groups are different by trait then we need all permutations
10:06:51 gibi otherwise we might not found the right one
10:07:16 gibi the trick here is that we have identical groups and identicaly PRs
10:07:16 sean-k-mooney[m] well an allocation candiate by defieniton meets the requirements
10:07:23 sean-k-mooney[m] so it depend on whyer this is happening
10:07:48 gibi to find a_c placement needs to iterate permutations I think in the general case
10:07:51 sean-k-mooney[m] anyway something to look into i guess
10:07:55 gibi yepp
10:08:36 sean-k-mooney[m] im going to grab a coffee and take my blood pressre meds and then check on freya be back in about 10 mins
10:09:02 gibi ack
10:09:22 gibi for me coffee is the blood pressure med
10:32:32 sean-k-mooney gibi: should fastener reintoudce the workaround they had
10:35:20 gibi that is a way too but I have less authority over that project
10:35:51 sean-k-mooney isnit it an oslo deliverable too
10:36:09 sean-k-mooney or has it move out of openstack
10:36:56 sean-k-mooney oh its not an openstack project
10:37:02 sean-k-mooney i tough it used to be at one point
10:38:21 sean-k-mooney i guess we can fix it in oslo but we should leth the know that under eventlet the rentrant guarentee is broken
10:38:25 sean-k-mooney https://github.com/harlowja/fasteners#-overview
10:38:35 sean-k-mooney then note that it should be reentrant
10:44:17 sean-k-mooney elif (self._writer == me) or not self.has_pending_writers:
10:44:17 sean-k-mooney elif not self.has_pending_writers:
10:44:17 sean-k-mooney https://github.com/harlowja/fasteners/pull/87/files
10:44:17 sean-k-mooney that also intereisng
10:44:17 sean-k-mooney gibi: https://github.com/harlowja/fasteners/issues/86
10:49:45 gibi I'm not sure I follow how this connects to your current problem.
10:49:52 gibi our
10:50:16 sean-k-mooney its relying on threading.current_thread
10:50:35 sean-k-mooney an now it allows you to reaquire the lock if tha tis the same
10:50:47 sean-k-mooney but with spwan_n
10:51:00 sean-k-mooney that means two greenthread coudl get the same lock if they run on the same os thread
10:51:17 gibi yes, ReaderWriterLock rely on current_thread for reentrancy
10:52:01 sean-k-mooney this was changed in januaary
10:52:16 sean-k-mooney could you try downgrading fastener to 0.17.2
10:52:30 gibi but the fact that ReaderWriterLock depends on current_thread was not introduced there, it was there before https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L169
10:52:32 sean-k-mooney gibi: it did not used to in the reader writer case
10:53:04 sean-k-mooney right but it used to prevent geting the reader lock if there were any writers
10:53:46 gibi olso depends on the writer lock it seems https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L288
10:53:56 gibi and the writer part had reentrancy before 0.17.2
10:54:34 sean-k-mooney im oging to try your oslo repoducer and downgrade it just to see
10:54:49 gibi and writer lock is affected independently from the 0.17.2 https://github.com/harlowja/fasteners/pull/87/files#diff-bdd827bd84626190e8a93d1a50782b998b426261511e653de5bb775e9082e1f3L208
10:56:36 sean-k-mooney synconise is takign a write_lock ya?
10:56:44 sean-k-mooney if so then its not related to that change
10:57:00 sean-k-mooney but the is_writer code is not eventlet safe
10:57:09 sean-k-mooney likely becuase of the workaround you mentioned they remvoed
10:57:51 sean-k-mooney https://github.com/harlowja/fasteners/commit/467ed75ee1e9465ebff8b5edf452770befb93913
10:58:31 sean-k-mooney so 0.15 dropped that
11:00:12 gibi yes it is broken since 0.15
11:01:10 gibi it is effecting master and yoga
11:01:15 gibi in xena we have < 0.15
11:12:53 sean-k-mooney https://github.com/harlowja/fasteners/issues/96
11:13:20 sean-k-mooney at leasst they can triage ^ and decied if its somethign they want to fix

Earlier   Later