| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-09-05 | |||
| 09:18:35 | gibi | so we need to decide which was we go. a) Store Row (or namedtuple) objects in caches and keep attribute access b) store dict and go with dict access | |
| 09:18:37 | sean-k-mooney[m] | https://github.com/openstack/placement/commit/b3fe04f081a096258468d032560f46cdfe77e144 stpehen tried to remove these assumtions in ^ | |
| 09:19:07 | sean-k-mooney[m] | well that will work as a named tuple | |
| 09:19:28 | gibi | Row and namedtuple is pretty much compatible, yes | |
| 09:19:48 | sean-k-mooney[m] | i would avoid random dicts personally | |
| 09:19:50 | gibi | ack | |
| 09:20:07 | sean-k-mooney[m] | and either store the row or named tuple | |
| 09:20:14 | sean-k-mooney[m] | im not sure if there is a reason not to sotre thr row object | |
| 09:20:21 | sean-k-mooney[m] | does it increase memory | |
| 09:20:31 | sean-k-mooney[m] | or have any other sideffect we would not want | |
| 09:23:38 | gibi | storing row is OK the doc said dict hence my statement that the caches is broken | |
| 09:23:55 | sean-k-mooney[m] | ack | |
| 09:24:34 | gibi | we need to mix Row and namedtuple as I can only create namedtuple here https://github.com/openstack/placement/blob/13bbdba06da19f85c05a2a9e1fbdb9d1813c3b47/placement/attribute_cache.py#L157-L168 | |
| 09:24:40 | gibi | but they are compatible | |
| 09:24:43 | gibi | so I only leave a note | |
| 09:24:55 | gibi | when I convert that to namedtuple | |
| 09:39:38 | sean-k-mooney[m] | if you use a named tuple on line 155 as well in refersh_from_db i thik we dont need to mix types but cool ill review when you push | |
| 09:47:01 | songwenping_ | sean-k-mooney[m],gibi: hi, nova-scheduler get allocation_candidates return 504 gateway timeout when create vm with 8gpus requests on our client's env, there are 13 gpu compute, and every compute has 8gpus. | |
| 09:47:02 | opendevreview | Balazs Gibizer proposed openstack/placement master: Make us compatible with oslo.db 12.1.0 https://review.opendev.org/c/openstack/placement/+/855862 | |
| 09:47:07 | gibi | sean-k-mooney[m]: ^^ | |
| 09:47:12 | gibi | stephenfin: ^^ | |
| 09:50:49 | gibi | sean-k-mooney[m]: also here is my stab at the nova only fair lock fix https://review.opendev.org/c/openstack/nova/+/855717 there is the oslo version of the fix https://review.opendev.org/c/openstack/oslo.concurrency/+/855714 | |
| 09:51:01 | songwenping_ | i imitate to insert some test datas on my devstack env, and the result is same, perhaps the api of allocation_candidates/limit... need to optimize. | |
| 09:51:27 | gibi | but I have to go back and think about the unit tests as it seems they are unstable | |
| 09:53:13 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check for VM snapshot fail while quiesce https://review.opendev.org/c/openstack/nova/+/852171 | |
| 09:54:22 | sean-k-mooney[m] | songwenping_: this is physical gpu passthoug? | |
| 09:54:29 | sean-k-mooney[m] | not vGPU correct | |
| 09:54:30 | songwenping_ | yes | |
| 09:54:41 | songwenping_ | pgpu passthough | |
| 09:55:05 | sean-k-mooney[m] | those are not tracked in placment then | |
| 09:55:18 | auniyal_ | Hi | |
| 09:55:23 | auniyal_ | please review these | |
| 09:55:25 | auniyal_ | https://review.opendev.org/c/openstack/nova/+/852171 | |
| 09:55:25 | auniyal_ | https://review.opendev.org/c/openstack/nova/+/854499 | |
| 09:55:25 | auniyal_ | backporting | |
| 09:55:25 | auniyal_ | https://review.opendev.org/c/openstack/nova/+/854979 | |
| 09:55:25 | auniyal_ | https://review.opendev.org/c/openstack/nova/+/854980 | |
| 09:56:17 | sean-k-mooney[m] | songwenping_: on master we now can track pci devics in placment but in any other release pci devices are not tracked in placment | |
| 09:56:34 | songwenping_ | sean-k-mooney[m]: we use cyborg to manage pgpu, the request url is :curl -g -i -X GET "http://10.7.20.73/placement/allocation_candidates?limit=1000&group_policy=none&required1=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required2=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required3=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required4=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&required5=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT | |
| 09:56:34 | songwenping_ | _ID_1DB6&required6=CUSTOM_GPU_NVIDIA%2CCUSTOM_GPU_PRODUCT_ID_1DB6&resources=MEMORY_MB%3A32%2CVCPU%3A1&resources1=PGPU%3A1&resources2=PGPU%3A1&resources3=PGPU%3A1&resources4=PGPU%3A1&resources5=PGPU%3A1&resources6=PGPU%3A1" -H "Accept: application/json" -H "OpenStack-API-Version: placement 1.29" -H "User-Agent: openstacksdk/0.99.0 keystoneauth1/4.6.0 python-requests/2.27.1 CPython/3.8.10" -H "X-Auth-Token: gAAAAABjFbkj8nz24Q7A6J0qmjpdZHfWM | |
| 09:56:35 | songwenping_ | vZidJOT9iYhV2MQCngcYQHhSQmjsGJofkYoT087tAISpf3IniDGwPTHXz_-8x-1nF60WavSYFgEd-5l3_ENrGumaHuU1yfhMJqZu06IR4SXacjA1g6ImSSEfLbfQ9zrPouB0roFokHPmPy3-UpnFZE" | |
| 09:56:46 | gibi | sean-k-mooney[m], sean-k-mooney[m]: is it via cyborg? because then it might be tracked in placement | |
| 09:56:53 | sean-k-mooney[m] | ok | |
| 09:58:05 | sean-k-mooney[m] | in that case perhaps they are hitting the compintorial explosion we were worried about with tracking VFs directly | |
| 09:58:11 | gibi | probably there too many possible candidates | |
| 09:58:19 | sean-k-mooney[m] | there are 13 chose 8 combinations | |
| 09:58:39 | sean-k-mooney[m] | that 1287 | |
| 09:58:46 | gibi | that is not that much | |
| 09:58:55 | gibi | but if there is 1000 computes | |
| 09:58:59 | sean-k-mooney[m] | 1287*1000 | |
| 09:58:59 | gibi | or just 100 | |
| 09:59:07 | gibi | then that is sizeable | |
| 09:59:16 | sean-k-mooney[m] | ya it will grow quickly | |
| 10:00:03 | sean-k-mooney[m] | sorry no | |
| 10:00:18 | songwenping_ | there are extra 8 computes without gpu. | |
| 10:00:22 | sean-k-mooney[m] | its 13 hosts each with 8 gpus and the vm is asking for 8 | |
| 10:00:53 | sean-k-mooney[m] | so it wont explode like that there is only 1 allocation pooible per host | |
| 10:01:01 | sean-k-mooney[m] | *possible | |
| 10:01:01 | gibi | nope | |
| 10:01:08 | gibi | if you have 8 groups and 8 gpus | |
| 10:01:19 | gibi | then each group can be satisfied by each gpu | |
| 10:01:25 | sean-k-mooney[m] | you think it would b n squared | |
| 10:01:41 | sean-k-mooney[m] | oh hum maybe | |
| 10:01:48 | gibi | 8! | |
| 10:02:48 | gibi | 40320 | |
| 10:03:00 | sean-k-mooney[m] | ya… that would be bad | |
| 10:03:33 | sean-k-mooney[m] | i hope thats not whats happening or that will cause issue for pci devices in placemnt too | |
| 10:03:43 | gibi | this is coming from the fact that placement treats groups and RPs individually. even if we have very similar RPs and very similar groups | |
| 10:04:02 | gibi | I can make a test case for 8 PCI devs :) | |
| 10:04:09 | gibi | in nova func test | |
| 10:04:12 | gibi | so we can prove it | |
| 10:04:32 | sean-k-mooney[m] | perhaps start with 4 | |
| 10:05:08 | sean-k-mooney[m] | i mean if its a gabbit test then sure 8 | |
| 10:05:13 | gibi | ack. first 1 will go and try to stabilize the fair lock unit tests then I will look at the 4-8 PCI issue | |
| 10:05:45 | sean-k-mooney[m] | we really need to not do all possible permuations on the placement side | |
| 10:06:11 | gibi | if this is real factorial then we need to introduce an per host limit for a_c query | |
| 10:06:28 | sean-k-mooney[m] | ya i was debating if that was the only option | |
| 10:06:33 | gibi | if the groups are different by trait then we need all permutations | |
| 10:06:51 | gibi | otherwise we might not found the right one | |
| 10:07:16 | sean-k-mooney[m] | well an allocation candiate by defieniton meets the requirements | |
| 10:07:16 | gibi | the trick here is that we have identical groups and identicaly PRs | |
| 10:07:23 | sean-k-mooney[m] | so it depend on whyer this is happening | |
| 10:07:48 | gibi | to find a_c placement needs to iterate permutations I think in the general case | |
| 10:07:51 | sean-k-mooney[m] | anyway something to look into i guess | |
| 10:07:55 | gibi | yepp | |
| 10:08:36 | sean-k-mooney[m] | im going to grab a coffee and take my blood pressre meds and then check on freya be back in about 10 mins | |
| 10:09:02 | gibi | ack | |
| 10:09:22 | gibi | for me coffee is the blood pressure med | |
| 10:32:32 | sean-k-mooney | gibi: should fastener reintoudce the workaround they had | |
| 10:35:20 | gibi | that is a way too but I have less authority over that project | |
| 10:35:51 | sean-k-mooney | isnit it an oslo deliverable too | |
| 10:36:09 | sean-k-mooney | or has it move out of openstack | |
| 10:36:56 | sean-k-mooney | oh its not an openstack project | |
| 10:37:02 | sean-k-mooney | i tough it used to be at one point | |
| 10:38:21 | sean-k-mooney | i guess we can fix it in oslo but we should leth the know that under eventlet the rentrant guarentee is broken | |
| 10:38:25 | sean-k-mooney | https://github.com/harlowja/fasteners#-overview | |
| 10:38:35 | sean-k-mooney | then note that it should be reentrant | |
| 10:44:17 | sean-k-mooney | gibi: https://github.com/harlowja/fasteners/issues/86 | |
| 10:44:17 | sean-k-mooney | that also intereisng | |
| 10:44:17 | sean-k-mooney | https://github.com/harlowja/fasteners/pull/87/files | |