| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-09-05 | |||
| 11:33:24 | gibi | added the fasteners issue link and hopefully stabilized the unit test ^^ | |
| 11:34:00 | opendevreview | Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718 | |
| 11:34:20 | sean-k-mooney | im going to propose two reverts to fasteneres and assocaite it with the issue link too incase they deciced that that is approicate | |
| 11:35:30 | gibi | ack | |
| 11:40:33 | sean-k-mooney | https://github.com/harlowja/fasteners/pull/97 | |
| 11:47:10 | gibi | thanks | |
| 13:00:34 | opendevreview | Balazs Gibizer proposed openstack/nova master: Show candidate combinatorial explosion by dev number https://review.opendev.org/c/openstack/nova/+/855885 | |
| 13:00:46 | gibi | sean-k-mooney: here are the numbers of the combinatorial explosion | |
| 13:00:48 | gibi | ^^ | |
| 13:01:16 | gibi | in case of single device per RP (ie PCI or PF) we have worst case factorial amount of candidates | |
| 13:01:48 | gibi | in case a single RP provides more than one devices (n VFs for a PF RP) then worst case we have exponential candidates | |
| 13:04:09 | gibi | and placement first generate all of them then limit the result based on the limit queryparam https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/research_context.py#L289-L292 | |
| 13:13:19 | gibi | so this probably needs a bit of (probably massive) refactoring if we want to avoid placement to blow up on 8 devices | |
| 13:13:34 | gibi | we need to inline the limit somehow | |
| 13:15:45 | gibi | but by that we would potentially loose viable candidates | |
| 13:16:12 | gibi | so placement alone cannot decide where to limiot | |
| 13:23:49 | gibi | So modeling similar PFs of PCI devs does not help as it would lead to the VF scenario. | |
| 13:24:39 | gibi | Also we cannot model count=n as a single group as placement never splits a suffixed group to fit it into multiple RP | |
| 13:26:31 | gibi | while for nova it would be enough to have a small number of candidates per compute host, while we have nova side PCI filtering we need all candidates from placement as we don't know which will fulfill the nova side filtering | |
| 13:44:29 | sean-k-mooney | gibi: ya so this is exactly why we did not want each VF to ba an RP | |
| 13:44:40 | sean-k-mooney | we were very concerned it would explode like this | |
| 13:46:04 | sean-k-mooney | gibi: this feels a bit like the numa node combintorial issue | |
| 13:46:20 | sean-k-mooney | is this happening in sql or in python | |
| 13:46:28 | gibi | this is in python | |
| 13:46:47 | gibi | in case of numa we re-tried host - guest numa mapping multiple times | |
| 13:46:50 | sean-k-mooney | ack i wond if we can use itertools.combinations there in stead of permuations in that case | |
| 13:47:07 | gibi | we use itertools.product to generate all mapping between RPs and groups | |
| 13:47:29 | sean-k-mooney | hum ya i wonder if we really need all | |
| 13:47:43 | gibi | from placement perspective we need all | |
| 13:47:53 | sean-k-mooney | do you have a pointer to the code | |
| 13:48:04 | gibi | nova might be able to limit it by providing al the PCI filtering information to placement | |
| 13:48:26 | sean-k-mooney | gibi: we likely can take a perhost limit and then use a genortator to limit the amount we return | |
| 13:49:00 | sean-k-mooney | gibi: to porvide all the info we would also need to pass the numa toplogy info which would change the tree structure | |
| 13:49:11 | sean-k-mooney | doable but a lot of work | |
| 13:49:16 | gibi | the perhost limit has the problem that if nova still filters out PCI devices after placement then we need to make sure that placement returns enough candidate to fulfill that extra filtering | |
| 13:49:40 | sean-k-mooney | https://github.com/openstack/placement/blob/c68d472dca6619055579831ad5464042f745557a/placement/objects/allocation_candidate.py#L364-L387 | |
| 13:50:01 | gibi | yeh I linked it above :) | |
| 13:50:05 | sean-k-mooney | gibi: ya its the same issue with the current request limit | |
| 13:50:38 | sean-k-mooney | you linked to the resarch context | |
| 13:50:49 | sean-k-mooney | unless i missed it | |
| 13:51:03 | gibi | ahh sorry yes | |
| 13:51:12 | gibi | in the commit message I linked to the product call | |
| 13:51:26 | gibi | https://review.opendev.org/c/openstack/nova/+/855885/1//COMMIT_MSG#26 | |
| 13:51:34 | sean-k-mooney | ack | |
| 13:54:16 | gibi | at the moment I don't think this is easy to fix and given my time allocation for the next months I won't start on it. | |
| 13:54:35 | gibi | songwenping_ might have time and ideas to hack on it | |
| 13:55:12 | gibi | I left the above functional test top of the PCI series so we will not forget that this needs to be fixed | |
| 13:55:35 | gibi | but this makes me question if we want to merge the scheduling support in AA | |
| 13:55:48 | gibi | it might be useful for small deployments (<8 devs per host) | |
| 13:56:00 | gibi | but it is dangerous for big deployments | |
| 13:56:42 | gibi | bauzas: do we have a PTG etherpad? | |
| 13:56:49 | sean-k-mooney | im wondering if we need to have a way to limit this form the api query | |
| 13:56:59 | bauzas | gibi: not yet, but I can create one | |
| 13:57:08 | gibi | bauzas: I could use one :) | |
| 13:57:19 | bauzas | as you want | |
| 13:58:55 | gibi | sean-k-mooney: if we limit the a_c query then we need to give hints to placement about which order to iterate the candidates to fill the limited response | |
| 13:59:20 | gibi | sean-k-mooney: but I'm not sure I can express what we need | |
| 13:59:59 | gibi | sean-k-mooney: it is skip those candidates that are "too similar" to an already found candidate | |
| 14:01:04 | sean-k-mooney | gibi: im wonderign if we can avoid it by generatting a suffictly diverse set of combinations | |
| 14:01:19 | gibi | i.e. in case RP1(2), RP2(2), G1(1), G2(1) -> (RP1-G1, RP2-G2) and (RP1-G2, RP2-G1) might be too similar if G1 an G2 asks for the same RC and traits | |
| 14:01:52 | gibi | yeah divers set of candidates == skip the too similar ones :) | |
| 14:02:03 | gibi | but what is divers might be not universal | |
| 14:02:55 | gibi | like ir RP2 is remote_managed=True in nova then placement still sees the same symmetry but the two RP is not equivalent from nova perspecgive | |
| 14:02:59 | gibi | perspective | |
| 14:04:12 | sean-k-mooney | right so we want to ignore order when lookign at equvialce provided the request group is the same | |
| 14:04:40 | sean-k-mooney | i feel likel there is definetly a way to optimise so that we dont generate a product | |
| 14:04:51 | sean-k-mooney | that is goign to over produce results | |
| 14:05:19 | sean-k-mooney | but off the top of my head im not sure the correct way to proceed | |
| 14:05:41 | gibi | bauzas: I've created https://etherpad.opendev.org/p/nova-antelope-ptg | |
| 14:06:34 | sean-k-mooney | lookingat https://docs.python.org/3/library/itertools.html#itertools-recipes | |
| 14:06:34 | gibi | sean-k-mooney: I agree on the second part "but off the top of my head im not sure the correct way to proceed" but I'm not sure we can have better than actually iterating the product | |
| 14:07:44 | sean-k-mooney | maybe take(per_host_limit, random_product(...)) | |
| 14:09:30 | gibi | yeah random is a way out even if a dirty one | |
| 14:09:38 | sean-k-mooney | gibi: if we cant avoid the need to generate the product im wonderign if we can break the implict lexegfacical ordering | |
| 14:10:14 | opendevreview | Amit Uniyal proposed openstack/nova master: add regression test case for bug 1552777 https://review.opendev.org/c/openstack/nova/+/855900 | |
| 14:10:14 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds check for instance resizing https://review.opendev.org/c/openstack/nova/+/855901 | |
| 14:10:50 | gibi | if nova could define a requested order based on information that nova has but placement doesnt, then yes, ordering can be a solution. that is basically nova asking placement to generate diverse candiates with a definition of divers provided by nova in the a_c query | |
| 14:18:30 | gibi | OK I tried to document this issue on the PTG etherpad and by that I will move this to background processing :) | |
| 14:34:55 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717 | |
| 14:36:38 | opendevreview | Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718 | |
| 16:22:06 | gibi | bauzas, sean-k-mooney: ML thread about the fair lock issue https://lists.openstack.org/pipermail/openstack-discuss/2022-September/030325.html | |
| 16:22:13 | bauzas | gibi: thanks | |
| 16:23:58 | sean-k-mooney | cool | |
| 16:26:43 | gibi | I did find any direct evidence that other projects than nova and taskflow are affected | |
| 16:26:47 | gibi | I did not | |
| 16:30:04 | opendevreview | Balazs Gibizer proposed openstack/nova master: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855717 | |
| 16:31:16 | opendevreview | Balazs Gibizer proposed openstack/nova stable/yoga: Fix fair internal lock used from eventlet.spawn_n https://review.opendev.org/c/openstack/nova/+/855718 | |
| 17:24:07 | sean-k-mooney | gibi bauzas https://blueprints.launchpad.net/nova/+spec/non-admin-hw-offloaded-ovs | |
| 17:24:47 | sean-k-mooney | i can proably hack up a poc of that this week i guess but im unsure if i will have time to take that to completion | |
| 17:26:07 | bauzas | cycle highlights ready to review https://review.opendev.org/c/openstack/releases/+/855974 | |
| 17:28:03 | sean-k-mooney | do we have any features to highlight for non libvirt drivers | |
| 17:28:15 | sean-k-mooney | were there any imporant ironic improvmements | |
| 17:28:19 | sean-k-mooney | or hyperv | |
| 17:29:25 | sean-k-mooney | based on https://docs.openstack.org/releasenotes/nova/unreleased.html#new-features i guess not | |
| 18:05:58 | sean-k-mooney | gibi: bauzas ... so there is alos a libvirt bug at play for hardwaore offloaded ovs | |
| 18:06:00 | sean-k-mooney | https://github.com/libvirt/libvirt/commit/8708ca01c0dd38764cad3e483405bdeb05ac2e96 | |
| 19:03:15 | whoami-rajat | bauzas, hey, we're past client freeze but my API feature is in and just wanted to mention the OSC and novaclient patches required by my feature | |
| 19:03:19 | whoami-rajat | novaclient https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 19:03:25 | whoami-rajat | OSC: https://review.opendev.org/c/openstack/python-openstackclient/+/831014 | |
| #openstack-nova - 2022-09-06 | |||
| 07:05:14 | bauzas | good morning Nova | |