| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-07-30 | |||
| 15:09:16 | sean-k-mooney | i need to check if the numa policy is in the alias or pci white list | |
| 15:09:21 | stephenfin | ralonsoh: Sorry, my time's been split multiple different ways so I haven't gotten to sunk my teeth into this before now. Just teasing things out before we merge something we can't undo :) | |
| 15:09:23 | sean-k-mooney | if its in the whitelist then no | |
| 15:09:31 | sean-k-mooney | * alias | |
| 15:09:37 | ralonsoh | stephenfin, sure | |
| 15:10:41 | sean-k-mooney | ok so the numa policy is part of the alias https://github.com/openstack/nova/blob/master/nova/pci/request.py#L93 | |
| 15:11:03 | sean-k-mooney | so we dont need to cater for reporting device as only consumable with a singel policy | |
| 15:11:10 | sean-k-mooney | its not defiend in the whitelist | |
| 15:11:15 | stephenfin | right, though it's only done that way because aliases are as granular as we get | |
| 15:12:13 | stephenfin | we didn't (don't) have a mechanism to say use the PCI device indicated by this alias and don't worry about NUMA affinity | |
| 15:12:26 | sean-k-mooney | yes we do | |
| 15:12:44 | sean-k-mooney | oh you mean per alias | |
| 15:12:46 | stephenfin | the flavor extra spec applies to all PCI devices, not just one | |
| 15:12:48 | sean-k-mooney | right we dont | |
| 15:12:49 | stephenfin | yes | |
| 15:12:56 | stephenfin | right | |
| 15:13:28 | stephenfin | so that's still a TODO after this but we'll be able to do it for SR-IOV ports at least | |
| 15:13:46 | stephenfin | but the ability to do that smells like a simple binary capability | |
| 15:14:12 | sean-k-mooney | stephenfin: we kind of hoped to stop using the alias and move pci deivce to the top level api or use cyborg | |
| 15:14:21 | ralonsoh | but with this information in the port we don't need to limit that to sriov ports only | |
| 15:14:27 | sean-k-mooney | which is why it was not added orginally | |
| 15:14:37 | stephenfin | i.e. I allow you to configure NUMA affinity of individual resources or I don't | |
| 15:14:47 | sean-k-mooney | ralonsoh: right we are not | |
| 15:15:20 | sean-k-mooney | the trait is not technially need we can do the enforcement in the numa toplogy filter without it | |
| 15:15:48 | sean-k-mooney | but with out the trait we do that the proablem that plamcnet may not return hosts with the capablity if the limit is set too low | |
| 15:15:52 | stephenfin | ralonsoh: So I guess you're thinking we could have multiple child RPs, e.g. one for the OVS network, one for the SR-IOV device, one for a generic PCI device etc., and each could support their own type of policy? | |
| 15:16:09 | ralonsoh | that's the goal | |
| 15:16:23 | sean-k-mooney | which iw why we should have a trait but i guess it does not need to be 3 | |
| 15:16:52 | stephenfin | Hmm, okay, that sounds reasonable. I'd been thinking the trait would be reported on the root provider (the compute node) | |
| 15:17:05 | stephenfin | which makes little sense because you provide no granularity that way | |
| 15:17:08 | sean-k-mooney | by the way this is part of https://bugzilla.redhat.com/show_bug.cgi?id=1780721 and https://bugzilla.redhat.com/show_bug.cgi?id=1791834 | |
| 15:17:08 | openstack | bugzilla.redhat.com bug 1780721 in openstack-nova "[RFE] Optional NUMA affinity for neutron ports" [High,New] - Assigned to smooney | |
| 15:17:09 | openstack | bugzilla.redhat.com bug 1791834 in openstack-neutron "[RFE] Optional NUMA affinity for neutron ports" [High,New] - Assigned to amuller | |
| 15:17:33 | ralonsoh | yes | |
| 15:17:49 | stephenfin | sean-k-mooney: oh yeah, I understand and agree with the feature as a whole. It's just this trait bit that's tripping me up | |
| 15:18:11 | sean-k-mooney | stephenfin: right now if we said we want strict affinity we have no way to guarrentee that | |
| 15:19:14 | sean-k-mooney | becasue placment if limit=10 could select 10 allocation candated form host that dont supprot numa affinity | |
| 15:19:33 | sean-k-mooney | the trait is basicaly to make it work in deployment like cerns where they have a really low limit | |
| 15:19:47 | stephenfin | the solution to that is NUMA in placement though, no? | |
| 15:19:52 | sean-k-mooney | no | |
| 15:20:12 | stephenfin | model NUMA nodes as RPs in placement. Say all resources must share the same root RP (NUMA node) | |
| 15:20:14 | sean-k-mooney | numa in placmnet wont advertise which host have numa aware vswitches | |
| 15:20:40 | sean-k-mooney | since we dont model netowrk backend ports as a resouce | |
| 15:21:32 | sean-k-mooney | stephenfin: we would some how have to model the pysnet numa affinity in the rp tree | |
| 15:21:58 | ralonsoh | we have this information in nova | |
| 15:22:09 | sean-k-mooney | yes but we cant model it in plamcnet | |
| 15:22:10 | ralonsoh | more or less: you have the numa nodes of a physnet | |
| 15:22:13 | stephenfin | sean-k-mooney: how does this help? | |
| 15:22:26 | sean-k-mooney | it just allow me to know the host has the feature | |
| 15:22:41 | sean-k-mooney | i guess we could have 3 different tratis. | |
| 15:23:02 | sean-k-mooney | one for numa-vsiwthc, one for sriov and one for numa affinity | |
| 15:23:06 | sean-k-mooney | but that feels messy | |
| 15:23:25 | stephenfin | You mean it has the NUMA-aware vSwitches feature enabled? | |
| 15:23:31 | sean-k-mooney | yes | |
| 15:23:40 | openstackgerrit | Jie Li proposed openstack/nova-specs master: add volume backed server rebuild spec https://review.opendev.org/739349 | |
| 15:23:56 | sean-k-mooney | if it was a non sriov port and it asked for strict affinity | |
| 15:24:04 | sean-k-mooney | it need to land on a host with numa aware vswtichs | |
| 15:24:08 | sean-k-mooney | or we shoudl reject it | |
| 15:24:31 | sean-k-mooney | legacy and prefer not require that | |
| 15:24:46 | sean-k-mooney | but strict means we have to guarrente numa affinity or not boot | |
| 15:25:13 | sean-k-mooney | we can do that in the numa toplogy filter proably | |
| 15:25:25 | sean-k-mooney | but then you have the cherry picking problem | |
| 15:25:57 | stephenfin | so you're worried about NUMA-aware vSwitches being configured on some hosts and not others, and you want to makes sure instances never land on the latter | |
| 15:26:08 | sean-k-mooney | yes | |
| 15:26:33 | stephenfin | The concern I have with that is that NUMA-aware vSwitches is a hack, and we knew it was a hack when we implemented it /o\ | |
| 15:26:55 | sean-k-mooney | yes we wanted to evnetually move it to placment | |
| 15:26:58 | stephenfin | and I'm concerned about encoding that hack into traits, which we can't ever drop | |
| 15:27:17 | sean-k-mooney | we dont actully need the traits right now | |
| 15:27:17 | stephenfin | couldn't we just solve that particular issue with documentation? | |
| 15:27:23 | sean-k-mooney | so we can defer it | |
| 15:27:39 | sean-k-mooney | i wanted neutron to include the trait in the port resouce request | |
| 15:27:51 | sean-k-mooney | but we can have nova append it in the future if we want | |
| 15:27:56 | stephenfin | i.e. if you really care about NUMA affinity, make sure to configure this on all hosts or use host aggregates to separate from hosts without the config | |
| 15:28:24 | sean-k-mooney | maybe it kind of feells like soemthing that should not need to be manual | |
| 15:28:49 | stephenfin | I agree, but you could say the same about the NUMA-aware vSwitches feature in general | |
| 15:29:00 | sean-k-mooney | we did | |
| 15:29:04 | stephenfin | True :) | |
| 15:29:10 | sean-k-mooney | be we also said we did not want to wait | |
| 15:29:13 | stephenfin | but it was a means to an end | |
| 15:29:14 | stephenfin | yup | |
| 15:29:36 | stephenfin | I don't really want to pile more on top of that shaky foundation though | |
| 15:29:48 | stephenfin | like I said, we're stuck with traits once they're there | |
| 15:29:52 | sean-k-mooney | ralonsoh: stephenfin ok so do we want to put the traits on pause untill we see if we will use them in the nova half of the feature | |
| 15:30:08 | ralonsoh | sean-k-mooney, np | |
| 15:30:14 | sean-k-mooney | ralonsoh: what that means for you is you just dont need to add a trait request to the port | |
| 15:30:15 | stephenfin | if that works for ralonsoh, I'd much rather that | |
| 15:30:35 | sean-k-mooney | and if nova want to use the traits in the futrue we can do it instead of neutorn | |
| 15:30:38 | ralonsoh | sean-k-mooney, but what should I add to the port info? | |
| 15:30:39 | stephenfin | if it turns out we want a way to advertise these capabilities, then I'm all for revisiting | |
| 15:30:52 | ralonsoh | because I don't know how nova is going to read it | |
| 15:31:02 | sean-k-mooney | ralonsoh: neutron need to add the polciy via the new extention | |
| 15:31:09 | sean-k-mooney | but it does not need to add the trait | |
| 15:31:26 | sean-k-mooney | the extenion was adding a new extended atibute to the port anyway right | |
| 15:31:26 | ralonsoh | sean-k-mooney, that means another parameter in the port dict | |
| 15:31:27 | ralonsoh | ok | |
| 15:31:41 | ralonsoh | I'll need to revisit the spec | |
| 15:31:42 | sean-k-mooney | ralonsoh: yes isnt that what the spec says | |
| 15:32:36 | stephenfin | ralonsoh: again, apologies for only getting to this now. I didn't know that this work was as far along as it is. You've been busy :) | |
| 15:32:39 | sean-k-mooney | https://specs.openstack.org/openstack/neutron-specs/specs/victoria/port-numa-affinity-policy.html#rest-api-impact | |
| 15:32:48 | ralonsoh | np at all | |