Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-02-07
15:16:22 sean-k-mooney when people start using mdev for subfuction or non VF usecase we might need to do that anyway
15:16:32 sean-k-mooney at which point we can test it properly in ci
15:17:18 dmitriis I heard that SFs were introduced to support container use-cases because of scalability issues, less so for VMs. Doesn't mean VMs cannot use them but still.
15:17:52 dmitriis scalability ~ small number of ports
15:18:03 sean-k-mooney depend on who you ask. i know intel were interestin in there version for contaienr but also saw it as a way to adress the vm case in terms of number of ports
15:18:37 sean-k-mooney dmitriis: really the use case is the same allow more then max_vf ports
15:19:15 sean-k-mooney that said i think recent connectx/bf2 cards have started to get into the 1000+ vf range
15:19:24 dmitriis sean-k-mooney: yeah, there are some parts of the SR-IOV spec that can be used to overcome the 256 limit
15:19:33 dmitriis some neat tricks with PCI address allocation
15:20:04 sean-k-mooney yes you multiple buses to card to work around that
15:20:09 dmitriis yep
15:21:15 sean-k-mooney contaienr and vm cloud system really prefer to pertend that the port limit is infinity
15:21:40 sean-k-mooney like k8s and opnestack really dont treat ports as a finite reqouce outside fo sriov
15:22:27 dmitriis sean-k-mooney: there's another problem: accounting and quotas on certain hw limits. There's a finite number of flows that can be programmed and not a lot of data on hw limits per VF/SF
15:22:37 dmitriis likewise, no APIs to query that AFAIK
15:22:38 sean-k-mooney in reality most customer proably wont hit that limit but they are woried fi they layer things like openshift running on top of openstack they might
15:23:25 dmitriis sean-k-mooney: yes, plus with layers on top of OpenStack people start running overlays on top of overlays
15:23:32 sean-k-mooney ya there are both limits on the number of tables and the table row count and then seperat lmits on but the hardwar flows offload
15:23:34 dmitriis and doing that using guest CPU
15:24:30 sean-k-mooney ok i better get back to reviewing your changes before my next meeting :)
15:24:43 dmitriis sean-k-mooney: right, I'll get to re-submitting too :^)
15:49:24 sean-k-mooney dmitriis: is there anyting in the port beyond the vnic_type that tells nova that we need a vf
15:49:56 sean-k-mooney dmitriis: gibi be might not be able to reuse vnic_type smartnic
15:50:11 opendevreview Tobias Urdin proposed openstack/nova master: Cleanup old resize instances dir before resize https://review.opendev.org/c/openstack/nova/+/827865
15:50:34 sean-k-mooney if we assume that all port that have vnic_type smartnic require a vf that would break ironics use of it yes?
15:50:55 opendevreview Balazs Gibizer proposed openstack/placement master: Fix perfload jobs after consumer_types https://review.opendev.org/c/openstack/placement/+/828167
15:51:22 gibi sean-k-mooney: I don't know how ironic uses the smartnic vnic_type
15:51:38 gibi melwitt: I think the perfload job is easy to fix https://review.opendev.org/c/openstack/placement/+/828167
15:51:49 dmitriis sean-k-mooney: (thinking)
15:51:55 sean-k-mooney https://specs.openstack.org/openstack/ironic-specs/specs/12.1/support-smart-nic.html
15:54:47 sean-k-mooney gibi: they bascially wanted to supprot ovs running on the smartnic also
15:54:54 sean-k-mooney but using ml2/ovs
15:55:12 sean-k-mooney with the neutron agetn deployed on the smartnic
15:55:14 dmitriis sean-k-mooney: they have a special config option for the neutron-openvswitch-agent as well
15:55:24 sean-k-mooney yes
15:55:55 sean-k-mooney dmitriis: the possible problem i am seing is when we go to scedule the vm we have not bound the port
15:56:08 sean-k-mooney since we have not selected a host yet
15:56:23 sean-k-mooney so i dont think we will be able to tell the difference between the ironic usage and the new usage
15:56:33 sean-k-mooney and we wont know if we shoudl request a VF or not
15:57:01 sean-k-mooney so we might need to use a differnt off-path vnic type
15:57:04 sean-k-mooney instead
15:57:29 dmitriis sean-k-mooney: in Nova we decide if we want to request a remote_managed port or not
15:57:36 dmitriis and add that to a InstancePCIRequest
15:57:48 sean-k-mooney dmitriis: yes based on the vnic_type right
15:57:51 dmitriis yes
15:58:01 sean-k-mooney right so if we only look at that we need a new vnic type
15:58:27 dmitriis but that's in Nova, trying to figure out how that affects ironic
15:58:27 sean-k-mooney since that decision need to be made before we schdule and therefor cannot depend on the host or driver
15:58:43 sean-k-mooney booting ironic service via nova api
15:58:47 sean-k-mooney *servers
15:59:05 sean-k-mooney in both cases the vnic_type would be the same
15:59:20 sean-k-mooney the falvor woudl be differnt but we would not know its an ironic flavor
16:00:21 sean-k-mooney dmitriis: so i think we need to quickly add a new vnic in neutron-lib and update the nova and neutron code to use that
16:01:13 opendevreview Takashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action https://review.opendev.org/c/openstack/nova/+/828168
16:01:39 opendevreview Takashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action https://review.opendev.org/c/openstack/nova/+/828168
16:03:00 dmitriis sean-k-mooney: we could, trying to think if there's anything we've missed that could allow us to avoid that. We originally wanted to add a new VNIC type but then decided to reuse VNIC_TYPE_SMARTNIC after some reviews.
16:03:36 sean-k-mooney yep i proably suggested the reuse :) but i think its proably required
16:03:46 sean-k-mooney *the new vnic
16:03:59 sean-k-mooney i dont think there is anything else on the port we can use to differenciate
16:05:31 sean-k-mooney oh same
16:10:32 opendevreview Takashi Kajinami proposed openstack/nova master: Allow authorization by user_id for server resume action https://review.opendev.org/c/openstack/nova/+/828168
16:38:19 gibi so if we always translate the smartnic vnic_type to InstancePCIRequest but ironic support smartnic with ml2/ovs then we might have a problem
16:38:39 gibi do we require ironic to use some custom resource in the flavor or it is just an option?
16:42:14 sean-k-mooney am we require cpu, ram and disk to be overriden to resources:cpu=0
16:46:33 gibi that could be a clue
16:46:44 sean-k-mooney im not sure if we want to rely on that
16:46:47 sean-k-mooney we could
16:46:51 gibi yeah it is hackins
16:46:52 gibi hackis
16:47:11 gibi when we have the lib freeze?
16:47:20 sean-k-mooney i belive in a week or two
16:47:25 sean-k-mooney ill check
16:47:41 gibi feb 18
16:47:53 sean-k-mooney ya
16:48:00 sean-k-mooney actully proably 17th
16:48:03 gibi yeah
16:48:32 sean-k-mooney so we still have time to add the value
16:48:40 gibi lets try the new vnic_type way then
16:48:54 gibi that would be clean
16:49:04 sean-k-mooney so vnic_type=off-path
16:49:18 sean-k-mooney actully maybe vnic_type=off-path-direct
16:49:47 sean-k-mooney so that we can have vnic_type=off-path-vdpa or off-path-mev or whaterver later if needed
16:49:50 sean-k-mooney but keep the pattern
16:50:24 gibi yes, off-path or remote-managed whathever floats the neturon team's boat
16:51:07 sean-k-mooney remote-managed might be better ya
16:51:16 sean-k-mooney keep the continuity with the config option
16:56:11 dmitriis sean-k-mooney, gibi: been trying to find something to use as a clue (such as image or flavor metadata)
16:56:19 dmitriis but, yes, it's tricky to rely on that
17:01:21 sean-k-mooney dmitriis: i dont think there is really anything and really it shoudl be soemthing on the port anyway
17:01:38 sean-k-mooney we could look at the flaovr but that is messy
17:02:39 dmitriis sean-k-mooney: ovs hardware offload uses binding:profile to differentiate between legacy SR-IOV and offload cases https://docs.openstack.org/neutron/latest/admin/config-ovs-offload.html#validate-open-vswitch-hardware-offloading
17:02:51 dmitriis been considering utilizing something like that as well
17:02:53 sean-k-mooney dmitriis: the port is not bound at this point
17:03:12 sean-k-mooney we cant use anything that depend on the host/neutron backend
17:03:27 sean-k-mooney since the host is only selected after the scheduling
17:03:58 sean-k-mooney dmitriis also --binding-profile '{"capabilities": ["switchdev"]}' i dot think was ever actully implemented
17:03:58 dmitriis sean-k-mooney: yeah, that's the unfortunate thing - I don't know the driver before scheduling happens
17:04:14 dmitriis sean-k-mooney: ah, right we discussed that way back
17:04:24 sean-k-mooney it also would be admin only since binind-profile is admin only
17:04:29 sean-k-mooney yep

Earlier   Later