| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-23 | |||
| 11:55:41 | sean-k-mooney | just that we model device in placment with teh pci address | |
| 11:56:16 | sean-k-mooney | when we start modeling sriov vfs in placmnet we will need to use the PF pci address as part of the name for the VF inventores | |
| 11:56:47 | sean-k-mooney | other but that will be challing with qos | |
| 11:57:42 | sean-k-mooney | we shoudl not have different behavour with nic vs other pci devices so i dont see using the devname as a valid option for nova reporting pci device in the future | |
| 11:59:06 | gibi | sean-k-mooney: I agree that when we model the PFs in placement we need to do it right. and make the QoS related RPs aligned. | |
| 12:13:52 | gibi | sean-k-mooney: it is not impossile rename the device RPs in placement to use pci address, we just need to add some netdev -> PCI address translation code in the neturon sriov agent, and keep conditional logic in nova to look for old netdev names and new pci address names for an extra cycle to support rolling upgrade | |
| 12:14:41 | sean-k-mooney | how did the RP get move by the way | |
| 12:14:45 | sean-k-mooney | you mentioned a bugfix? | |
| 12:14:54 | sean-k-mooney | i assume that has been reverted? | |
| 12:15:17 | sean-k-mooney | and yes that is one way to adress that in the future | |
| 12:15:29 | gibi | sean-k-mooney: this fixed a bug https://review.opendev.org/c/openstack/neutron/+/696600 but broke the tree | |
| 12:15:40 | sean-k-mooney | the other way woul dbe to add a kind fo symlink or alias feature to placment | |
| 12:15:49 | sean-k-mooney | so we can refer to the same RP with different names | |
| 12:16:47 | sean-k-mooney | ah i rememebr this | |
| 12:17:27 | sean-k-mooney | gibi: what is currently broken by having them under the compute node RP | |
| 12:18:25 | gibi | sean-k-mooney: if you upgrade from Train to Ussuri in a way that you had QoS configured already in Train then the Ussuri neutron will error out | |
| 12:18:38 | gibi | if you deploy a new Ussuri then no visible problem seen | |
| 12:18:45 | sean-k-mooney | got it | |
| 12:18:56 | sean-k-mooney | so we dont have min bandwith testing in grenade | |
| 12:18:59 | gibi | this is why the problem went undetected for so long | |
| 12:19:02 | sean-k-mooney | well neutron does not | |
| 12:19:23 | gibi | I can even think that schduling works after the upgrade | |
| 12:19:33 | gibi | but i have to reproduce it and try | |
| 12:19:43 | sean-k-mooney | i dont think it woudl break schduling | |
| 12:19:48 | gibi | new deployments are not affected just have a wrong tree sturcture | |
| 12:19:50 | sean-k-mooney | nova is not currently relying on the structure | |
| 12:20:16 | sean-k-mooney | i proposed using that stucure because i wante dot model other networking requirments | |
| 12:20:27 | sean-k-mooney | like make port count support by a vswtich | |
| 12:20:49 | sean-k-mooney | or trait on a agent provider for offloads or network type supprot e.g. vxlan | |
| 12:21:00 | sean-k-mooney | then evneutally add that to the query with same subtree | |
| 12:21:24 | sean-k-mooney | but we did nto have same subtree at the time so the agent RPs are not actuly adding any benifit currently | |
| 12:22:05 | sean-k-mooney | anyway so the fix for this would go into neutron right | |
| 12:22:11 | gibi | yes and yes | |
| 12:22:37 | gibi | and the whole think came out as I started looking into modeling OVS packet processing capacity on the ovs agent RP | |
| 12:23:05 | sean-k-mooney | as in packet per second? | |
| 12:23:11 | gibi | yepp | |
| 12:23:38 | sean-k-mooney | you an i both know that that depends on the type of packet you use and the ip pipline/match rules | |
| 12:24:06 | sean-k-mooney | its very hard to get that right without a lot of testing on your excat hardware | |
| 12:24:30 | sean-k-mooney | i assume neutron will be dumb and just have the operator say the capastity is X in a config like bandwith | |
| 12:24:34 | gibi | I'm not well educated in OVS | |
| 12:24:56 | gibi | yes the inventory would be config driven like bandwidth | |
| 12:25:26 | sean-k-mooney | gibi: the pps number is differnt for vlan network vs vxlan network on the same ovs on the same host | |
| 12:25:55 | sean-k-mooney | e.g. l3 tunneled networks use more cpu cycle to decap and encap then vlan | |
| 12:26:23 | sean-k-mooney | in the case of ovs-dpdk vm to vm traffic on the same host uses more cycle then vm to physical network | |
| 12:26:38 | sean-k-mooney | since vm to vm can levgerage as many hardware offloads | |
| 12:27:01 | sean-k-mooney | so depening on the direction of traffic and the overlay use the capsity will change | |
| 12:27:33 | sean-k-mooney | you can obvioulsy deploy and test for this but it also will change based on other factors | |
| 12:27:56 | sean-k-mooney | such as security group rules implemnte in contrack and the traffic profile. | |
| 12:28:43 | sean-k-mooney | e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking | |
| 12:29:11 | sean-k-mooney | so the pps will be lower with lots of new connection being established vs steady state | |
| 12:29:29 | sean-k-mooney | multicast/broadcast vs unicast also is a factor | |
| 12:29:47 | sean-k-mooney | gibi: so im glad figuring that out will be the operators problem | |
| 12:33:54 | lyarwood | https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now. | |
| 12:34:15 | lyarwood | I'm still working on the grenade change ontop of it | |
| 12:36:51 | gibi | sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details | |
| 12:37:22 | gibi | sean-k-mooney: I agree that configuring this will be really deployment specific | |
| 12:38:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463 | |
| 12:38:55 | gibi | sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns. | |
| 12:38:56 | sean-k-mooney | i worked with other teams at intel that worked on ovs and vsperf in opnfv | |
| 12:39:31 | sean-k-mooney | gibi: yep they do | |
| 12:39:45 | sean-k-mooney | but for public clouds this gets harder as you dont know the workload | |
| 12:39:53 | gibi | yes | |
| 12:39:58 | sean-k-mooney | i assume as part of this there is an enforcemnt elemnt | |
| 12:40:12 | sean-k-mooney | e.g. a max pps qos policy | |
| 12:40:30 | sean-k-mooney | i think tc can enforce that | |
| 12:40:34 | gibi | yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst | |
| 12:41:14 | gibi | at the moment I'm looking into min pps only for the scheduling decision but later there might be enforcement for that as well | |
| 12:41:51 | sean-k-mooney | good because we dont currenlty allow enforcing that in nova flavor extra specs with quota:vif_* and i kind of want to kill those | |
| 12:42:20 | sean-k-mooney | well we can do what we do for bandwith today | |
| 12:42:32 | sean-k-mooney | e.g. set min = max | |
| 12:42:42 | sean-k-mooney | and just schulde on min and enfroce max | |
| 12:42:57 | sean-k-mooney | which sice we dont over subscibe means we could effectivly enforce min | |
| 12:43:10 | gibi | yes, that is possible here too | |
| 12:43:19 | sean-k-mooney | i dont think ovs support min pps | |
| 12:43:27 | sean-k-mooney | even for sriov that is a challange | |
| 12:45:02 | gibi | three are some differences with BW. The packet rate inventory seems to be directionless in nature | |
| 12:45:48 | gibi | and in case of OVS it more belongs to the softswitch level than to the bridge level | |
| 12:46:15 | gibi | and for SRIOV packet rate is not really a limiting factor | |
| 12:46:19 | gibi | afaik | |
| 12:46:23 | sean-k-mooney | the same is true for pps form an ovs point of view | |
| 12:46:40 | sean-k-mooney | bridges are meaningless for the most part | |
| 12:47:03 | sean-k-mooney | gibi: it kind fo still is | |
| 12:47:23 | sean-k-mooney | even with sriov 64b packets will limit the bandwith fo the nic | |
| 12:47:44 | sean-k-mooney | typeically they can handel 1 or 2 VF at that rate but not multiple | |
| 12:48:06 | gibi | sean-k-mooney: interesting, I did not know that | |
| 12:49:06 | gibi | so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small? | |
| 12:49:16 | sean-k-mooney | normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps | |
| 12:49:53 | sean-k-mooney | so it depens where the limit will come in would be vm to vm traffic on the same host | |
| 12:50:22 | sean-k-mooney | the other place its a proble is prot mirroing | |
| 12:50:35 | sean-k-mooney | our running vf in trusted mode | |
| 12:51:27 | sean-k-mooney | setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them | |
| 12:51:54 | sean-k-mooney | it forcs the nic to copy a subset of packets to multiple vfs | |
| 12:53:24 | gibi | thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent | |
| 12:53:27 | sean-k-mooney | gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning | |
| 12:54:12 | gibi | good to know | |
| 12:55:16 | sean-k-mooney | https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov | |
| 12:55:42 | sean-k-mooney | they are using isolcpus=1-19,21-39 on the kernel in part 4 | |
| 12:55:49 | sean-k-mooney | 6.2.4 | |
| 12:57:13 | gibi | yeah we also separate tenat cpus from host cpus and from ovs cpus | |
| 12:57:20 | gibi | with isolcpu | |