| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-23 | |||
| 12:15:49 | sean-k-mooney | so we can refer to the same RP with different names | |
| 12:16:47 | sean-k-mooney | ah i rememebr this | |
| 12:17:27 | sean-k-mooney | gibi: what is currently broken by having them under the compute node RP | |
| 12:18:25 | gibi | sean-k-mooney: if you upgrade from Train to Ussuri in a way that you had QoS configured already in Train then the Ussuri neutron will error out | |
| 12:18:38 | gibi | if you deploy a new Ussuri then no visible problem seen | |
| 12:18:45 | sean-k-mooney | got it | |
| 12:18:56 | sean-k-mooney | so we dont have min bandwith testing in grenade | |
| 12:18:59 | gibi | this is why the problem went undetected for so long | |
| 12:19:02 | sean-k-mooney | well neutron does not | |
| 12:19:23 | gibi | I can even think that schduling works after the upgrade | |
| 12:19:33 | gibi | but i have to reproduce it and try | |
| 12:19:43 | sean-k-mooney | i dont think it woudl break schduling | |
| 12:19:48 | gibi | new deployments are not affected just have a wrong tree sturcture | |
| 12:19:50 | sean-k-mooney | nova is not currently relying on the structure | |
| 12:20:16 | sean-k-mooney | i proposed using that stucure because i wante dot model other networking requirments | |
| 12:20:27 | sean-k-mooney | like make port count support by a vswtich | |
| 12:20:49 | sean-k-mooney | or trait on a agent provider for offloads or network type supprot e.g. vxlan | |
| 12:21:00 | sean-k-mooney | then evneutally add that to the query with same subtree | |
| 12:21:24 | sean-k-mooney | but we did nto have same subtree at the time so the agent RPs are not actuly adding any benifit currently | |
| 12:22:05 | sean-k-mooney | anyway so the fix for this would go into neutron right | |
| 12:22:11 | gibi | yes and yes | |
| 12:22:37 | gibi | and the whole think came out as I started looking into modeling OVS packet processing capacity on the ovs agent RP | |
| 12:23:05 | sean-k-mooney | as in packet per second? | |
| 12:23:11 | gibi | yepp | |
| 12:23:38 | sean-k-mooney | you an i both know that that depends on the type of packet you use and the ip pipline/match rules | |
| 12:24:06 | sean-k-mooney | its very hard to get that right without a lot of testing on your excat hardware | |
| 12:24:30 | sean-k-mooney | i assume neutron will be dumb and just have the operator say the capastity is X in a config like bandwith | |
| 12:24:34 | gibi | I'm not well educated in OVS | |
| 12:24:56 | gibi | yes the inventory would be config driven like bandwidth | |
| 12:25:26 | sean-k-mooney | gibi: the pps number is differnt for vlan network vs vxlan network on the same ovs on the same host | |
| 12:25:55 | sean-k-mooney | e.g. l3 tunneled networks use more cpu cycle to decap and encap then vlan | |
| 12:26:23 | sean-k-mooney | in the case of ovs-dpdk vm to vm traffic on the same host uses more cycle then vm to physical network | |
| 12:26:38 | sean-k-mooney | since vm to vm can levgerage as many hardware offloads | |
| 12:27:01 | sean-k-mooney | so depening on the direction of traffic and the overlay use the capsity will change | |
| 12:27:33 | sean-k-mooney | you can obvioulsy deploy and test for this but it also will change based on other factors | |
| 12:27:56 | sean-k-mooney | such as security group rules implemnte in contrack and the traffic profile. | |
| 12:28:43 | sean-k-mooney | e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking | |
| 12:29:11 | sean-k-mooney | so the pps will be lower with lots of new connection being established vs steady state | |
| 12:29:29 | sean-k-mooney | multicast/broadcast vs unicast also is a factor | |
| 12:29:47 | sean-k-mooney | gibi: so im glad figuring that out will be the operators problem | |
| 12:33:54 | lyarwood | https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now. | |
| 12:34:15 | lyarwood | I'm still working on the grenade change ontop of it | |
| 12:36:51 | gibi | sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details | |
| 12:37:22 | gibi | sean-k-mooney: I agree that configuring this will be really deployment specific | |
| 12:38:45 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463 | |
| 12:38:55 | gibi | sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns. | |
| 12:38:56 | sean-k-mooney | i worked with other teams at intel that worked on ovs and vsperf in opnfv | |
| 12:39:31 | sean-k-mooney | gibi: yep they do | |
| 12:39:45 | sean-k-mooney | but for public clouds this gets harder as you dont know the workload | |
| 12:39:53 | gibi | yes | |
| 12:39:58 | sean-k-mooney | i assume as part of this there is an enforcemnt elemnt | |
| 12:40:12 | sean-k-mooney | e.g. a max pps qos policy | |
| 12:40:30 | sean-k-mooney | i think tc can enforce that | |
| 12:40:34 | gibi | yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst | |
| 12:41:14 | gibi | at the moment I'm looking into min pps only for the scheduling decision but later there might be enforcement for that as well | |
| 12:41:51 | sean-k-mooney | good because we dont currenlty allow enforcing that in nova flavor extra specs with quota:vif_* and i kind of want to kill those | |
| 12:42:20 | sean-k-mooney | well we can do what we do for bandwith today | |
| 12:42:32 | sean-k-mooney | e.g. set min = max | |
| 12:42:42 | sean-k-mooney | and just schulde on min and enfroce max | |
| 12:42:57 | sean-k-mooney | which sice we dont over subscibe means we could effectivly enforce min | |
| 12:43:10 | gibi | yes, that is possible here too | |
| 12:43:19 | sean-k-mooney | i dont think ovs support min pps | |
| 12:43:27 | sean-k-mooney | even for sriov that is a challange | |
| 12:45:02 | gibi | three are some differences with BW. The packet rate inventory seems to be directionless in nature | |
| 12:45:48 | gibi | and in case of OVS it more belongs to the softswitch level than to the bridge level | |
| 12:46:15 | gibi | and for SRIOV packet rate is not really a limiting factor | |
| 12:46:19 | gibi | afaik | |
| 12:46:23 | sean-k-mooney | the same is true for pps form an ovs point of view | |
| 12:46:40 | sean-k-mooney | bridges are meaningless for the most part | |
| 12:47:03 | sean-k-mooney | gibi: it kind fo still is | |
| 12:47:23 | sean-k-mooney | even with sriov 64b packets will limit the bandwith fo the nic | |
| 12:47:44 | sean-k-mooney | typeically they can handel 1 or 2 VF at that rate but not multiple | |
| 12:48:06 | gibi | sean-k-mooney: interesting, I did not know that | |
| 12:49:06 | gibi | so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small? | |
| 12:49:16 | sean-k-mooney | normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps | |
| 12:49:53 | sean-k-mooney | so it depens where the limit will come in would be vm to vm traffic on the same host | |
| 12:50:22 | sean-k-mooney | the other place its a proble is prot mirroing | |
| 12:50:35 | sean-k-mooney | our running vf in trusted mode | |
| 12:51:27 | sean-k-mooney | setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them | |
| 12:51:54 | sean-k-mooney | it forcs the nic to copy a subset of packets to multiple vfs | |
| 12:53:24 | gibi | thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent | |
| 12:53:27 | sean-k-mooney | gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning | |
| 12:54:12 | gibi | good to know | |
| 12:55:16 | sean-k-mooney | https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov | |
| 12:55:42 | sean-k-mooney | they are using isolcpus=1-19,21-39 on the kernel in part 4 | |
| 12:55:49 | sean-k-mooney | 6.2.4 | |
| 12:57:13 | gibi | yeah we also separate tenat cpus from host cpus and from ovs cpus | |
| 12:57:20 | gibi | with isolcpu | |
| 12:57:40 | sean-k-mooney | really i dont think people should use that on non realtime hosts | |
| 12:57:59 | sean-k-mooney | its dprecated upstream in the kernel and its kind fo a blunt hammer | |
| 12:58:04 | gibi | isolcpu helps avouding host cpu load to affect the pinned VMs | |
| 12:58:15 | sean-k-mooney | in many cases you can achive the same thing without using it | |
| 12:58:23 | sean-k-mooney | it does yes | |
| 12:58:42 | sean-k-mooney | but people enable it and then deploy unpinned vms on the host | |
| 12:58:43 | gibi | cgroups would be the alternative I guess | |
| 12:59:01 | sean-k-mooney | yes tuned has support | |
| 12:59:29 | sean-k-mooney | we wanted to entirely remove supprot for isolcpus form our product and move people to use tuned instead | |
| 12:59:40 | sean-k-mooney | but we have not been able to convice the nfv dfg to do that yet | |
| 13:00:20 | sean-k-mooney | https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf | |
| 13:00:50 | sean-k-mooney | you can use tuned to configure the cgoups for you including isolating the cores form irqs | |