Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-23
12:24:30 sean-k-mooney i assume neutron will be dumb and just have the operator say the capastity is X in a config like bandwith
12:24:34 gibi I'm not well educated in OVS
12:24:56 gibi yes the inventory would be config driven like bandwidth
12:25:26 sean-k-mooney gibi: the pps number is differnt for vlan network vs vxlan network on the same ovs on the same host
12:25:55 sean-k-mooney e.g. l3 tunneled networks use more cpu cycle to decap and encap then vlan
12:26:23 sean-k-mooney in the case of ovs-dpdk vm to vm traffic on the same host uses more cycle then vm to physical network
12:26:38 sean-k-mooney since vm to vm can levgerage as many hardware offloads
12:27:01 sean-k-mooney so depening on the direction of traffic and the overlay use the capsity will change
12:27:33 sean-k-mooney you can obvioulsy deploy and test for this but it also will change based on other factors
12:27:56 sean-k-mooney such as security group rules implemnte in contrack and the traffic profile.
12:28:43 sean-k-mooney e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking
12:29:11 sean-k-mooney so the pps will be lower with lots of new connection being established vs steady state
12:29:29 sean-k-mooney multicast/broadcast vs unicast also is a factor
12:29:47 sean-k-mooney gibi: so im glad figuring that out will be the operators problem
12:33:54 lyarwood https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now.
12:34:15 lyarwood I'm still working on the grenade change ontop of it
12:36:51 gibi sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details
12:37:22 gibi sean-k-mooney: I agree that configuring this will be really deployment specific
12:38:45 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463
12:38:55 gibi sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns.
12:38:56 sean-k-mooney i worked with other teams at intel that worked on ovs and vsperf in opnfv
12:39:31 sean-k-mooney gibi: yep they do
12:39:45 sean-k-mooney but for public clouds this gets harder as you dont know the workload
12:39:53 gibi yes
12:39:58 sean-k-mooney i assume as part of this there is an enforcemnt elemnt
12:40:12 sean-k-mooney e.g. a max pps qos policy
12:40:30 sean-k-mooney i think tc can enforce that
12:40:34 gibi yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst
12:41:14 gibi at the moment I'm looking into min pps only for the scheduling decision but later there might be enforcement for that as well
12:41:51 sean-k-mooney good because we dont currenlty allow enforcing that in nova flavor extra specs with quota:vif_* and i kind of want to kill those
12:42:20 sean-k-mooney well we can do what we do for bandwith today
12:42:32 sean-k-mooney e.g. set min = max
12:42:42 sean-k-mooney and just schulde on min and enfroce max
12:42:57 sean-k-mooney which sice we dont over subscibe means we could effectivly enforce min
12:43:10 gibi yes, that is possible here too
12:43:19 sean-k-mooney i dont think ovs support min pps
12:43:27 sean-k-mooney even for sriov that is a challange
12:45:02 gibi three are some differences with BW. The packet rate inventory seems to be directionless in nature
12:45:48 gibi and in case of OVS it more belongs to the softswitch level than to the bridge level
12:46:15 gibi and for SRIOV packet rate is not really a limiting factor
12:46:19 gibi afaik
12:46:23 sean-k-mooney the same is true for pps form an ovs point of view
12:46:40 sean-k-mooney bridges are meaningless for the most part
12:47:03 sean-k-mooney gibi: it kind fo still is
12:47:23 sean-k-mooney even with sriov 64b packets will limit the bandwith fo the nic
12:47:44 sean-k-mooney typeically they can handel 1 or 2 VF at that rate but not multiple
12:48:06 gibi sean-k-mooney: interesting, I did not know that
12:49:06 gibi so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small?
12:49:16 sean-k-mooney normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps
12:49:53 sean-k-mooney so it depens where the limit will come in would be vm to vm traffic on the same host
12:50:22 sean-k-mooney the other place its a proble is prot mirroing
12:50:35 sean-k-mooney our running vf in trusted mode
12:51:27 sean-k-mooney setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them
12:51:54 sean-k-mooney it forcs the nic to copy a subset of packets to multiple vfs
12:53:24 gibi thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent
12:53:27 sean-k-mooney gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning
12:54:12 gibi good to know
12:55:16 sean-k-mooney https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov
12:55:42 sean-k-mooney they are using isolcpus=1-19,21-39 on the kernel in part 4
12:55:49 sean-k-mooney 6.2.4
12:57:13 gibi yeah we also separate tenat cpus from host cpus and from ovs cpus
12:57:20 gibi with isolcpu
12:57:40 sean-k-mooney really i dont think people should use that on non realtime hosts
12:57:59 sean-k-mooney its dprecated upstream in the kernel and its kind fo a blunt hammer
12:58:04 gibi isolcpu helps avouding host cpu load to affect the pinned VMs
12:58:15 sean-k-mooney in many cases you can achive the same thing without using it
12:58:23 sean-k-mooney it does yes
12:58:42 sean-k-mooney but people enable it and then deploy unpinned vms on the host
12:58:43 gibi cgroups would be the alternative I guess
12:59:01 sean-k-mooney yes tuned has support
12:59:29 sean-k-mooney we wanted to entirely remove supprot for isolcpus form our product and move people to use tuned instead
12:59:40 sean-k-mooney but we have not been able to convice the nfv dfg to do that yet
13:00:20 sean-k-mooney https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf
13:00:50 sean-k-mooney you can use tuned to configure the cgoups for you including isolating the cores form irqs
13:01:18 sean-k-mooney thats what we use by default for non realtime hosts now
13:03:44 gibi I see
13:04:55 sean-k-mooney in the cgroup way you can prevent the kernel schduling vms to the core indepently of balancing the vms on cores
13:05:03 sean-k-mooney isolcpus does both
13:05:28 sean-k-mooney so you can use the tuned/cgrop method with floating vms too not jsut pinned
13:06:22 sean-k-mooney but isolcpu does a few things that you cant quite do form userspace fully or at least with tuned at this point so they still use and support itn for nfv usecases
13:06:30 gibi I see. I think we only support pinned vms by default
13:06:48 sean-k-mooney that both complicates and simplcies your life :)
13:06:53 gibi yeah
13:06:55 gibi soo telco
13:07:01 gibi :)
13:07:04 sean-k-mooney complciates becasue numa and simplfies because only numa
13:07:28 gibi yeah
13:07:44 sean-k-mooney if you dont have to deal with peopel mixing numa and non numa vms its nice
13:08:25 gibi yes we sort of force numa with pinned cpus and huge pages
13:19:59 openstackgerrit Merged openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834
13:29:58 bauzas folks, if you need to discuss with kashyap, he's off IRC as he has an Internet issue with his ISP
13:31:08 sean-k-mooney i did have one thing that came up last night
13:32:06 sean-k-mooney stars here http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-03-22.log.html#t2021-03-22T19:34:19
13:32:49 sean-k-mooney basically qemu change the cpu flags that are set in a specific cpu model and if you dont use version machine types that will break live migration
13:33:29 sean-k-mooney we may or may not be able to work around that in nova via the migration xml
13:34:25 sean-k-mooney but the end effect is that eypc-ibrs when the vm was booted on the source host nolgere results in the same cpu feature enabel if you boot a vm on the same host or a different one with the same libvirt xml
13:34:40 bauzas sean-k-mooney: kashyap dunno when he will be back
13:34:40 sean-k-mooney wehn you are using unversion machine types
13:35:08 sean-k-mooney its not supper urgent at least for me
13:35:58 sean-k-mooney but rouk has already confrim that setting an explit machine type in nova or using kashyap recent patch to allow removing feature flags both cannot fix the issue

Earlier   Later