Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-23
12:26:38 sean-k-mooney since vm to vm can levgerage as many hardware offloads
12:27:01 sean-k-mooney so depening on the direction of traffic and the overlay use the capsity will change
12:27:33 sean-k-mooney you can obvioulsy deploy and test for this but it also will change based on other factors
12:27:56 sean-k-mooney such as security group rules implemnte in contrack and the traffic profile.
12:28:43 sean-k-mooney e.g. setting up and tearing downs lots of tcp connection requires all the inital packets to go through contrack to do the state tracking
12:29:11 sean-k-mooney so the pps will be lower with lots of new connection being established vs steady state
12:29:29 sean-k-mooney multicast/broadcast vs unicast also is a factor
12:29:47 sean-k-mooney gibi: so im glad figuring that out will be the operators problem
12:33:54 lyarwood https://review.opendev.org/c/openstack/nova/+/768466 - anyone able to review this change introducing a nova-live-migration-ceph job? Everything has finally merged so we should be good to go now.
12:34:15 lyarwood I'm still working on the grenade change ontop of it
12:36:51 gibi sean-k-mooney: thanks for the info, I don't have this deep networking knowledge, so I appreciate these details
12:37:22 gibi sean-k-mooney: I agree that configuring this will be really deployment specific
12:38:45 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor https://review.opendev.org/c/openstack/nova/+/778463
12:38:55 gibi sean-k-mooney: as far as I understand some of our big deployers tend use a unified network setup, like all vlan. Also the these deployers tend to do deep performance testing with their config, so I assume they will figure out a pps number for their specific deployment and traffic patterns.
12:38:56 sean-k-mooney i worked with other teams at intel that worked on ovs and vsperf in opnfv
12:39:31 sean-k-mooney gibi: yep they do
12:39:45 sean-k-mooney but for public clouds this gets harder as you dont know the workload
12:39:53 gibi yes
12:39:58 sean-k-mooney i assume as part of this there is an enforcemnt elemnt
12:40:12 sean-k-mooney e.g. a max pps qos policy
12:40:30 sean-k-mooney i think tc can enforce that
12:40:34 gibi yeah the max pps is already proposed https://review.opendev.org/c/openstack/neutron-specs/+/779940/1/specs/wallaby/qos_pps_rule.rst
12:41:14 gibi at the moment I'm looking into min pps only for the scheduling decision but later there might be enforcement for that as well
12:41:51 sean-k-mooney good because we dont currenlty allow enforcing that in nova flavor extra specs with quota:vif_* and i kind of want to kill those
12:42:20 sean-k-mooney well we can do what we do for bandwith today
12:42:32 sean-k-mooney e.g. set min = max
12:42:42 sean-k-mooney and just schulde on min and enfroce max
12:42:57 sean-k-mooney which sice we dont over subscibe means we could effectivly enforce min
12:43:10 gibi yes, that is possible here too
12:43:19 sean-k-mooney i dont think ovs support min pps
12:43:27 sean-k-mooney even for sriov that is a challange
12:45:02 gibi three are some differences with BW. The packet rate inventory seems to be directionless in nature
12:45:48 gibi and in case of OVS it more belongs to the softswitch level than to the bridge level
12:46:15 gibi and for SRIOV packet rate is not really a limiting factor
12:46:19 gibi afaik
12:46:23 sean-k-mooney the same is true for pps form an ovs point of view
12:46:40 sean-k-mooney bridges are meaningless for the most part
12:47:03 sean-k-mooney gibi: it kind fo still is
12:47:23 sean-k-mooney even with sriov 64b packets will limit the bandwith fo the nic
12:47:44 sean-k-mooney typeically they can handel 1 or 2 VF at that rate but not multiple
12:48:06 gibi sean-k-mooney: interesting, I did not know that
12:49:06 gibi so the pps will be the limit before the bandwidth if there is a lot of VFs and the packets are small?
12:49:16 sean-k-mooney normlaly the north south driction can handel full line rate at 64b packets for 10G thats about 14.4mpps
12:49:53 sean-k-mooney so it depens where the limit will come in would be vm to vm traffic on the same host
12:50:22 sean-k-mooney the other place its a proble is prot mirroing
12:50:35 sean-k-mooney our running vf in trusted mode
12:51:27 sean-k-mooney setting a vf to trusted mode consumes a lot of bandwith ignorign the security implications espcially if there are multipel of them
12:51:54 sean-k-mooney it forcs the nic to copy a subset of packets to multiple vfs
12:53:24 gibi thanks, these make sense. This also points to the direction that keeping bandwidth and packet rate inventory configurable as it is really deployment and traffic pattern dependent
12:53:27 sean-k-mooney gibi: you tend to hit the pcie bottelneck before the pps bottelneck in most cases but if you wnat to get full line rate with sriov and small packet i know our nfv folks had to a lot fo tuning
12:54:12 gibi good to know
12:55:16 sean-k-mooney https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/network_functions_virtualization_planning_and_configuration_guide/part-sriov-nfv-configuration#sect-configuring-sriov
12:55:42 sean-k-mooney they are using isolcpus=1-19,21-39 on the kernel in part 4
12:55:49 sean-k-mooney 6.2.4
12:57:13 gibi yeah we also separate tenat cpus from host cpus and from ovs cpus
12:57:20 gibi with isolcpu
12:57:40 sean-k-mooney really i dont think people should use that on non realtime hosts
12:57:59 sean-k-mooney its dprecated upstream in the kernel and its kind fo a blunt hammer
12:58:04 gibi isolcpu helps avouding host cpu load to affect the pinned VMs
12:58:15 sean-k-mooney in many cases you can achive the same thing without using it
12:58:23 sean-k-mooney it does yes
12:58:42 sean-k-mooney but people enable it and then deploy unpinned vms on the host
12:58:43 gibi cgroups would be the alternative I guess
12:59:01 sean-k-mooney yes tuned has support
12:59:29 sean-k-mooney we wanted to entirely remove supprot for isolcpus form our product and move people to use tuned instead
12:59:40 sean-k-mooney but we have not been able to convice the nfv dfg to do that yet
13:00:20 sean-k-mooney https://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf
13:00:50 sean-k-mooney you can use tuned to configure the cgoups for you including isolating the cores form irqs
13:01:18 sean-k-mooney thats what we use by default for non realtime hosts now
13:03:44 gibi I see
13:04:55 sean-k-mooney in the cgroup way you can prevent the kernel schduling vms to the core indepently of balancing the vms on cores
13:05:03 sean-k-mooney isolcpus does both
13:05:28 sean-k-mooney so you can use the tuned/cgrop method with floating vms too not jsut pinned
13:06:22 sean-k-mooney but isolcpu does a few things that you cant quite do form userspace fully or at least with tuned at this point so they still use and support itn for nfv usecases
13:06:30 gibi I see. I think we only support pinned vms by default
13:06:48 sean-k-mooney that both complicates and simplcies your life :)
13:06:53 gibi yeah
13:06:55 gibi soo telco
13:07:01 gibi :)
13:07:04 sean-k-mooney complciates becasue numa and simplfies because only numa
13:07:28 gibi yeah
13:07:44 sean-k-mooney if you dont have to deal with peopel mixing numa and non numa vms its nice
13:08:25 gibi yes we sort of force numa with pinned cpus and huge pages
13:19:59 openstackgerrit Merged openstack/nova master: Dynamically archive FK related records in archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/773834
13:29:58 bauzas folks, if you need to discuss with kashyap, he's off IRC as he has an Internet issue with his ISP
13:31:08 sean-k-mooney i did have one thing that came up last night
13:32:06 sean-k-mooney stars here http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2021-03-22.log.html#t2021-03-22T19:34:19
13:32:49 sean-k-mooney basically qemu change the cpu flags that are set in a specific cpu model and if you dont use version machine types that will break live migration
13:33:29 sean-k-mooney we may or may not be able to work around that in nova via the migration xml
13:34:25 sean-k-mooney but the end effect is that eypc-ibrs when the vm was booted on the source host nolgere results in the same cpu feature enabel if you boot a vm on the same host or a different one with the same libvirt xml
13:34:40 sean-k-mooney wehn you are using unversion machine types
13:34:40 bauzas sean-k-mooney: kashyap dunno when he will be back
13:35:08 sean-k-mooney its not supper urgent at least for me
13:35:58 sean-k-mooney but rouk has already confrim that setting an explit machine type in nova or using kashyap recent patch to allow removing feature flags both cannot fix the issue
13:36:12 sean-k-mooney since neither ake effect on live migration
13:36:53 sean-k-mooney it wont break use downstream since we use versioned machine types but it will break anyone that uses an unversioned machine type after a qemu update
13:37:22 sean-k-mooney that include our downstream customer if they set an unversion machine type in the nova config or in the glance image
14:30:47 lyarwood gibi / stephenfin ; https://review.opendev.org/c/openstack/nova/+/733627 - could either of you look at this sometime this week, wsgi fix when launched by mod_wsgi
14:33:23 gibi lyarwood: ack
14:33:27 lyarwood thanks

Earlier   Later