Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2025-11-13
16:23:08 vsaienko switchdev - sriov + asap2 hardware offload, vdpa-packed - its packed ring + 8 queues, virtio def - is default flavor, no multiqueue or packed ring, virtio-packed - packed ring + 8 queues
16:23:20 vsaienko vdpa with hardware offload as well
16:23:45 vsaienko between hosts VXLAN network, mtu 1450 (no jumboframes) on all configurations
16:23:46 sean-k-mooney ok because the defautl is usign vdpa so its not correct to call it virtio
16:24:06 sean-k-mooney ah you also adding vxlan overhead
16:24:45 sean-k-mooney so is this inter host traffic i.e. 2 vms on diffent hosts or 2 vms on the same host
16:25:01 vsaienko virtio def is default neutron port =normal which is virtio
16:25:16 sean-k-mooney its technaily kernel vhost
16:25:18 vsaienko all scenarios between VMs on different hosts
16:26:01 sean-k-mooney virtio woudl be the virto stack in ovs without the kernel vhost-net module offloading it
16:27:29 sean-k-mooney this looks to me like you are hittig a code path that is not supproted for hardware offload
16:28:00 sean-k-mooney for example the conenction tracker used for security groups is not fully offloadable to hardware
16:28:08 sean-k-mooney are you using ml2/ovs?
16:28:35 vsaienko its ovs, no qos no security groups, in TC I see that traffic is offloaded
16:29:25 sean-k-mooney well in https://jumpshare.com/s/xHU2UTQoKoeKwAWDFsa6
16:29:50 sean-k-mooney you are not hitting line rate env in the swtichdev case which is effectivly jsut using sriov for the dataplane
16:29:59 sean-k-mooney with ovs providing the contol plane
16:30:10 vsaienko yes
16:30:29 sean-k-mooney that to me suggest you are hittign bottel necks in teh trafic generateion as well
16:31:13 vsaienko not sure between hosts on same hypervisor iperf can show more, its new version with multithreads
16:31:35 sean-k-mooney well ierf it not really godo for doing real testing
16:31:43 sean-k-mooney you can use -p to have it use multipel cores
16:32:02 sean-k-mooney but if you want ot test this properly you eigher need a hardware traffic generator or somethign like trex
16:32:17 vsaienko yes, there are scenarios with p1 - its -P 1, p30 - -P 30
16:32:23 sean-k-mooney https://trex-tgn.cisco.com/
16:33:13 sean-k-mooney right but for 1400 by packets you shoudl be able to hit 10G even with 1 thread
16:34:03 sean-k-mooney if the hsot are other wise idel with 1400 by packets even standar kernel ovs shoudl be able to hit 10G
16:34:26 sean-k-mooney so to me the performac eof this host/vms is very much below what i woudl expect
16:36:00 sean-k-mooney vsaienko: what version of opesntack are you using
16:36:20 vsaienko but why with sriov it provides 6+gbps
16:36:27 vsaienko its epoxy
16:36:42 sean-k-mooney so 6GBPS is also too low for sriov
16:36:42 vsaienko 3.x openvswitch
16:37:01 vsaienko maybe indeed its a limitation of iperf
16:37:26 sean-k-mooney so if your using epoxy you shoudl have https://github.com/openstack/os-vif/blob/master/vif_plug_ovs/ovs.py#L107-L126
16:37:45 sean-k-mooney but can you check in ovs what the qos policy on the ovs port is set too
16:37:53 sean-k-mooney and confirm its linux-noop
16:37:57 vsaienko qos plugin is disabled on my env
16:38:06 sean-k-mooney that is not what im asking
16:38:27 vsaienko but thanks for pointing this out, I haven't seen this option
16:38:30 sean-k-mooney in new version fo systemd or the kernel the default qdisg was changed to fq_codel
16:38:55 sean-k-mooney now we added this option to make sure tha tthat was disabled on teh tap devices
16:39:08 vsaienko interesting
16:39:11 vsaienko let me check this
16:39:11 sean-k-mooney because if we dont ti results in massive perfroamce hti
16:39:21 vsaienko it should be default
16:39:24 sean-k-mooney yep
16:39:47 sean-k-mooney so we need ot check that 1 the policy in ovs set by os-vif is linux-noop
16:40:23 sean-k-mooney but we shoudl also check that the representor netdev added to ovs does not have a qos policy applied with tc
16:40:53 vsaienko its not explicitly set, so default is picked
16:41:39 sean-k-mooney right so im askign you to conrimf this because this was never tested for vdpa or hardware offloaded ovs
16:41:46 sean-k-mooney it shoudl get appled but it may not be
16:42:01 sean-k-mooney and if its not beign set in the ovs db then it coudl reduce the performacne
16:43:43 vsaienko https://paste.openstack.org/show/bZeQE1UxDhnNWIpJmiCb/
16:44:31 sean-k-mooney yep that the problem
16:44:35 sean-k-mooney its using fqcodel
16:44:57 sean-k-mooney create /etc/sysctl.conf.d/99_qdisk.conf and add net.core.default_qdisc = pfifo
16:45:24 sean-k-mooney thatthen apply it and try doing it again with a new vm
16:45:39 vsaienko ok thanks, let me try
16:46:33 sean-k-mooney well that might work but lets confirm something
16:46:40 sean-k-mooney what is eth1 in this case
16:46:49 sean-k-mooney is it teh respreentor netdev for the vdpa device
16:47:10 sean-k-mooney i expect it si based on the altname
16:47:17 sean-k-mooney enp33s0f1npf1vf1
16:47:48 sean-k-mooney if that is the case and the perfoamce impves when you set your defautl qdisc ot pfifo
16:48:18 sean-k-mooney the the issue you are hittig is the os-vif change is not properly handelign this edge case
16:49:09 sean-k-mooney it was intened to fix https://bugs.launchpad.net/os-vif/zed/+bug/2017868 for standard kernel ovs
16:54:19 sean-k-mooney vsaienko: if your using iperf to test you shousl also ensure that you have disable fqcodel i nteh guest. https://trex-tgn.cisco.com/ is a lot more work to set up but it woudl elimiante any worry that the traffic genreator is the bottel neck. maybe book mark that for future reading
16:57:58 vsaienko ack let me try this
16:58:00 vsaienko thanks
17:29:15 vsaienko sean-k-mooney: pfifo improves situation https://jumpshare.com/s/GkgTEynhPdHlvWptjih5
17:31:28 gmaan gibi: RE default value change of config options (966016): yes it is fine as long as we are adding it in upgrade release notes. oslo.config have way to respect both old and new when config name is changed but for default value change, yes upgrade notes is right things.
17:33:15 sean-k-mooney vsaienko: ack it will likely improve the other results too if you ened up testing them
17:33:47 sean-k-mooney we know it can limit perforamce to <10% form some reports
17:34:01 sean-k-mooney diffent backend are affected more or less
17:34:26 sean-k-mooney os-vif obviouly has a blind spot for hardware offloaded ports that we shoudl fix
17:34:47 sean-k-mooney but the root of the issue was the changeing fo behviro in ovs and systemd
17:35:20 sean-k-mooney the defuatl qdisk was change about 3-4 year ago it used to be pfifo by default
17:35:41 sean-k-mooney ovs also change to not ignoring/removing the qdisc on port added to it by defualt
17:36:19 gibi gmaan: thanks! I added the upgrade reno
17:37:50 sean-k-mooney vsaienko: https://wiki.archlinux.org/title/Advanced_traffic_control#CoDel_and_Fair_Queueing_CoDel
17:38:09 sean-k-mooney that is some of the history on this changes it hapspen din systemd 217
17:41:45 opendevreview Nicolai Ruckel proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/959682
17:44:45 gmaan anyone face/know the issue of not seeing 'comoute.<hostname>' queue in devstack env ? I am listing all queues (sudo rabiitmqctl list_queues) and it show all queues except compute service queues (oslo.message create three queues per compute service 1. 'compute' 2. 'compute.<hostname>' 3. compute fanout with uuids)
17:45:32 gmaan it list all other queues for example sch, conductor, cinder queues
17:47:07 opendevreview Nicolai Ruckel proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.opendev.org/c/openstack/nova/+/959682
17:48:10 nicolairuckel sean-k-mooney, I found a few more places with code that wasn't necessary anymore
17:48:29 nicolairuckel I think it should be a lot cleaner now
17:48:37 sean-k-mooney yep im +1 on your patch after skimming it
17:48:51 sean-k-mooney ill upgrade to +2 after i have time to test it and read over it again
17:48:59 nicolairuckel perfect :)
17:50:51 sean-k-mooney gmaan: so it shoudl be "compute.{CONF.host}" but if the queue is not there it normally is a netwrokign or auth issue, i belive it can also happen if we exit because fo say the missing compute node id file
17:51:05 sean-k-mooney i.e. if the agent stop runing and the queue gets deleted
17:51:26 gmaan compute service is up, instance are booting fine so no issue on that side
17:51:53 sean-k-mooney are you lookig at the correct exchange?
17:52:06 gmaan if you have devstack running up, can you try if you see the comoute queues?
17:52:20 gmaan yes, nove and even i checked all exhange
17:52:34 sean-k-mooney sure i have one
17:52:49 gmaan before i boot new VM and install devstack i just want to cofirm if something in my machine
17:52:51 gmaan thanks

Earlier   Later