| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-13 | |||
| 19:38:42 | openstackgerrit | Merged openstack/nova stable/rocky: docs: Don't version links to reno docs https://review.opendev.org/662178 | |
| 19:38:50 | openstackgerrit | Merged openstack/nova stable/rocky: Include all network devices in nova diagnostics https://review.opendev.org/661962 | |
| 20:01:07 | openstackgerrit | Merged openstack/nova master: Api-guide: Add Block Device Mapping https://review.opendev.org/522084 | |
| 20:07:38 | spatel | sean-k-mooney: Question | |
| 20:08:45 | spatel | building new cloud so should i go with linuxbridge or OVS ( any performance difference here? other than features) | |
| 20:15:07 | mriedem | efried: fwiw i'm going through stephen's metadata docs change now | |
| 20:15:22 | efried | mriedem: cool, thanks for that. | |
| 20:15:32 | efried | mriedem: It still hasn't rendered, right? | |
| 20:15:44 | efried | I guess I could pull it down and build it myself... | |
| 20:15:49 | mriedem | that's what i did | |
| 20:17:57 | mriedem | the docs are failing to build locally for me | |
| 20:32:52 | mriedem | i see why now | |
| 20:52:42 | mriedem | gdi i've only made it 10 of 22 files in this change | |
| 21:02:35 | sean-k-mooney | spatel: linuxbirdge is not that well tested i personally am much more comfortable with ovs however some large public cloud have use linux birdge like rackspace | |
| 21:03:12 | sean-k-mooney | spatel: in generall linuxbrige will have fewer fancy networking feature and tends not to scale beyond 10G | |
| 21:03:26 | sean-k-mooney | but its also simpler | |
| 21:04:07 | sean-k-mooney | so depending on your netwroking backgorund and the team that will be maintaining the clould you could be trading performace for lower operationgs costs | |
| 21:04:36 | spatel | sean-k-mooney: thank you, i am already using LinuxBridge in my cloud but yes its very simple, first time i am going to play with OVS so not sure what and where i am going to face complexity | |
| 21:05:12 | spatel | I may planning to use DPDK so i have to learn OVS | |
| 21:06:03 | sean-k-mooney | if your starting out with ovs its simpler to start with kernel ovs and learn how to use ovs-dpdk after you are comfortable with openflow and ovs in general | |
| 21:06:23 | sean-k-mooney | it just helps not to add dpdk to the learning curve | |
| 21:06:59 | sean-k-mooney | ovs is the default netrok backend for like 80% of all deployments in one form or another | |
| 21:07:45 | spatel | sean-k-mooney: hmm | |
| 21:08:48 | spatel | Is it possible i deploy ovs with general configuration and then slowly rollout ovs-dpdk compute by compute | |
| 21:09:18 | spatel | Can i have mix environment like currently i have SR-IOV with general compute | |
| 21:10:58 | sean-k-mooney | spatel: yes you can mix ovs and ovs-dpdk in the same cloud and you can also have sr-iov in that cloud | |
| 21:11:20 | sean-k-mooney | generally you will use a different host aggage to seperate your dpdk nodes that need hugepages form the rest that dont | |
| 21:11:30 | spatel | I think i should go with OVS so i can have all kind of function and feature available.. | |
| 21:12:00 | spatel | what are the basic requirement for dpdk? | |
| 21:12:11 | mriedem | efried: please don't fix the docs build break | |
| 21:12:16 | mriedem | i just dumped 42 comments on that change | |
| 21:12:34 | efried | mriedem: okay | |
| 21:12:37 | spatel | NIC support, hugepages etc.. | |
| 21:13:19 | sean-k-mooney | most nicst that support sriov support dpdk but there is a list here | |
| 21:13:20 | sean-k-mooney | https://core.dpdk.org/supported/ | |
| 21:13:47 | sean-k-mooney | that is more a list of driver but you likely have a nic that is supported | |
| 21:14:17 | sean-k-mooney | hugepages are needed for the guest and for the vswtich | |
| 21:14:25 | sean-k-mooney | other then that there are no special requiremetns | |
| 21:14:45 | mriedem | with that i think i'm done for the day | |
| 21:14:58 | spatel | I have - 04:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Backplane Connection (rev 01) | |
| 21:15:29 | sean-k-mooney | spatel: yep they use the ixgbe dirver and are supported | |
| 21:15:29 | spatel | How much hugepage required for vswitch? | |
| 21:15:46 | spatel | My all compute nodes are 32G memory ( some 64G) | |
| 21:16:25 | sean-k-mooney | dpending on your workload 2-4 G if you are using jumbo frams 1-2G is fine for standard 1500b packets | |
| 21:16:34 | sean-k-mooney | i have tunned it to less in the past | |
| 21:16:54 | sean-k-mooney | it jsut depends on your data rates and how long your queuse are going to be | |
| 21:17:36 | spatel | When you saying hugepage for vswitch does that means i have to give some dedicated hugepage to vSwitch right? | |
| 21:17:50 | sean-k-mooney | spatel: https://developers.redhat.com/blog/2018/03/16/ovs-dpdk-hugepage-memory/ | |
| 21:17:58 | sean-k-mooney | spatel: yes | |
| 21:18:40 | spatel | Nice doc.. | |
| 21:18:54 | sean-k-mooney | ovs-dpdk uses hugepage memroy to process packets so that it can DMA transfer packets directly to and from the nic bypassing the kernel | |
| 21:18:58 | spatel | sean-k-mooney: do i also need to give some dedicated CPU to vSwitch? | |
| 21:19:56 | sean-k-mooney | yes ideally 1 PMD(poll mode driver) core per numa node | |
| 21:20:41 | sean-k-mooney | ovs-dpdk scalse effectivly liniarly with the number of cpu cores you give it so you can tune it up or down depending on your data rates | |
| 21:20:52 | spatel | I have two numa zone so i have to give 2 CPU core right? is that CPU core will be pin with vSwitch? | |
| 21:21:30 | sean-k-mooney | yes dpdk will run in a bussy loop waiting for packets and will consume the full core | |
| 21:21:45 | sean-k-mooney | even when idel | |
| 21:22:15 | sean-k-mooney | it does that to miniumese latency but it basically means you cant run other thing on the core at the same time | |
| 21:23:06 | sean-k-mooney | per packet ovs-dpdk is much more efficent then kernel ovs however kernel ovs use kernel thread to process the packet and does not consume cpus cycles when idel | |
| 21:23:12 | sean-k-mooney | so its a trade off | |
| 21:23:25 | sean-k-mooney | ovs-dpdk cpu usage will be fixed based on the cores you allcoate it | |
| 21:24:00 | spatel | core allocation process is automatic or part of configuration? | |
| 21:24:01 | sean-k-mooney | kernel ovs will use less or more cpus as needed includeing stealing cpu cycles form the guests if you do not tune the kernel to prevent that | |
| 21:24:10 | sean-k-mooney | spatel: part of the configuration | |
| 21:24:32 | spatel | ah! ok.. | |
| 21:24:52 | sean-k-mooney | you set a bit mask in the ovs db to choose what core dpdk is allows to use | |
| 21:25:01 | spatel | Do i need to create neutron port for dpdk (like i am create currently for SR-IOV)? | |
| 21:25:32 | sean-k-mooney | you can but no. it used vnic_type=normal which is the default | |
| 21:25:39 | sean-k-mooney | so you dont need to precreate them | |
| 21:25:55 | sean-k-mooney | form a user point of view its identiacl to linux bridge or kernel ovs | |
| 21:26:05 | sean-k-mooney | other then the fact you need a flaovr with hugepages | |
| 21:26:12 | spatel | Perfect!! i love that part, currently in SR-IOV i have to create port in advance to map with vms | |
| 21:26:21 | sean-k-mooney | yes | |
| 21:26:48 | sean-k-mooney | which is annoying but there are reason why we require that | |
| 21:26:51 | spatel | anyway my all vms running with hugepages at moment | |
| 21:27:21 | spatel | Question why does vms required hugepage for dpdk? | |
| 21:27:49 | sean-k-mooney | it has to do with how the packeets are transfered between the vswitch and the vm | |
| 21:27:58 | sean-k-mooney | we use a technology called vhost-user | |
| 21:28:35 | sean-k-mooney | which allows the vswitch to mapp the virtio-rings( tx and rx queues) of the guest nic into the memory of the vswitch | |
| 21:29:10 | spatel | Interesting.. | |
| 21:29:17 | sean-k-mooney | to do that we need to use gest memroy that is mmap shared with an open file discripto, is contiguaon and preallcoated | |
| 21:29:26 | sean-k-mooney | hugepages do all of the above | |
| 21:29:40 | sean-k-mooney | technically you can acive the same wiht out hugepages | |
| 21:29:55 | sean-k-mooney | its just slower, more work and not supported by openstack by defualt | |
| 21:30:17 | spatel | Got it.. so hugepage has big role here | |
| 21:30:34 | sean-k-mooney | if you are using a new enought ovs-dpdk you can actully get 0 copy packet transmistion betwwen the guest and the nic | |
| 21:31:17 | openstackgerrit | Merged openstack/nova master: update constraints url https://review.opendev.org/664771 | |
| 21:31:19 | sean-k-mooney | e.g. when the guest kernel rights the packet to the tx queue it is nver copyied and is drieectly trasmited by the nic similar to sriov | |
| 21:31:34 | sean-k-mooney | 0 copy recive i think is also posibel but experimental | |
| 21:32:12 | sean-k-mooney | that obviorsly depend on if you are using vxlan or vlans feature that may or may not be supported in your nic hardware | |
| 21:32:20 | spatel | ah! ok.. in short it bypass the kernel | |
| 21:32:47 | sean-k-mooney | yep and does everything in userspacce or in hardaware | |
| 21:33:07 | spatel | Do you guys running dpdk in production? | |
| 21:33:31 | spatel | very few people i found using dpdk in prod cloud | |
| 21:33:36 | sean-k-mooney | well i dont run a production cloud. i just work on upstream stuff | |
| 21:33:56 | sean-k-mooney | spatel: but redhat has customer using it in production yest | |
| 21:34:01 | sean-k-mooney | *yes | |
| 21:34:19 | sean-k-mooney | it is much less common then kernel ovs or sriov | |
| 21:34:28 | spatel | Let see how successful i can run in my production. | |
| 21:35:05 | sean-k-mooney | if you want the full intel pitch then https://software.intel.com/en-us/articles/yahoo-japan-accelerates-l7-performance-using-open-vswitch-with-dpdk | |