| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 17:39:10 | sean-k-mooney | by setting hw:numa_nodes=2 you will have half the cpus on one numa node and half on the other | |
| 17:39:23 | sean-k-mooney | memory will also be equally split | |
| 17:40:13 | sean-k-mooney | provided there is a pci device free on at least one of the 2 numa nodes assocaicated with the vm vcpus then we will allow the vm to boot | |
| 17:40:26 | spatel | if i remove numa_node=2 then it will pin down all vCPU on same node right? | |
| 17:40:57 | sean-k-mooney | correct addint hw:cpu_policy=dedicated implictly adds hw:numa_nodes=1 | |
| 17:41:25 | spatel | hmm! interesting.. | |
| 17:41:53 | sean-k-mooney | by explcitly setting hw:numa_nodes=2 it will allow both numa nodes on the host to be used but it will also limit the vm to host with 2+ numa nodes | |
| 17:41:53 | spatel | using numa_node=2 will have some performing issue right? | |
| 17:42:25 | sean-k-mooney | spatel: it can if the application in the gust itself does not understand numa affinity | |
| 17:43:02 | sean-k-mooney | it can also improve the performacne as you doubles your memory bandwith as the vm will now use memroy form 2 host numa noes/memory controlers | |
| 17:43:26 | spatel | I think time to run some test... | |
| 17:43:38 | spatel | We are media company and using VoIP base application | |
| 17:43:57 | sean-k-mooney | testing is always a good idea :) | |
| 17:44:19 | spatel | First i build openstack without SR-IOV and found performance was horrible (PPS rate was only 50k after that it start dropping packets) | |
| 17:44:20 | sean-k-mooney | as a comunity we have done a lot of work to improve numa affinity over the years | |
| 17:45:14 | spatel | I have just started learning numa stuff so i am new but it looks interesting.. | |
| 17:45:18 | sean-k-mooney | spatel: the stict pci numa affinity was added for telco usescases wehre they could not tolerate cross numa pci/sriov | |
| 17:45:56 | sean-k-mooney | spatel: it certenly is .... interesting. its also a pain in the ass but give better performace when you get it right | |
| 17:46:11 | spatel | I have some legacy hardware and i have to stick to them | |
| 17:46:27 | spatel | other side i am planning to test DPDK if its better | |
| 17:46:31 | sean-k-mooney | numa is not going away infact its become more common | |
| 17:46:44 | sean-k-mooney | dpdk is much better then kernel ovs | |
| 17:46:51 | sean-k-mooney | but its more complicated too | |
| 17:47:05 | spatel | but it doesn't need hardware dependency atleast | |
| 17:47:37 | spatel | I spent thousand of $$$$ to get SR-IOV supported card | |
| 17:47:38 | sean-k-mooney | spatel: not in the same way it requires that the guests use hugepages and that there is a dpdk driver for your nic | |
| 17:48:19 | spatel | Does it perform like SR-IOV ? | |
| 17:48:22 | sean-k-mooney | spatel: ya dpdk will be cheaeper in that sense but you will have to dedicate 1-2 cores to handel trafic for ovs-dpdk | |
| 17:48:50 | sean-k-mooney | spatel: in some cases yes. in general not quite | |
| 17:49:14 | sean-k-mooney | what data rates / traffic profiles are you targeting? | |
| 17:49:31 | spatel | currently i am deploying VoIP application on 1U server with 32 core / 32G memory. and i have 1000 servers... | |
| 17:49:34 | sean-k-mooney | 10G small packets 40G jumbo frames? a mix | |
| 17:50:01 | spatel | my peak in production 200 to 230kpps UDP packet rate | |
| 17:50:21 | sean-k-mooney | on well dpdk can handel that easilly | |
| 17:50:30 | spatel | really??? | |
| 17:50:46 | spatel | if that is the case then it will be win win solution | |
| 17:51:09 | sean-k-mooney | ya dpdk was desinged to hit 10G line rate with 64byte packets which is 14mpps | |
| 17:51:26 | spatel | we have lots of server in AWS (with sr-iov) support | |
| 17:51:40 | spatel | that is really cool! | |
| 17:51:46 | sean-k-mooney | with the right hardware it can hit 32mpps on a singel core but in generall you will see more like 6mpps | |
| 17:52:01 | spatel | we are using LinuxBridge + VLAN so i need to upgrade to OVS | |
| 17:52:18 | sean-k-mooney | its mroe or less like this | |
| 17:52:43 | sean-k-mooney | lb<ovs<sriov+macvtap<ovs-dpdk<sriov direct | |
| 17:53:08 | spatel | I have tried macvtap but that didn't work too | |
| 17:54:39 | sean-k-mooney | spatel: checkout https://dpdksummit.com/Archive/pdf/2016USA/Day02-Session04-ThomasHerbert-DPDKUSASummit2016.pdf slides 16-19 | |
| 17:55:21 | spatel | reading.. | |
| 17:56:04 | sean-k-mooney | spatel: im alittle biased as im one of the people that added ovs-dpdk support to openstack but for your data rate i think it would work quite well | |
| 17:56:42 | spatel | I need to find out how to migrate LinuxBridge to OVS | |
| 17:57:15 | sean-k-mooney | spatel: today cold migrate works. im working on fixing livemigrate | |
| 17:57:31 | spatel | cool! | |
| 17:57:40 | spatel | in SR-IOV i am not able to get that function too | |
| 17:57:48 | spatel | even bonding isn't supported | |
| 17:57:55 | sean-k-mooney | live migrate almost works we just dont update the bridge name correctly im hoping to backport that | |
| 17:58:27 | spatel | nice! if that work | |
| 17:58:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 17:58:50 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_qfd_by_cell_and_project() https://review.openstack.org/607663 | |
| 17:58:50 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | |
| 17:58:52 | sean-k-mooney | spatel: haha i think im working on all your missing features :) https://review.openstack.org/#/c/605116/ | |
| 17:59:14 | openstackgerrit | Merged openstack/nova stable/pike: nova-manage - fix online_data_migrations counts https://review.openstack.org/605840 | |
| 17:59:25 | spatel | :) | |
| 17:59:51 | spatel | i have lots of requirement :) these are just starting | |
| 18:00:02 | sean-k-mooney | my main focus this realease at least initally is livemigraton hardenign. e.g fixing edgecase like lb->ovs or sriov | |
| 18:00:02 | spatel | sean-k-mooney: thanks for help!!! | |
| 18:00:25 | nicolasbock | <freenode_sea "nicolasbock: if you are still ar"> Thanks for the tip! | |
| 18:00:26 | spatel | i didn't know freenode will be very helpful.. last 2 days i am chasing google | |
| 18:01:12 | spatel | what you use to deploy your openstack? I am using openstack-ansible | |
| 18:01:14 | sean-k-mooney | spatel: no worries. im usually hear so feel free to ping me if you have issues | |
| 18:01:44 | spatel | I am going to spend next 6 month here :) until my cloud is ready!! | |
| 18:01:53 | sean-k-mooney | spatel: for developement devstack. i used to use kolla-ansible but recently joined redhat so i proably shoudl suggest OSP | |
| 18:02:07 | spatel | we spent million dollar last year in AWS so my boss want to build own AWS :) | |
| 18:02:39 | sean-k-mooney | spatel: that is how alot of compaines end up running opnestack clouds yes | |
| 18:02:56 | spatel | indeed | |
| 18:03:04 | spatel | OSP using tripleO right? | |
| 18:03:12 | spatel | i tried and found very complicated | |
| 18:03:23 | sean-k-mooney | yes that is the officaly supprot installer from redhat | |
| 18:03:30 | sean-k-mooney | spatel: and yes it can be | |
| 18:03:42 | mnaser | does anyone know if daniel berrange hangs out on irc much? | |
| 18:03:54 | mnaser | i'm looking at this old abandoned review and i'm wondering if this is still an issue -- https://review.openstack.org/#/c/241401/ | |
| 18:04:14 | sean-k-mooney | mnaser: not on this irc but he is usally on the libvirt one | |
| 18:04:29 | mnaser | i'll try to ping him there | |
| 18:04:38 | spatel | sean-k-mooney: going to eat something, will catch you again, if any issue :) thanks again | |
| 18:06:49 | sean-k-mooney | mnaser: i think that is a bug that has been forgoten about but not nessacarily fixed | |
| 18:07:14 | mnaser | sean-k-mooney: yeah, it's not fixed, but it's been a while so i'm wondering if the whole "it doesnt working with backing" argument is no longer valid | |
| 18:07:40 | mnaser | we're setting up some really fast hardware (pci-e nvme drives) and want to squeeze the best performance out of it.. short of going to something like lvm | |
| 18:07:58 | openstack | Launchpad bug 1510328 in OpenStack Compute (nova) "Nova pre-allocation of qcow2 is flawed" [Low,Confirmed] | |
| 18:07:58 | sean-k-mooney | mdbooth: and lyarwood would like be able to comment better then i on https://bugs.launchpad.net/nova/+bug/1510328 | |
| 18:08:15 | openstackgerrit | Jack Ding proposed openstack/nova master: Add HPET timer support for x86 guests https://review.openstack.org/605902 | |
| 18:08:48 | sean-k-mooney | mnaser: right am in that case would you be better with a raw image instead of qcow if your always preallocating | |
| 18:09:57 | mnaser | sean-k-mooney: right, i'm thinking that might be the next path, raw files on disk | |
| 18:10:03 | mnaser | but then we lose a lot of qcow2 features | |
| 18:10:03 | sean-k-mooney | mnaser: if you are also supporting ceph or boot form volume raw can often be better too even if you are using more space for glance / image cache | |
| 18:10:27 | sean-k-mooney | mnaser: like live snapshot | |
| 18:10:36 | mnaser | yeah, a lot.. unfortunately | |
| 18:11:02 | sean-k-mooney | mnaser: i dont think anyone would object if you had a way to fix the bug but did not cause others | |
| 18:11:20 | mnaser | sean-k-mooney: yep.. its just that unfortunately there was no documentation as to why that was an issue with backing images | |
| 18:11:24 | mnaser | so thats what im trying to research | |
| 18:13:33 | sean-k-mooney | mnaser: its got to have something to do with the overlays that we create | |
| 18:13:50 | mnaser | yeah it looks like it's not really a possibility | |
| 18:13:51 | mnaser | :< | |
| 18:14:14 | sean-k-mooney | mnaser: mdbooth and kashyap should be able to confirm tomorow when they are back online | |