| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 17:46:31 | sean-k-mooney | numa is not going away infact its become more common | |
| 17:46:44 | sean-k-mooney | dpdk is much better then kernel ovs | |
| 17:46:51 | sean-k-mooney | but its more complicated too | |
| 17:47:05 | spatel | but it doesn't need hardware dependency atleast | |
| 17:47:37 | spatel | I spent thousand of $$$$ to get SR-IOV supported card | |
| 17:47:38 | sean-k-mooney | spatel: not in the same way it requires that the guests use hugepages and that there is a dpdk driver for your nic | |
| 17:48:19 | spatel | Does it perform like SR-IOV ? | |
| 17:48:22 | sean-k-mooney | spatel: ya dpdk will be cheaeper in that sense but you will have to dedicate 1-2 cores to handel trafic for ovs-dpdk | |
| 17:48:50 | sean-k-mooney | spatel: in some cases yes. in general not quite | |
| 17:49:14 | sean-k-mooney | what data rates / traffic profiles are you targeting? | |
| 17:49:31 | spatel | currently i am deploying VoIP application on 1U server with 32 core / 32G memory. and i have 1000 servers... | |
| 17:49:34 | sean-k-mooney | 10G small packets 40G jumbo frames? a mix | |
| 17:50:01 | spatel | my peak in production 200 to 230kpps UDP packet rate | |
| 17:50:21 | sean-k-mooney | on well dpdk can handel that easilly | |
| 17:50:30 | spatel | really??? | |
| 17:50:46 | spatel | if that is the case then it will be win win solution | |
| 17:51:09 | sean-k-mooney | ya dpdk was desinged to hit 10G line rate with 64byte packets which is 14mpps | |
| 17:51:26 | spatel | we have lots of server in AWS (with sr-iov) support | |
| 17:51:40 | spatel | that is really cool! | |
| 17:51:46 | sean-k-mooney | with the right hardware it can hit 32mpps on a singel core but in generall you will see more like 6mpps | |
| 17:52:01 | spatel | we are using LinuxBridge + VLAN so i need to upgrade to OVS | |
| 17:52:18 | sean-k-mooney | its mroe or less like this | |
| 17:52:43 | sean-k-mooney | lb<ovs<sriov+macvtap<ovs-dpdk<sriov direct | |
| 17:53:08 | spatel | I have tried macvtap but that didn't work too | |
| 17:54:39 | sean-k-mooney | spatel: checkout https://dpdksummit.com/Archive/pdf/2016USA/Day02-Session04-ThomasHerbert-DPDKUSASummit2016.pdf slides 16-19 | |
| 17:55:21 | spatel | reading.. | |
| 17:56:04 | sean-k-mooney | spatel: im alittle biased as im one of the people that added ovs-dpdk support to openstack but for your data rate i think it would work quite well | |
| 17:56:42 | spatel | I need to find out how to migrate LinuxBridge to OVS | |
| 17:57:15 | sean-k-mooney | spatel: today cold migrate works. im working on fixing livemigrate | |
| 17:57:31 | spatel | cool! | |
| 17:57:40 | spatel | in SR-IOV i am not able to get that function too | |
| 17:57:48 | spatel | even bonding isn't supported | |
| 17:57:55 | sean-k-mooney | live migrate almost works we just dont update the bridge name correctly im hoping to backport that | |
| 17:58:27 | spatel | nice! if that work | |
| 17:58:49 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 17:58:50 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | |
| 17:58:50 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_qfd_by_cell_and_project() https://review.openstack.org/607663 | |
| 17:58:52 | sean-k-mooney | spatel: haha i think im working on all your missing features :) https://review.openstack.org/#/c/605116/ | |
| 17:59:14 | openstackgerrit | Merged openstack/nova stable/pike: nova-manage - fix online_data_migrations counts https://review.openstack.org/605840 | |
| 17:59:25 | spatel | :) | |
| 17:59:51 | spatel | i have lots of requirement :) these are just starting | |
| 18:00:02 | spatel | sean-k-mooney: thanks for help!!! | |
| 18:00:02 | sean-k-mooney | my main focus this realease at least initally is livemigraton hardenign. e.g fixing edgecase like lb->ovs or sriov | |
| 18:00:25 | nicolasbock | <freenode_sea "nicolasbock: if you are still ar"> Thanks for the tip! | |
| 18:00:26 | spatel | i didn't know freenode will be very helpful.. last 2 days i am chasing google | |
| 18:01:12 | spatel | what you use to deploy your openstack? I am using openstack-ansible | |
| 18:01:14 | sean-k-mooney | spatel: no worries. im usually hear so feel free to ping me if you have issues | |
| 18:01:44 | spatel | I am going to spend next 6 month here :) until my cloud is ready!! | |
| 18:01:53 | sean-k-mooney | spatel: for developement devstack. i used to use kolla-ansible but recently joined redhat so i proably shoudl suggest OSP | |
| 18:02:07 | spatel | we spent million dollar last year in AWS so my boss want to build own AWS :) | |
| 18:02:39 | sean-k-mooney | spatel: that is how alot of compaines end up running opnestack clouds yes | |
| 18:02:56 | spatel | indeed | |
| 18:03:04 | spatel | OSP using tripleO right? | |
| 18:03:12 | spatel | i tried and found very complicated | |
| 18:03:23 | sean-k-mooney | yes that is the officaly supprot installer from redhat | |
| 18:03:30 | sean-k-mooney | spatel: and yes it can be | |
| 18:03:42 | mnaser | does anyone know if daniel berrange hangs out on irc much? | |
| 18:03:54 | mnaser | i'm looking at this old abandoned review and i'm wondering if this is still an issue -- https://review.openstack.org/#/c/241401/ | |
| 18:04:14 | sean-k-mooney | mnaser: not on this irc but he is usally on the libvirt one | |
| 18:04:29 | mnaser | i'll try to ping him there | |
| 18:04:38 | spatel | sean-k-mooney: going to eat something, will catch you again, if any issue :) thanks again | |
| 18:06:49 | sean-k-mooney | mnaser: i think that is a bug that has been forgoten about but not nessacarily fixed | |
| 18:07:14 | mnaser | sean-k-mooney: yeah, it's not fixed, but it's been a while so i'm wondering if the whole "it doesnt working with backing" argument is no longer valid | |
| 18:07:40 | mnaser | we're setting up some really fast hardware (pci-e nvme drives) and want to squeeze the best performance out of it.. short of going to something like lvm | |
| 18:07:58 | sean-k-mooney | mdbooth: and lyarwood would like be able to comment better then i on https://bugs.launchpad.net/nova/+bug/1510328 | |
| 18:07:58 | openstack | Launchpad bug 1510328 in OpenStack Compute (nova) "Nova pre-allocation of qcow2 is flawed" [Low,Confirmed] | |
| 18:08:15 | openstackgerrit | Jack Ding proposed openstack/nova master: Add HPET timer support for x86 guests https://review.openstack.org/605902 | |
| 18:08:48 | sean-k-mooney | mnaser: right am in that case would you be better with a raw image instead of qcow if your always preallocating | |
| 18:09:57 | mnaser | sean-k-mooney: right, i'm thinking that might be the next path, raw files on disk | |
| 18:10:03 | sean-k-mooney | mnaser: if you are also supporting ceph or boot form volume raw can often be better too even if you are using more space for glance / image cache | |
| 18:10:03 | mnaser | but then we lose a lot of qcow2 features | |
| 18:10:27 | sean-k-mooney | mnaser: like live snapshot | |
| 18:10:36 | mnaser | yeah, a lot.. unfortunately | |
| 18:11:02 | sean-k-mooney | mnaser: i dont think anyone would object if you had a way to fix the bug but did not cause others | |
| 18:11:20 | mnaser | sean-k-mooney: yep.. its just that unfortunately there was no documentation as to why that was an issue with backing images | |
| 18:11:24 | mnaser | so thats what im trying to research | |
| 18:13:33 | sean-k-mooney | mnaser: its got to have something to do with the overlays that we create | |
| 18:13:50 | mnaser | yeah it looks like it's not really a possibility | |
| 18:13:51 | mnaser | :< | |
| 18:14:14 | sean-k-mooney | mnaser: mdbooth and kashyap should be able to confirm tomorow when they are back online | |
| 18:14:22 | mnaser | i'll wait to hear | |
| 18:14:40 | mnaser | now to find ways to benchmark this server | |
| 18:14:44 | mnaser | server/vm that is | |
| 18:15:23 | mnaser | http://paste.openstack.org/show/731425/ | |
| 18:16:16 | sean-k-mooney | is that a vm with 468G of ram | |
| 18:16:34 | sean-k-mooney | sorry 472 | |
| 18:17:00 | sean-k-mooney | i also like the insane amount of gpus | |
| 18:17:34 | sean-k-mooney | may i sugges you use it to play minecrat tootally how you should benchmark | |
| 18:17:46 | sean-k-mooney | mnaser: also i hear bitcoin is a thing :) | |
| 18:18:19 | mnaser | sean-k-mooney: aha, we're rolling out gpus and we have instances with 472G of ram, 48 (dedicated) threads, 1.8T of PCI-e NVMe storage.. | |
| 18:18:33 | sean-k-mooney | mnaser: actully on a serious note you are an operator of a cloud with vgpus correct | |
| 18:18:51 | mnaser | sean-k-mooney: no vgpu support, only dedicated gpus (as far as we've planned) | |
| 18:19:17 | mnaser | part of this is MAYBE seeing if we can get some vGPU CI.. if possible, but i hear there are some more complicated reasons why its not possible | |
| 18:19:19 | sean-k-mooney | ah well does the lack of vgpu numa affinity effect your decision to use vgpus or deploy gpus in the cloud in general | |
| 18:19:46 | sean-k-mooney | mnaser: actully it might be useing complicated trick | |
| 18:20:36 | sean-k-mooney | e.g. nested virt + q35 chipset + viommu + pci passthoug of phyical gpu PF to host vm | |
| 18:20:58 | mnaser | i think we're starting to roll things out by having dedicated gpus to see market demand for it (we've had some). unfortunately the other thing that's coming to mind is i'm thinking that users who need gpu levels of performance probably would want 100% of it | |
| 18:21:25 | mnaser | we can make nested virt available for gpu instances so maybe thats possible | |
| 18:21:33 | sean-k-mooney | mnaser: have you talked to bauzas about possible vgpu ci? | |
| 18:21:57 | mnaser | sean-k-mooney: we briefly talked about it.. dansmith mentioned concerns about iommu and stuff that's beyond my level of comprehension :) | |