| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-05 | |||
| 16:26:14 | spatel | I totally get that point AWS != openstack in AZ context :) | |
| 16:26:38 | sean-k-mooney | spatel: so looking at https://ibb.co/j9kgUK can you discibe your intient | |
| 16:26:53 | sean-k-mooney | you have 2 tors | |
| 16:26:58 | spatel | yes | |
| 16:27:06 | sean-k-mooney | and 2 AZs one per tor | |
| 16:27:10 | spatel | yes | |
| 16:27:38 | sean-k-mooney | half of your servers use TOR-1 for there SRIOV dataplane and TOR-2 for managmente | |
| 16:27:52 | spatel | yup! that is correct | |
| 16:28:08 | sean-k-mooney | the other half swap which TOR is used form management and data | |
| 16:28:22 | spatel | yup! | |
| 16:29:07 | sean-k-mooney | so if TOR-1's PSU explodes half the servers will loose data conenctivity and the other half will loose manament capablity | |
| 16:29:21 | spatel | yes! that is right | |
| 16:29:37 | sean-k-mooney | am is that what you want? | |
| 16:30:19 | spatel | yes in PSU failure still my 50% instance will be live and providing service but openstack will loose management function | |
| 16:30:20 | sean-k-mooney | i assume you do not have the option to seperate management and data traffic | |
| 16:31:36 | sean-k-mooney | spatel: do you intend to pass the PF or VF to the vms running your voip applications? | |
| 16:31:36 | spatel | I don't have option because i have to purchase NIC to install on compute node also SR-IOV doesn't do bonding so i am going to loose some customer if TOR switch blowup | |
| 16:32:10 | spatel | yes my VF is attach to instance | |
| 16:32:31 | spatel | i don't have any kind of bonding anywhere at present because of limitation of SR-IOV | |
| 16:33:02 | spatel | http://paste.openstack.org/show/731595/ | |
| 16:33:24 | spatel | This is what my compute look, i am running 2 instance with dual NIC on two different VLAN | |
| 16:34:31 | sean-k-mooney | right so what i was goint to suggest is that you could either reserve a VF or use the pf of your sriov nic as a bond pair for your manament network. | |
| 16:35:34 | spatel | oh!! wait is that possible? | |
| 16:35:52 | sean-k-mooney | so in your case you can create a bound between eno1 and eno2 for managment traffic and then openstack can use just the vfs | |
| 16:36:00 | sean-k-mooney | spatel: yes | |
| 16:36:01 | spatel | i thought if you configure NIC for sr-iov then you can't bond them | |
| 16:36:13 | sean-k-mooney | it depends on the nic | |
| 16:36:36 | spatel | I have tried that and found my loop in my switch | |
| 16:36:44 | spatel | may be it was config issue.. | |
| 16:37:18 | spatel | i have all Cisco nexus switch running vPC (active+active bonding mode=4) | |
| 16:37:33 | sean-k-mooney | well it defintely depens on how you create the bond (e.g. just mac loadbalcing with out swich supprot or lacp with it) | |
| 16:37:33 | spatel | i have Qlogic NIC | |
| 16:38:41 | spatel | did you every run that kind of test? | |
| 16:38:59 | spatel | i would like to know if you did and what config you configure | |
| 16:39:13 | spatel | I will run same experiment on my LAB and try come combination | |
| 16:39:33 | spatel | that was great if it work | |
| 16:39:42 | sean-k-mooney | i have never used Qlogic NIC but i have created bonds with intel nics in active failover mode | |
| 16:39:47 | spatel | s/was/would be/ | |
| 16:40:06 | spatel | did you configure with SR-IOV ? | |
| 16:40:24 | spatel | i meant SR-IOV + Mgmt (in bonding) | |
| 16:40:47 | spatel | I think i understand what you trying to explain.. i will give it a try.. | |
| 16:41:13 | spatel | i wish if you can create bonding between two VF :) | |
| 16:41:17 | sean-k-mooney | that is a good questiong the host had both sriov and a manament bound but i cant remeber if the manament bound was using the pf that had the sriov vf allocated from it | |
| 16:42:01 | spatel | i remember last time when i try to did that i got interface error (BPDU guard disable my nic) | |
| 16:42:10 | sean-k-mooney | i think i had 10G nic for tenat data/vfs 1 1G nic and a second 10G nic bonded for manament and ceph traffic | |
| 16:42:33 | spatel | oh!! you have PF level isolation | |
| 16:42:57 | spatel | I have plan in future to install 1G nic for Mgmt traffic but not this year | |
| 16:43:10 | sean-k-mooney | spatel: yes i did in that case but im tring to remeber if i tried to create a bound with the pf used for the VFs | |
| 16:44:05 | spatel | hmm! worth running some test :) | |
| 16:44:12 | sean-k-mooney | spatel: see the thng was will i had vm using VFs allocated form the PF named eth1 i also added eth1 to ovs | |
| 16:44:30 | spatel | hmm!! | |
| 16:44:47 | sean-k-mooney | so i think that is why i did not add it to the storage/manament bond | |
| 16:44:56 | sean-k-mooney | e.g. ovs was using the PF | |
| 16:45:27 | spatel | i will try if i can setup bonding with PF then it would be great | |
| 16:45:54 | spatel | i can protect mgmt traffic but still i will loose dataplan | |
| 16:46:18 | sean-k-mooney | anyway my 1st concern with https://ibb.co/j9kgUK would be that while the datapane woudl remain up on half the node that same half would no lonbger be manageable | |
| 16:47:41 | sean-k-mooney | personally if i could not have a ha manament plane i would put the manament and datapane for my hosts to the same TOR | |
| 16:48:42 | spatel | hmm! ok that also make sense | |
| 16:48:48 | sean-k-mooney | provide you deploy your app to both TORs you will still have half your applications service coustomers but you wil be able to also mange those hosts | |
| 16:48:53 | spatel | data + mgmt on same TOR | |
| 16:49:27 | spatel | If we loose mgmt for few hour i don't think its going to be a big deal.. | |
| 16:49:54 | spatel | these compute nodes are going to run only 1 or 2 instance ( because they are CPU hungry ) | |
| 16:50:17 | sean-k-mooney | well it depens on your useage model. with the split deployemnt you were goning tohave that anyway right since you did not have manamnet ha | |
| 16:50:38 | spatel | Here i am just trying to minimize failure (50%) | |
| 16:51:15 | spatel | i have good remote hand in datacenter and they will replace TOR within 1 hour :) | |
| 16:51:59 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Fix nits in choices documentation https://review.openstack.org/608310 | |
| 16:52:06 | spatel | my main question was if i split server between two AZ then how i can automate instance build so they spread our between A | |
| 16:52:06 | finucannot | fried_rice: ^ | |
| 16:52:08 | spatel | AZ | |
| 16:52:56 | sean-k-mooney | :) well just one other note. if you do end up with ovs-dpdk instead of sriov in that case you would be better just bonding eno1 and eno2 and then using ovs to deivde up the bandwith betwwen manament and teant use. | |
| 16:53:17 | sean-k-mooney | am waht you are asking for is how can i express an antiafinity policy | |
| 16:53:25 | spatel | currently i have script which i manually specify vm-1 go to AZ-1 and vm-2 go to AZ-2 (I am doing by hand ) | |
| 16:54:07 | sean-k-mooney | SteelyDan: ^ we do not have a way to do AZ anti afinity with server groups correct | |
| 16:54:14 | spatel | DPDK is next month todo list :) | |
| 16:54:50 | SteelyDan | sean-k-mooney: I don't even understand those words | |
| 16:55:28 | spatel | I am reading this line from one of openstack doc: The Nova scheduler provides filters that you can use to ensure that OpenStack instances are automatically placed on the same host (affinity) or separate hosts (anti-affinity). | |
| 16:55:38 | sean-k-mooney | SteelyDan: spatel is trying to desing a layout that will allow https://ibb.co/j9kgUK one of the two tors to fail and have openstack loadbalnce vms across them | |
| 16:55:58 | SteelyDan | spatel: that sentence means you can say "put this instance on the same host as this other instance" | |
| 16:56:51 | spatel | e.g - if i tell openstack build 50 instance then it will load 25 on TOR1 server and 25 in TOR2 server | |
| 16:57:35 | SteelyDan | spatel: the doc line you quoted above is purely about affinity between instances, not about affinity or load balancing between two hosts | |
| 16:57:47 | SteelyDan | I don't think we have any way to do that with the existing set of filters, no. | |
| 16:57:52 | spatel | oh! | |
| 16:57:56 | sean-k-mooney | SteelyDan: we had been discussing using AZ to model the network falut domain(TOR) | |
| 16:58:42 | sean-k-mooney | spatel: you might be able to do this with heat. on top of nova | |
| 16:59:10 | sean-k-mooney | spatel: there is one other way to do it which is to use neutron physnets | |
| 16:59:15 | SteelyDan | sean-k-mooney: if nova can't represent this, I'm not sure how you could get heat to do it for you | |
| 16:59:56 | spatel | hmm! i think time to exploration | |
| 16:59:58 | sean-k-mooney | SteelyDan: i was specalting that you may be able to tell heat to spit this request between AZ a and AZ b | |
| 17:00:30 | SteelyDan | obviously if you want to hardcode alternating boxes in the rack to two different AZs and then boot your instances into AZs with your own spread then you can do this | |
| 17:00:37 | SteelyDan | which is all heat could do | |
| 17:00:54 | SteelyDan | but what I mean is, I don't see a way to specify this generically to nova with existing stuff | |
| 17:01:05 | spatel | we can do that by script also right? script will take count and divide them by number of AZ and feed them to nova boot | |
| 17:01:14 | SteelyDan | sure | |
| 17:01:35 | sean-k-mooney | spatel: yep | |
| 17:01:55 | sean-k-mooney | the othere way to model this is via neutron physnets | |
| 17:01:58 | spatel | or write more smart script will will collect data from AZ and then compare which AZ has more room to handle more instance | |
| 17:02:18 | sean-k-mooney | on all the hosts connected to tor1 lable the physnet as top1 | |
| 17:02:26 | sean-k-mooney | and on the others tor2 | |
| 17:02:30 | spatel | can you explain about neutron physnets? | |
| 17:02:44 | spatel | oh!! i know what you saying | |