| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-05 | |||
| 16:03:12 | PapaOurs | spatel: there was an excellent talk about AZs in a previous Summit | |
| 16:03:26 | PapaOurs | spatel: lemme help you by giving you the session link | |
| 16:04:00 | spatel | PapaOurs: great they must have video in youtube then | |
| 16:04:10 | PapaOurs | spatel: https://www.openstack.org/videos/vancouver-2018/curse-your-bones-availability-zones-1 | |
| 16:04:25 | spatel | sweet!! | |
| 16:04:36 | leakypipes | fried_rice: :) | |
| 16:05:29 | spatel | trying to understand mechanic behind AZ and scheduler, how drive logic to spread machine out between two zone? | |
| 16:05:38 | spatel | is that scheduler job ? | |
| 16:06:24 | PapaOurs | spatel: that's a filter | |
| 16:06:35 | PapaOurs | but see the video, it explains it :) | |
| 16:06:43 | spatel | i am watching it.. | |
| 16:06:50 | spatel | PapaOurs: thanks | |
| 16:08:15 | PapaOurs | 6pm here, time for opening a beer | |
| 16:08:21 | PapaOurs | have a nice week-end folks | |
| 16:08:23 | PapaOurs | \o | |
| 16:09:18 | sean-k-mooney | spatel: sorry just looking at teh diagram now | |
| 16:10:12 | spatel | no worry! take your time | |
| 16:10:38 | spatel | SR-IOV doesn't support bonding so i can't create bond between two TOR switches | |
| 16:11:05 | spatel | SR-IOV does support bonding but inside VM (i think i am not ready for that) | |
| 16:11:35 | sean-k-mooney | so availablity zones are a user facing grouping mechaniums. from the schduler point of view they are just a metadata tag on a host aggaragte | |
| 16:12:18 | sean-k-mooney | so if you requst a vm in zone A the scudler will only look at host that are in a host aggragte with avaiablity_zone=a set | |
| 16:13:05 | sean-k-mooney | it then allpies the rest of the filters to reduce the set of host. e.g. ram, cpus, disk or in your case pci deivces for sriov | |
| 16:13:10 | spatel | but that would be hardwire right? | |
| 16:13:40 | sean-k-mooney | that is a bit of a simplicfation as there is another service called placement that is also invoved. | |
| 16:13:46 | sean-k-mooney | so yes and no | |
| 16:14:03 | spatel | currently we are using AWS with terraform and we tell tf to create instance between A and B zone and it does it job | |
| 16:14:10 | sean-k-mooney | the operator e.g. you in this case has to decalre what hosts are in the A avaliablity zone | |
| 16:14:32 | sean-k-mooney | spatel: AWS availablity zones ar enot the same as openstacks | |
| 16:15:03 | spatel | understand but is there way i can get functionality close to AWS | |
| 16:15:08 | sean-k-mooney | in AWS an availablity zone is a seperate falut domain so if you have an ouate in zone A zone by is still fine | |
| 16:16:25 | spatel | Let's in openstack i create two AZ (a, b) and i want to tell openstack go build 10 vm then does openstack spread machine out between two AZ ? or this is not possible? | |
| 16:16:44 | sean-k-mooney | spatel: yes unfortunetely we leave that as an exerciest to the operator to configure cooreltys. 1.) you can actully map your AZ to seprate falut domains 2.) you can use cells + availahity zone or 3.) you can use regions | |
| 16:16:57 | spatel | if not possible then i have to specify AZ by hand | |
| 16:17:51 | sean-k-mooney | spatel: if they are different requires e.g. 10 vm requires they can be spread. if you use the multi create api im not certine if it will spread or scdule them all to the same AZ | |
| 16:18:08 | spatel | currently i have TOR level AZ (i am not much worried about power failure) | |
| 16:18:54 | spatel | just want to make sure if i loose TOR it won't knock out all my customer server | |
| 16:19:15 | spatel | so trying to spread out VM between two tor | |
| 16:19:18 | sean-k-mooney | ok the other thing that you are sharing with openstack AZ are your controler service. e.g. your db and api seriveces | |
| 16:19:51 | spatel | hmm?? | |
| 16:20:47 | sean-k-mooney | so in a standard openstack deployment you have contoler nodes that run the api services, schduler and databases | |
| 16:21:03 | spatel | Yes i have 3 controller node | |
| 16:21:15 | sean-k-mooney | thoes contoll plane nodes manage the compute nodes that reside in AZs | |
| 16:22:11 | spatel | yup | |
| 16:22:29 | sean-k-mooney | yes so in your 2 AZ config as long as the compute node can sitll acess one of the 3 controllers and you can also acess the same controler then you will be able to manage that host | |
| 16:23:12 | sean-k-mooney | if you have a network partition that breaks that link then the guests workload running on the compute nodes may still work fine but you will lose the ablity to manage them | |
| 16:23:57 | spatel | i am totally fine if i loose management access!! as far as application serving my customer | |
| 16:24:17 | spatel | replacing TOR switch will take may be 1 hour or so!! | |
| 16:24:17 | sean-k-mooney | in aws the api/dataables/storage and networking uplinks are deployed per AZ so the failrue more for openstack is slightly different as the contolplane is not per aZ | |
| 16:24:37 | spatel | as far as i meet my customer SLA we are fine with that | |
| 16:25:45 | sean-k-mooney | ok :) many people coming form AWS expect openstacks availablity zone to map exactly to the same concept as AWS. you seam to have done more reasearch or given it more taught then most | |
| 16:25:46 | spatel | In my design if you see i break total server between two TOR switch and if either one down i will loose 50% machine | |
| 16:26:14 | spatel | I totally get that point AWS != openstack in AZ context :) | |
| 16:26:38 | sean-k-mooney | spatel: so looking at https://ibb.co/j9kgUK can you discibe your intient | |
| 16:26:53 | sean-k-mooney | you have 2 tors | |
| 16:26:58 | spatel | yes | |
| 16:27:06 | sean-k-mooney | and 2 AZs one per tor | |
| 16:27:10 | spatel | yes | |
| 16:27:38 | sean-k-mooney | half of your servers use TOR-1 for there SRIOV dataplane and TOR-2 for managmente | |
| 16:27:52 | spatel | yup! that is correct | |
| 16:28:08 | sean-k-mooney | the other half swap which TOR is used form management and data | |
| 16:28:22 | spatel | yup! | |
| 16:29:07 | sean-k-mooney | so if TOR-1's PSU explodes half the servers will loose data conenctivity and the other half will loose manament capablity | |
| 16:29:21 | spatel | yes! that is right | |
| 16:29:37 | sean-k-mooney | am is that what you want? | |
| 16:30:19 | spatel | yes in PSU failure still my 50% instance will be live and providing service but openstack will loose management function | |
| 16:30:20 | sean-k-mooney | i assume you do not have the option to seperate management and data traffic | |
| 16:31:36 | sean-k-mooney | spatel: do you intend to pass the PF or VF to the vms running your voip applications? | |
| 16:31:36 | spatel | I don't have option because i have to purchase NIC to install on compute node also SR-IOV doesn't do bonding so i am going to loose some customer if TOR switch blowup | |
| 16:32:10 | spatel | yes my VF is attach to instance | |
| 16:32:31 | spatel | i don't have any kind of bonding anywhere at present because of limitation of SR-IOV | |
| 16:33:02 | spatel | http://paste.openstack.org/show/731595/ | |
| 16:33:24 | spatel | This is what my compute look, i am running 2 instance with dual NIC on two different VLAN | |
| 16:34:31 | sean-k-mooney | right so what i was goint to suggest is that you could either reserve a VF or use the pf of your sriov nic as a bond pair for your manament network. | |
| 16:35:34 | spatel | oh!! wait is that possible? | |
| 16:35:52 | sean-k-mooney | so in your case you can create a bound between eno1 and eno2 for managment traffic and then openstack can use just the vfs | |
| 16:36:00 | sean-k-mooney | spatel: yes | |
| 16:36:01 | spatel | i thought if you configure NIC for sr-iov then you can't bond them | |
| 16:36:13 | sean-k-mooney | it depends on the nic | |
| 16:36:36 | spatel | I have tried that and found my loop in my switch | |
| 16:36:44 | spatel | may be it was config issue.. | |
| 16:37:18 | spatel | i have all Cisco nexus switch running vPC (active+active bonding mode=4) | |
| 16:37:33 | sean-k-mooney | well it defintely depens on how you create the bond (e.g. just mac loadbalcing with out swich supprot or lacp with it) | |
| 16:37:33 | spatel | i have Qlogic NIC | |
| 16:38:41 | spatel | did you every run that kind of test? | |
| 16:38:59 | spatel | i would like to know if you did and what config you configure | |
| 16:39:13 | spatel | I will run same experiment on my LAB and try come combination | |
| 16:39:33 | spatel | that was great if it work | |
| 16:39:42 | sean-k-mooney | i have never used Qlogic NIC but i have created bonds with intel nics in active failover mode | |
| 16:39:47 | spatel | s/was/would be/ | |
| 16:40:06 | spatel | did you configure with SR-IOV ? | |
| 16:40:24 | spatel | i meant SR-IOV + Mgmt (in bonding) | |
| 16:40:47 | spatel | I think i understand what you trying to explain.. i will give it a try.. | |
| 16:41:13 | spatel | i wish if you can create bonding between two VF :) | |
| 16:41:17 | sean-k-mooney | that is a good questiong the host had both sriov and a manament bound but i cant remeber if the manament bound was using the pf that had the sriov vf allocated from it | |
| 16:42:01 | spatel | i remember last time when i try to did that i got interface error (BPDU guard disable my nic) | |
| 16:42:10 | sean-k-mooney | i think i had 10G nic for tenat data/vfs 1 1G nic and a second 10G nic bonded for manament and ceph traffic | |
| 16:42:33 | spatel | oh!! you have PF level isolation | |
| 16:42:57 | spatel | I have plan in future to install 1G nic for Mgmt traffic but not this year | |
| 16:43:10 | sean-k-mooney | spatel: yes i did in that case but im tring to remeber if i tried to create a bound with the pf used for the VFs | |
| 16:44:05 | spatel | hmm! worth running some test :) | |