| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 18:26:30 | mordred | "hey, we actually don't find the stuff in X precious, DNR" :) | |
| 18:26:34 | erhudy | "ephemeral" in nova-ese is different from "ephemeral" in AWS-ese anyway, which confused a lot of our users | |
| 18:26:35 | cburgess | mordred right. Its def a valid use case. We should make it possible for a user to request a local disk as well *if* the compute node supports it. | |
| 18:26:41 | mordred | cburgess: ++ | |
| 18:27:20 | cburgess | erhudy Its very confusing and something it would be good to clear up some as we think about what would be involved in "fixing" storage in nova. | |
| 18:27:23 | mordred | there are definitely places where we're using cinder volumes for things that are totally non-precious because it's the only way we can get our hands on the disk | |
| 18:27:51 | mgagne | cburgess: isn't this use case already supported? like have a flavor with local root + 1 ephemeral + 1 attached volume for your data. | |
| 18:28:01 | erhudy | sounds like the sort of dilemma where you can ask 10 people and get 11 answers | |
| 18:28:02 | cburgess | mordred We are seeing more and more client deploying there clouds that way. Some cinder integrated storage device and no local storage at all. | |
| 18:28:45 | mordred | cburgess: yah. I mean, in those cases we're fine using cinder- I just mostly don't want to stress out someone's storage backplane if there's cheaper/crappier disk available too | |
| 18:28:49 | erhudy | we are leaning that way in our current greenfield setup, doing it all on ceph, but we may keep nova local storage around for certain use cases | |
| 18:28:51 | mordred | for _some_ things | |
| 18:29:09 | cburgess | mgagne Its is. The question is, if we are looking at designing a more modular imagebackend with a local, ceph, lvm, and now cinder plugin do we want to require all backends to support the use case or say "if you want this use case you must deploy this backend with this configuration". | |
| 18:29:25 | mordred | cause if I'm doing, say, ceph _in_ the cloud - but I can only get disk via cinder which is on ceph _under_ the cloud | |
| 18:29:31 | mgagne | right, I have no answer to that one | |
| 18:29:35 | erhudy | the downside of the local storage setup is, i can give somebody /dev/vda on local storage, if i make a special flavor with ephemeral defined i can give them /dev/vdb as well | |
| 18:29:36 | mordred | then _wow_ is that data super extra redundant | |
| 18:29:40 | cburgess | mordred Oh I agree completely. Not passing judgement on good or bad, just saying its a trend we are seeing with our clients. | |
| 18:29:43 | erhudy | but no way to let them provision more local volumes after that | |
| 18:29:45 | mordred | cburgess: ++ | |
| 18:30:17 | mgagne | erhudy: sounds like a good idea | |
| 18:30:35 | erhudy | it's only a good idea because nova storage doesn't get tracked in quota | |
| 18:30:36 | mgagne | I mean, do you want them to get more local disks or not? | |
| 18:30:50 | erhudy | i want to be able to provide them the same thing they can get on AWS with instance store volumes | |
| 18:31:19 | erhudy | and me providing RBD via cinder is the analogue to EBS | |
| 18:32:54 | cburgess | mgagne So the hypervisor files aren't counting toward quota today, but they do count towards the available/used storage of the backend. | |
| 18:33:14 | mgagne | for those that wonder what's instance store volumes: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html | |
| 18:33:31 | cburgess | mgagne So *if* we split all that out we need to think about how that might impact scheduling decisions. | |
| 18:33:42 | erhudy | that was why i originally asked about whether the idea was to eventually eliminate nova's block storage and just turn it all over to cinder | |
| 18:34:10 | mgagne | cburgess: sure. more or less implementation details at this point. | |
| 18:34:35 | cburgess | erhudy I'm not advocating removing the other backends. I'm simply advocating creating a clean separation so that a cinder backend is viable. | |
| 18:34:38 | cburgess | mgagne Agreed | |
| 18:35:07 | cburgess | As mordred pointed out there are viable use cases for the pure local option that nova offers today. | |
| 18:35:26 | cburgess | But given how overload things like instance_path are today, it makes a clean separation difficult. | |
| 18:35:45 | mgagne | erhudy: wow "data in the instance store is lost under the following circumstances: The instance stops" | |
| 18:36:06 | erhudy | mgagne: okay, not _exactly_ the same thing | |
| 18:36:23 | mgagne | just reading about the concept for now | |
| 18:36:31 | cburgess | mgagne Yeah instance storage on AWS is *super* ephemeral. | |
| 18:36:33 | erhudy | but fundamentally the ability to be able to provision extra volumes that will land on local storage versus going to some replicated backend | |
| 18:36:55 | jaypipes | holy channel traffic... man, I need to read back a whole bunch... | |
| 18:37:09 | erhudy | without having to create new flavors where the only difference is some have ephemeral disk sizes specified and some don't | |
| 18:37:09 | mgagne | however it's only available at creation time right? | |
| 18:37:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 18:37:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 18:37:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 18:37:32 | cburgess | jaypipes Don't worry we made it all a placement problem. | |
| 18:37:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 18:37:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 18:37:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 18:37:43 | jaypipes | thx Chet :) | |
| 18:37:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 18:37:56 | cburgess | jaypipes Just felt like the right thing to do. | |
| 18:38:19 | erhudy | mgagne: yes, they have to be added at launch time | |
| 18:38:26 | erhudy | "they" being instance store volumes | |
| 18:38:48 | erhudy | so it's effectively letting the user specify the number and size of ephemeral disks they want instead of the operator encoding it into the flavor | |
| 18:39:47 | mgagne | erhudy: right. can you explain what's the purpose of the table found on this page? Is it some kind of limitations/specs per flavor? Like you can create "up to" the limits mentioned? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes | |
| 18:41:14 | erhudy | yeah, i don't know where the particular restrictions on each instance type come from, i assume it has to do with the physical characteristics of the SKUs they run on | |
| 18:43:36 | mgagne | for example, the C3 instance types here: https://aws.amazon.com/ec2/instance-types/ The listed SSD storage matches the ones found in the table above. So I feel it's a bunch of instance store volume provided by default, you don't have the choice of the number or size? | |
| 18:44:51 | erhudy | i will walk through instance launch for a c3.large and tell you in 3 minutes | |
| 18:45:12 | mgagne | you are a winner | |
| 18:46:08 | mgagne | because if they are ALL provided by default, it would be the same as ephemeral disks provided by Nova (missing the fact that you lose your data if you stop the instance) | |
| 18:47:20 | mriedem | dansmith: haven't heard anything about that test spiking | |
| 18:47:51 | dansmith | mriedem: okay I figure it must be something in my stuff, I just don't know what.. I'm stacking now locally to try and poke at ti | |
| 18:47:57 | erhudy | mgagne: the way it looks on the add storage step for the default amazon linux AMI (which is on EBS) is i see root, /dev/xvda | |
| 18:48:03 | erhudy | then i can add instance store 0, and then instance store 1 | |
| 18:48:06 | erhudy | they are not there by default | |
| 18:48:45 | mgagne | does the limit match the table? Can you add 20 disks if you want? | |
| 18:48:50 | erhudy | no, i can only add those 2 | |
| 18:48:51 | mgagne | table mentions 2 x 16 GB (32 GB) | |
| 18:48:58 | erhudy | so it matches that quantity given | |
| 18:48:58 | mgagne | with no choice of size? | |
| 18:49:09 | erhudy | no choice of size, it doesn't even show a size in the table | |
| 18:49:09 | mgagne | like it's 16GB or nothing | |
| 18:49:14 | mgagne | ok | |
| 18:49:50 | mgagne | so this could be done with bdm where destination=ephemeral in Nova? | |
| 18:50:04 | erhudy | it could, yeah | |
| 18:50:14 | mgagne | and find a way to limit/define the number/size of ephemeral disks per flavor | |
| 18:50:53 | erhudy | so expanding the existing ephemeral functionality and making it an "up to" sort of thing on the flavor? | |
| 18:51:15 | mgagne | erhudy: AFAIK, the number of ephemeral disks is hardcoded no? you can't have 20 for example | |
| 18:51:33 | erhudy | the max number is part of the EC2 instance type, yeah | |
| 18:52:01 | erhudy | so if i back this launch request out and try another one, one sec | |
| 18:52:09 | mgagne | I mean, in Nova, I think the limit is 1 ephemeral disk as per ephemeral_gb field | |
| 18:52:52 | erhudy | so like, if i pick f1.16xlarge, i'm now allowed to provision up to 4 instance store volumes | |
| 18:52:55 | erhudy | per that table | |
| 18:53:42 | mgagne | logic/behavior looks sane so far | |
| 18:54:22 | erhudy | the main thing i like about the AWS model is by default you get 0, but you can go up to however many are allowed by the instance type, and you don't need to have separate flavors for 0 versus 1+ | |
| 18:54:34 | openstackgerrit | Moshe Levi proposed openstack/os-vif master: WIP Revert "Revert "hardware offload support for openvswitch"" Don't merge until OVS patches are upstream This reverts commit f154ef9cc8f9a14273b7c48ac29cb828cd25109a. https://review.openstack.org/460278 | |
| 18:54:57 | mgagne | erhudy: well, there is a sane policy/default provided per flavor. Some just requires you to have EBS, no local disks. | |
| 18:55:06 | erhudy | yes | |
| 18:55:36 | erhudy | which probably goes back to the SKUs that instance type runs on | |
| 18:55:50 | erhudy | the EC2 guys probably curse local storage as much as openstack operators do | |
| 18:56:04 | erhudy | and said "let's have some SKUs where we don't have to deal with local storage" | |
| 18:58:06 | mgagne | yes + less complaints from people screaming: where did my data go? | |
| 18:58:14 | eandersson | Hey - is having redundant memcached server for nova-novnc an issue in Mitaka? | |
| 18:58:36 | eandersson | Every other request fails with two memcached servers enabled. | |
| 18:59:02 | erhudy | do you have memcached binding to an external interface and not localhost? | |
| 18:59:29 | erhudy | mgagne: i don't necessarily advocate the AWS model of ephemeral/instance store volumes vaporizing everything on reboot | |
| 19:00:01 | erhudy | i'd have to ask around to find why it works that way | |
| 19:00:08 | mgagne | sure, no judgement for or against the behavior here =) | |
| 19:00:55 | mgagne | erhudy: could be that after an instance is stopped, there is no guaranty you will start on the same host later? | |