| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 18:18:21 | cburgess | mgagne #2 we should focus on a 1-2 cycle solution, for #1 we should take a step back and say whats need to really do this right (aka the cinder imagebackend). | |
| 18:18:35 | mgagne | 100% agree | |
| 18:19:04 | mgagne | what about ephemeral disks or swap? what's your take? | |
| 18:19:23 | cburgess | Personally I love the idea of a cinder imagebackend but to do that right I think its going to require first rewritting the existing backends to have a much more modular and cleaner seperation. | |
| 18:20:08 | mgagne | yes, I found the task to be near impossible without major refactor when I checked years ago, hence the middleware hack | |
| 18:20:14 | cburgess | mgagne I don't understand the use case for either of those. Honestly I would vote to remove them from nova entirely but I know that wouldn't fly. The question is should we require all backends to support them or should we make those optional. | |
| 18:20:39 | mordred | mgagne, cburgess: in the name of requesting absurd features from people while the local vs. remote disk discussion is taking place ... | |
| 18:20:59 | cburgess | mgagne Right. We also have cases where things like instance_path are used both for the imagebackend and for non-imagebackend things like console logs, libvirt xml etc. So we have to get a clean separation there as well. | |
| 18:21:12 | cburgess | mordred Yes......? | |
| 18:21:16 | mordred | no-local-disk is really useful in both keeping data protected and making sure live migration can happen well ... | |
| 18:21:29 | erhudy | is the goal to eventually turn over all responsibility for block storage to cinder and remove it from nova entirely? | |
| 18:21:40 | cburgess | erhudy Thats is not my goal. | |
| 18:21:55 | cburgess | erhudy My goal is to provide an improved integration between nova and cinder. | |
| 18:22:07 | mordred | but depending on workload, the user might also know "this node is part of an application-level high availablity set, killing it and its data rather than live-migrating it is fine" | |
| 18:22:38 | mordred | especially when for those things locally attached things that behave like ephemeral disk does today is nice becaues of the lack of network latency hop between the vm and disk | |
| 18:22:59 | mgagne | cburgess: So having the ability to split imagebackend/disk from "hypervisor" (logs, xml) files is something an operator should be able to do? | |
| 18:23:10 | cburgess | mordred Hmm hand't thought about that. data volume on cinder, but a local ephemeral disk for "fast" scratch. | |
| 18:23:14 | mordred | yah | |
| 18:23:27 | mordred | but as a user opt-in sort fo thing | |
| 18:23:40 | mordred | since it's definitely a choice witha tradeoff involved | |
| 18:23:43 | cburgess | mgagne Well.. I think the only way to truely make imagebackend modular enough to support cinder we would have to have that clean seperation since cinder isn't going to provide us a way of storing that data. | |
| 18:23:45 | erhudy | cburgess: that can be done now | |
| 18:24:02 | cburgess | erhudy Which thing? | |
| 18:24:07 | mgagne | cburgess: that's the use case I can think of, like on SSD or NVMe | |
| 18:24:18 | erhudy | data/root volumes on cinder with local storage ephemeral disk | |
| 18:24:46 | erhudy | what i keep coming back to as part of our architecture discussions here is wanting to be able to do something like AWS instance store volumes, which afaik doesn't really have an analogue in nova | |
| 18:24:53 | cburgess | erhudy Yes I'm aware. mordred was pointing out that my state of "kill ephemeral in a fire" didn't take into acount that very valid use case. | |
| 18:25:07 | mordred | cburgess: the mirror.*.openstack.org farm is a good example- the only things on the local disk are cache - everything else is stored somewhere else | |
| 18:25:13 | erhudy | you can pass whatever you like to nova in the BDM but unless it matches your flavor it gets rejected | |
| 18:26:00 | mordred | but we'd be totally fine with the datain the ephemeral going away for those nodes, and could opt-in at boot to that and _maybe_ that makes something better for someone somewhere | |
| 18:26:30 | mordred | "hey, we actually don't find the stuff in X precious, DNR" :) | |
| 18:26:34 | erhudy | "ephemeral" in nova-ese is different from "ephemeral" in AWS-ese anyway, which confused a lot of our users | |
| 18:26:35 | cburgess | mordred right. Its def a valid use case. We should make it possible for a user to request a local disk as well *if* the compute node supports it. | |
| 18:26:41 | mordred | cburgess: ++ | |
| 18:27:20 | cburgess | erhudy Its very confusing and something it would be good to clear up some as we think about what would be involved in "fixing" storage in nova. | |
| 18:27:23 | mordred | there are definitely places where we're using cinder volumes for things that are totally non-precious because it's the only way we can get our hands on the disk | |
| 18:27:51 | mgagne | cburgess: isn't this use case already supported? like have a flavor with local root + 1 ephemeral + 1 attached volume for your data. | |
| 18:28:01 | erhudy | sounds like the sort of dilemma where you can ask 10 people and get 11 answers | |
| 18:28:02 | cburgess | mordred We are seeing more and more client deploying there clouds that way. Some cinder integrated storage device and no local storage at all. | |
| 18:28:45 | mordred | cburgess: yah. I mean, in those cases we're fine using cinder- I just mostly don't want to stress out someone's storage backplane if there's cheaper/crappier disk available too | |
| 18:28:49 | erhudy | we are leaning that way in our current greenfield setup, doing it all on ceph, but we may keep nova local storage around for certain use cases | |
| 18:28:51 | mordred | for _some_ things | |
| 18:29:09 | cburgess | mgagne Its is. The question is, if we are looking at designing a more modular imagebackend with a local, ceph, lvm, and now cinder plugin do we want to require all backends to support the use case or say "if you want this use case you must deploy this backend with this configuration". | |
| 18:29:25 | mordred | cause if I'm doing, say, ceph _in_ the cloud - but I can only get disk via cinder which is on ceph _under_ the cloud | |
| 18:29:31 | mgagne | right, I have no answer to that one | |
| 18:29:35 | erhudy | the downside of the local storage setup is, i can give somebody /dev/vda on local storage, if i make a special flavor with ephemeral defined i can give them /dev/vdb as well | |
| 18:29:36 | mordred | then _wow_ is that data super extra redundant | |
| 18:29:40 | cburgess | mordred Oh I agree completely. Not passing judgement on good or bad, just saying its a trend we are seeing with our clients. | |
| 18:29:43 | erhudy | but no way to let them provision more local volumes after that | |
| 18:29:45 | mordred | cburgess: ++ | |
| 18:30:17 | mgagne | erhudy: sounds like a good idea | |
| 18:30:35 | erhudy | it's only a good idea because nova storage doesn't get tracked in quota | |
| 18:30:36 | mgagne | I mean, do you want them to get more local disks or not? | |
| 18:30:50 | erhudy | i want to be able to provide them the same thing they can get on AWS with instance store volumes | |
| 18:31:19 | erhudy | and me providing RBD via cinder is the analogue to EBS | |
| 18:32:54 | cburgess | mgagne So the hypervisor files aren't counting toward quota today, but they do count towards the available/used storage of the backend. | |
| 18:33:14 | mgagne | for those that wonder what's instance store volumes: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html | |
| 18:33:31 | cburgess | mgagne So *if* we split all that out we need to think about how that might impact scheduling decisions. | |
| 18:33:42 | erhudy | that was why i originally asked about whether the idea was to eventually eliminate nova's block storage and just turn it all over to cinder | |
| 18:34:10 | mgagne | cburgess: sure. more or less implementation details at this point. | |
| 18:34:35 | cburgess | erhudy I'm not advocating removing the other backends. I'm simply advocating creating a clean separation so that a cinder backend is viable. | |
| 18:34:38 | cburgess | mgagne Agreed | |
| 18:35:07 | cburgess | As mordred pointed out there are viable use cases for the pure local option that nova offers today. | |
| 18:35:26 | cburgess | But given how overload things like instance_path are today, it makes a clean separation difficult. | |
| 18:35:45 | mgagne | erhudy: wow "data in the instance store is lost under the following circumstances: The instance stops" | |
| 18:36:06 | erhudy | mgagne: okay, not _exactly_ the same thing | |
| 18:36:23 | mgagne | just reading about the concept for now | |
| 18:36:31 | cburgess | mgagne Yeah instance storage on AWS is *super* ephemeral. | |
| 18:36:33 | erhudy | but fundamentally the ability to be able to provision extra volumes that will land on local storage versus going to some replicated backend | |
| 18:36:55 | jaypipes | holy channel traffic... man, I need to read back a whole bunch... | |
| 18:37:09 | erhudy | without having to create new flavors where the only difference is some have ephemeral disk sizes specified and some don't | |
| 18:37:09 | mgagne | however it's only available at creation time right? | |
| 18:37:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 18:37:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 18:37:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 18:37:32 | cburgess | jaypipes Don't worry we made it all a placement problem. | |
| 18:37:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 18:37:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 18:37:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 18:37:43 | jaypipes | thx Chet :) | |
| 18:37:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 18:37:56 | cburgess | jaypipes Just felt like the right thing to do. | |
| 18:38:19 | erhudy | mgagne: yes, they have to be added at launch time | |
| 18:38:26 | erhudy | "they" being instance store volumes | |
| 18:38:48 | erhudy | so it's effectively letting the user specify the number and size of ephemeral disks they want instead of the operator encoding it into the flavor | |
| 18:39:47 | mgagne | erhudy: right. can you explain what's the purpose of the table found on this page? Is it some kind of limitations/specs per flavor? Like you can create "up to" the limits mentioned? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes | |
| 18:41:14 | erhudy | yeah, i don't know where the particular restrictions on each instance type come from, i assume it has to do with the physical characteristics of the SKUs they run on | |
| 18:43:36 | mgagne | for example, the C3 instance types here: https://aws.amazon.com/ec2/instance-types/ The listed SSD storage matches the ones found in the table above. So I feel it's a bunch of instance store volume provided by default, you don't have the choice of the number or size? | |
| 18:44:51 | erhudy | i will walk through instance launch for a c3.large and tell you in 3 minutes | |
| 18:45:12 | mgagne | you are a winner | |
| 18:46:08 | mgagne | because if they are ALL provided by default, it would be the same as ephemeral disks provided by Nova (missing the fact that you lose your data if you stop the instance) | |
| 18:47:20 | mriedem | dansmith: haven't heard anything about that test spiking | |
| 18:47:51 | dansmith | mriedem: okay I figure it must be something in my stuff, I just don't know what.. I'm stacking now locally to try and poke at ti | |
| 18:47:57 | erhudy | mgagne: the way it looks on the add storage step for the default amazon linux AMI (which is on EBS) is i see root, /dev/xvda | |
| 18:48:03 | erhudy | then i can add instance store 0, and then instance store 1 | |
| 18:48:06 | erhudy | they are not there by default | |
| 18:48:45 | mgagne | does the limit match the table? Can you add 20 disks if you want? | |
| 18:48:50 | erhudy | no, i can only add those 2 | |
| 18:48:51 | mgagne | table mentions 2 x 16 GB (32 GB) | |
| 18:48:58 | erhudy | so it matches that quantity given | |