| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 17:37:49 | mriedem | because then i don't need to look at gabbits | |
| 17:38:23 | cdent | i regret slinging that name around | |
| 17:43:57 | openstackgerrit | Steven Webster proposed openstack/nova master: Improve comment for PCI port binding update https://review.openstack.org/460233 | |
| 17:51:02 | mriedem | mgagne: cburgess: johnthetubaguy: stackempty: here you go https://etherpad.openstack.org/p/BOS-forum-using-cinder-for-nova-ephemeral-storage | |
| 17:51:47 | cburgess | mriedem Are we open to having a 'short term' and a 'long term' set of goals and solutions? | |
| 17:52:12 | mriedem | i think you could tag things that way | |
| 17:52:24 | mriedem | i'm open to eating lunch | |
| 17:52:27 | mriedem | so bbiab | |
| 17:55:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Attach and detach encryptors during swap_volume https://review.openstack.org/460244 | |
| 17:55:39 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Refactor encryptor attach and detach calls https://review.openstack.org/460243 | |
| 17:57:51 | cburgess | mriedem Just doing that myself | |
| 18:04:37 | mgagne | cburgess: so there are 2 use cases? : 1) Provide a default BDM 2) Prevent/forbid source=local in BDM | |
| 18:05:19 | mgagne | in fact, destination=local | |
| 18:06:38 | dansmith | does anyone know if test_delete_server_while_in_building_state is known to be flaky at the moment? logstash doesn't seem to think it's spiking but I see plenty of failures that aren't in my patch, like even on periodic stable jobs | |
| 18:06:40 | dansmith | mriedem: sdague ^ | |
| 18:12:02 | mgagne | k policy. | |
| 18:12:02 | mgagne | cburgess: what do you think about the idea of having a "default bdm" for flavor-root-disk-none ? If you provide no BDM, the root_gb size is used to create a volume for you for your root disk. (flavor could also provide default volume type) This means no 400 HTTP error by default as a sane behavior/default is provided. An other extra spec or flavor attribute could forbid destination=local so if BDM is provided, it still respect the no-local-dis | |
| 18:12:46 | cburgess | mgagne Thats a decent idea. Pull size from root_gb and image_id from request spec. | |
| 18:12:52 | mgagne | yes | |
| 18:13:04 | mgagne | that's basically what we did on our side | |
| 18:13:44 | mgagne | also this could allow Horizon or dashboard to present appropriate UI (default to BFV view or something like that) | |
| 18:14:59 | mgagne | I also see 2 sub use cases: 1) flavor itself *never* support local disk. 2) specific compute nodes *never* want local disk. // One can be implemented without the other. | |
| 18:16:21 | cburgess | Right so #2 is my biggest concern and the biggest pain right now. Ensuring it can never happen. Because if it does happen you end up with data a user might care about living on a single local disk on a compute node, and thats very bad. | |
| 18:16:54 | mgagne | ok, can I take the liberty of splitting the use cases on etherpad? | |
| 18:17:02 | cburgess | mgagne Sure | |
| 18:17:33 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/451049 | |
| 18:17:40 | cburgess | mgagne I prefer that actually as I think there are 2 orthogonal needs 1) Making BDM/cinder integration better, 2) Making sure we can block local disk. | |
| 18:17:51 | mgagne | yes | |
| 18:18:21 | cburgess | mgagne #2 we should focus on a 1-2 cycle solution, for #1 we should take a step back and say whats need to really do this right (aka the cinder imagebackend). | |
| 18:18:35 | mgagne | 100% agree | |
| 18:19:04 | mgagne | what about ephemeral disks or swap? what's your take? | |
| 18:19:23 | cburgess | Personally I love the idea of a cinder imagebackend but to do that right I think its going to require first rewritting the existing backends to have a much more modular and cleaner seperation. | |
| 18:20:08 | mgagne | yes, I found the task to be near impossible without major refactor when I checked years ago, hence the middleware hack | |
| 18:20:14 | cburgess | mgagne I don't understand the use case for either of those. Honestly I would vote to remove them from nova entirely but I know that wouldn't fly. The question is should we require all backends to support them or should we make those optional. | |
| 18:20:39 | mordred | mgagne, cburgess: in the name of requesting absurd features from people while the local vs. remote disk discussion is taking place ... | |
| 18:20:59 | cburgess | mgagne Right. We also have cases where things like instance_path are used both for the imagebackend and for non-imagebackend things like console logs, libvirt xml etc. So we have to get a clean separation there as well. | |
| 18:21:12 | cburgess | mordred Yes......? | |
| 18:21:16 | mordred | no-local-disk is really useful in both keeping data protected and making sure live migration can happen well ... | |
| 18:21:29 | erhudy | is the goal to eventually turn over all responsibility for block storage to cinder and remove it from nova entirely? | |
| 18:21:40 | cburgess | erhudy Thats is not my goal. | |
| 18:21:55 | cburgess | erhudy My goal is to provide an improved integration between nova and cinder. | |
| 18:22:07 | mordred | but depending on workload, the user might also know "this node is part of an application-level high availablity set, killing it and its data rather than live-migrating it is fine" | |
| 18:22:38 | mordred | especially when for those things locally attached things that behave like ephemeral disk does today is nice becaues of the lack of network latency hop between the vm and disk | |
| 18:22:59 | mgagne | cburgess: So having the ability to split imagebackend/disk from "hypervisor" (logs, xml) files is something an operator should be able to do? | |
| 18:23:10 | cburgess | mordred Hmm hand't thought about that. data volume on cinder, but a local ephemeral disk for "fast" scratch. | |
| 18:23:14 | mordred | yah | |
| 18:23:27 | mordred | but as a user opt-in sort fo thing | |
| 18:23:40 | mordred | since it's definitely a choice witha tradeoff involved | |
| 18:23:43 | cburgess | mgagne Well.. I think the only way to truely make imagebackend modular enough to support cinder we would have to have that clean seperation since cinder isn't going to provide us a way of storing that data. | |
| 18:23:45 | erhudy | cburgess: that can be done now | |
| 18:24:02 | cburgess | erhudy Which thing? | |
| 18:24:07 | mgagne | cburgess: that's the use case I can think of, like on SSD or NVMe | |
| 18:24:18 | erhudy | data/root volumes on cinder with local storage ephemeral disk | |
| 18:24:46 | erhudy | what i keep coming back to as part of our architecture discussions here is wanting to be able to do something like AWS instance store volumes, which afaik doesn't really have an analogue in nova | |
| 18:24:53 | cburgess | erhudy Yes I'm aware. mordred was pointing out that my state of "kill ephemeral in a fire" didn't take into acount that very valid use case. | |
| 18:25:07 | mordred | cburgess: the mirror.*.openstack.org farm is a good example- the only things on the local disk are cache - everything else is stored somewhere else | |
| 18:25:13 | erhudy | you can pass whatever you like to nova in the BDM but unless it matches your flavor it gets rejected | |
| 18:26:00 | mordred | but we'd be totally fine with the datain the ephemeral going away for those nodes, and could opt-in at boot to that and _maybe_ that makes something better for someone somewhere | |
| 18:26:30 | mordred | "hey, we actually don't find the stuff in X precious, DNR" :) | |
| 18:26:34 | erhudy | "ephemeral" in nova-ese is different from "ephemeral" in AWS-ese anyway, which confused a lot of our users | |
| 18:26:35 | cburgess | mordred right. Its def a valid use case. We should make it possible for a user to request a local disk as well *if* the compute node supports it. | |
| 18:26:41 | mordred | cburgess: ++ | |
| 18:27:20 | cburgess | erhudy Its very confusing and something it would be good to clear up some as we think about what would be involved in "fixing" storage in nova. | |
| 18:27:23 | mordred | there are definitely places where we're using cinder volumes for things that are totally non-precious because it's the only way we can get our hands on the disk | |
| 18:27:51 | mgagne | cburgess: isn't this use case already supported? like have a flavor with local root + 1 ephemeral + 1 attached volume for your data. | |
| 18:28:01 | erhudy | sounds like the sort of dilemma where you can ask 10 people and get 11 answers | |
| 18:28:02 | cburgess | mordred We are seeing more and more client deploying there clouds that way. Some cinder integrated storage device and no local storage at all. | |
| 18:28:45 | mordred | cburgess: yah. I mean, in those cases we're fine using cinder- I just mostly don't want to stress out someone's storage backplane if there's cheaper/crappier disk available too | |
| 18:28:49 | erhudy | we are leaning that way in our current greenfield setup, doing it all on ceph, but we may keep nova local storage around for certain use cases | |
| 18:28:51 | mordred | for _some_ things | |
| 18:29:09 | cburgess | mgagne Its is. The question is, if we are looking at designing a more modular imagebackend with a local, ceph, lvm, and now cinder plugin do we want to require all backends to support the use case or say "if you want this use case you must deploy this backend with this configuration". | |
| 18:29:25 | mordred | cause if I'm doing, say, ceph _in_ the cloud - but I can only get disk via cinder which is on ceph _under_ the cloud | |
| 18:29:31 | mgagne | right, I have no answer to that one | |
| 18:29:35 | erhudy | the downside of the local storage setup is, i can give somebody /dev/vda on local storage, if i make a special flavor with ephemeral defined i can give them /dev/vdb as well | |
| 18:29:36 | mordred | then _wow_ is that data super extra redundant | |
| 18:29:40 | cburgess | mordred Oh I agree completely. Not passing judgement on good or bad, just saying its a trend we are seeing with our clients. | |
| 18:29:43 | erhudy | but no way to let them provision more local volumes after that | |
| 18:29:45 | mordred | cburgess: ++ | |
| 18:30:17 | mgagne | erhudy: sounds like a good idea | |
| 18:30:35 | erhudy | it's only a good idea because nova storage doesn't get tracked in quota | |
| 18:30:36 | mgagne | I mean, do you want them to get more local disks or not? | |
| 18:30:50 | erhudy | i want to be able to provide them the same thing they can get on AWS with instance store volumes | |
| 18:31:19 | erhudy | and me providing RBD via cinder is the analogue to EBS | |
| 18:32:54 | cburgess | mgagne So the hypervisor files aren't counting toward quota today, but they do count towards the available/used storage of the backend. | |
| 18:33:14 | mgagne | for those that wonder what's instance store volumes: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html | |
| 18:33:31 | cburgess | mgagne So *if* we split all that out we need to think about how that might impact scheduling decisions. | |
| 18:33:42 | erhudy | that was why i originally asked about whether the idea was to eventually eliminate nova's block storage and just turn it all over to cinder | |
| 18:34:10 | mgagne | cburgess: sure. more or less implementation details at this point. | |
| 18:34:35 | cburgess | erhudy I'm not advocating removing the other backends. I'm simply advocating creating a clean separation so that a cinder backend is viable. | |
| 18:34:38 | cburgess | mgagne Agreed | |
| 18:35:07 | cburgess | As mordred pointed out there are viable use cases for the pure local option that nova offers today. | |
| 18:35:26 | cburgess | But given how overload things like instance_path are today, it makes a clean separation difficult. | |
| 18:35:45 | mgagne | erhudy: wow "data in the instance store is lost under the following circumstances: The instance stops" | |
| 18:36:06 | erhudy | mgagne: okay, not _exactly_ the same thing | |
| 18:36:23 | mgagne | just reading about the concept for now | |
| 18:36:31 | cburgess | mgagne Yeah instance storage on AWS is *super* ephemeral. | |
| 18:36:33 | erhudy | but fundamentally the ability to be able to provision extra volumes that will land on local storage versus going to some replicated backend | |
| 18:36:55 | jaypipes | holy channel traffic... man, I need to read back a whole bunch... | |
| 18:37:09 | mgagne | however it's only available at creation time right? | |
| 18:37:09 | erhudy | without having to create new flavors where the only difference is some have ephemeral disk sizes specified and some don't | |