| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 18:29:35 | erhudy | the downside of the local storage setup is, i can give somebody /dev/vda on local storage, if i make a special flavor with ephemeral defined i can give them /dev/vdb as well | |
| 18:29:36 | mordred | then _wow_ is that data super extra redundant | |
| 18:29:40 | cburgess | mordred Oh I agree completely. Not passing judgement on good or bad, just saying its a trend we are seeing with our clients. | |
| 18:29:43 | erhudy | but no way to let them provision more local volumes after that | |
| 18:29:45 | mordred | cburgess: ++ | |
| 18:30:17 | mgagne | erhudy: sounds like a good idea | |
| 18:30:35 | erhudy | it's only a good idea because nova storage doesn't get tracked in quota | |
| 18:30:36 | mgagne | I mean, do you want them to get more local disks or not? | |
| 18:30:50 | erhudy | i want to be able to provide them the same thing they can get on AWS with instance store volumes | |
| 18:31:19 | erhudy | and me providing RBD via cinder is the analogue to EBS | |
| 18:32:54 | cburgess | mgagne So the hypervisor files aren't counting toward quota today, but they do count towards the available/used storage of the backend. | |
| 18:33:14 | mgagne | for those that wonder what's instance store volumes: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html | |
| 18:33:31 | cburgess | mgagne So *if* we split all that out we need to think about how that might impact scheduling decisions. | |
| 18:33:42 | erhudy | that was why i originally asked about whether the idea was to eventually eliminate nova's block storage and just turn it all over to cinder | |
| 18:34:10 | mgagne | cburgess: sure. more or less implementation details at this point. | |
| 18:34:35 | cburgess | erhudy I'm not advocating removing the other backends. I'm simply advocating creating a clean separation so that a cinder backend is viable. | |
| 18:34:38 | cburgess | mgagne Agreed | |
| 18:35:07 | cburgess | As mordred pointed out there are viable use cases for the pure local option that nova offers today. | |
| 18:35:26 | cburgess | But given how overload things like instance_path are today, it makes a clean separation difficult. | |
| 18:35:45 | mgagne | erhudy: wow "data in the instance store is lost under the following circumstances: The instance stops" | |
| 18:36:06 | erhudy | mgagne: okay, not _exactly_ the same thing | |
| 18:36:23 | mgagne | just reading about the concept for now | |
| 18:36:31 | cburgess | mgagne Yeah instance storage on AWS is *super* ephemeral. | |
| 18:36:33 | erhudy | but fundamentally the ability to be able to provision extra volumes that will land on local storage versus going to some replicated backend | |
| 18:36:55 | jaypipes | holy channel traffic... man, I need to read back a whole bunch... | |
| 18:37:09 | mgagne | however it's only available at creation time right? | |
| 18:37:09 | erhudy | without having to create new flavors where the only difference is some have ephemeral disk sizes specified and some don't | |
| 18:37:24 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 18:37:26 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 18:37:31 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 18:37:32 | cburgess | jaypipes Don't worry we made it all a placement problem. | |
| 18:37:35 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 18:37:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 18:37:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 18:37:43 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |
| 18:37:43 | jaypipes | thx Chet :) | |
| 18:37:56 | cburgess | jaypipes Just felt like the right thing to do. | |
| 18:38:19 | erhudy | mgagne: yes, they have to be added at launch time | |
| 18:38:26 | erhudy | "they" being instance store volumes | |
| 18:38:48 | erhudy | so it's effectively letting the user specify the number and size of ephemeral disks they want instead of the operator encoding it into the flavor | |
| 18:39:47 | mgagne | erhudy: right. can you explain what's the purpose of the table found on this page? Is it some kind of limitations/specs per flavor? Like you can create "up to" the limits mentioned? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes | |
| 18:41:14 | erhudy | yeah, i don't know where the particular restrictions on each instance type come from, i assume it has to do with the physical characteristics of the SKUs they run on | |
| 18:43:36 | mgagne | for example, the C3 instance types here: https://aws.amazon.com/ec2/instance-types/ The listed SSD storage matches the ones found in the table above. So I feel it's a bunch of instance store volume provided by default, you don't have the choice of the number or size? | |
| 18:44:51 | erhudy | i will walk through instance launch for a c3.large and tell you in 3 minutes | |
| 18:45:12 | mgagne | you are a winner | |
| 18:46:08 | mgagne | because if they are ALL provided by default, it would be the same as ephemeral disks provided by Nova (missing the fact that you lose your data if you stop the instance) | |
| 18:47:20 | mriedem | dansmith: haven't heard anything about that test spiking | |
| 18:47:51 | dansmith | mriedem: okay I figure it must be something in my stuff, I just don't know what.. I'm stacking now locally to try and poke at ti | |
| 18:47:57 | erhudy | mgagne: the way it looks on the add storage step for the default amazon linux AMI (which is on EBS) is i see root, /dev/xvda | |
| 18:48:03 | erhudy | then i can add instance store 0, and then instance store 1 | |
| 18:48:06 | erhudy | they are not there by default | |
| 18:48:45 | mgagne | does the limit match the table? Can you add 20 disks if you want? | |
| 18:48:50 | erhudy | no, i can only add those 2 | |
| 18:48:51 | mgagne | table mentions 2 x 16 GB (32 GB) | |
| 18:48:58 | mgagne | with no choice of size? | |
| 18:48:58 | erhudy | so it matches that quantity given | |
| 18:49:09 | mgagne | like it's 16GB or nothing | |
| 18:49:09 | erhudy | no choice of size, it doesn't even show a size in the table | |
| 18:49:14 | mgagne | ok | |
| 18:49:50 | mgagne | so this could be done with bdm where destination=ephemeral in Nova? | |
| 18:50:04 | erhudy | it could, yeah | |
| 18:50:14 | mgagne | and find a way to limit/define the number/size of ephemeral disks per flavor | |
| 18:50:53 | erhudy | so expanding the existing ephemeral functionality and making it an "up to" sort of thing on the flavor? | |
| 18:51:15 | mgagne | erhudy: AFAIK, the number of ephemeral disks is hardcoded no? you can't have 20 for example | |
| 18:51:33 | erhudy | the max number is part of the EC2 instance type, yeah | |
| 18:52:01 | erhudy | so if i back this launch request out and try another one, one sec | |
| 18:52:09 | mgagne | I mean, in Nova, I think the limit is 1 ephemeral disk as per ephemeral_gb field | |
| 18:52:52 | erhudy | so like, if i pick f1.16xlarge, i'm now allowed to provision up to 4 instance store volumes | |
| 18:52:55 | erhudy | per that table | |
| 18:53:42 | mgagne | logic/behavior looks sane so far | |
| 18:54:22 | erhudy | the main thing i like about the AWS model is by default you get 0, but you can go up to however many are allowed by the instance type, and you don't need to have separate flavors for 0 versus 1+ | |
| 18:54:34 | openstackgerrit | Moshe Levi proposed openstack/os-vif master: WIP Revert "Revert "hardware offload support for openvswitch"" Don't merge until OVS patches are upstream This reverts commit f154ef9cc8f9a14273b7c48ac29cb828cd25109a. https://review.openstack.org/460278 | |
| 18:54:57 | mgagne | erhudy: well, there is a sane policy/default provided per flavor. Some just requires you to have EBS, no local disks. | |
| 18:55:06 | erhudy | yes | |
| 18:55:36 | erhudy | which probably goes back to the SKUs that instance type runs on | |
| 18:55:50 | erhudy | the EC2 guys probably curse local storage as much as openstack operators do | |
| 18:56:04 | erhudy | and said "let's have some SKUs where we don't have to deal with local storage" | |
| 18:58:06 | mgagne | yes + less complaints from people screaming: where did my data go? | |
| 18:58:14 | eandersson | Hey - is having redundant memcached server for nova-novnc an issue in Mitaka? | |
| 18:58:36 | eandersson | Every other request fails with two memcached servers enabled. | |
| 18:59:02 | erhudy | do you have memcached binding to an external interface and not localhost? | |
| 18:59:29 | erhudy | mgagne: i don't necessarily advocate the AWS model of ephemeral/instance store volumes vaporizing everything on reboot | |
| 19:00:01 | erhudy | i'd have to ask around to find why it works that way | |
| 19:00:08 | mgagne | sure, no judgement for or against the behavior here =) | |
| 19:00:55 | mgagne | erhudy: could be that after an instance is stopped, there is no guaranty you will start on the same host later? | |
| 19:01:01 | erhudy | mgagne: that's my guess | |
| 19:01:19 | erhudy | part of why i want to retain local storage despite the inconveniences is because our distributed storage backend is ceph, which is great, but without per-client metrics and being able to provision IOPS on each client a single noisy neighbor can make the entire ceph cluster miserable | |
| 19:01:29 | erhudy | that's a separate discussion we're having with red hat | |
| 19:02:13 | openstackgerrit | Chris Dent proposed openstack/nova master: Add pbr-installed wsgi application for metadata api https://review.openstack.org/459413 | |
| 19:03:23 | mriedem | dansmith: did you see this? https://review.openstack.org/#/c/433476/ | |
| 19:05:41 | openstackgerrit | Rick Bartra proposed openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | |
| 19:09:24 | dansmith | mriedem: hah no | |
| 19:09:41 | jaypipes | edleafe, dansmith, cdent, mriedem: https://review.openstack.org/460281 | |
| 19:09:57 | eandersson | erhudy, you mean localhost+second memcached? | |
| 19:10:06 | eandersson | or just a single memcached? | |
| 19:10:15 | dansmith | mriedem: I agree, it seems silly to not allow deleting the cell0 record.. it's the one cell record you _can_ delete and recreate if needed | |
| 19:10:30 | mriedem | the bug report is pretty thin, | |
| 19:10:38 | mriedem | it just says "you shouldn't be able to do this" | |
| 19:10:43 | mriedem | entirely opinionated | |
| 19:10:52 | dansmith | mriedem: yeah.. | |