| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 18:38:26 | erhudy | "they" being instance store volumes | |
| 18:38:48 | erhudy | so it's effectively letting the user specify the number and size of ephemeral disks they want instead of the operator encoding it into the flavor | |
| 18:39:47 | mgagne | erhudy: right. can you explain what's the purpose of the table found on this page? Is it some kind of limitations/specs per flavor? Like you can create "up to" the limits mentioned? http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes | |
| 18:41:14 | erhudy | yeah, i don't know where the particular restrictions on each instance type come from, i assume it has to do with the physical characteristics of the SKUs they run on | |
| 18:43:36 | mgagne | for example, the C3 instance types here: https://aws.amazon.com/ec2/instance-types/ The listed SSD storage matches the ones found in the table above. So I feel it's a bunch of instance store volume provided by default, you don't have the choice of the number or size? | |
| 18:44:51 | erhudy | i will walk through instance launch for a c3.large and tell you in 3 minutes | |
| 18:45:12 | mgagne | you are a winner | |
| 18:46:08 | mgagne | because if they are ALL provided by default, it would be the same as ephemeral disks provided by Nova (missing the fact that you lose your data if you stop the instance) | |
| 18:47:20 | mriedem | dansmith: haven't heard anything about that test spiking | |
| 18:47:51 | dansmith | mriedem: okay I figure it must be something in my stuff, I just don't know what.. I'm stacking now locally to try and poke at ti | |
| 18:47:57 | erhudy | mgagne: the way it looks on the add storage step for the default amazon linux AMI (which is on EBS) is i see root, /dev/xvda | |
| 18:48:03 | erhudy | then i can add instance store 0, and then instance store 1 | |
| 18:48:06 | erhudy | they are not there by default | |
| 18:48:45 | mgagne | does the limit match the table? Can you add 20 disks if you want? | |
| 18:48:50 | erhudy | no, i can only add those 2 | |
| 18:48:51 | mgagne | table mentions 2 x 16 GB (32 GB) | |
| 18:48:58 | mgagne | with no choice of size? | |
| 18:48:58 | erhudy | so it matches that quantity given | |
| 18:49:09 | mgagne | like it's 16GB or nothing | |
| 18:49:09 | erhudy | no choice of size, it doesn't even show a size in the table | |
| 18:49:14 | mgagne | ok | |
| 18:49:50 | mgagne | so this could be done with bdm where destination=ephemeral in Nova? | |
| 18:50:04 | erhudy | it could, yeah | |
| 18:50:14 | mgagne | and find a way to limit/define the number/size of ephemeral disks per flavor | |
| 18:50:53 | erhudy | so expanding the existing ephemeral functionality and making it an "up to" sort of thing on the flavor? | |
| 18:51:15 | mgagne | erhudy: AFAIK, the number of ephemeral disks is hardcoded no? you can't have 20 for example | |
| 18:51:33 | erhudy | the max number is part of the EC2 instance type, yeah | |
| 18:52:01 | erhudy | so if i back this launch request out and try another one, one sec | |
| 18:52:09 | mgagne | I mean, in Nova, I think the limit is 1 ephemeral disk as per ephemeral_gb field | |
| 18:52:52 | erhudy | so like, if i pick f1.16xlarge, i'm now allowed to provision up to 4 instance store volumes | |
| 18:52:55 | erhudy | per that table | |
| 18:53:42 | mgagne | logic/behavior looks sane so far | |
| 18:54:22 | erhudy | the main thing i like about the AWS model is by default you get 0, but you can go up to however many are allowed by the instance type, and you don't need to have separate flavors for 0 versus 1+ | |
| 18:54:34 | openstackgerrit | Moshe Levi proposed openstack/os-vif master: WIP Revert "Revert "hardware offload support for openvswitch"" Don't merge until OVS patches are upstream This reverts commit f154ef9cc8f9a14273b7c48ac29cb828cd25109a. https://review.openstack.org/460278 | |
| 18:54:57 | mgagne | erhudy: well, there is a sane policy/default provided per flavor. Some just requires you to have EBS, no local disks. | |
| 18:55:06 | erhudy | yes | |
| 18:55:36 | erhudy | which probably goes back to the SKUs that instance type runs on | |
| 18:55:50 | erhudy | the EC2 guys probably curse local storage as much as openstack operators do | |
| 18:56:04 | erhudy | and said "let's have some SKUs where we don't have to deal with local storage" | |
| 18:58:06 | mgagne | yes + less complaints from people screaming: where did my data go? | |
| 18:58:14 | eandersson | Hey - is having redundant memcached server for nova-novnc an issue in Mitaka? | |
| 18:58:36 | eandersson | Every other request fails with two memcached servers enabled. | |
| 18:59:02 | erhudy | do you have memcached binding to an external interface and not localhost? | |
| 18:59:29 | erhudy | mgagne: i don't necessarily advocate the AWS model of ephemeral/instance store volumes vaporizing everything on reboot | |
| 19:00:01 | erhudy | i'd have to ask around to find why it works that way | |
| 19:00:08 | mgagne | sure, no judgement for or against the behavior here =) | |
| 19:00:55 | mgagne | erhudy: could be that after an instance is stopped, there is no guaranty you will start on the same host later? | |
| 19:01:01 | erhudy | mgagne: that's my guess | |
| 19:01:19 | erhudy | part of why i want to retain local storage despite the inconveniences is because our distributed storage backend is ceph, which is great, but without per-client metrics and being able to provision IOPS on each client a single noisy neighbor can make the entire ceph cluster miserable | |
| 19:01:29 | erhudy | that's a separate discussion we're having with red hat | |
| 19:02:13 | openstackgerrit | Chris Dent proposed openstack/nova master: Add pbr-installed wsgi application for metadata api https://review.openstack.org/459413 | |
| 19:03:23 | mriedem | dansmith: did you see this? https://review.openstack.org/#/c/433476/ | |
| 19:05:41 | openstackgerrit | Rick Bartra proposed openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | |
| 19:09:24 | dansmith | mriedem: hah no | |
| 19:09:41 | jaypipes | edleafe, dansmith, cdent, mriedem: https://review.openstack.org/460281 | |
| 19:09:57 | eandersson | erhudy, you mean localhost+second memcached? | |
| 19:10:06 | eandersson | or just a single memcached? | |
| 19:10:15 | dansmith | mriedem: I agree, it seems silly to not allow deleting the cell0 record.. it's the one cell record you _can_ delete and recreate if needed | |
| 19:10:30 | mriedem | the bug report is pretty thin, | |
| 19:10:38 | mriedem | it just says "you shouldn't be able to do this" | |
| 19:10:43 | mriedem | entirely opinionated | |
| 19:10:52 | dansmith | mriedem: yeah.. | |
| 19:10:54 | mriedem | dansmith: i'd +2 if you revert | |
| 19:11:01 | cdent | jaypipes: that's quite the name. will look more closely soon (which might include tomorrow morning) | |
| 19:11:12 | dansmith | mriedem: cool, I think it's probably worth reverting | |
| 19:12:46 | edleafe | jaypipes: why not build that on top of the automated import patch? | |
| 19:13:39 | openstackgerrit | Dan Smith proposed openstack/nova master: Revert "Prevent delete cell0 in nova-manage command" https://review.openstack.org/460284 | |
| 19:13:42 | dansmith | mriedem: ^ | |
| 19:15:04 | mriedem | jaypipes: comments / questions inline | |
| 19:15:33 | erhudy | eandersson: if every other request is failing it sounds like it's trying to round-robin between the memcached instances and can hit one but not the other | |
| 19:15:38 | jaypipes | edleafe: because it's not related... it's a different branch. | |
| 19:15:59 | jaypipes | edleafe: if the auto-import merges first, I'll quickly rebase. no biggie. | |
| 19:16:01 | erhudy | so, iptables or memcached configuration seem like possible culprits | |
| 19:16:17 | mriedem | melwitt: https://review.openstack.org/#/c/460284/ | |
| 19:16:43 | eandersson | erhudy, every other nova service works fine though | |
| 19:16:56 | eandersson | it's just nova-novnc that is having issues with this configuration | |
| 19:17:17 | edleafe | jaypipes: just thought I'd save ya some of that ugly typing... :-P | |
| 19:17:40 | eandersson | but yes it looks like it is round-robin | |
| 19:18:16 | erhudy | maybe strace nova-novncproxy and see what it's sad about? strace is usually what i turn to in Mysterious Situations | |
| 19:19:05 | erhudy | or strace memcached, even | |
| 19:19:25 | erhudy | once upon a time i had strange issues after going kilo->liberty and strace revealed it was because liberty was using memcached more aggressively and memcached had silently run out of connections | |
| 19:19:54 | erhudy | we can take this over to #openstack though instead of filling up dev chat with ops stuff | |
| 19:21:45 | melwitt | mriedem: +W | |
| 19:27:13 | melwitt | jaypipes: thanks for braving the quota patches. working on replies | |
| 19:32:01 | jaypipes | mriedem: answerd. | |
| 19:32:16 | jaypipes | melwitt: sure, no prob. it's a mess in there :) | |
| 19:34:47 | melwitt | jaypipes: I tried to make it make sense but I failed, so sorry about that :( | |
| 19:35:00 | jaypipes | melwitt: and you're not the cause of the indecipherability :) | |
| 19:35:33 | melwitt | my goal was to code comment it enough to where laypersons could understand it | |
| 19:36:47 | melwitt | because learning it myself was extremely painful and wanted to make it easier for others | |
| 19:37:41 | melwitt | jaypipes: the dicts returned from count() are evil and that was me, I will try to think of another way | |
| 19:38:06 | jaypipes | melwitt: yeah, I hear ya. definitely not bemoaning your code. it's just a pit of vipers in general. | |
| 19:38:35 | melwitt | I know. not taken that way :) | |
| 19:38:44 | jaypipes | k, cool :) | |
| 19:43:00 | mriedem | jaypipes: ok, replied in https://review.openstack.org/#/c/460281/ as well, | |
| 19:43:05 | melwitt | mriedem: been meaning to tell you about this, I found a bug in the BackOffLoopingCall that was causing that ironic test to timeout intermittently (160s test timeout whereas the backoff timeout should have expired well before that) https://review.openstack.org/#/c/459790/ | |
| 19:43:22 | mriedem | jaypipes: also mentioned at a higher level that i don't know that the plan as you describe it in there is actually written down anywhere | |
| 19:44:25 | mriedem | melwitt: heh, nice, i guess that really bugged you | |
| 19:45:31 | jaypipes | mriedem: I got dinged for being too verbose in specs, then dinged for not having everything written down ahead of time... | |
| 19:45:38 | jaypipes | can't win for losin. | |