| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 18:58:06 | mgagne | yes + less complaints from people screaming: where did my data go? | |
| 18:58:14 | eandersson | Hey - is having redundant memcached server for nova-novnc an issue in Mitaka? | |
| 18:58:36 | eandersson | Every other request fails with two memcached servers enabled. | |
| 18:59:02 | erhudy | do you have memcached binding to an external interface and not localhost? | |
| 18:59:29 | erhudy | mgagne: i don't necessarily advocate the AWS model of ephemeral/instance store volumes vaporizing everything on reboot | |
| 19:00:01 | erhudy | i'd have to ask around to find why it works that way | |
| 19:00:08 | mgagne | sure, no judgement for or against the behavior here =) | |
| 19:00:55 | mgagne | erhudy: could be that after an instance is stopped, there is no guaranty you will start on the same host later? | |
| 19:01:01 | erhudy | mgagne: that's my guess | |
| 19:01:19 | erhudy | part of why i want to retain local storage despite the inconveniences is because our distributed storage backend is ceph, which is great, but without per-client metrics and being able to provision IOPS on each client a single noisy neighbor can make the entire ceph cluster miserable | |
| 19:01:29 | erhudy | that's a separate discussion we're having with red hat | |
| 19:02:13 | openstackgerrit | Chris Dent proposed openstack/nova master: Add pbr-installed wsgi application for metadata api https://review.openstack.org/459413 | |
| 19:03:23 | mriedem | dansmith: did you see this? https://review.openstack.org/#/c/433476/ | |
| 19:05:41 | openstackgerrit | Rick Bartra proposed openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | |
| 19:09:24 | dansmith | mriedem: hah no | |
| 19:09:41 | jaypipes | edleafe, dansmith, cdent, mriedem: https://review.openstack.org/460281 | |
| 19:09:57 | eandersson | erhudy, you mean localhost+second memcached? | |
| 19:10:06 | eandersson | or just a single memcached? | |
| 19:10:15 | dansmith | mriedem: I agree, it seems silly to not allow deleting the cell0 record.. it's the one cell record you _can_ delete and recreate if needed | |
| 19:10:30 | mriedem | the bug report is pretty thin, | |
| 19:10:38 | mriedem | it just says "you shouldn't be able to do this" | |
| 19:10:43 | mriedem | entirely opinionated | |
| 19:10:52 | dansmith | mriedem: yeah.. | |
| 19:10:54 | mriedem | dansmith: i'd +2 if you revert | |
| 19:11:01 | cdent | jaypipes: that's quite the name. will look more closely soon (which might include tomorrow morning) | |
| 19:11:12 | dansmith | mriedem: cool, I think it's probably worth reverting | |
| 19:12:46 | edleafe | jaypipes: why not build that on top of the automated import patch? | |
| 19:13:39 | openstackgerrit | Dan Smith proposed openstack/nova master: Revert "Prevent delete cell0 in nova-manage command" https://review.openstack.org/460284 | |
| 19:13:42 | dansmith | mriedem: ^ | |
| 19:15:04 | mriedem | jaypipes: comments / questions inline | |
| 19:15:33 | erhudy | eandersson: if every other request is failing it sounds like it's trying to round-robin between the memcached instances and can hit one but not the other | |
| 19:15:38 | jaypipes | edleafe: because it's not related... it's a different branch. | |
| 19:15:59 | jaypipes | edleafe: if the auto-import merges first, I'll quickly rebase. no biggie. | |
| 19:16:01 | erhudy | so, iptables or memcached configuration seem like possible culprits | |
| 19:16:17 | mriedem | melwitt: https://review.openstack.org/#/c/460284/ | |
| 19:16:43 | eandersson | erhudy, every other nova service works fine though | |
| 19:16:56 | eandersson | it's just nova-novnc that is having issues with this configuration | |
| 19:17:17 | edleafe | jaypipes: just thought I'd save ya some of that ugly typing... :-P | |
| 19:17:40 | eandersson | but yes it looks like it is round-robin | |
| 19:18:16 | erhudy | maybe strace nova-novncproxy and see what it's sad about? strace is usually what i turn to in Mysterious Situations | |
| 19:19:05 | erhudy | or strace memcached, even | |
| 19:19:25 | erhudy | once upon a time i had strange issues after going kilo->liberty and strace revealed it was because liberty was using memcached more aggressively and memcached had silently run out of connections | |
| 19:19:54 | erhudy | we can take this over to #openstack though instead of filling up dev chat with ops stuff | |
| 19:21:45 | melwitt | mriedem: +W | |
| 19:27:13 | melwitt | jaypipes: thanks for braving the quota patches. working on replies | |
| 19:32:01 | jaypipes | mriedem: answerd. | |
| 19:32:16 | jaypipes | melwitt: sure, no prob. it's a mess in there :) | |
| 19:34:47 | melwitt | jaypipes: I tried to make it make sense but I failed, so sorry about that :( | |
| 19:35:00 | jaypipes | melwitt: and you're not the cause of the indecipherability :) | |
| 19:35:33 | melwitt | my goal was to code comment it enough to where laypersons could understand it | |
| 19:36:47 | melwitt | because learning it myself was extremely painful and wanted to make it easier for others | |
| 19:37:41 | melwitt | jaypipes: the dicts returned from count() are evil and that was me, I will try to think of another way | |
| 19:38:06 | jaypipes | melwitt: yeah, I hear ya. definitely not bemoaning your code. it's just a pit of vipers in general. | |
| 19:38:35 | melwitt | I know. not taken that way :) | |
| 19:38:44 | jaypipes | k, cool :) | |
| 19:43:00 | mriedem | jaypipes: ok, replied in https://review.openstack.org/#/c/460281/ as well, | |
| 19:43:05 | melwitt | mriedem: been meaning to tell you about this, I found a bug in the BackOffLoopingCall that was causing that ironic test to timeout intermittently (160s test timeout whereas the backoff timeout should have expired well before that) https://review.openstack.org/#/c/459790/ | |
| 19:43:22 | mriedem | jaypipes: also mentioned at a higher level that i don't know that the plan as you describe it in there is actually written down anywhere | |
| 19:44:25 | mriedem | melwitt: heh, nice, i guess that really bugged you | |
| 19:45:31 | jaypipes | mriedem: I got dinged for being too verbose in specs, then dinged for not having everything written down ahead of time... | |
| 19:45:38 | jaypipes | can't win for losin. | |
| 19:46:02 | melwitt | mriedem: yup, I became obsessed with it | |
| 19:46:17 | mriedem | jaypipes: i know | |
| 19:46:32 | jaypipes | mriedem: I'll try to write something up tomorrow. | |
| 19:46:46 | mriedem | jaypipes: i'm +2 on the change after you explained the plan, i just wouldn't have known otherwise | |
| 19:46:54 | jaypipes | understood. | |
| 19:52:37 | mordred | jaypipes: write down more things, but then delete hide them | |
| 19:52:47 | jaypipes | heh | |
| 19:59:54 | mriedem | jaypipes: dansmith: anything off the top of your head that we already know is going to be a focus for queens? besides just anything that doesn't make pike? things that come to mind are volume multiattach, cinder ephemeral / no local disk | |
| 19:59:58 | mriedem | removing caching scheduler isn't really a feature | |
| 20:00:08 | mriedem | but claims in scheduler would be required in queens | |
| 20:00:22 | jaypipes | mriedem: VGPU. | |
| 20:00:34 | dansmith | mriedem: full shared storage, shared network reporting and stuff | |
| 20:00:41 | jaypipes | mriedem: port pair allocations with SR-IOV. | |
| 20:00:41 | dansmith | mriedem: hopefully some locality/affinity in placement | |
| 20:00:50 | openstackgerrit | Abhishek Sharma M proposed openstack/nova master: Catching OverQuota Exception https://review.openstack.org/455118 | |
| 20:00:50 | openstackgerrit | Abhishek Sharma M proposed openstack/nova master: Catching OverQuota Exception https://review.openstack.org/460293 | |
| 20:00:54 | dansmith | yeah, GPU and SRIOV love in placement | |
| 20:00:57 | jaypipes | dansmith: yeah. ++ on affinity stuff. | |
| 20:01:06 | jaypipes | heh, much love. | |
| 20:04:24 | cburgess | dansmith mriedem Thinking fill shared storage and cinder ephemeral both probably depend on a re-write of the imagebackend interface and creating cleaner seperation and modularization of that area of the code. | |
| 20:05:44 | dansmith | that's at least a cycle of work probably | |
| 20:05:47 | dansmith | unless it's all we do | |
| 20:07:13 | cburgess | dansmith Probably right. Just trying to figure out how we could do shared storage or a cinder backend properly without fixing that dumpster fire induced miasma. | |
| 20:07:35 | dansmith | TIL a new word | |
| 20:07:57 | cburgess | lol | |
| 20:08:03 | cburgess | miasma? | |
| 20:08:37 | dansmith | no "shared" | |
| 20:08:39 | dansmith | yes miasma :P | |
| 20:09:00 | cburgess | I try and educate as well as provide bad ideas, and donuts, to the nova community. | |
| 20:09:13 | cburgess | Its just one of my many redeeming qualities. | |
| 20:09:13 | dansmith | and we thank you for it | |
| 20:10:42 | sean-k-mooney | jaypipes: do you think the user controlled port pair allocations will slip to queens? | |
| 20:11:10 | jaypipes | sean-k-mooney: considering it didn't land for Pike... yes. | |
| 20:11:13 | sean-k-mooney | jaypipes: i have been getting some question lately about doing bonding with sriov for ha and performacne which would need that | |
| 20:11:30 | sean-k-mooney | oh ya we are past spec deadline | |
| 20:15:35 | openstackgerrit | Rafael Folco proposed openstack/nova master: placement: Specific error for inventory in use https://review.openstack.org/460231 | |
| 20:29:41 | mriedem | jaypipes: i don't know what "port pair allocations with SR-IOV" means | |
| 20:30:44 | mriedem | dansmith: isn't gpu/sriov love in placement == nested resource providers == pike? | |
| 20:31:13 | dansmith | mriedem: there's virt driver work that needs to be done that I don't think is going to happen in pike | |