| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 20:41:13 | dansmith | hah | |
| 20:41:52 | mriedem | b/c we're always coding 18 months ahead at least | |
| 20:41:54 | mriedem | and have no idea | |
| 20:51:57 | mriedem | "While the majority of users aren’t currently using Nova cells, about 146 deployments reported using cells, which is substantially more than the 67 users reporting using Nova cells one year ago. " | |
| 20:53:36 | abhi89 | johnthetubaguy: Hi | |
| 20:55:07 | abhi89 | Hi all.. can someone please review https://review.openstack.org/#/c/460305/ | |
| 20:55:19 | mordred | mriedem: I fully expect to start consuming microversions in the next 6 months | |
| 20:55:57 | mriedem | oh right this was the question | |
| 20:55:58 | mriedem | "How important is it to be able to customize Nova in your deployment? e.g. classload your own managers/drivers, use hooks, plug in API extensions, etc." | |
| 20:56:18 | mordred | I was very happy to see the low numbers of "we use this extensively" | |
| 20:56:23 | mriedem | 51% not important, 39% somewhat, 10% very | |
| 20:56:41 | mordred | s/low/lowish/ | |
| 20:57:10 | jaypipes | dansmith: you're gonna love all this shared resource provider SQL... | |
| 20:57:49 | mriedem | i see that neutron asked about actual feature usage/adoption | |
| 20:58:08 | mriedem | that's something we could breakdown at some point, and have wondered about, i.e. does anyone actually use shelve? | |
| 20:58:19 | mordred | edleafe: I'll try to start with an amuse bouche, but that's probably not too far off | |
| 20:58:49 | mordred | mriedem: I don't | |
| 20:59:55 | mriedem | page 60 https://www.openstack.org/assets/survey/April2017SurveyReport.pdf | |
| 21:00:21 | mriedem | breaking something down like that in nova wouldn't be trivial | |
| 21:00:33 | mordred | no - but it would be worthwhile | |
| 21:02:11 | dansmith | more than twice as many people patch in an unmaintained docker driver than use the in-tree xen support | |
| 21:02:21 | dansmith | I'll remember that next time I have to fix something in there | |
| 21:02:52 | cfriesen | I'm curious...are we expecting that eventually nova-api won't need a connection to the main nova db? | |
| 21:03:19 | mordred | dansmith: that also highlights one of the big disconnects between what users want and what developers are funded to work on/maintain | |
| 21:03:35 | dansmith | mordred: well, might've been a disconnect a couple weeks ago, but yeah | |
| 21:05:51 | openstackgerrit | Steve Noyes proposed openstack/nova master: Add Cinder v3 detach to local_cleanup https://review.openstack.org/456851 | |
| 21:07:03 | mriedem | dansmith: heh, and lxc more than both of those | |
| 21:07:13 | dansmith | yeah | |
| 21:07:23 | mriedem | which if people are using lxc, | |
| 21:07:35 | mriedem | i'd sure like them to upstream their patch that makes the CI not completely fall over itself when deleting an instance | |
| 21:07:45 | dansmith | heh yeah | |
| 21:12:23 | mriedem | i'd like to ask for feedback on newish things like placement and cells v2 but with a bunch of deployments on kilo, that's going to be hard to get | |
| 21:14:08 | openstackgerrit | Rick Bartra proposed openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | |
| 21:14:10 | mriedem | 1. Are you using REST API microversions > 2.1? | |
| 21:14:11 | mriedem | 2. Have you started evaluating or testing the Placement service? | |
| 21:14:11 | mriedem | 3. Have you started evaluating or testing Cells v2? | |
| 21:14:13 | mriedem | is my list right now | |
| 21:14:51 | cfriesen | mriedem: speaking of cells v2, the Newton release notes don't talk about the increase in the number of DB connections due to the need for services to talk to both DBs. | |
| 21:15:06 | mriedem | cfriesen: sorry i saw your earlier question about the api | |
| 21:15:22 | mriedem | cfriesen: i think you can disable the api from going directly to the db | |
| 21:15:35 | dansmith | mriedem: you can't | |
| 21:15:36 | mriedem | i was trying to answer this for someone internally last night | |
| 21:15:41 | dansmith | and I have no plans to head that way | |
| 21:15:46 | mriedem | dansmith: i thought you could make the api use the conductor indirection api? | |
| 21:15:58 | dansmith | not unless someone added it and I didn't know | |
| 21:15:59 | mriedem | like https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L53 | |
| 21:16:03 | dansmith | nope | |
| 21:16:05 | mriedem | no it's not in tree upstream | |
| 21:16:18 | dansmith | I can't imagine that not sucking balls | |
| 21:16:38 | dansmith | of course, apparently we're fine with ~10s nova-list operations, so.. maybe nobody cares | |
| 21:17:35 | mriedem | cfriesen: have you enabled the cells v2 stuff in newton? because it's not required there yet | |
| 21:17:43 | mriedem | cfriesen: bully if you have | |
| 21:17:47 | mriedem | i'm interested in feedback | |
| 21:17:50 | dansmith | mriedem: well, api db was required in newton right? | |
| 21:17:53 | mriedem | we don't really get any sort of performance feedback upstream | |
| 21:18:07 | mriedem | dansmith: api db has been required for several releases before newton | |
| 21:18:11 | mriedem | i think liberty? | |
| 21:18:13 | mriedem | for reqspec | |
| 21:18:21 | dansmith | yeah, he was asking about increased connections for api/nova db right? | |
| 21:18:24 | cfriesen | mriedem: no, our clouds are small enough we don't use cells. but we did see nova basically doubling the number of DB connections going from Mitaka to Newton. | |
| 21:19:45 | mriedem | actually i guess reqspec was mitaka https://review.openstack.org/#/c/211753/ | |
| 21:20:20 | mriedem | so what's hitting the api db in newton? build request right? | |
| 21:20:39 | cfriesen | mriedem: yeah, I see the api db in mitaka | |
| 21:21:04 | clarkb | dansmith: we cache the list results for that reason | |
| 21:21:09 | mriedem | in newton i suppose we are using build request and looking up instance, host and cellmappings even if we're just getting NotFounds | |
| 21:21:15 | dansmith | clarkb: yeah, it's sad | |
| 21:21:17 | clarkb | (as well as caching everyting else) | |
| 21:21:31 | cfriesen | mriedem: aggregates maybe? | |
| 21:21:39 | dansmith | cfriesen: no | |
| 21:21:47 | mriedem | flavors are in the api db | |
| 21:22:35 | mriedem | and keypairs, and aggregates, and server groups | |
| 21:22:57 | mriedem | and those are doing online data migrations | |
| 21:23:28 | cfriesen | we use aggregates and flavor extra specs a lot | |
| 21:23:37 | mriedem | so we might have some overhead for checking for online data migrations which are not necessary once you've migrated everything | |
| 21:24:00 | melwitt | for any target_cell, a new oslo.db _TransactionContextManager will be created to make the connection | |
| 21:24:09 | clarkb | dansmith: that said we peaked booting at 30k instances a day across multiple clouds/regions so 10s isn't the end of the world | |
| 21:24:22 | clarkb | dansmith: and I'm guessing more users do far less so its not a major headache | |
| 21:24:42 | dansmith | clarkb: well, it still sucks, don't try to convince me :) | |
| 21:24:54 | clarkb | oh I agree, just trying to point out why people probably don' | |
| 21:24:56 | clarkb | t scream at you | |
| 21:24:57 | melwitt | dansmith: is the 10s sans novaclient or is it novaclient? | |
| 21:25:44 | mriedem | yeah doesn't novaclient look up the image for every instance to show the image name in the output? | |
| 21:26:16 | mriedem | cfriesen: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L602 | |
| 21:26:29 | mriedem | should give you an idea of online data migrations added in newton | |
| 21:26:55 | mriedem | which is a lot | |
| 21:26:56 | cfriesen | mriedem: thanks. so basically most nova services would be expected to have connections open to both DBs now? | |
| 21:27:08 | mriedem | cfriesen: only nova-api, conductor and scheduler | |
| 21:27:34 | cfriesen | right, the heavyduty ones other than nova-compute | |
| 21:28:02 | mriedem | cfriesen: have you guys done any profiling of where the worst hits are? | |
| 21:28:28 | cfriesen | not yet...someone was just complaining about maybe needing to bump up the limit on overall DB connections. | |
| 21:28:29 | mriedem | dansmith: we should maybe look at blocker migrations for pike so we can stop dropping these older online data migration routines | |
| 21:28:42 | mriedem | s/stop/start/ | |
| 21:28:43 | mriedem | :) | |
| 21:28:49 | dansmith | hah, was confused | |
| 21:29:13 | dansmith | sure, but they're not that expensive so I've been working on other stuff | |
| 21:30:41 | mriedem | wonder if any of the versioned notification stuff is blowing up perf by doing lazy-loads | |
| 21:33:19 | mriedem | yeah i guess with like the online flavor migrations, those are all of the style where we looking in the api db first, then fallback to looking in the main cell db | |
| 21:33:29 | mriedem | we don't migrate those in the object code, we leave that for the online_data_migrations CLI | |
| 21:33:44 | mriedem | cfriesen: have you guys been running the online_data_migrations CLIs in newton? | |
| 21:36:23 | mriedem | dansmith: so say i wanted to remove all of the api->cell_db checking code in the flavor object during the lookup methods, are the steps add a blocker db migration, then drop the compat code in the flavor object, but leave https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L604 since you need that to get through the blocker migration | |