| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 21:07:03 | mriedem | dansmith: heh, and lxc more than both of those | |
| 21:07:13 | dansmith | yeah | |
| 21:07:23 | mriedem | which if people are using lxc, | |
| 21:07:35 | mriedem | i'd sure like them to upstream their patch that makes the CI not completely fall over itself when deleting an instance | |
| 21:07:45 | dansmith | heh yeah | |
| 21:12:23 | mriedem | i'd like to ask for feedback on newish things like placement and cells v2 but with a bunch of deployments on kilo, that's going to be hard to get | |
| 21:14:08 | openstackgerrit | Rick Bartra proposed openstack/nova master: Add policy granularity to the Flavors API https://review.openstack.org/449288 | |
| 21:14:10 | mriedem | 1. Are you using REST API microversions > 2.1? | |
| 21:14:11 | mriedem | 2. Have you started evaluating or testing the Placement service? | |
| 21:14:11 | mriedem | 3. Have you started evaluating or testing Cells v2? | |
| 21:14:13 | mriedem | is my list right now | |
| 21:14:51 | cfriesen | mriedem: speaking of cells v2, the Newton release notes don't talk about the increase in the number of DB connections due to the need for services to talk to both DBs. | |
| 21:15:06 | mriedem | cfriesen: sorry i saw your earlier question about the api | |
| 21:15:22 | mriedem | cfriesen: i think you can disable the api from going directly to the db | |
| 21:15:35 | dansmith | mriedem: you can't | |
| 21:15:36 | mriedem | i was trying to answer this for someone internally last night | |
| 21:15:41 | dansmith | and I have no plans to head that way | |
| 21:15:46 | mriedem | dansmith: i thought you could make the api use the conductor indirection api? | |
| 21:15:58 | dansmith | not unless someone added it and I didn't know | |
| 21:15:59 | mriedem | like https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L53 | |
| 21:16:03 | dansmith | nope | |
| 21:16:05 | mriedem | no it's not in tree upstream | |
| 21:16:18 | dansmith | I can't imagine that not sucking balls | |
| 21:16:38 | dansmith | of course, apparently we're fine with ~10s nova-list operations, so.. maybe nobody cares | |
| 21:17:35 | mriedem | cfriesen: have you enabled the cells v2 stuff in newton? because it's not required there yet | |
| 21:17:43 | mriedem | cfriesen: bully if you have | |
| 21:17:47 | mriedem | i'm interested in feedback | |
| 21:17:50 | dansmith | mriedem: well, api db was required in newton right? | |
| 21:17:53 | mriedem | we don't really get any sort of performance feedback upstream | |
| 21:18:07 | mriedem | dansmith: api db has been required for several releases before newton | |
| 21:18:11 | mriedem | i think liberty? | |
| 21:18:13 | mriedem | for reqspec | |
| 21:18:21 | dansmith | yeah, he was asking about increased connections for api/nova db right? | |
| 21:18:24 | cfriesen | mriedem: no, our clouds are small enough we don't use cells. but we did see nova basically doubling the number of DB connections going from Mitaka to Newton. | |
| 21:19:45 | mriedem | actually i guess reqspec was mitaka https://review.openstack.org/#/c/211753/ | |
| 21:20:20 | mriedem | so what's hitting the api db in newton? build request right? | |
| 21:20:39 | cfriesen | mriedem: yeah, I see the api db in mitaka | |
| 21:21:04 | clarkb | dansmith: we cache the list results for that reason | |
| 21:21:09 | mriedem | in newton i suppose we are using build request and looking up instance, host and cellmappings even if we're just getting NotFounds | |
| 21:21:15 | dansmith | clarkb: yeah, it's sad | |
| 21:21:17 | clarkb | (as well as caching everyting else) | |
| 21:21:31 | cfriesen | mriedem: aggregates maybe? | |
| 21:21:39 | dansmith | cfriesen: no | |
| 21:21:47 | mriedem | flavors are in the api db | |
| 21:22:35 | mriedem | and keypairs, and aggregates, and server groups | |
| 21:22:57 | mriedem | and those are doing online data migrations | |
| 21:23:28 | cfriesen | we use aggregates and flavor extra specs a lot | |
| 21:23:37 | mriedem | so we might have some overhead for checking for online data migrations which are not necessary once you've migrated everything | |
| 21:24:00 | melwitt | for any target_cell, a new oslo.db _TransactionContextManager will be created to make the connection | |
| 21:24:09 | clarkb | dansmith: that said we peaked booting at 30k instances a day across multiple clouds/regions so 10s isn't the end of the world | |
| 21:24:22 | clarkb | dansmith: and I'm guessing more users do far less so its not a major headache | |
| 21:24:42 | dansmith | clarkb: well, it still sucks, don't try to convince me :) | |
| 21:24:54 | clarkb | oh I agree, just trying to point out why people probably don' | |
| 21:24:56 | clarkb | t scream at you | |
| 21:24:57 | melwitt | dansmith: is the 10s sans novaclient or is it novaclient? | |
| 21:25:44 | mriedem | yeah doesn't novaclient look up the image for every instance to show the image name in the output? | |
| 21:26:16 | mriedem | cfriesen: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L602 | |
| 21:26:29 | mriedem | should give you an idea of online data migrations added in newton | |
| 21:26:55 | mriedem | which is a lot | |
| 21:26:56 | cfriesen | mriedem: thanks. so basically most nova services would be expected to have connections open to both DBs now? | |
| 21:27:08 | mriedem | cfriesen: only nova-api, conductor and scheduler | |
| 21:27:34 | cfriesen | right, the heavyduty ones other than nova-compute | |
| 21:28:02 | mriedem | cfriesen: have you guys done any profiling of where the worst hits are? | |
| 21:28:28 | cfriesen | not yet...someone was just complaining about maybe needing to bump up the limit on overall DB connections. | |
| 21:28:29 | mriedem | dansmith: we should maybe look at blocker migrations for pike so we can stop dropping these older online data migration routines | |
| 21:28:42 | mriedem | s/stop/start/ | |
| 21:28:43 | mriedem | :) | |
| 21:28:49 | dansmith | hah, was confused | |
| 21:29:13 | dansmith | sure, but they're not that expensive so I've been working on other stuff | |
| 21:30:41 | mriedem | wonder if any of the versioned notification stuff is blowing up perf by doing lazy-loads | |
| 21:33:19 | mriedem | yeah i guess with like the online flavor migrations, those are all of the style where we looking in the api db first, then fallback to looking in the main cell db | |
| 21:33:29 | mriedem | we don't migrate those in the object code, we leave that for the online_data_migrations CLI | |
| 21:33:44 | mriedem | cfriesen: have you guys been running the online_data_migrations CLIs in newton? | |
| 21:36:23 | mriedem | dansmith: so say i wanted to remove all of the api->cell_db checking code in the flavor object during the lookup methods, are the steps add a blocker db migration, then drop the compat code in the flavor object, but leave https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L604 since you need that to get through the blocker migration | |
| 21:36:47 | mriedem | this doesn't buy us anything for perf, but it would drop all of that compat code in the flavor object | |
| 21:37:20 | dansmith | mriedem: historically we've done the blocker and then removed it, saying you had to use N-1 code to do the migration | |
| 21:37:25 | dansmith | but those were thigns we really wanted to drop | |
| 21:37:32 | dansmith | and it was all intertwined with object code | |
| 21:37:41 | dansmith | and, we always do blockers as the first migration | |
| 21:37:46 | mriedem | yeah i figured we wouldn't have to here b/c the object code isn't doing a migration | |
| 21:37:54 | dansmith | potentially | |
| 21:38:24 | mriedem | before you start using multiple cells you should really have your flavors in the api db | |
| 21:38:29 | dansmith | yes :) | |
| 21:38:36 | mriedem | in fact.... | |
| 21:38:55 | mriedem | multi-cell basically breaks the lookup routine since we don't know which cell db to fallback to if the flavor isn't in the api db | |
| 21:39:14 | mriedem | so we kind of need to drop all of that code anyway i'd think | |
| 21:39:44 | dansmith | I'm confinced | |
| 21:39:47 | dansmith | convinced even | |
| 21:40:24 | mriedem | are we unable to do this if the blocker migration isn't the first migration for pike? | |
| 21:40:44 | dansmith | no, we did that to avoid you being able to keep running on N-1 code | |
| 21:40:54 | dansmith | like, we tweak your db, then tell you you can't run N without migrating in N-1, | |
| 21:40:59 | dansmith | but N-1 might choke on the tweaks | |
| 21:41:01 | dansmith | that would be evil | |
| 21:41:15 | dansmith | if you're not dropping the migration code, then I guess you could do it | |
| 21:41:33 | mriedem | right that's why i said we still keep https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L604 | |
| 21:42:07 | mriedem | but you really need to purge this stuff out of your main cell db and get it into api before you can do multiple cells | |
| 21:42:20 | mriedem | maybe a nice reminder for your summit talk :) | |
| 21:44:28 | dansmith | heh | |
| 21:45:17 | mriedem | so https://review.openstack.org/#/c/374307/ | |
| 21:45:23 | mriedem | we already enforced this in ocata :) | |