| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-26 | |||
| 21:18:11 | mriedem | i think liberty? | |
| 21:18:13 | mriedem | for reqspec | |
| 21:18:21 | dansmith | yeah, he was asking about increased connections for api/nova db right? | |
| 21:18:24 | cfriesen | mriedem: no, our clouds are small enough we don't use cells. but we did see nova basically doubling the number of DB connections going from Mitaka to Newton. | |
| 21:19:45 | mriedem | actually i guess reqspec was mitaka https://review.openstack.org/#/c/211753/ | |
| 21:20:20 | mriedem | so what's hitting the api db in newton? build request right? | |
| 21:20:39 | cfriesen | mriedem: yeah, I see the api db in mitaka | |
| 21:21:04 | clarkb | dansmith: we cache the list results for that reason | |
| 21:21:09 | mriedem | in newton i suppose we are using build request and looking up instance, host and cellmappings even if we're just getting NotFounds | |
| 21:21:15 | dansmith | clarkb: yeah, it's sad | |
| 21:21:17 | clarkb | (as well as caching everyting else) | |
| 21:21:31 | cfriesen | mriedem: aggregates maybe? | |
| 21:21:39 | dansmith | cfriesen: no | |
| 21:21:47 | mriedem | flavors are in the api db | |
| 21:22:35 | mriedem | and keypairs, and aggregates, and server groups | |
| 21:22:57 | mriedem | and those are doing online data migrations | |
| 21:23:28 | cfriesen | we use aggregates and flavor extra specs a lot | |
| 21:23:37 | mriedem | so we might have some overhead for checking for online data migrations which are not necessary once you've migrated everything | |
| 21:24:00 | melwitt | for any target_cell, a new oslo.db _TransactionContextManager will be created to make the connection | |
| 21:24:09 | clarkb | dansmith: that said we peaked booting at 30k instances a day across multiple clouds/regions so 10s isn't the end of the world | |
| 21:24:22 | clarkb | dansmith: and I'm guessing more users do far less so its not a major headache | |
| 21:24:42 | dansmith | clarkb: well, it still sucks, don't try to convince me :) | |
| 21:24:54 | clarkb | oh I agree, just trying to point out why people probably don' | |
| 21:24:56 | clarkb | t scream at you | |
| 21:24:57 | melwitt | dansmith: is the 10s sans novaclient or is it novaclient? | |
| 21:25:44 | mriedem | yeah doesn't novaclient look up the image for every instance to show the image name in the output? | |
| 21:26:16 | mriedem | cfriesen: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L602 | |
| 21:26:29 | mriedem | should give you an idea of online data migrations added in newton | |
| 21:26:55 | mriedem | which is a lot | |
| 21:26:56 | cfriesen | mriedem: thanks. so basically most nova services would be expected to have connections open to both DBs now? | |
| 21:27:08 | mriedem | cfriesen: only nova-api, conductor and scheduler | |
| 21:27:34 | cfriesen | right, the heavyduty ones other than nova-compute | |
| 21:28:02 | mriedem | cfriesen: have you guys done any profiling of where the worst hits are? | |
| 21:28:28 | cfriesen | not yet...someone was just complaining about maybe needing to bump up the limit on overall DB connections. | |
| 21:28:29 | mriedem | dansmith: we should maybe look at blocker migrations for pike so we can stop dropping these older online data migration routines | |
| 21:28:42 | mriedem | s/stop/start/ | |
| 21:28:43 | mriedem | :) | |
| 21:28:49 | dansmith | hah, was confused | |
| 21:29:13 | dansmith | sure, but they're not that expensive so I've been working on other stuff | |
| 21:30:41 | mriedem | wonder if any of the versioned notification stuff is blowing up perf by doing lazy-loads | |
| 21:33:19 | mriedem | yeah i guess with like the online flavor migrations, those are all of the style where we looking in the api db first, then fallback to looking in the main cell db | |
| 21:33:29 | mriedem | we don't migrate those in the object code, we leave that for the online_data_migrations CLI | |
| 21:33:44 | mriedem | cfriesen: have you guys been running the online_data_migrations CLIs in newton? | |
| 21:36:23 | mriedem | dansmith: so say i wanted to remove all of the api->cell_db checking code in the flavor object during the lookup methods, are the steps add a blocker db migration, then drop the compat code in the flavor object, but leave https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L604 since you need that to get through the blocker migration | |
| 21:36:47 | mriedem | this doesn't buy us anything for perf, but it would drop all of that compat code in the flavor object | |
| 21:37:20 | dansmith | mriedem: historically we've done the blocker and then removed it, saying you had to use N-1 code to do the migration | |
| 21:37:25 | dansmith | but those were thigns we really wanted to drop | |
| 21:37:32 | dansmith | and it was all intertwined with object code | |
| 21:37:41 | dansmith | and, we always do blockers as the first migration | |
| 21:37:46 | mriedem | yeah i figured we wouldn't have to here b/c the object code isn't doing a migration | |
| 21:37:54 | dansmith | potentially | |
| 21:38:24 | mriedem | before you start using multiple cells you should really have your flavors in the api db | |
| 21:38:29 | dansmith | yes :) | |
| 21:38:36 | mriedem | in fact.... | |
| 21:38:55 | mriedem | multi-cell basically breaks the lookup routine since we don't know which cell db to fallback to if the flavor isn't in the api db | |
| 21:39:14 | mriedem | so we kind of need to drop all of that code anyway i'd think | |
| 21:39:44 | dansmith | I'm confinced | |
| 21:39:47 | dansmith | convinced even | |
| 21:40:24 | mriedem | are we unable to do this if the blocker migration isn't the first migration for pike? | |
| 21:40:44 | dansmith | no, we did that to avoid you being able to keep running on N-1 code | |
| 21:40:54 | dansmith | like, we tweak your db, then tell you you can't run N without migrating in N-1, | |
| 21:40:59 | dansmith | but N-1 might choke on the tweaks | |
| 21:41:01 | dansmith | that would be evil | |
| 21:41:15 | dansmith | if you're not dropping the migration code, then I guess you could do it | |
| 21:41:33 | mriedem | right that's why i said we still keep https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L604 | |
| 21:42:07 | mriedem | but you really need to purge this stuff out of your main cell db and get it into api before you can do multiple cells | |
| 21:42:20 | mriedem | maybe a nice reminder for your summit talk :) | |
| 21:44:28 | dansmith | heh | |
| 21:45:17 | mriedem | so https://review.openstack.org/#/c/374307/ | |
| 21:45:23 | mriedem | we already enforced this in ocata :) | |
| 21:46:26 | dansmith | flavors yeah | |
| 21:46:36 | mriedem | and keypairs, aggregates and server groups | |
| 21:46:44 | dansmith | and those yeah | |
| 21:48:44 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: De-complicate some of the instance delete path https://review.openstack.org/458636 | |
| 21:48:44 | openstackgerrit | Dan Smith proposed openstack/nova master: Move to proper target_cell calling convention https://review.openstack.org/458634 | |
| 21:48:45 | openstackgerrit | Dan Smith proposed openstack/nova master: Make target_cell() yield a new context https://review.openstack.org/458635 | |
| 21:48:45 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary test against multi-cell devstack https://review.openstack.org/458537 | |
| 22:09:11 | efried | mordred Still around? | |
| 23:05:39 | mordred | efried: hey! not really - EODing - catch you in the morning? | |
| #openstack-nova - 2017-04-27 | |||
| 00:39:48 | mriedem | dansmith: i just got thinking that maybe we shouldn't have removed the online data migration hook in https://review.openstack.org/#/c/448579/ but then again that was added in mitaka, and the blocker migration was added in newton. the thing that makes me 2nd guess it is people skipping releases for upgrades. | |
| 00:39:54 | mriedem | which we don't support, but people do it | |
| 00:40:02 | mriedem | skipping from newton to pike though, would be nuts | |
| 00:41:14 | mriedem | the online flavor migration was added in newton, the blocker migration was added in ocata, so theoretically we could remove all of that in pike, but if people skip ocata, and we removed the online data migration CLI for flavors, they'd be screwed | |
| 00:47:13 | mordred | mriedem: skip-upgrades seem like a completely crazy thing if they're not actually supported and planned for | |
| 00:48:57 | efried | mordred Yes, mañana. Want to make sure I'm on the right track with https://review.openstack.org/#/c/460337/ and ask some questions about same. | |
| 00:49:39 | mordred | cool- looking forward to it | |
| 00:51:02 | mordred | efried: oh great - I missed that you'd pushed that up - I'll read it first thing | |
| 00:58:11 | mriedem | mordred: yeah, except ops are always talking about doing it | |
| 00:58:18 | mriedem | because we don't have LTS or whatever | |
| 01:16:55 | mriedem | dansmith: oh fun, uncovered a bug removing all of this flavor object compat code for the main cell db, | |
| 01:17:13 | mriedem | when sending a flavor delete notification, we lazy-load the projects which results in a FlavorNotFound b/c we just deleted that flavor | |
| 01:23:42 | mriedem | looks like that's due to https://review.openstack.org/#/c/445697/ | |
| 02:53:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Remove compatibility code for flavors https://review.openstack.org/460377 | |
| 02:55:46 | morgan | huh I saw my name here | |
| 02:55:51 | morgan | what did I miss? | |
| 02:56:00 | morgan | can't find it in the backlog ATM. | |
| 03:48:31 | RiteshPaiboina | who jaypipes | |
| 04:18:02 | openstackgerrit | falseuser proposed openstack/nova master: [BugFix] Check the instance task status to release the memory quota for vram https://review.openstack.org/456050 | |
| 04:18:03 | openstackgerrit | falseuser proposed openstack/nova master: [BugFix]Include the video ram when comparing the difference of two flavors https://review.openstack.org/460388 | |
| 05:15:37 | openstackgerrit | Kiseok Kim proposed openstack/nova master: Fixed a broken link in API Plugins document https://review.openstack.org/460398 | |