| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-08 | |||
| 18:13:27 | openstackgerrit | Merged openstack/nova master: trivial: Remove dead 'ALIAS' constant https://review.openstack.org/649565 | |
| 18:13:38 | openstackgerrit | Merged openstack/nova master: trivial: Remove unused constants, functions https://review.openstack.org/649567 | |
| 18:13:47 | openstackgerrit | Merged openstack/nova master: trivial: Remove dead resource tracker code https://review.openstack.org/649569 | |
| 18:36:10 | mriedem | dansmith: a quick hot take if you will on the path to dropping this online data migration added in ocata: https://github.com/openstack/nova/blob/master/nova/objects/build_request.py#L484 - options: (1) drop it and just assume people are ok (2) add a nova-status upgrade check and drop it or (3) add an api migrations script which just does the query and delete (not a blocker migration, just query for those records and delete th | |
| 18:36:31 | mriedem | i'm partial to #3 which if #1 is true, should be a no-op | |
| 18:39:27 | dansmith | surely we can delete with extreme prejudice | |
| 18:39:47 | dansmith | the case would be what? something created in mitaka that didn't boot that people still want to be able to see in nova-list? | |
| 18:43:00 | dansmith | mriedem: ^ | |
| 18:43:49 | mriedem | yeah i guess, but we're already deleting with prejudice in the online data migration | |
| 18:44:23 | mriedem | because you literally can't load the thing otherwise | |
| 18:44:26 | mriedem | so nova-list won't work anyway | |
| 18:44:28 | dansmith | mriedem: I mean delete all of the code | |
| 18:44:50 | mriedem | so #1 | |
| 18:45:01 | dansmith | if we really need a delete in a schema migration, then I guess, but I'd rather not create the confusion about why it was okay in this case, personally | |
| 18:46:07 | mriedem | i think the only thing that scares me is people doing skip upgrades from let's say mitaka in this case to train, but we've said time and again we don't support that and you have to roll through, | |
| 18:46:20 | mriedem | so if you do that and hit this, your action would be to just drop those records from your api db manually | |
| 18:47:29 | dansmith | yeah, that's a really long skip | |
| 19:50:26 | cfriesen | portdirect: thought I'd give you a heads-up on something weird I saw. was booting up controller nodes and one of them didn't come up due to disk pressure. The other mariadb nodes were stuck in /tmp/start.py because check_if_cluster_data_is_fresh() kept evaluating to false (because the one node wasn't updating its timestamp) | |
| 20:00:43 | cfriesen | whoops...^ was supposed to be in another channel. | |
| 20:12:13 | mnaser | I mentioned this earlier but.. this does a full table scan, even with max rows.. https://github.com/openstack/nova/blob/73edcfae7d5ea498fd17fa8e548ae1bd690f408a/nova/db/sqlalchemy/api.py#L5508-L5537 | |
| 20:12:18 | mnaser | I'm not sure what the best way to work around it is.. | |
| 20:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 20:17:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | |
| 20:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 20:17:53 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 20:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 20:17:54 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 20:17:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 20:17:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 20:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 20:17:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 20:17:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method https://review.openstack.org/642590 | |
| 20:17:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | |
| 20:17:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.openstack.org/635646 | |
| 20:17:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.openstack.org/635080 | |
| 20:17:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.openstack.org/635668 | |
| 20:17:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field https://review.openstack.org/614035 | |
| 20:18:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.openstack.org/636224 | |
| 20:18:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.openstack.org/635684 | |
| 20:18:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.openstack.org/636253 | |
| 20:18:01 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change HostManager to allow scheduling to other cells https://review.openstack.org/614037 | |
| 20:18:52 | mriedem | mnaser: are we missing indexes? | |
| 20:20:36 | mriedem | i don't see any indexes on instance_actions_events | |
| 20:20:48 | mriedem | update_statement = table.update().values(deleted=table.c.id).\ where(table.c.action_id.in_(deleted_actions)) | |
| 20:20:48 | mriedem | but that does: | |
| 20:21:45 | mriedem | i don't know if adding an index on instance_actions_events.action_id would help though | |
| 20:22:04 | mriedem | probably need jaypipes_ to tell us what sucks about those queries | |
| 20:26:25 | mriedem | that's also in the mainline code https://github.com/openstack/nova/blob/fb1fee6772bb101eac83845bac9022df77113aaa/nova/db/sqlalchemy/api.py#L5288 | |
| 20:28:30 | mnaser | Yeah I don’t know much. I had to kind orchestrate this outside nova and shortcircuit that code path | |
| 20:31:13 | mriedem | mnaser: do you archive/purge on a cron or just during each upgrade? | |
| 20:31:40 | mriedem | and tbc this just makes that archive/purge slower right? | |
| 20:33:27 | mnaser | mriedem: we’ve kinda not done this for a while so we ended up with huge databases. | |
| 20:33:39 | mnaser | so this may be a case of our fault not doing it as often | |
| 20:35:04 | mriedem | ok, was just trying to figure out if it's worth opening a bug report, i.e. if archive takes like 5 minutes or something, then maybe | |
| 20:43:09 | efried | stephenfin, bauzas, dansmith: Do we need to get those placeholder thingies in before the real release? I.e. modify the release patch? | |
| 20:45:01 | efried | mriedem, melwitt: ^ | |
| 20:49:43 | mriedem | no the placeholder migrations are post-release | |
| 20:49:57 | mriedem | so if we needed to add a schema migration in stable/stein, we have a placeholder for it in train | |
| 20:50:51 | efried | mriedem: Okay, so... is there some reason we couldn't backport placeholder migrations for rocky? | |
| 20:51:12 | mriedem | we don't backport placeholder migrations, | |
| 20:51:18 | mriedem | but apparently forgot to add them for rocky | |
| 20:51:21 | mriedem | *in* stein | |
| 20:51:38 | efried | I'm confused | |
| 20:51:46 | efried | then why is https://review.openstack.org/#/c/650964/ in master and not in stable/stein? | |
| 20:51:58 | mriedem | for example, latest cell db migration in stein is 391 https://github.com/openstack/nova/tree/stable/stein/nova/db/sqlalchemy/migrate_repo/versions so if we needed something in stable/stein, we'd have to add 392 in stable/stein and then copy that to the 392_placeholder in train | |
| 20:52:11 | mriedem | it's essentially a buffer, | |
| 20:52:20 | mriedem | because otherwise the first schema migration in train is 392 | |
| 20:52:30 | mriedem | so if we need something in stein but already used 392 in train, we're stuck | |
| 20:53:26 | efried | oh, so if we hadn't done any migrations in stein, we could set up the buffer for rocky. | |
| 20:53:28 | jaypipes | mriedem: instance_action_events.action_id will automatically have an index on it since it's an FK to instance_actions | |
| 20:53:29 | mriedem | rocky ended at 390 and there was one cell db migration in stein, 391 | |
| 20:53:33 | mriedem | so if we need something on rocky we're f'ed | |
| 20:53:51 | jaypipes | at least for mysql and pgsql it will. | |
| 20:54:06 | mriedem | jaypipes: ah ok, thanks. i seem to remember asking about this same query a year or more ago and came to the same conclusion (or you told me the same at the time) | |
| 20:54:07 | jaypipes | I think :) | |
| 20:54:35 | jaypipes | might want to have mnaser do a quick SHOW TABLE instance_action_events to verify | |
| 20:54:38 | mriedem | else i'm thinking action_id/deleted or something | |
| 20:55:11 | jaypipes | hmm, maybe not since we don't use SQLalchemy create_tables() | |
| 20:55:14 | mnaser | be back in my machine in a sec | |
| 20:56:21 | jaypipes | but if the FOREIGN KEY .. REFERENCES statement/clause is executed at any time (either by SQLAlchemy's declarative_base or using sqlalchemy-migrate) then there is indeed an index created on action_id | |
| 20:56:56 | jaypipes | lemme log into an ocata prod DB and check... | |
| 20:58:40 | jaypipes | mriedem, mnaser: verified. there is indeed an index on (action_id) | |
| 20:59:12 | mriedem | jaypipes: ok cool | |
| 20:59:15 | mriedem | efried: https://review.openstack.org/#/q/project:openstack/nova+status:open+path:%255Enova/db/sqlalchemy/migrate_repo/versions/.* :) | |
| 21:02:51 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899 | |
| 21:03:35 | efried | dustinc: That looks like a sane delta ^ | |
| 21:03:45 | efried | were you planning to push other patches on top of that at the same time? | |
| 21:03:58 | dustinc | not yet, I have them local but was just pushing that one for now | |
| 21:04:06 | mnaser | jaypipes: so the thing is I think it might be the deleted field | |
| 21:05:14 | mnaser | the query looked a something like this: UPDATE instance_actions_events SET deleted=instance_action_events.id WHERE action_id IN (SELECT id FROM instance_actions WHERE instance_uuid IN (SELECT uuid FROM instances WHERE deleted!=0)) | |
| 21:05:35 | mnaser | so you grab a list of all non deleted instances, then you grab all their instance actions, then you grab all the action ids associated to them | |
| 21:05:37 | mriedem | weeee https://github.com/openstack/nova/blob/fb1fee6772bb101eac83845bac9022df77113aaa/nova/db/sqlalchemy/api.py#L174 | |
| 21:06:11 | mriedem | mnaser: other way around - it's soft-deleting actions and events for deleted instances | |
| 21:06:16 | mriedem | deleted!=0 == soft-deleted | |
| 21:06:28 | mriedem | archive is the only thing that actually deletes actions/events | |
| 21:06:33 | mnaser | right, yeah, so it soft deletes them because for some reason we don't soft delete them by default? yeah | |
| 21:06:45 | mnaser | so if you haven't done that in a while, that can be.. a big dataset | |
| 21:06:52 | mriedem | mnaser: corret https://github.com/openstack/nova/blob/fb1fee6772bb101eac83845bac9022df77113aaa/nova/db/sqlalchemy/api.py#L1832 | |
| 21:06:54 | mriedem | *correct | |
| 21:07:15 | mnaser | oh neat | |