Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-21
20:18:06 imacdonn mriedem: can I ask a "dumb" question or two about upgrade checks?
20:18:24 mriedem yes
20:19:11 imacdonn so ... my understanding of "nova-status upgrade check" is that it's something that would be run *after* upgrading (db sync, etc), to check that everything is OK to proceed
20:19:25 imacdonn now it sounds like you want to do checks for things that should be done BEFORE the upgrade in there
20:19:43 imacdonn which is it?
20:22:10 mriedem it's meant to be run before starting new code
20:22:24 mriedem but that's not really a hard and fast rule
20:22:36 imacdonn but ... before or after upgrading the database ?
20:22:37 mriedem i.e. you can also run the nova-status upgrade check as a post-install step for a fresh install
20:22:44 imacdonn (databases)
20:22:49 mriedem after
20:23:09 imacdonn OK, so then ... it seems like it's not the right place for pre-upgrade checks
20:23:27 imacdonn there could be things you'd want to check before you try to upgrade the databases
20:23:48 mriedem well, depends on what your db upgrades are doing
20:23:58 mriedem are they doing data migrations during db sync?
20:24:01 mriedem b/c they really shouldn't
20:24:53 imacdonn hmm, I guess I don't know enough about the db internals ... I kinda thought that that's what db sync does
20:25:27 imacdonn if not where, where should the migrations happen? on first startup after upgrade ?
20:25:28 mriedem nova's db sync routines which lay down schema are primarily only for additive schema changes, like adding tables, columns and indexes/constraints
20:25:39 mriedem not for things that involve data migrations
20:25:49 mriedem during runtime
20:26:08 mriedem e.g. if you need to change the format for some value stored in the db, check on read and update on write
20:26:37 mriedem nova has moved several things from the 'cell' db to the api db, and during that process our routine is to read from the api db first, if not there, read from the cell db, and then migrate
20:26:51 mriedem we also have commands for performing those data migration in batches
20:26:56 mriedem nova-manage db online_data_migrations
20:28:34 imacdonn hmmm ... that's probably documented somewhere, but I haven't run across it yet ... is there a good description of the right sequence of events for upgrades somewhere?
20:28:49 mriedem have you read through https://docs.openstack.org/nova/latest/reference/upgrade-checks.html ?
20:29:44 imacdonn don't think I'd seen that one ... will study it
20:30:16 mriedem so having said that, this is how nova does upgrades,
20:30:17 imacdonn but back to the checks ... I think it may be a good idea, especially if you're trying to make this a generic framework, so have places for pre- and post-upgrade checking
20:30:22 mriedem which isn't going to be how everyone does it
20:30:37 mriedem well, pre and post upgrade are pretty fuzzy,
20:30:54 mriedem the general rule i try to follow is how idempotent can i write the check such that anyone can run it at anytime
20:31:17 mriedem be that before the db sync with new code, after that but before new code starts running, after new code is running, etc
20:31:26 mriedem i know that doesn't help you much here...
20:31:54 mriedem it makes more sense when you have something you actually need to write a check for,
20:32:21 mriedem e.g. if your release notes say, "make sure you do x before upgrading to stein" then that's a pretty obvious thing to check (if you can)
20:32:38 imacdonn I can sortof see that ... but it seems that what constitutes "OK to start the upgrade" vs. "the upgrade did everything it was supposed to, and you can start services now" may be different
20:38:03 imacdonn Looking at the output of 'nova-status upgrade check', I see one of the items is "Request Spec Migration"
20:38:47 imacdonn haven't looked to see exactly what that is, but if it's a migration, and migrations don't happen during sync, and you're supposed to do the upgrade check before running the new code ..............
20:39:50 mriedem so the story on that one is,
20:39:59 mriedem nova added this thing called a request spec in newton,
20:40:13 mriedem and has compatibility code all over the place for instances created before newton that don't have a related request spec record in the db,
20:40:27 mriedem we have an online data migration that operators can run to create records for old instances,
20:40:41 mriedem the upgrade check was added because i want to drop that compatibility code in stein,
20:41:16 mriedem which means, before you get to stein and restart services with stein code (which could have dropped that compatibility), you can run that check and see, oh i better finish my homework - it's been due for 2 years
20:42:54 imacdonn so ... if we're running this check after doing the db sync with the stein code, but before we can start up the services with the new code, can we still do the online migration ?
20:43:07 mriedem so the way upgrades work in grenade for nova is: lay down new code, sync the db schema for the new models, run the online data migrations, restart the services
20:43:19 mriedem *with nova-status upgrade check in between those last 2
20:43:31 mriedem sure you can
20:43:56 mriedem the online data migrations CLI in nova just needs access to the config and db
20:44:02 mriedem none of that goes over rpc or rest api
20:44:12 mriedem otherwise you can't FFU
20:45:35 imacdonn so "online" means that it's OK to do online, not that it has to be done online? or am I completely misunderstanding what "online" means ?
20:46:58 mriedem the 'online_data_migrations' name might be a bad name
20:47:08 mriedem it just means,
20:47:16 mriedem not offline like during a db sync when the controller is down
20:47:24 mriedem like how data migrations were done during db sync in the long ago
20:47:34 mriedem which meant operators could have substantial downtime in the control plane
20:48:39 imacdonn ok
20:49:28 mriedem which project are you looking to do the upgrade checks for btw?
20:49:40 imacdonn so I need to digest the whole upgrade flow a bit more (maybe after I digest lunch) .... but I still think it's worth at least considering pre/post-upgrade checks in general
20:50:21 imacdonn I'm trying to be a good citizen and 'do the right thing' for all projects .. or all of the ones that I uses, which is mostly the core ones
20:51:40 imacdonn based on your ML posts a while back, I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services
20:52:15 imacdonn maybe mostly because currently placement is considered a nova service, at least in my deployment mechanism
20:53:03 imacdonn I may try hacking in sometihng to start placement individually before doing the upgrade check, then start the rest of the (nova) services
20:53:47 imacdonn or maybe I'll worry about that for stein
20:54:47 mriedem "I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services" - as i said, that's also acceptable in the case of nova and it's even in our install docs: https://docs.openstack.org/nova/latest/install/verify.html
20:54:55 mriedem for a fresh install i mean
20:55:29 mriedem flexibility and idempotency are kind of the key things here
20:55:54 imacdonn Well, the description does say "Performs a release-specific readiness check before restarting services with new code"
20:56:01 imacdonn https://docs.openstack.org/nova/latest/cli/nova-status.html
20:56:39 imacdonn but yeah ... I was basically using it as a validation that "everything went OK with the nova upgrade"
20:56:40 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423
20:58:14 imacdonn re fresh install vs upgrade, I basically perform the same steps in both cases (that idempotence thing)
21:01:54 mriedem yeah some of the wording is a bit dated and was written when the initial change in ocata was really for before the new code was started up
21:02:12 mriedem but that's still how we do it in grenade
21:03:36 imacdonn wondering if there might be a case where you really do want to check for something before starting the new code, because otherwise the new code might try to do something that would turn out to be catastrophic
21:04:15 mriedem well, yes, you want to run it after starting new code
21:05:17 mriedem guh, sorry, typo
21:05:21 mriedem before starting new code
21:05:22 mriedem that's the idea
21:05:33 mriedem put it in a venv or container and run it
21:47:53 openstackgerrit Matt Riedemann proposed openstack/nova master: Add some more docs for upgrade checkers https://review.openstack.org/604486
22:26:28 imacdonn mriedem_away: On a slightly related note; is there a reason that this do like .where(compute_nodes.c.deleted==0) ? https://github.com/openstack/nova/blob/master/nova/cmd/status.py#L111-L130
22:29:43 imacdonn erm, I mean; ... that this doesn't* do like .....
22:32:41 openstack Launchpad bug 1757207 in OpenStack Compute (nova) "compute resource providers not equal to compute nodes in deployment" [Undecided,Incomplete]
22:32:41 imacdonn see also https://bugs.launchpad.net/nova/+bug/1757207
#openstack-nova - 2018-09-22
00:10:24 karimull Hi mriedem : a qk question , can we add plugins in nova-compute , which can be called before libvirt is executed for special processing during instance launch?
00:59:18 openstackgerrit iain MacDonnell proposed openstack/nova master: nova-status - don't count deleted compute_nodes https://review.openstack.org/604495
01:19:05 openstackgerrit Merged openstack/nova stable/rocky: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604367
04:30:36 openstackgerrit Merged openstack/nova master: remove virt driver requires_allocation_refresh https://review.openstack.org/602481
04:30:43 openstackgerrit Merged openstack/nova master: Nix update_instance_allocation, _allocate_for_instance https://review.openstack.org/604186
06:50:21 openstackgerrit OpenStack Proposal Bot proposed openstack/nova stable/rocky: Imported Translations from Zanata https://review.openstack.org/604260
11:29:17 openstackgerrit Brin Zhang proposed openstack/nova master: Specifies the storage backend to boot instance https://review.openstack.org/579360
16:06:25 mriedem imacdonn: replied
16:07:03 mriedem karimull: there are "hooks" so you could write something out of tree to run before/after the driver spawn in the compute service, however, hooks are deprecated and your out of tree code could be broken at any time
16:07:46 mriedem https://github.com/openstack/nova/blob/master/nova/hooks.py
16:24:52 imacdonn mriedem: thanks!
16:30:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Ignore VirtDriverNotReady in _sync_power_states periodic task https://review.openstack.org/604376

Earlier   Later