Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-21
16:55:17 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733
16:55:17 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
16:55:17 openstackgerrit Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846
16:55:51 openstackgerrit Matthew Booth proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423
17:03:20 kashyap mriedem: Shall I bump the MIN_VERSION_* to the advertized NEXT_MIN_* here: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L230
17:03:35 kashyap (Will get to it on Monday; but just wanted to double-check here.)
17:06:17 openstackgerrit Merged openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134
17:17:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
17:17:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993
17:21:46 openstackgerrit Merged openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138
17:21:53 openstackgerrit Merged openstack/nova master: Remove deprecated nova-consoleauth reference from doc https://review.openstack.org/604277
17:24:47 openstackgerrit Merged openstack/nova master: ironic: stop hammering ironic API in power sync loop https://review.openstack.org/602127
17:25:16 kashyap mriedem: When you're around, I should we should update it. Since we advertized them as the NEXT_MIN.
17:26:00 kashyap (That means, some more tedious, but necessary work, about selecting the NEXT_MIN_* for 'T' release.)
17:31:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999
17:59:36 openstackgerrit Merged openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158
18:47:10 openstackgerrit Mohammed Naser proposed openstack/nova stable/queens: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604448
18:48:02 openstackgerrit Mohammed Naser proposed openstack/nova stable/pike: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604449
18:52:30 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604451
19:37:27 cfriesen is there an API equivalent to "nova-manage cell_v2 discover_hosts"?
19:38:52 mriedem cfriesen: no
19:39:03 mriedem there is a config option to run it periodically in the scheduler
20:00:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397
20:00:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
20:18:06 imacdonn mriedem: can I ask a "dumb" question or two about upgrade checks?
20:18:24 mriedem yes
20:19:11 imacdonn so ... my understanding of "nova-status upgrade check" is that it's something that would be run *after* upgrading (db sync, etc), to check that everything is OK to proceed
20:19:25 imacdonn now it sounds like you want to do checks for things that should be done BEFORE the upgrade in there
20:19:43 imacdonn which is it?
20:22:10 mriedem it's meant to be run before starting new code
20:22:24 mriedem but that's not really a hard and fast rule
20:22:36 imacdonn but ... before or after upgrading the database ?
20:22:37 mriedem i.e. you can also run the nova-status upgrade check as a post-install step for a fresh install
20:22:44 imacdonn (databases)
20:22:49 mriedem after
20:23:09 imacdonn OK, so then ... it seems like it's not the right place for pre-upgrade checks
20:23:27 imacdonn there could be things you'd want to check before you try to upgrade the databases
20:23:48 mriedem well, depends on what your db upgrades are doing
20:23:58 mriedem are they doing data migrations during db sync?
20:24:01 mriedem b/c they really shouldn't
20:24:53 imacdonn hmm, I guess I don't know enough about the db internals ... I kinda thought that that's what db sync does
20:25:27 imacdonn if not where, where should the migrations happen? on first startup after upgrade ?
20:25:28 mriedem nova's db sync routines which lay down schema are primarily only for additive schema changes, like adding tables, columns and indexes/constraints
20:25:39 mriedem not for things that involve data migrations
20:25:49 mriedem during runtime
20:26:08 mriedem e.g. if you need to change the format for some value stored in the db, check on read and update on write
20:26:37 mriedem nova has moved several things from the 'cell' db to the api db, and during that process our routine is to read from the api db first, if not there, read from the cell db, and then migrate
20:26:51 mriedem we also have commands for performing those data migration in batches
20:26:56 mriedem nova-manage db online_data_migrations
20:28:34 imacdonn hmmm ... that's probably documented somewhere, but I haven't run across it yet ... is there a good description of the right sequence of events for upgrades somewhere?
20:28:49 mriedem have you read through https://docs.openstack.org/nova/latest/reference/upgrade-checks.html ?
20:29:44 imacdonn don't think I'd seen that one ... will study it
20:30:16 mriedem so having said that, this is how nova does upgrades,
20:30:17 imacdonn but back to the checks ... I think it may be a good idea, especially if you're trying to make this a generic framework, so have places for pre- and post-upgrade checking
20:30:22 mriedem which isn't going to be how everyone does it
20:30:37 mriedem well, pre and post upgrade are pretty fuzzy,
20:30:54 mriedem the general rule i try to follow is how idempotent can i write the check such that anyone can run it at anytime
20:31:17 mriedem be that before the db sync with new code, after that but before new code starts running, after new code is running, etc
20:31:26 mriedem i know that doesn't help you much here...
20:31:54 mriedem it makes more sense when you have something you actually need to write a check for,
20:32:21 mriedem e.g. if your release notes say, "make sure you do x before upgrading to stein" then that's a pretty obvious thing to check (if you can)
20:32:38 imacdonn I can sortof see that ... but it seems that what constitutes "OK to start the upgrade" vs. "the upgrade did everything it was supposed to, and you can start services now" may be different
20:38:03 imacdonn Looking at the output of 'nova-status upgrade check', I see one of the items is "Request Spec Migration"
20:38:47 imacdonn haven't looked to see exactly what that is, but if it's a migration, and migrations don't happen during sync, and you're supposed to do the upgrade check before running the new code ..............
20:39:50 mriedem so the story on that one is,
20:39:59 mriedem nova added this thing called a request spec in newton,
20:40:13 mriedem and has compatibility code all over the place for instances created before newton that don't have a related request spec record in the db,
20:40:27 mriedem we have an online data migration that operators can run to create records for old instances,
20:40:41 mriedem the upgrade check was added because i want to drop that compatibility code in stein,
20:41:16 mriedem which means, before you get to stein and restart services with stein code (which could have dropped that compatibility), you can run that check and see, oh i better finish my homework - it's been due for 2 years
20:42:54 imacdonn so ... if we're running this check after doing the db sync with the stein code, but before we can start up the services with the new code, can we still do the online migration ?
20:43:07 mriedem so the way upgrades work in grenade for nova is: lay down new code, sync the db schema for the new models, run the online data migrations, restart the services
20:43:19 mriedem *with nova-status upgrade check in between those last 2
20:43:31 mriedem sure you can
20:43:56 mriedem the online data migrations CLI in nova just needs access to the config and db
20:44:02 mriedem none of that goes over rpc or rest api
20:44:12 mriedem otherwise you can't FFU
20:45:35 imacdonn so "online" means that it's OK to do online, not that it has to be done online? or am I completely misunderstanding what "online" means ?
20:46:58 mriedem the 'online_data_migrations' name might be a bad name
20:47:08 mriedem it just means,
20:47:16 mriedem not offline like during a db sync when the controller is down
20:47:24 mriedem like how data migrations were done during db sync in the long ago
20:47:34 mriedem which meant operators could have substantial downtime in the control plane
20:48:39 imacdonn ok
20:49:28 mriedem which project are you looking to do the upgrade checks for btw?
20:49:40 imacdonn so I need to digest the whole upgrade flow a bit more (maybe after I digest lunch) .... but I still think it's worth at least considering pre/post-upgrade checks in general
20:50:21 imacdonn I'm trying to be a good citizen and 'do the right thing' for all projects .. or all of the ones that I uses, which is mostly the core ones
20:51:40 imacdonn based on your ML posts a while back, I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services
20:52:15 imacdonn maybe mostly because currently placement is considered a nova service, at least in my deployment mechanism
20:53:03 imacdonn I may try hacking in sometihng to start placement individually before doing the upgrade check, then start the rest of the (nova) services
20:53:47 imacdonn or maybe I'll worry about that for stein
20:54:47 mriedem "I added 'nova-status upgrade check', but it's not quite right, as I'm running it after starting the services" - as i said, that's also acceptable in the case of nova and it's even in our install docs: https://docs.openstack.org/nova/latest/install/verify.html
20:54:55 mriedem for a fresh install i mean
20:55:29 mriedem flexibility and idempotency are kind of the key things here
20:55:54 imacdonn Well, the description does say "Performs a release-specific readiness check before restarting services with new code"
20:56:01 imacdonn https://docs.openstack.org/nova/latest/cli/nova-status.html
20:56:39 imacdonn but yeah ... I was basically using it as a validation that "everything went OK with the nova upgrade"
20:56:40 openstackgerrit Matt Riedemann proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423
20:58:14 imacdonn re fresh install vs upgrade, I basically perform the same steps in both cases (that idempotence thing)
21:01:54 mriedem yeah some of the wording is a bit dated and was written when the initial change in ocata was really for before the new code was started up

Earlier   Later