Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-09-21
16:32:05 mdbooth It's quite a matrix :/
16:32:25 mdbooth mriedem: Add in with/without kernel/ramdisk
16:32:33 mriedem no one cares about those
16:32:50 mdbooth Hehe, they were a bug in my evacuate patch, though :)
16:34:05 mdbooth The fake imageservice was non-deterministically returning an image with kernel/ramdisk, which caused a failure in my functional test. The bug was real.
16:35:33 mdbooth mriedem: How much of test_evacuate.sh is boilerplate, btw?
16:35:43 mriedem as in virt agnostic?
16:35:58 mdbooth As in copied almost verbatim from another well-tested script
16:36:07 mriedem 0
16:36:10 mdbooth k
16:36:15 mriedem i wrote it while at the ptg last week
16:36:27 mriedem in between bouts of yelling "argh" at people
16:36:40 mriedem you were my muse you know
16:36:46 mdbooth You must have typed quickly in those gaps
16:36:50 mriedem heh
16:37:09 mriedem btw, assuming these evacuate test patches pass, we should run your fix on top of them,
16:37:16 mriedem could probably do that without rebasing
16:37:24 mriedem if you put a change on top of your fix that depends-on my stack
16:37:28 mdbooth mriedem: ack. I'll add a Depends-on to it now
16:37:37 mriedem don't put the depends-on your fix directly,
16:37:42 mriedem put it on a DNM patch on top of yours
16:37:50 mdbooth Oh, ok. Got it.
16:38:45 mdbooth I'll have to leave the detailed review for Monday now, though. I need to be somewhere in 50 minutes, and I need to both get home, eat, and travel there first.
16:38:53 mriedem yup, sure
16:38:54 mriedem ttyl
16:39:42 mriedem SteelyDan: stacking up a new devstack atm to test your placement db copy script thing, should hopefully be able to start writing grenade changes this afternoon
16:40:20 SteelyDan mriedem: ack.. did the projects thing for the grenade job get resolved?
16:40:25 SteelyDan I haven't really been paying attention
16:40:32 mriedem don't know
16:55:17 openstackgerrit Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733
16:55:17 openstack bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9)
16:55:17 openstackgerrit Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846
16:55:51 openstackgerrit Matthew Booth proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423
17:03:20 kashyap mriedem: Shall I bump the MIN_VERSION_* to the advertized NEXT_MIN_* here: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L230
17:03:35 kashyap (Will get to it on Monday; but just wanted to double-check here.)
17:06:17 openstackgerrit Merged openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134
17:17:10 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992
17:17:44 openstackgerrit Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993
17:21:46 openstackgerrit Merged openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138
17:21:53 openstackgerrit Merged openstack/nova master: Remove deprecated nova-consoleauth reference from doc https://review.openstack.org/604277
17:24:47 openstackgerrit Merged openstack/nova master: ironic: stop hammering ironic API in power sync loop https://review.openstack.org/602127
17:25:16 kashyap mriedem: When you're around, I should we should update it. Since we advertized them as the NEXT_MIN.
17:26:00 kashyap (That means, some more tedious, but necessary work, about selecting the NEXT_MIN_* for 'T' release.)
17:31:37 openstackgerrit sean mooney proposed openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999
17:59:36 openstackgerrit Merged openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158
18:47:10 openstackgerrit Mohammed Naser proposed openstack/nova stable/queens: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604448
18:48:02 openstackgerrit Mohammed Naser proposed openstack/nova stable/pike: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604449
18:52:30 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604451
19:37:27 cfriesen is there an API equivalent to "nova-manage cell_v2 discover_hosts"?
19:38:52 mriedem cfriesen: no
19:39:03 mriedem there is a config option to run it periodically in the scheduler
20:00:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397
20:00:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400
20:18:06 imacdonn mriedem: can I ask a "dumb" question or two about upgrade checks?
20:18:24 mriedem yes
20:19:11 imacdonn so ... my understanding of "nova-status upgrade check" is that it's something that would be run *after* upgrading (db sync, etc), to check that everything is OK to proceed
20:19:25 imacdonn now it sounds like you want to do checks for things that should be done BEFORE the upgrade in there
20:19:43 imacdonn which is it?
20:22:10 mriedem it's meant to be run before starting new code
20:22:24 mriedem but that's not really a hard and fast rule
20:22:36 imacdonn but ... before or after upgrading the database ?
20:22:37 mriedem i.e. you can also run the nova-status upgrade check as a post-install step for a fresh install
20:22:44 imacdonn (databases)
20:22:49 mriedem after
20:23:09 imacdonn OK, so then ... it seems like it's not the right place for pre-upgrade checks
20:23:27 imacdonn there could be things you'd want to check before you try to upgrade the databases
20:23:48 mriedem well, depends on what your db upgrades are doing
20:23:58 mriedem are they doing data migrations during db sync?
20:24:01 mriedem b/c they really shouldn't
20:24:53 imacdonn hmm, I guess I don't know enough about the db internals ... I kinda thought that that's what db sync does
20:25:27 imacdonn if not where, where should the migrations happen? on first startup after upgrade ?
20:25:28 mriedem nova's db sync routines which lay down schema are primarily only for additive schema changes, like adding tables, columns and indexes/constraints
20:25:39 mriedem not for things that involve data migrations
20:25:49 mriedem during runtime
20:26:08 mriedem e.g. if you need to change the format for some value stored in the db, check on read and update on write
20:26:37 mriedem nova has moved several things from the 'cell' db to the api db, and during that process our routine is to read from the api db first, if not there, read from the cell db, and then migrate
20:26:51 mriedem we also have commands for performing those data migration in batches
20:26:56 mriedem nova-manage db online_data_migrations
20:28:34 imacdonn hmmm ... that's probably documented somewhere, but I haven't run across it yet ... is there a good description of the right sequence of events for upgrades somewhere?
20:28:49 mriedem have you read through https://docs.openstack.org/nova/latest/reference/upgrade-checks.html ?
20:29:44 imacdonn don't think I'd seen that one ... will study it
20:30:16 mriedem so having said that, this is how nova does upgrades,
20:30:17 imacdonn but back to the checks ... I think it may be a good idea, especially if you're trying to make this a generic framework, so have places for pre- and post-upgrade checking
20:30:22 mriedem which isn't going to be how everyone does it
20:30:37 mriedem well, pre and post upgrade are pretty fuzzy,
20:30:54 mriedem the general rule i try to follow is how idempotent can i write the check such that anyone can run it at anytime
20:31:17 mriedem be that before the db sync with new code, after that but before new code starts running, after new code is running, etc
20:31:26 mriedem i know that doesn't help you much here...
20:31:54 mriedem it makes more sense when you have something you actually need to write a check for,
20:32:21 mriedem e.g. if your release notes say, "make sure you do x before upgrading to stein" then that's a pretty obvious thing to check (if you can)
20:32:38 imacdonn I can sortof see that ... but it seems that what constitutes "OK to start the upgrade" vs. "the upgrade did everything it was supposed to, and you can start services now" may be different
20:38:03 imacdonn Looking at the output of 'nova-status upgrade check', I see one of the items is "Request Spec Migration"
20:38:47 imacdonn haven't looked to see exactly what that is, but if it's a migration, and migrations don't happen during sync, and you're supposed to do the upgrade check before running the new code ..............
20:39:50 mriedem so the story on that one is,
20:39:59 mriedem nova added this thing called a request spec in newton,
20:40:13 mriedem and has compatibility code all over the place for instances created before newton that don't have a related request spec record in the db,
20:40:27 mriedem we have an online data migration that operators can run to create records for old instances,
20:40:41 mriedem the upgrade check was added because i want to drop that compatibility code in stein,
20:41:16 mriedem which means, before you get to stein and restart services with stein code (which could have dropped that compatibility), you can run that check and see, oh i better finish my homework - it's been due for 2 years
20:42:54 imacdonn so ... if we're running this check after doing the db sync with the stein code, but before we can start up the services with the new code, can we still do the online migration ?

Earlier   Later