| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-21 | |||
| 16:35:58 | mdbooth | As in copied almost verbatim from another well-tested script | |
| 16:36:07 | mriedem | 0 | |
| 16:36:10 | mdbooth | k | |
| 16:36:15 | mriedem | i wrote it while at the ptg last week | |
| 16:36:27 | mriedem | in between bouts of yelling "argh" at people | |
| 16:36:40 | mriedem | you were my muse you know | |
| 16:36:46 | mdbooth | You must have typed quickly in those gaps | |
| 16:36:50 | mriedem | heh | |
| 16:37:09 | mriedem | btw, assuming these evacuate test patches pass, we should run your fix on top of them, | |
| 16:37:16 | mriedem | could probably do that without rebasing | |
| 16:37:24 | mriedem | if you put a change on top of your fix that depends-on my stack | |
| 16:37:28 | mdbooth | mriedem: ack. I'll add a Depends-on to it now | |
| 16:37:37 | mriedem | don't put the depends-on your fix directly, | |
| 16:37:42 | mriedem | put it on a DNM patch on top of yours | |
| 16:37:50 | mdbooth | Oh, ok. Got it. | |
| 16:38:45 | mdbooth | I'll have to leave the detailed review for Monday now, though. I need to be somewhere in 50 minutes, and I need to both get home, eat, and travel there first. | |
| 16:38:53 | mriedem | yup, sure | |
| 16:38:54 | mriedem | ttyl | |
| 16:39:42 | mriedem | SteelyDan: stacking up a new devstack atm to test your placement db copy script thing, should hopefully be able to start writing grenade changes this afternoon | |
| 16:40:20 | SteelyDan | mriedem: ack.. did the projects thing for the grenade job get resolved? | |
| 16:40:25 | SteelyDan | I haven't really been paying attention | |
| 16:40:32 | mriedem | don't know | |
| 16:55:17 | openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | |
| 16:55:17 | openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | |
| 16:55:17 | openstackgerrit | Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733 | |
| 16:55:51 | openstackgerrit | Matthew Booth proposed openstack/nova master: DNM: Test against mdbooth's evacuate patch https://review.openstack.org/604423 | |
| 17:03:20 | kashyap | mriedem: Shall I bump the MIN_VERSION_* to the advertized NEXT_MIN_* here: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L230 | |
| 17:03:35 | kashyap | (Will get to it on Monday; but just wanted to double-check here.) | |
| 17:06:17 | openstackgerrit | Merged openstack/nova stable/queens: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604134 | |
| 17:17:10 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | |
| 17:17:44 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | |
| 17:21:46 | openstackgerrit | Merged openstack/nova stable/pike: Add tempest-slow job to run the tempest slow tests https://review.openstack.org/604138 | |
| 17:21:53 | openstackgerrit | Merged openstack/nova master: Remove deprecated nova-consoleauth reference from doc https://review.openstack.org/604277 | |
| 17:24:47 | openstackgerrit | Merged openstack/nova master: ironic: stop hammering ironic API in power sync loop https://review.openstack.org/602127 | |
| 17:25:16 | kashyap | mriedem: When you're around, I should we should update it. Since we advertized them as the NEXT_MIN. | |
| 17:26:00 | kashyap | (That means, some more tedious, but necessary work, about selecting the NEXT_MIN_* for 'T' release.) | |
| 17:31:37 | openstackgerrit | sean mooney proposed openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 17:59:36 | openstackgerrit | Merged openstack/os-vif master: Fix upper-constraints link in tox file https://review.openstack.org/604158 | |
| 18:47:10 | openstackgerrit | Mohammed Naser proposed openstack/nova stable/queens: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604448 | |
| 18:48:02 | openstackgerrit | Mohammed Naser proposed openstack/nova stable/pike: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604449 | |
| 18:52:30 | openstackgerrit | Mohammed Naser proposed openstack/nova stable/ocata: Filter deleted computes from get_all_by_uuids() https://review.openstack.org/604451 | |
| 19:37:27 | cfriesen | is there an API equivalent to "nova-manage cell_v2 discover_hosts"? | |
| 19:38:52 | mriedem | cfriesen: no | |
| 19:39:03 | mriedem | there is a config option to run it periodically in the scheduler | |
| 20:00:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Run evacuate tests with local/lvm and shared/rbd storage https://review.openstack.org/604400 | |
| 20:00:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add volume-backed evacuate test https://review.openstack.org/604397 | |
| 20:18:06 | imacdonn | mriedem: can I ask a "dumb" question or two about upgrade checks? | |
| 20:18:24 | mriedem | yes | |
| 20:19:11 | imacdonn | so ... my understanding of "nova-status upgrade check" is that it's something that would be run *after* upgrading (db sync, etc), to check that everything is OK to proceed | |
| 20:19:25 | imacdonn | now it sounds like you want to do checks for things that should be done BEFORE the upgrade in there | |
| 20:19:43 | imacdonn | which is it? | |
| 20:22:10 | mriedem | it's meant to be run before starting new code | |
| 20:22:24 | mriedem | but that's not really a hard and fast rule | |
| 20:22:36 | imacdonn | but ... before or after upgrading the database ? | |
| 20:22:37 | mriedem | i.e. you can also run the nova-status upgrade check as a post-install step for a fresh install | |
| 20:22:44 | imacdonn | (databases) | |
| 20:22:49 | mriedem | after | |
| 20:23:09 | imacdonn | OK, so then ... it seems like it's not the right place for pre-upgrade checks | |
| 20:23:27 | imacdonn | there could be things you'd want to check before you try to upgrade the databases | |
| 20:23:48 | mriedem | well, depends on what your db upgrades are doing | |
| 20:23:58 | mriedem | are they doing data migrations during db sync? | |
| 20:24:01 | mriedem | b/c they really shouldn't | |
| 20:24:53 | imacdonn | hmm, I guess I don't know enough about the db internals ... I kinda thought that that's what db sync does | |
| 20:25:27 | imacdonn | if not where, where should the migrations happen? on first startup after upgrade ? | |
| 20:25:28 | mriedem | nova's db sync routines which lay down schema are primarily only for additive schema changes, like adding tables, columns and indexes/constraints | |
| 20:25:39 | mriedem | not for things that involve data migrations | |
| 20:25:49 | mriedem | during runtime | |
| 20:26:08 | mriedem | e.g. if you need to change the format for some value stored in the db, check on read and update on write | |
| 20:26:37 | mriedem | nova has moved several things from the 'cell' db to the api db, and during that process our routine is to read from the api db first, if not there, read from the cell db, and then migrate | |
| 20:26:51 | mriedem | we also have commands for performing those data migration in batches | |
| 20:26:56 | mriedem | nova-manage db online_data_migrations | |
| 20:28:34 | imacdonn | hmmm ... that's probably documented somewhere, but I haven't run across it yet ... is there a good description of the right sequence of events for upgrades somewhere? | |
| 20:28:49 | mriedem | have you read through https://docs.openstack.org/nova/latest/reference/upgrade-checks.html ? | |
| 20:29:44 | imacdonn | don't think I'd seen that one ... will study it | |
| 20:30:16 | mriedem | so having said that, this is how nova does upgrades, | |
| 20:30:17 | imacdonn | but back to the checks ... I think it may be a good idea, especially if you're trying to make this a generic framework, so have places for pre- and post-upgrade checking | |
| 20:30:22 | mriedem | which isn't going to be how everyone does it | |
| 20:30:37 | mriedem | well, pre and post upgrade are pretty fuzzy, | |
| 20:30:54 | mriedem | the general rule i try to follow is how idempotent can i write the check such that anyone can run it at anytime | |
| 20:31:17 | mriedem | be that before the db sync with new code, after that but before new code starts running, after new code is running, etc | |
| 20:31:26 | mriedem | i know that doesn't help you much here... | |
| 20:31:54 | mriedem | it makes more sense when you have something you actually need to write a check for, | |
| 20:32:21 | mriedem | e.g. if your release notes say, "make sure you do x before upgrading to stein" then that's a pretty obvious thing to check (if you can) | |
| 20:32:38 | imacdonn | I can sortof see that ... but it seems that what constitutes "OK to start the upgrade" vs. "the upgrade did everything it was supposed to, and you can start services now" may be different | |
| 20:38:03 | imacdonn | Looking at the output of 'nova-status upgrade check', I see one of the items is "Request Spec Migration" | |
| 20:38:47 | imacdonn | haven't looked to see exactly what that is, but if it's a migration, and migrations don't happen during sync, and you're supposed to do the upgrade check before running the new code .............. | |
| 20:39:50 | mriedem | so the story on that one is, | |
| 20:39:59 | mriedem | nova added this thing called a request spec in newton, | |
| 20:40:13 | mriedem | and has compatibility code all over the place for instances created before newton that don't have a related request spec record in the db, | |
| 20:40:27 | mriedem | we have an online data migration that operators can run to create records for old instances, | |
| 20:40:41 | mriedem | the upgrade check was added because i want to drop that compatibility code in stein, | |
| 20:41:16 | mriedem | which means, before you get to stein and restart services with stein code (which could have dropped that compatibility), you can run that check and see, oh i better finish my homework - it's been due for 2 years | |
| 20:42:54 | imacdonn | so ... if we're running this check after doing the db sync with the stein code, but before we can start up the services with the new code, can we still do the online migration ? | |
| 20:43:07 | mriedem | so the way upgrades work in grenade for nova is: lay down new code, sync the db schema for the new models, run the online data migrations, restart the services | |
| 20:43:19 | mriedem | *with nova-status upgrade check in between those last 2 | |
| 20:43:31 | mriedem | sure you can | |
| 20:43:56 | mriedem | the online data migrations CLI in nova just needs access to the config and db | |
| 20:44:02 | mriedem | none of that goes over rpc or rest api | |
| 20:44:12 | mriedem | otherwise you can't FFU | |
| 20:45:35 | imacdonn | so "online" means that it's OK to do online, not that it has to be done online? or am I completely misunderstanding what "online" means ? | |