Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-01
17:22:42 andymccr mriedem: OpenStack-Ansible
17:22:49 andymccr although im sure the tripleo ppl have already figured this one out :P
17:22:50 mriedem do you already have flavors created?
17:23:02 mriedem yes owalsh hit the same thing
17:23:05 mriedem in tripleo
17:23:14 mriedem and we relaxed something in the api db schema migration
17:23:17 andymccr mriedem: in testing, no - in practice yes
17:23:47 cfriesen bauzas: I don't think changing reset_forced_destinations() is sufficient by itself given how we handle ignore_hosts in ConductorManager.rebuild_instance().
17:24:20 mriedem andymccr: assuming you have this change? https://github.com/openstack/nova/commit/f781409f366904362b36bb8047fc371c17301c5e
17:24:52 melwitt I thought upgrade scenario meant not a greenfield?
17:25:26 mriedem yeah i'm confused
17:25:27 melwitt like, you're upgrading a deployment that already has a api db with a schema in it
17:25:36 mriedem if you've synced the api db in newton, you have the cell mappings table and can run map_cell0
17:25:44 mriedem to create an entry in the cell_mappings table in the nova_api db
17:26:06 andymccr mriedem: melwitt: sure but the issue is the process is then diff for greenfield vs upgrade, which means i need to have 2 code paths to manage 1 thing
17:26:13 mriedem if it's a completely fresh install, the api db schema migration checks for that by seeing if you have created flavors yet
17:26:16 andymccr e.g. in greenfield i do api_db sync --> cell_create --> the other things
17:26:27 owalsh andymccr: yes, we had to re-order the db sync for upgrades in tripleo
17:26:27 andymccr and in upgrade im going to have to do cell_create --> api_db sync --> the other things
17:26:38 andymccr owalsh: so do you do different ordering for upgrade vs greenfield?
17:27:11 digvijay2016 how to restrict the nova service from changing hostname of created instance ?
17:27:58 melwitt I see. I think maybe you could have the same process for both if you have potentially an extra sync in there. I'd have to think about it more
17:28:05 mriedem andymccr: to be clear, on the newton side the api db is already all synced up yes?
17:28:08 mriedem before upgrading to ocata
17:28:09 melwitt mriedem: does it hurt to do a sync twice? I guess it could
17:28:14 andymccr mriedem: yes
17:28:15 mriedem melwitt: no
17:28:21 mriedem melwitt: db sync is idempotent
17:28:31 mriedem if you're already at the latest version, it's a noop
17:28:41 mriedem there is a migrations table created in the db that tracks the version you're synced to
17:28:51 mriedem i.e. you can do something like nova-manage db sync --version
17:29:07 melwitt okay, so maybe it could be api_db sync --> cell_create --> api_db sync --> other things for both?
17:29:14 andymccr is there anyway we could get the rc for nova-manage api db_sync to be something that isn't 1 if it fails because the cell doesnt exist?
17:29:34 mriedem melwitt: that's how it works for grenade https://github.com/openstack-dev/grenade/blob/stable/ocata/projects/60_nova/from-newton/upgrade-nova
17:29:44 owalsh melwitt: no, can't run api_db sync for upgrades until the end
17:29:59 andymccr ^ yeah the issue is in upgrade you cant run the api_db sync unless the cell exists
17:30:04 mriedem newton is all done, then in grenade we create the cell0 db, map_cell0, run simple_cell_setup to create the main cell and host mappings
17:30:08 andymccr but in greenfield you cant run the cell unless api_db has been run
17:30:29 owalsh andymccr: this is the tripleo upgrade (ansible) tasks https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L225
17:30:43 melwitt I guess I don't understand why you can't run it twice
17:30:49 mriedem andymccr: no, the whole point of the blocker migration is to make sure you've setup cells before starting ocata services
17:31:07 mriedem melwitt: yeah i also don't see how this is working in grenade but not for anything else
17:32:08 andymccr mriedem: so is there a status command we can check that could be a decider around "hey you haven't run api_db sync yet so you need to before cell create"
17:32:52 andymccr mriedem: i imagine its working in grenade because you arent using the same path for upgrade vs greenfield - to me it seems silly to have to carry a separate "upgrade" play and it also means we have to determine whether you are upgrading or not - which is tricky. ideally it will just get you to the correct state
17:32:55 mriedem andymccr: the nova-status upgrade check command will tell you what's missing yeah
17:34:58 andymccr mriedem: ok will try that out - im guessing its meant to be human readable and wont give an rc that i can automate on?
17:36:43 mriedem andymccr: nova-manage db sync? it's a pass/fail, we don't have logic built in based on how specific migration scripts behave
17:37:04 mriedem and the entire point of that blocker migration is that you can't proceed w/o doing some other stuff first
17:38:19 andymccr mriedem: i get that - but the problem here is that api_db sync is not idempotent? i can run it fine on a brand new database with no cells, and it works fine - but if i run it again straight after it fails because the cell doesnt exist
17:39:22 andymccr end goal is that i want to (at least) be able to determine in an automated fashion where i need to run the cell create then the api_db sync or vice versa.
17:40:39 cfriesen separate question about the scheduler. why does live-migration add nodes to request_spec.ignore_hosts on RescheduledException but cold migration doesn't?
17:41:40 openstackgerrit MarkMielke proposed openstack/nova master: DiskFilter handling for free space less than zero https://review.openstack.org/430091
17:44:00 owalsh andymccr: we couldn't do that with puppet for the tripleo undercloud, resorted to using an upgrade flag
17:44:50 andymccr owalsh: yeah i want to avoid that if possible :)
17:47:17 melwitt andymccr: from what I can tell in the code, the return code for the 'cell doesn't exist' should be 2. but you said it was 1 when you did it?
17:48:31 andymccr melwitt: let me do another test - if that's the case then that would be perfect
17:50:12 mriedem melwitt: he's talking about nova-manage db sync right?
17:50:17 mriedem that's either pass or fail i believe
17:50:33 melwitt the return code from the status check is 2, does that not pass through the db sync?
17:50:39 owalsh mriedem: api db sync I expect
17:50:45 mriedem andymccr: you can tell if it's a fresh install by telling if there are flavors in the api db can't you? that's what the api schema migration does to tell if it's a fresh install or an upgrade
17:50:57 mriedem melwitt: nova-status is not the same as nova-manage
17:51:02 mriedem the status check is a separate thing
17:51:04 mriedem pre-upgrade
17:51:22 melwitt mriedem: nova-manage doesn't call nova-status functions to do the blocker?
17:51:45 mriedem no
17:52:00 melwitt sigh ... okay. nevermind andymccr
17:52:13 andymccr mriedem: do you mean by doing a db query to check if it exists?
17:52:26 mriedem andymccr: if it's an upgrade isn't nova-api already running?
17:52:48 mriedem this is the api db migration thats hitting you https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L28
17:52:57 andymccr mriedem: ok so check if the service is running, if so then do the cell create before the api_db?
17:53:03 mriedem that migration checks the api db for flavors, if there are none it assumes fresh install and exists
17:53:25 mriedem *exits
17:53:42 mriedem i don't know how the osa stuff works to tell if you're upgrading or not
17:54:45 owalsh I doubt using service status would be idempotent
17:54:46 melwitt they just need to be able to differentiate between a regular db sync fail and a validation db sync fail. I wonder if we could do something like return 2 if we catch ValidationError
17:56:07 andymccr mriedem: ideally we dont tell the difference between upgrade or not - it should run through in an order that would work for both, or do some logic to ensure an appropriate state is configured
17:56:22 andymccr melwitt: that sounds like it would work
17:56:36 owalsh melwitt: another nova-status command might be cleaner
17:56:55 andymccr yeah if a nova-status command could tell us before hand without failing, that would work fine too
17:57:57 melwitt owalsh: how does the current nova-status command not work for you? does it return a fail on an empty api db, or?
18:01:24 owalsh melwitt: no idea... garbage collected my brain since then
18:05:42 owalsh melwitt: possibly because the placement api status checks could give false negatives
18:07:20 owalsh melwitt: I think we would at least need the ability to run individual checks with the current command
18:09:53 openstackgerrit Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754
18:13:25 jbernard do we support booting from an encrypted volume?
18:13:36 mriedem jbernard: should be able to
18:13:36 jbernard docs does not say explicitly, which makes me wonder
18:13:45 kevinbenton sdague, johnthetubaguy: want to send https://review.openstack.org/#/c/439560/ in?
18:14:03 jbernard mriedem: cool thanks, we have also encryptors in os-brick,
18:14:07 kevinbenton that last statement i added will allow us to see right what's in the http response
18:14:21 jbernard mriedem: do we plan to make use of those at some point, or is that for something else?
18:16:23 mriedem jbernard: i thought lyarwood added the encryptors to os-brick
18:16:27 mriedem i'm not really familiar
18:16:42 mriedem i thought the idea was to put them in os-brick so that nova/cinder were talking the same language
18:16:52 mriedem because i believe the encryptor class in the volume type in cinder is just a string today
18:16:58 mriedem so you can throw whatever in there and it's not validated
18:17:04 mriedem until it doesn't work on the nova side
18:17:05 jbernard ok, that was my impression as well
18:17:39 mriedem kevinbenton: which job would be running with that on so we can see the results?

Earlier   Later