Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-01
17:01:33 bauzas cfriesen: yup
17:01:43 dansmith oops
17:02:01 bauzas cfriesen: But I need to dig into code
17:02:29 bauzas Because I remember a similar issue that I fixed
17:03:12 sneti johnthetubaguy, can you please review this when you have time: https://review.openstack.org/#/c/396420/
17:03:54 johnthetubaguy sneti: ah, yes, opened that, hopefully will get there soon
17:04:21 sneti thanks johnthetubaguy
17:10:22 mriedem sdague: i want to reuse your FakeResponse thing https://github.com/openstack/nova/blob/master/nova/tests/unit/test_identity.py#L25
17:10:27 mriedem but need to adjust the nonzero
17:10:36 mriedem to be True for status_code < 400
17:10:49 mriedem to match requests.Response https://github.com/kennethreitz/requests/blob/v2.12.5/requests/models.py#L658
17:10:53 mriedem sdague: you ok with that?
17:15:37 cfriesen bauzas: bug 1669054 has been opened to track the issue
17:15:37 openstack bug 1669054 in OpenStack Compute (nova) "RequestSpec.ignore_hosts from resize is reused in subsequent evacuate" [Undecided,New] https://launchpad.net/bugs/1669054
17:16:36 sdague mriedem: sure
17:18:18 andymccr what is the correct order for nova-manage (api_db/db sync & cell_v2 commands) in an upgrade? i'm running into an issue where greenfield works, but upgrades don't unless the order is slightly different
17:19:44 andymccr or rather - is there an order that works for both upgrades and greenfield? :)
17:20:24 mriedem andymccr: https://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2
17:20:32 mriedem and https://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases
17:21:32 andymccr mriedem: those docs only cover the cell setup bits - which is working fine afaict. the issue im running into is that in greenfield i have to have done an api_db sync before i can do the cell_v2 map_cell0 - otherwise it fails because certain tables are missing in the nova_api database
17:22:05 andymccr but in an upgrade scenario (from a non-cell_v2 --> cell_v2 deploy, or say N->O) if i run the api_db sync before the cell exists it will bomb saying that the cell needs to exist
17:22:10 mriedem yes you need the api db synced to get the cell mapping model
17:22:15 andymccr which means that the approach for upgrades doesn't work for greenfield
17:22:19 mriedem map_cell0 creates a cell_mappings record in the nova_api db
17:22:35 andymccr so then should api_db sync fail if cell0 doesn't exist yet?
17:22:37 mriedem andymccr: tripleo?
17:22:42 andymccr mriedem: OpenStack-Ansible
17:22:49 andymccr although im sure the tripleo ppl have already figured this one out :P
17:22:50 mriedem do you already have flavors created?
17:23:02 mriedem yes owalsh hit the same thing
17:23:05 mriedem in tripleo
17:23:14 mriedem and we relaxed something in the api db schema migration
17:23:17 andymccr mriedem: in testing, no - in practice yes
17:23:47 cfriesen bauzas: I don't think changing reset_forced_destinations() is sufficient by itself given how we handle ignore_hosts in ConductorManager.rebuild_instance().
17:24:20 mriedem andymccr: assuming you have this change? https://github.com/openstack/nova/commit/f781409f366904362b36bb8047fc371c17301c5e
17:24:52 melwitt I thought upgrade scenario meant not a greenfield?
17:25:26 mriedem yeah i'm confused
17:25:27 melwitt like, you're upgrading a deployment that already has a api db with a schema in it
17:25:36 mriedem if you've synced the api db in newton, you have the cell mappings table and can run map_cell0
17:25:44 mriedem to create an entry in the cell_mappings table in the nova_api db
17:26:06 andymccr mriedem: melwitt: sure but the issue is the process is then diff for greenfield vs upgrade, which means i need to have 2 code paths to manage 1 thing
17:26:13 mriedem if it's a completely fresh install, the api db schema migration checks for that by seeing if you have created flavors yet
17:26:16 andymccr e.g. in greenfield i do api_db sync --> cell_create --> the other things
17:26:27 andymccr and in upgrade im going to have to do cell_create --> api_db sync --> the other things
17:26:27 owalsh andymccr: yes, we had to re-order the db sync for upgrades in tripleo
17:26:38 andymccr owalsh: so do you do different ordering for upgrade vs greenfield?
17:27:11 digvijay2016 how to restrict the nova service from changing hostname of created instance ?
17:27:58 melwitt I see. I think maybe you could have the same process for both if you have potentially an extra sync in there. I'd have to think about it more
17:28:05 mriedem andymccr: to be clear, on the newton side the api db is already all synced up yes?
17:28:08 mriedem before upgrading to ocata
17:28:09 melwitt mriedem: does it hurt to do a sync twice? I guess it could
17:28:14 andymccr mriedem: yes
17:28:15 mriedem melwitt: no
17:28:21 mriedem melwitt: db sync is idempotent
17:28:31 mriedem if you're already at the latest version, it's a noop
17:28:41 mriedem there is a migrations table created in the db that tracks the version you're synced to
17:28:51 mriedem i.e. you can do something like nova-manage db sync --version
17:29:07 melwitt okay, so maybe it could be api_db sync --> cell_create --> api_db sync --> other things for both?
17:29:14 andymccr is there anyway we could get the rc for nova-manage api db_sync to be something that isn't 1 if it fails because the cell doesnt exist?
17:29:34 mriedem melwitt: that's how it works for grenade https://github.com/openstack-dev/grenade/blob/stable/ocata/projects/60_nova/from-newton/upgrade-nova
17:29:44 owalsh melwitt: no, can't run api_db sync for upgrades until the end
17:29:59 andymccr ^ yeah the issue is in upgrade you cant run the api_db sync unless the cell exists
17:30:04 mriedem newton is all done, then in grenade we create the cell0 db, map_cell0, run simple_cell_setup to create the main cell and host mappings
17:30:08 andymccr but in greenfield you cant run the cell unless api_db has been run
17:30:29 owalsh andymccr: this is the tripleo upgrade (ansible) tasks https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L225
17:30:43 melwitt I guess I don't understand why you can't run it twice
17:30:49 mriedem andymccr: no, the whole point of the blocker migration is to make sure you've setup cells before starting ocata services
17:31:07 mriedem melwitt: yeah i also don't see how this is working in grenade but not for anything else
17:32:08 andymccr mriedem: so is there a status command we can check that could be a decider around "hey you haven't run api_db sync yet so you need to before cell create"
17:32:52 andymccr mriedem: i imagine its working in grenade because you arent using the same path for upgrade vs greenfield - to me it seems silly to have to carry a separate "upgrade" play and it also means we have to determine whether you are upgrading or not - which is tricky. ideally it will just get you to the correct state
17:32:55 mriedem andymccr: the nova-status upgrade check command will tell you what's missing yeah
17:34:58 andymccr mriedem: ok will try that out - im guessing its meant to be human readable and wont give an rc that i can automate on?
17:36:43 mriedem andymccr: nova-manage db sync? it's a pass/fail, we don't have logic built in based on how specific migration scripts behave
17:37:04 mriedem and the entire point of that blocker migration is that you can't proceed w/o doing some other stuff first
17:38:19 andymccr mriedem: i get that - but the problem here is that api_db sync is not idempotent? i can run it fine on a brand new database with no cells, and it works fine - but if i run it again straight after it fails because the cell doesnt exist
17:39:22 andymccr end goal is that i want to (at least) be able to determine in an automated fashion where i need to run the cell create then the api_db sync or vice versa.
17:40:39 cfriesen separate question about the scheduler. why does live-migration add nodes to request_spec.ignore_hosts on RescheduledException but cold migration doesn't?
17:41:40 openstackgerrit MarkMielke proposed openstack/nova master: DiskFilter handling for free space less than zero https://review.openstack.org/430091
17:44:00 owalsh andymccr: we couldn't do that with puppet for the tripleo undercloud, resorted to using an upgrade flag
17:44:50 andymccr owalsh: yeah i want to avoid that if possible :)
17:47:17 melwitt andymccr: from what I can tell in the code, the return code for the 'cell doesn't exist' should be 2. but you said it was 1 when you did it?
17:48:31 andymccr melwitt: let me do another test - if that's the case then that would be perfect
17:50:12 mriedem melwitt: he's talking about nova-manage db sync right?
17:50:17 mriedem that's either pass or fail i believe
17:50:33 melwitt the return code from the status check is 2, does that not pass through the db sync?
17:50:39 owalsh mriedem: api db sync I expect
17:50:45 mriedem andymccr: you can tell if it's a fresh install by telling if there are flavors in the api db can't you? that's what the api schema migration does to tell if it's a fresh install or an upgrade
17:50:57 mriedem melwitt: nova-status is not the same as nova-manage
17:51:02 mriedem the status check is a separate thing
17:51:04 mriedem pre-upgrade
17:51:22 melwitt mriedem: nova-manage doesn't call nova-status functions to do the blocker?
17:51:45 mriedem no
17:52:00 melwitt sigh ... okay. nevermind andymccr
17:52:13 andymccr mriedem: do you mean by doing a db query to check if it exists?
17:52:26 mriedem andymccr: if it's an upgrade isn't nova-api already running?
17:52:48 mriedem this is the api db migration thats hitting you https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L28
17:52:57 andymccr mriedem: ok so check if the service is running, if so then do the cell create before the api_db?
17:53:03 mriedem that migration checks the api db for flavors, if there are none it assumes fresh install and exists
17:53:25 mriedem *exits
17:53:42 mriedem i don't know how the osa stuff works to tell if you're upgrading or not

Earlier   Later