| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 17:20:32 | mriedem | and https://docs.openstack.org/developer/nova/cells.html#step-by-step-for-common-use-cases | |
| 17:21:32 | andymccr | mriedem: those docs only cover the cell setup bits - which is working fine afaict. the issue im running into is that in greenfield i have to have done an api_db sync before i can do the cell_v2 map_cell0 - otherwise it fails because certain tables are missing in the nova_api database | |
| 17:22:05 | andymccr | but in an upgrade scenario (from a non-cell_v2 --> cell_v2 deploy, or say N->O) if i run the api_db sync before the cell exists it will bomb saying that the cell needs to exist | |
| 17:22:10 | mriedem | yes you need the api db synced to get the cell mapping model | |
| 17:22:15 | andymccr | which means that the approach for upgrades doesn't work for greenfield | |
| 17:22:19 | mriedem | map_cell0 creates a cell_mappings record in the nova_api db | |
| 17:22:35 | andymccr | so then should api_db sync fail if cell0 doesn't exist yet? | |
| 17:22:37 | mriedem | andymccr: tripleo? | |
| 17:22:42 | andymccr | mriedem: OpenStack-Ansible | |
| 17:22:49 | andymccr | although im sure the tripleo ppl have already figured this one out :P | |
| 17:22:50 | mriedem | do you already have flavors created? | |
| 17:23:02 | mriedem | yes owalsh hit the same thing | |
| 17:23:05 | mriedem | in tripleo | |
| 17:23:14 | mriedem | and we relaxed something in the api db schema migration | |
| 17:23:17 | andymccr | mriedem: in testing, no - in practice yes | |
| 17:23:47 | cfriesen | bauzas: I don't think changing reset_forced_destinations() is sufficient by itself given how we handle ignore_hosts in ConductorManager.rebuild_instance(). | |
| 17:24:20 | mriedem | andymccr: assuming you have this change? https://github.com/openstack/nova/commit/f781409f366904362b36bb8047fc371c17301c5e | |
| 17:24:52 | melwitt | I thought upgrade scenario meant not a greenfield? | |
| 17:25:26 | mriedem | yeah i'm confused | |
| 17:25:27 | melwitt | like, you're upgrading a deployment that already has a api db with a schema in it | |
| 17:25:36 | mriedem | if you've synced the api db in newton, you have the cell mappings table and can run map_cell0 | |
| 17:25:44 | mriedem | to create an entry in the cell_mappings table in the nova_api db | |
| 17:26:06 | andymccr | mriedem: melwitt: sure but the issue is the process is then diff for greenfield vs upgrade, which means i need to have 2 code paths to manage 1 thing | |
| 17:26:13 | mriedem | if it's a completely fresh install, the api db schema migration checks for that by seeing if you have created flavors yet | |
| 17:26:16 | andymccr | e.g. in greenfield i do api_db sync --> cell_create --> the other things | |
| 17:26:27 | owalsh | andymccr: yes, we had to re-order the db sync for upgrades in tripleo | |
| 17:26:27 | andymccr | and in upgrade im going to have to do cell_create --> api_db sync --> the other things | |
| 17:26:38 | andymccr | owalsh: so do you do different ordering for upgrade vs greenfield? | |
| 17:27:11 | digvijay2016 | how to restrict the nova service from changing hostname of created instance ? | |
| 17:27:58 | melwitt | I see. I think maybe you could have the same process for both if you have potentially an extra sync in there. I'd have to think about it more | |
| 17:28:05 | mriedem | andymccr: to be clear, on the newton side the api db is already all synced up yes? | |
| 17:28:08 | mriedem | before upgrading to ocata | |
| 17:28:09 | melwitt | mriedem: does it hurt to do a sync twice? I guess it could | |
| 17:28:14 | andymccr | mriedem: yes | |
| 17:28:15 | mriedem | melwitt: no | |
| 17:28:21 | mriedem | melwitt: db sync is idempotent | |
| 17:28:31 | mriedem | if you're already at the latest version, it's a noop | |
| 17:28:41 | mriedem | there is a migrations table created in the db that tracks the version you're synced to | |
| 17:28:51 | mriedem | i.e. you can do something like nova-manage db sync --version | |
| 17:29:07 | melwitt | okay, so maybe it could be api_db sync --> cell_create --> api_db sync --> other things for both? | |
| 17:29:14 | andymccr | is there anyway we could get the rc for nova-manage api db_sync to be something that isn't 1 if it fails because the cell doesnt exist? | |
| 17:29:34 | mriedem | melwitt: that's how it works for grenade https://github.com/openstack-dev/grenade/blob/stable/ocata/projects/60_nova/from-newton/upgrade-nova | |
| 17:29:44 | owalsh | melwitt: no, can't run api_db sync for upgrades until the end | |
| 17:29:59 | andymccr | ^ yeah the issue is in upgrade you cant run the api_db sync unless the cell exists | |
| 17:30:04 | mriedem | newton is all done, then in grenade we create the cell0 db, map_cell0, run simple_cell_setup to create the main cell and host mappings | |
| 17:30:08 | andymccr | but in greenfield you cant run the cell unless api_db has been run | |
| 17:30:29 | owalsh | andymccr: this is the tripleo upgrade (ansible) tasks https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L225 | |
| 17:30:43 | melwitt | I guess I don't understand why you can't run it twice | |
| 17:30:49 | mriedem | andymccr: no, the whole point of the blocker migration is to make sure you've setup cells before starting ocata services | |
| 17:31:07 | mriedem | melwitt: yeah i also don't see how this is working in grenade but not for anything else | |
| 17:32:08 | andymccr | mriedem: so is there a status command we can check that could be a decider around "hey you haven't run api_db sync yet so you need to before cell create" | |
| 17:32:52 | andymccr | mriedem: i imagine its working in grenade because you arent using the same path for upgrade vs greenfield - to me it seems silly to have to carry a separate "upgrade" play and it also means we have to determine whether you are upgrading or not - which is tricky. ideally it will just get you to the correct state | |
| 17:32:55 | mriedem | andymccr: the nova-status upgrade check command will tell you what's missing yeah | |
| 17:34:58 | andymccr | mriedem: ok will try that out - im guessing its meant to be human readable and wont give an rc that i can automate on? | |
| 17:36:43 | mriedem | andymccr: nova-manage db sync? it's a pass/fail, we don't have logic built in based on how specific migration scripts behave | |
| 17:37:04 | mriedem | and the entire point of that blocker migration is that you can't proceed w/o doing some other stuff first | |
| 17:38:19 | andymccr | mriedem: i get that - but the problem here is that api_db sync is not idempotent? i can run it fine on a brand new database with no cells, and it works fine - but if i run it again straight after it fails because the cell doesnt exist | |
| 17:39:22 | andymccr | end goal is that i want to (at least) be able to determine in an automated fashion where i need to run the cell create then the api_db sync or vice versa. | |
| 17:40:39 | cfriesen | separate question about the scheduler. why does live-migration add nodes to request_spec.ignore_hosts on RescheduledException but cold migration doesn't? | |
| 17:41:40 | openstackgerrit | MarkMielke proposed openstack/nova master: DiskFilter handling for free space less than zero https://review.openstack.org/430091 | |
| 17:44:00 | owalsh | andymccr: we couldn't do that with puppet for the tripleo undercloud, resorted to using an upgrade flag | |
| 17:44:50 | andymccr | owalsh: yeah i want to avoid that if possible :) | |
| 17:47:17 | melwitt | andymccr: from what I can tell in the code, the return code for the 'cell doesn't exist' should be 2. but you said it was 1 when you did it? | |
| 17:48:31 | andymccr | melwitt: let me do another test - if that's the case then that would be perfect | |
| 17:50:12 | mriedem | melwitt: he's talking about nova-manage db sync right? | |
| 17:50:17 | mriedem | that's either pass or fail i believe | |
| 17:50:33 | melwitt | the return code from the status check is 2, does that not pass through the db sync? | |
| 17:50:39 | owalsh | mriedem: api db sync I expect | |
| 17:50:45 | mriedem | andymccr: you can tell if it's a fresh install by telling if there are flavors in the api db can't you? that's what the api schema migration does to tell if it's a fresh install or an upgrade | |
| 17:50:57 | mriedem | melwitt: nova-status is not the same as nova-manage | |
| 17:51:02 | mriedem | the status check is a separate thing | |
| 17:51:04 | mriedem | pre-upgrade | |
| 17:51:22 | melwitt | mriedem: nova-manage doesn't call nova-status functions to do the blocker? | |
| 17:51:45 | mriedem | no | |
| 17:52:00 | melwitt | sigh ... okay. nevermind andymccr | |
| 17:52:13 | andymccr | mriedem: do you mean by doing a db query to check if it exists? | |
| 17:52:26 | mriedem | andymccr: if it's an upgrade isn't nova-api already running? | |
| 17:52:48 | mriedem | this is the api db migration thats hitting you https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L28 | |
| 17:52:57 | andymccr | mriedem: ok so check if the service is running, if so then do the cell create before the api_db? | |
| 17:53:03 | mriedem | that migration checks the api db for flavors, if there are none it assumes fresh install and exists | |
| 17:53:25 | mriedem | *exits | |
| 17:53:42 | mriedem | i don't know how the osa stuff works to tell if you're upgrading or not | |
| 17:54:45 | owalsh | I doubt using service status would be idempotent | |
| 17:54:46 | melwitt | they just need to be able to differentiate between a regular db sync fail and a validation db sync fail. I wonder if we could do something like return 2 if we catch ValidationError | |
| 17:56:07 | andymccr | mriedem: ideally we dont tell the difference between upgrade or not - it should run through in an order that would work for both, or do some logic to ensure an appropriate state is configured | |
| 17:56:22 | andymccr | melwitt: that sounds like it would work | |
| 17:56:36 | owalsh | melwitt: another nova-status command might be cleaner | |
| 17:56:55 | andymccr | yeah if a nova-status command could tell us before hand without failing, that would work fine too | |
| 17:57:57 | melwitt | owalsh: how does the current nova-status command not work for you? does it return a fail on an empty api db, or? | |
| 18:01:24 | owalsh | melwitt: no idea... garbage collected my brain since then | |
| 18:05:42 | owalsh | melwitt: possibly because the placement api status checks could give false negatives | |
| 18:07:20 | owalsh | melwitt: I think we would at least need the ability to run individual checks with the current command | |
| 18:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 18:13:25 | jbernard | do we support booting from an encrypted volume? | |
| 18:13:36 | mriedem | jbernard: should be able to | |
| 18:13:36 | jbernard | docs does not say explicitly, which makes me wonder | |
| 18:13:45 | kevinbenton | sdague, johnthetubaguy: want to send https://review.openstack.org/#/c/439560/ in? | |
| 18:14:03 | jbernard | mriedem: cool thanks, we have also encryptors in os-brick, | |
| 18:14:07 | kevinbenton | that last statement i added will allow us to see right what's in the http response | |
| 18:14:21 | jbernard | mriedem: do we plan to make use of those at some point, or is that for something else? | |