| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 17:26:06 | andymccr | mriedem: melwitt: sure but the issue is the process is then diff for greenfield vs upgrade, which means i need to have 2 code paths to manage 1 thing | |
| 17:26:13 | mriedem | if it's a completely fresh install, the api db schema migration checks for that by seeing if you have created flavors yet | |
| 17:26:16 | andymccr | e.g. in greenfield i do api_db sync --> cell_create --> the other things | |
| 17:26:27 | andymccr | and in upgrade im going to have to do cell_create --> api_db sync --> the other things | |
| 17:26:27 | owalsh | andymccr: yes, we had to re-order the db sync for upgrades in tripleo | |
| 17:26:38 | andymccr | owalsh: so do you do different ordering for upgrade vs greenfield? | |
| 17:27:11 | digvijay2016 | how to restrict the nova service from changing hostname of created instance ? | |
| 17:27:58 | melwitt | I see. I think maybe you could have the same process for both if you have potentially an extra sync in there. I'd have to think about it more | |
| 17:28:05 | mriedem | andymccr: to be clear, on the newton side the api db is already all synced up yes? | |
| 17:28:08 | mriedem | before upgrading to ocata | |
| 17:28:09 | melwitt | mriedem: does it hurt to do a sync twice? I guess it could | |
| 17:28:14 | andymccr | mriedem: yes | |
| 17:28:15 | mriedem | melwitt: no | |
| 17:28:21 | mriedem | melwitt: db sync is idempotent | |
| 17:28:31 | mriedem | if you're already at the latest version, it's a noop | |
| 17:28:41 | mriedem | there is a migrations table created in the db that tracks the version you're synced to | |
| 17:28:51 | mriedem | i.e. you can do something like nova-manage db sync --version | |
| 17:29:07 | melwitt | okay, so maybe it could be api_db sync --> cell_create --> api_db sync --> other things for both? | |
| 17:29:14 | andymccr | is there anyway we could get the rc for nova-manage api db_sync to be something that isn't 1 if it fails because the cell doesnt exist? | |
| 17:29:34 | mriedem | melwitt: that's how it works for grenade https://github.com/openstack-dev/grenade/blob/stable/ocata/projects/60_nova/from-newton/upgrade-nova | |
| 17:29:44 | owalsh | melwitt: no, can't run api_db sync for upgrades until the end | |
| 17:29:59 | andymccr | ^ yeah the issue is in upgrade you cant run the api_db sync unless the cell exists | |
| 17:30:04 | mriedem | newton is all done, then in grenade we create the cell0 db, map_cell0, run simple_cell_setup to create the main cell and host mappings | |
| 17:30:08 | andymccr | but in greenfield you cant run the cell unless api_db has been run | |
| 17:30:29 | owalsh | andymccr: this is the tripleo upgrade (ansible) tasks https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L225 | |
| 17:30:43 | melwitt | I guess I don't understand why you can't run it twice | |
| 17:30:49 | mriedem | andymccr: no, the whole point of the blocker migration is to make sure you've setup cells before starting ocata services | |
| 17:31:07 | mriedem | melwitt: yeah i also don't see how this is working in grenade but not for anything else | |
| 17:32:08 | andymccr | mriedem: so is there a status command we can check that could be a decider around "hey you haven't run api_db sync yet so you need to before cell create" | |
| 17:32:52 | andymccr | mriedem: i imagine its working in grenade because you arent using the same path for upgrade vs greenfield - to me it seems silly to have to carry a separate "upgrade" play and it also means we have to determine whether you are upgrading or not - which is tricky. ideally it will just get you to the correct state | |
| 17:32:55 | mriedem | andymccr: the nova-status upgrade check command will tell you what's missing yeah | |
| 17:34:58 | andymccr | mriedem: ok will try that out - im guessing its meant to be human readable and wont give an rc that i can automate on? | |
| 17:36:43 | mriedem | andymccr: nova-manage db sync? it's a pass/fail, we don't have logic built in based on how specific migration scripts behave | |
| 17:37:04 | mriedem | and the entire point of that blocker migration is that you can't proceed w/o doing some other stuff first | |
| 17:38:19 | andymccr | mriedem: i get that - but the problem here is that api_db sync is not idempotent? i can run it fine on a brand new database with no cells, and it works fine - but if i run it again straight after it fails because the cell doesnt exist | |
| 17:39:22 | andymccr | end goal is that i want to (at least) be able to determine in an automated fashion where i need to run the cell create then the api_db sync or vice versa. | |
| 17:40:39 | cfriesen | separate question about the scheduler. why does live-migration add nodes to request_spec.ignore_hosts on RescheduledException but cold migration doesn't? | |
| 17:41:40 | openstackgerrit | MarkMielke proposed openstack/nova master: DiskFilter handling for free space less than zero https://review.openstack.org/430091 | |
| 17:44:00 | owalsh | andymccr: we couldn't do that with puppet for the tripleo undercloud, resorted to using an upgrade flag | |
| 17:44:50 | andymccr | owalsh: yeah i want to avoid that if possible :) | |
| 17:47:17 | melwitt | andymccr: from what I can tell in the code, the return code for the 'cell doesn't exist' should be 2. but you said it was 1 when you did it? | |
| 17:48:31 | andymccr | melwitt: let me do another test - if that's the case then that would be perfect | |
| 17:50:12 | mriedem | melwitt: he's talking about nova-manage db sync right? | |
| 17:50:17 | mriedem | that's either pass or fail i believe | |
| 17:50:33 | melwitt | the return code from the status check is 2, does that not pass through the db sync? | |
| 17:50:39 | owalsh | mriedem: api db sync I expect | |
| 17:50:45 | mriedem | andymccr: you can tell if it's a fresh install by telling if there are flavors in the api db can't you? that's what the api schema migration does to tell if it's a fresh install or an upgrade | |
| 17:50:57 | mriedem | melwitt: nova-status is not the same as nova-manage | |
| 17:51:02 | mriedem | the status check is a separate thing | |
| 17:51:04 | mriedem | pre-upgrade | |
| 17:51:22 | melwitt | mriedem: nova-manage doesn't call nova-status functions to do the blocker? | |
| 17:51:45 | mriedem | no | |
| 17:52:00 | melwitt | sigh ... okay. nevermind andymccr | |
| 17:52:13 | andymccr | mriedem: do you mean by doing a db query to check if it exists? | |
| 17:52:26 | mriedem | andymccr: if it's an upgrade isn't nova-api already running? | |
| 17:52:48 | mriedem | this is the api db migration thats hitting you https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L28 | |
| 17:52:57 | andymccr | mriedem: ok so check if the service is running, if so then do the cell create before the api_db? | |
| 17:53:03 | mriedem | that migration checks the api db for flavors, if there are none it assumes fresh install and exists | |
| 17:53:25 | mriedem | *exits | |
| 17:53:42 | mriedem | i don't know how the osa stuff works to tell if you're upgrading or not | |
| 17:54:45 | owalsh | I doubt using service status would be idempotent | |
| 17:54:46 | melwitt | they just need to be able to differentiate between a regular db sync fail and a validation db sync fail. I wonder if we could do something like return 2 if we catch ValidationError | |
| 17:56:07 | andymccr | mriedem: ideally we dont tell the difference between upgrade or not - it should run through in an order that would work for both, or do some logic to ensure an appropriate state is configured | |
| 17:56:22 | andymccr | melwitt: that sounds like it would work | |
| 17:56:36 | owalsh | melwitt: another nova-status command might be cleaner | |
| 17:56:55 | andymccr | yeah if a nova-status command could tell us before hand without failing, that would work fine too | |
| 17:57:57 | melwitt | owalsh: how does the current nova-status command not work for you? does it return a fail on an empty api db, or? | |
| 18:01:24 | owalsh | melwitt: no idea... garbage collected my brain since then | |
| 18:05:42 | owalsh | melwitt: possibly because the placement api status checks could give false negatives | |
| 18:07:20 | owalsh | melwitt: I think we would at least need the ability to run individual checks with the current command | |
| 18:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 18:13:25 | jbernard | do we support booting from an encrypted volume? | |
| 18:13:36 | jbernard | docs does not say explicitly, which makes me wonder | |
| 18:13:36 | mriedem | jbernard: should be able to | |
| 18:13:45 | kevinbenton | sdague, johnthetubaguy: want to send https://review.openstack.org/#/c/439560/ in? | |
| 18:14:03 | jbernard | mriedem: cool thanks, we have also encryptors in os-brick, | |
| 18:14:07 | kevinbenton | that last statement i added will allow us to see right what's in the http response | |
| 18:14:21 | jbernard | mriedem: do we plan to make use of those at some point, or is that for something else? | |
| 18:16:23 | mriedem | jbernard: i thought lyarwood added the encryptors to os-brick | |
| 18:16:27 | mriedem | i'm not really familiar | |
| 18:16:42 | mriedem | i thought the idea was to put them in os-brick so that nova/cinder were talking the same language | |
| 18:16:52 | mriedem | because i believe the encryptor class in the volume type in cinder is just a string today | |
| 18:16:58 | mriedem | so you can throw whatever in there and it's not validated | |
| 18:17:04 | mriedem | until it doesn't work on the nova side | |
| 18:17:05 | jbernard | ok, that was my impression as well | |
| 18:17:39 | mriedem | kevinbenton: which job would be running with that on so we can see the results? | |
| 18:17:50 | smcginnis | mriedem: +1 | |
| 18:17:55 | mriedem | i forget which ones run the metadata api | |
| 18:18:03 | kevinbenton | mriedem: it seems quite a few more are now | |
| 18:18:22 | kevinbenton | mriedem: even gate-tempest-dsvm-neutron-full-ubuntu-xenial | |
| 18:19:06 | mriedem | yeah i suppose that makes sense, it's just not a separate service | |
| 18:19:10 | mriedem | like in the old pg job | |
| 18:19:15 | mriedem | it's running under nova-api | |
| 18:23:03 | mriedem | kevinbenton: comments inline | |
| 18:25:31 | mriedem | melwitt: andymccr: so on the idea of returning a different error code from nova-manage api_db sync, if we return 2 for a ValidationError or something, what would you do? basically always sync and ignore any rc=2 result, then try to map_cell0 and then sync again and if that fails you fail for real? that still seems shitty because at some point we could be raising validation error for something else you're not handling, and ignoring the va | |
| 18:26:48 | andymccr | mriedem: yeah the idea would be to know what the rc means - e.g. rc 2 == the cell db needs to be mapped, run that then sync later. but you're right if there are other cases that is problematic | |
| 18:28:28 | melwitt | mriedem: yeah, I guess it's still messy. I was thinking in general it might be nice to know if a sync failed because of a validation failure vs something else | |
| 18:29:10 | openstackgerrit | Kevin Benton proposed openstack/nova master: Add some metadata logging to root cause ssh failure https://review.openstack.org/439560 | |
| 18:29:18 | kevinbenton | mriedem, johnthetubaguy: ^^ | |
| 18:29:29 | mriedem | andymccr: have you hit issues like this in any other project? or do other projects just fail at runtime? | |