| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 17:38:19 | andymccr | mriedem: i get that - but the problem here is that api_db sync is not idempotent? i can run it fine on a brand new database with no cells, and it works fine - but if i run it again straight after it fails because the cell doesnt exist | |
| 17:39:22 | andymccr | end goal is that i want to (at least) be able to determine in an automated fashion where i need to run the cell create then the api_db sync or vice versa. | |
| 17:40:39 | cfriesen | separate question about the scheduler. why does live-migration add nodes to request_spec.ignore_hosts on RescheduledException but cold migration doesn't? | |
| 17:41:40 | openstackgerrit | MarkMielke proposed openstack/nova master: DiskFilter handling for free space less than zero https://review.openstack.org/430091 | |
| 17:44:00 | owalsh | andymccr: we couldn't do that with puppet for the tripleo undercloud, resorted to using an upgrade flag | |
| 17:44:50 | andymccr | owalsh: yeah i want to avoid that if possible :) | |
| 17:47:17 | melwitt | andymccr: from what I can tell in the code, the return code for the 'cell doesn't exist' should be 2. but you said it was 1 when you did it? | |
| 17:48:31 | andymccr | melwitt: let me do another test - if that's the case then that would be perfect | |
| 17:50:12 | mriedem | melwitt: he's talking about nova-manage db sync right? | |
| 17:50:17 | mriedem | that's either pass or fail i believe | |
| 17:50:33 | melwitt | the return code from the status check is 2, does that not pass through the db sync? | |
| 17:50:39 | owalsh | mriedem: api db sync I expect | |
| 17:50:45 | mriedem | andymccr: you can tell if it's a fresh install by telling if there are flavors in the api db can't you? that's what the api schema migration does to tell if it's a fresh install or an upgrade | |
| 17:50:57 | mriedem | melwitt: nova-status is not the same as nova-manage | |
| 17:51:02 | mriedem | the status check is a separate thing | |
| 17:51:04 | mriedem | pre-upgrade | |
| 17:51:22 | melwitt | mriedem: nova-manage doesn't call nova-status functions to do the blocker? | |
| 17:51:45 | mriedem | no | |
| 17:52:00 | melwitt | sigh ... okay. nevermind andymccr | |
| 17:52:13 | andymccr | mriedem: do you mean by doing a db query to check if it exists? | |
| 17:52:26 | mriedem | andymccr: if it's an upgrade isn't nova-api already running? | |
| 17:52:48 | mriedem | this is the api db migration thats hitting you https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/030_require_cell_setup.py#L28 | |
| 17:52:57 | andymccr | mriedem: ok so check if the service is running, if so then do the cell create before the api_db? | |
| 17:53:03 | mriedem | that migration checks the api db for flavors, if there are none it assumes fresh install and exists | |
| 17:53:25 | mriedem | *exits | |
| 17:53:42 | mriedem | i don't know how the osa stuff works to tell if you're upgrading or not | |
| 17:54:45 | owalsh | I doubt using service status would be idempotent | |
| 17:54:46 | melwitt | they just need to be able to differentiate between a regular db sync fail and a validation db sync fail. I wonder if we could do something like return 2 if we catch ValidationError | |
| 17:56:07 | andymccr | mriedem: ideally we dont tell the difference between upgrade or not - it should run through in an order that would work for both, or do some logic to ensure an appropriate state is configured | |
| 17:56:22 | andymccr | melwitt: that sounds like it would work | |
| 17:56:36 | owalsh | melwitt: another nova-status command might be cleaner | |
| 17:56:55 | andymccr | yeah if a nova-status command could tell us before hand without failing, that would work fine too | |
| 17:57:57 | melwitt | owalsh: how does the current nova-status command not work for you? does it return a fail on an empty api db, or? | |
| 18:01:24 | owalsh | melwitt: no idea... garbage collected my brain since then | |
| 18:05:42 | owalsh | melwitt: possibly because the placement api status checks could give false negatives | |
| 18:07:20 | owalsh | melwitt: I think we would at least need the ability to run individual checks with the current command | |
| 18:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 18:13:25 | jbernard | do we support booting from an encrypted volume? | |
| 18:13:36 | jbernard | docs does not say explicitly, which makes me wonder | |
| 18:13:36 | mriedem | jbernard: should be able to | |
| 18:13:45 | kevinbenton | sdague, johnthetubaguy: want to send https://review.openstack.org/#/c/439560/ in? | |
| 18:14:03 | jbernard | mriedem: cool thanks, we have also encryptors in os-brick, | |
| 18:14:07 | kevinbenton | that last statement i added will allow us to see right what's in the http response | |
| 18:14:21 | jbernard | mriedem: do we plan to make use of those at some point, or is that for something else? | |
| 18:16:23 | mriedem | jbernard: i thought lyarwood added the encryptors to os-brick | |
| 18:16:27 | mriedem | i'm not really familiar | |
| 18:16:42 | mriedem | i thought the idea was to put them in os-brick so that nova/cinder were talking the same language | |
| 18:16:52 | mriedem | because i believe the encryptor class in the volume type in cinder is just a string today | |
| 18:16:58 | mriedem | so you can throw whatever in there and it's not validated | |
| 18:17:04 | mriedem | until it doesn't work on the nova side | |
| 18:17:05 | jbernard | ok, that was my impression as well | |
| 18:17:39 | mriedem | kevinbenton: which job would be running with that on so we can see the results? | |
| 18:17:50 | smcginnis | mriedem: +1 | |
| 18:17:55 | mriedem | i forget which ones run the metadata api | |
| 18:18:03 | kevinbenton | mriedem: it seems quite a few more are now | |
| 18:18:22 | kevinbenton | mriedem: even gate-tempest-dsvm-neutron-full-ubuntu-xenial | |
| 18:19:06 | mriedem | yeah i suppose that makes sense, it's just not a separate service | |
| 18:19:10 | mriedem | like in the old pg job | |
| 18:19:15 | mriedem | it's running under nova-api | |
| 18:23:03 | mriedem | kevinbenton: comments inline | |
| 18:25:31 | mriedem | melwitt: andymccr: so on the idea of returning a different error code from nova-manage api_db sync, if we return 2 for a ValidationError or something, what would you do? basically always sync and ignore any rc=2 result, then try to map_cell0 and then sync again and if that fails you fail for real? that still seems shitty because at some point we could be raising validation error for something else you're not handling, and ignoring the va | |
| 18:26:48 | andymccr | mriedem: yeah the idea would be to know what the rc means - e.g. rc 2 == the cell db needs to be mapped, run that then sync later. but you're right if there are other cases that is problematic | |
| 18:28:28 | melwitt | mriedem: yeah, I guess it's still messy. I was thinking in general it might be nice to know if a sync failed because of a validation failure vs something else | |
| 18:29:10 | openstackgerrit | Kevin Benton proposed openstack/nova master: Add some metadata logging to root cause ssh failure https://review.openstack.org/439560 | |
| 18:29:18 | kevinbenton | mriedem, johnthetubaguy: ^^ | |
| 18:29:29 | mriedem | andymccr: have you hit issues like this in any other project? or do other projects just fail at runtime? | |
| 18:29:59 | melwitt | mriedem: owalsh mentioned being able to run individual checks with nova-status would help. so when you're automating, you can tell which things to check and respond to. currently I think the nova-status just checks all the things at the same time and they can't tell which thing failed to know which thing to do next? | |
| 18:29:59 | mriedem | +2 | |
| 18:30:24 | andymccr | melwitt: yeah that would work too and may be cleaner - e.g. you can do a specific check for the one thing you need | |
| 18:30:30 | mriedem | melwitt: yes it runs through a series of checks today and dumps a summary table with results and the most severe rc | |
| 18:30:48 | andymccr | mriedem: we do some logic around keystone rolling upgrades and rc responses afaik | |
| 18:32:30 | andymccr | so i think keystone-manage db_sync has rc codes for 2 3 4 with 1 being a flat out fail | |
| 18:33:34 | dansmith | I'm not really in favor of making db_sync indicate levels of fail | |
| 18:34:01 | dansmith | if you run db_sync, look for rc=2 and if so, do something and try to run it again, you're doing it wrong | |
| 18:34:17 | andymccr | dansmith: agree - in keystone its a --check | |
| 18:34:27 | andymccr | which basically tells you what is needed - you do those things, if they fail its a fail | |
| 18:34:43 | dansmith | ah, okay | |
| 18:34:46 | andymccr | keystone-manage db_sync --check - can return 2,3,4 (or 1 if it fails i guess) | |
| 18:34:47 | mriedem | andymccr: that's basically the same idea with nova-status upgrade check | |
| 18:34:52 | dansmith | right | |
| 18:35:19 | andymccr | mriedem: with rc's included? or just a table output - if so would it be a reasonable idea to have rc's that indicate what the table says? | |
| 18:35:24 | melwitt | yeah, I think we should make it possible to check individual checks via subcommand or something | |
| 18:35:27 | mriedem | andymccr: rc's | |
| 18:35:31 | andymccr | mriedem: ahh interesting | |
| 18:35:32 | mriedem | 0 is good, 1 is warning, 2 is fail | |
| 18:35:33 | andymccr | in that case | |
| 18:35:50 | andymccr | mriedem: ok i'll play around and see if i can get a spot that basically says "if 2 on status do these, otherwise do these" | |
| 18:35:58 | mriedem | melwitt: "check" is the subcommand, i don't think it goes lower than that, but we could build in an option | |
| 18:36:00 | andymccr | (for example) | |
| 18:36:03 | dansmith | mriedem: if they want to be able to run single checks from nova-status, that's probably okay too, but they have to run it once without that restriction otherwise they won't notice when we add things | |
| 18:36:06 | mriedem | nova-status upgrade check --cells | |
| 18:36:08 | mriedem | --placement | |
| 18:36:09 | mriedem | etc | |
| 18:36:17 | melwitt | I see | |
| 18:36:22 | mriedem | dansmith: yup | |
| 18:36:54 | mriedem | andymccr: v | |
| 18:36:55 | mriedem | https://docs.openstack.org/developer/nova/man/nova-status.html | |
| 18:36:57 | mriedem | before you ask | |
| 18:37:00 | dansmith | heh | |
| 18:37:15 | mriedem | god isn't it fucking awesome everything is documented | |