| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 17:54:46 | melwitt | they just need to be able to differentiate between a regular db sync fail and a validation db sync fail. I wonder if we could do something like return 2 if we catch ValidationError | |
| 17:56:07 | andymccr | mriedem: ideally we dont tell the difference between upgrade or not - it should run through in an order that would work for both, or do some logic to ensure an appropriate state is configured | |
| 17:56:22 | andymccr | melwitt: that sounds like it would work | |
| 17:56:36 | owalsh | melwitt: another nova-status command might be cleaner | |
| 17:56:55 | andymccr | yeah if a nova-status command could tell us before hand without failing, that would work fine too | |
| 17:57:57 | melwitt | owalsh: how does the current nova-status command not work for you? does it return a fail on an empty api db, or? | |
| 18:01:24 | owalsh | melwitt: no idea... garbage collected my brain since then | |
| 18:05:42 | owalsh | melwitt: possibly because the placement api status checks could give false negatives | |
| 18:07:20 | owalsh | melwitt: I think we would at least need the ability to run individual checks with the current command | |
| 18:09:53 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Get instance availability_zone without hitting the api db https://review.openstack.org/439754 | |
| 18:13:25 | jbernard | do we support booting from an encrypted volume? | |
| 18:13:36 | mriedem | jbernard: should be able to | |
| 18:13:36 | jbernard | docs does not say explicitly, which makes me wonder | |
| 18:13:45 | kevinbenton | sdague, johnthetubaguy: want to send https://review.openstack.org/#/c/439560/ in? | |
| 18:14:03 | jbernard | mriedem: cool thanks, we have also encryptors in os-brick, | |
| 18:14:07 | kevinbenton | that last statement i added will allow us to see right what's in the http response | |
| 18:14:21 | jbernard | mriedem: do we plan to make use of those at some point, or is that for something else? | |
| 18:16:23 | mriedem | jbernard: i thought lyarwood added the encryptors to os-brick | |
| 18:16:27 | mriedem | i'm not really familiar | |
| 18:16:42 | mriedem | i thought the idea was to put them in os-brick so that nova/cinder were talking the same language | |
| 18:16:52 | mriedem | because i believe the encryptor class in the volume type in cinder is just a string today | |
| 18:16:58 | mriedem | so you can throw whatever in there and it's not validated | |
| 18:17:04 | mriedem | until it doesn't work on the nova side | |
| 18:17:05 | jbernard | ok, that was my impression as well | |
| 18:17:39 | mriedem | kevinbenton: which job would be running with that on so we can see the results? | |
| 18:17:50 | smcginnis | mriedem: +1 | |
| 18:17:55 | mriedem | i forget which ones run the metadata api | |
| 18:18:03 | kevinbenton | mriedem: it seems quite a few more are now | |
| 18:18:22 | kevinbenton | mriedem: even gate-tempest-dsvm-neutron-full-ubuntu-xenial | |
| 18:19:06 | mriedem | yeah i suppose that makes sense, it's just not a separate service | |
| 18:19:10 | mriedem | like in the old pg job | |
| 18:19:15 | mriedem | it's running under nova-api | |
| 18:23:03 | mriedem | kevinbenton: comments inline | |
| 18:25:31 | mriedem | melwitt: andymccr: so on the idea of returning a different error code from nova-manage api_db sync, if we return 2 for a ValidationError or something, what would you do? basically always sync and ignore any rc=2 result, then try to map_cell0 and then sync again and if that fails you fail for real? that still seems shitty because at some point we could be raising validation error for something else you're not handling, and ignoring the va | |
| 18:26:48 | andymccr | mriedem: yeah the idea would be to know what the rc means - e.g. rc 2 == the cell db needs to be mapped, run that then sync later. but you're right if there are other cases that is problematic | |
| 18:28:28 | melwitt | mriedem: yeah, I guess it's still messy. I was thinking in general it might be nice to know if a sync failed because of a validation failure vs something else | |
| 18:29:10 | openstackgerrit | Kevin Benton proposed openstack/nova master: Add some metadata logging to root cause ssh failure https://review.openstack.org/439560 | |
| 18:29:18 | kevinbenton | mriedem, johnthetubaguy: ^^ | |
| 18:29:29 | mriedem | andymccr: have you hit issues like this in any other project? or do other projects just fail at runtime? | |
| 18:29:59 | mriedem | +2 | |
| 18:29:59 | melwitt | mriedem: owalsh mentioned being able to run individual checks with nova-status would help. so when you're automating, you can tell which things to check and respond to. currently I think the nova-status just checks all the things at the same time and they can't tell which thing failed to know which thing to do next? | |
| 18:30:24 | andymccr | melwitt: yeah that would work too and may be cleaner - e.g. you can do a specific check for the one thing you need | |
| 18:30:30 | mriedem | melwitt: yes it runs through a series of checks today and dumps a summary table with results and the most severe rc | |
| 18:30:48 | andymccr | mriedem: we do some logic around keystone rolling upgrades and rc responses afaik | |
| 18:32:30 | andymccr | so i think keystone-manage db_sync has rc codes for 2 3 4 with 1 being a flat out fail | |
| 18:33:34 | dansmith | I'm not really in favor of making db_sync indicate levels of fail | |
| 18:34:01 | dansmith | if you run db_sync, look for rc=2 and if so, do something and try to run it again, you're doing it wrong | |
| 18:34:17 | andymccr | dansmith: agree - in keystone its a --check | |
| 18:34:27 | andymccr | which basically tells you what is needed - you do those things, if they fail its a fail | |
| 18:34:43 | dansmith | ah, okay | |
| 18:34:46 | andymccr | keystone-manage db_sync --check - can return 2,3,4 (or 1 if it fails i guess) | |
| 18:34:47 | mriedem | andymccr: that's basically the same idea with nova-status upgrade check | |
| 18:34:52 | dansmith | right | |
| 18:35:19 | andymccr | mriedem: with rc's included? or just a table output - if so would it be a reasonable idea to have rc's that indicate what the table says? | |
| 18:35:24 | melwitt | yeah, I think we should make it possible to check individual checks via subcommand or something | |
| 18:35:27 | mriedem | andymccr: rc's | |
| 18:35:31 | andymccr | mriedem: ahh interesting | |
| 18:35:32 | mriedem | 0 is good, 1 is warning, 2 is fail | |
| 18:35:33 | andymccr | in that case | |
| 18:35:50 | andymccr | mriedem: ok i'll play around and see if i can get a spot that basically says "if 2 on status do these, otherwise do these" | |
| 18:35:58 | mriedem | melwitt: "check" is the subcommand, i don't think it goes lower than that, but we could build in an option | |
| 18:36:00 | andymccr | (for example) | |
| 18:36:03 | dansmith | mriedem: if they want to be able to run single checks from nova-status, that's probably okay too, but they have to run it once without that restriction otherwise they won't notice when we add things | |
| 18:36:06 | mriedem | nova-status upgrade check --cells | |
| 18:36:08 | mriedem | --placement | |
| 18:36:09 | mriedem | etc | |
| 18:36:17 | melwitt | I see | |
| 18:36:22 | mriedem | dansmith: yup | |
| 18:36:54 | mriedem | andymccr: v | |
| 18:36:55 | mriedem | https://docs.openstack.org/developer/nova/man/nova-status.html | |
| 18:36:57 | mriedem | before you ask | |
| 18:37:00 | dansmith | heh | |
| 18:37:15 | mriedem | god isn't it fucking awesome everything is documented | |
| 18:37:19 | andymccr | ah ok so status 2 is just "anything happened" | |
| 18:37:38 | andymccr | and 1 is "random warning" so there is no real way to tell what that means unless you inspect the output | |
| 18:37:40 | mriedem | andymccr: yeah it's not granular per check right now | |
| 18:37:43 | andymccr | ok | |
| 18:37:50 | andymccr | maybe i can work around that somehow | |
| 18:37:51 | mriedem | 2 means, stop, shit won't start if you proceed | |
| 18:38:03 | melwitt | is the only return item a table? maybe we need a json option for parsing by automations? | |
| 18:38:09 | mriedem | 1 means, you should investigate, but things will start and shouldn't cause huge issues | |
| 18:38:10 | dansmith | mriedem: well, to be fair "shit" will start [happening] if you proceed | |
| 18:38:29 | mriedem | well, e.g. n-cpu won't start | |
| 18:38:40 | dansmith | mriedem: right.. the "shit" will start [hitting the fan] | |
| 18:38:41 | mriedem | in ocata if placement isn't configured | |
| 18:38:42 | dansmith | nevermind. | |
| 18:38:44 | mriedem | yeah | |
| 18:39:10 | mriedem | melwitt: it's a table in stdout and a return code right now yeah | |
| 18:39:34 | mriedem | we could build in a json output option if needed | |
| 18:39:53 | dansmith | if you know what to do with rich per-test output from that, | |
| 18:40:02 | dansmith | then it means you already know what to do, you should do it, and fail if the test says you fail | |
| 18:40:29 | mriedem | yeah it does feel like a lot of bending over backwards to hack around something | |
| 18:40:53 | dansmith | with the exception of anything that really isn't idempotent, but the only thing i can think of that fits that is cell creation, which should be obvious when to do | |
| 18:41:01 | melwitt | I was thinking the problem was, if they get a fail, then they don't know which of the things placement or cells they need to set up | |
| 18:41:14 | dansmith | melwitt: they have to set up both things | |
| 18:41:26 | dansmith | I'm not sure why there's a question of when or if to do those | |
| 18:41:43 | andymccr | dansmith: the issue is in upgrades you have to make sure you do a cell create before a api_db sync - in greenfield you have to do the reverse order | |
| 18:41:48 | andymccr | that means the upgrade vs new install path is different | |
| 18:42:06 | dansmith | andymccr: how is it opposite? | |
| 18:42:06 | andymccr | so having a mechanism to determine what needs to happen is important unless we want to carry an "upgrade" flag which is really not a great way to do deploys | |