| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 18:17:05 | jbernard | ok, that was my impression as well | |
| 18:17:39 | mriedem | kevinbenton: which job would be running with that on so we can see the results? | |
| 18:17:50 | smcginnis | mriedem: +1 | |
| 18:17:55 | mriedem | i forget which ones run the metadata api | |
| 18:18:03 | kevinbenton | mriedem: it seems quite a few more are now | |
| 18:18:22 | kevinbenton | mriedem: even gate-tempest-dsvm-neutron-full-ubuntu-xenial | |
| 18:19:06 | mriedem | yeah i suppose that makes sense, it's just not a separate service | |
| 18:19:10 | mriedem | like in the old pg job | |
| 18:19:15 | mriedem | it's running under nova-api | |
| 18:23:03 | mriedem | kevinbenton: comments inline | |
| 18:25:31 | mriedem | melwitt: andymccr: so on the idea of returning a different error code from nova-manage api_db sync, if we return 2 for a ValidationError or something, what would you do? basically always sync and ignore any rc=2 result, then try to map_cell0 and then sync again and if that fails you fail for real? that still seems shitty because at some point we could be raising validation error for something else you're not handling, and ignoring the va | |
| 18:26:48 | andymccr | mriedem: yeah the idea would be to know what the rc means - e.g. rc 2 == the cell db needs to be mapped, run that then sync later. but you're right if there are other cases that is problematic | |
| 18:28:28 | melwitt | mriedem: yeah, I guess it's still messy. I was thinking in general it might be nice to know if a sync failed because of a validation failure vs something else | |
| 18:29:10 | openstackgerrit | Kevin Benton proposed openstack/nova master: Add some metadata logging to root cause ssh failure https://review.openstack.org/439560 | |
| 18:29:18 | kevinbenton | mriedem, johnthetubaguy: ^^ | |
| 18:29:29 | mriedem | andymccr: have you hit issues like this in any other project? or do other projects just fail at runtime? | |
| 18:29:59 | mriedem | +2 | |
| 18:29:59 | melwitt | mriedem: owalsh mentioned being able to run individual checks with nova-status would help. so when you're automating, you can tell which things to check and respond to. currently I think the nova-status just checks all the things at the same time and they can't tell which thing failed to know which thing to do next? | |
| 18:30:24 | andymccr | melwitt: yeah that would work too and may be cleaner - e.g. you can do a specific check for the one thing you need | |
| 18:30:30 | mriedem | melwitt: yes it runs through a series of checks today and dumps a summary table with results and the most severe rc | |
| 18:30:48 | andymccr | mriedem: we do some logic around keystone rolling upgrades and rc responses afaik | |
| 18:32:30 | andymccr | so i think keystone-manage db_sync has rc codes for 2 3 4 with 1 being a flat out fail | |
| 18:33:34 | dansmith | I'm not really in favor of making db_sync indicate levels of fail | |
| 18:34:01 | dansmith | if you run db_sync, look for rc=2 and if so, do something and try to run it again, you're doing it wrong | |
| 18:34:17 | andymccr | dansmith: agree - in keystone its a --check | |
| 18:34:27 | andymccr | which basically tells you what is needed - you do those things, if they fail its a fail | |
| 18:34:43 | dansmith | ah, okay | |
| 18:34:46 | andymccr | keystone-manage db_sync --check - can return 2,3,4 (or 1 if it fails i guess) | |
| 18:34:47 | mriedem | andymccr: that's basically the same idea with nova-status upgrade check | |
| 18:34:52 | dansmith | right | |
| 18:35:19 | andymccr | mriedem: with rc's included? or just a table output - if so would it be a reasonable idea to have rc's that indicate what the table says? | |
| 18:35:24 | melwitt | yeah, I think we should make it possible to check individual checks via subcommand or something | |
| 18:35:27 | mriedem | andymccr: rc's | |
| 18:35:31 | andymccr | mriedem: ahh interesting | |
| 18:35:32 | mriedem | 0 is good, 1 is warning, 2 is fail | |
| 18:35:33 | andymccr | in that case | |
| 18:35:50 | andymccr | mriedem: ok i'll play around and see if i can get a spot that basically says "if 2 on status do these, otherwise do these" | |
| 18:35:58 | mriedem | melwitt: "check" is the subcommand, i don't think it goes lower than that, but we could build in an option | |
| 18:36:00 | andymccr | (for example) | |
| 18:36:03 | dansmith | mriedem: if they want to be able to run single checks from nova-status, that's probably okay too, but they have to run it once without that restriction otherwise they won't notice when we add things | |
| 18:36:06 | mriedem | nova-status upgrade check --cells | |
| 18:36:08 | mriedem | --placement | |
| 18:36:09 | mriedem | etc | |
| 18:36:17 | melwitt | I see | |
| 18:36:22 | mriedem | dansmith: yup | |
| 18:36:54 | mriedem | andymccr: v | |
| 18:36:55 | mriedem | https://docs.openstack.org/developer/nova/man/nova-status.html | |
| 18:36:57 | mriedem | before you ask | |
| 18:37:00 | dansmith | heh | |
| 18:37:15 | mriedem | god isn't it fucking awesome everything is documented | |
| 18:37:19 | andymccr | ah ok so status 2 is just "anything happened" | |
| 18:37:38 | andymccr | and 1 is "random warning" so there is no real way to tell what that means unless you inspect the output | |
| 18:37:40 | mriedem | andymccr: yeah it's not granular per check right now | |
| 18:37:43 | andymccr | ok | |
| 18:37:50 | andymccr | maybe i can work around that somehow | |
| 18:37:51 | mriedem | 2 means, stop, shit won't start if you proceed | |
| 18:38:03 | melwitt | is the only return item a table? maybe we need a json option for parsing by automations? | |
| 18:38:09 | mriedem | 1 means, you should investigate, but things will start and shouldn't cause huge issues | |
| 18:38:10 | dansmith | mriedem: well, to be fair "shit" will start [happening] if you proceed | |
| 18:38:29 | mriedem | well, e.g. n-cpu won't start | |
| 18:38:40 | dansmith | mriedem: right.. the "shit" will start [hitting the fan] | |
| 18:38:41 | mriedem | in ocata if placement isn't configured | |
| 18:38:42 | dansmith | nevermind. | |
| 18:38:44 | mriedem | yeah | |
| 18:39:10 | mriedem | melwitt: it's a table in stdout and a return code right now yeah | |
| 18:39:34 | mriedem | we could build in a json output option if needed | |
| 18:39:53 | dansmith | if you know what to do with rich per-test output from that, | |
| 18:40:02 | dansmith | then it means you already know what to do, you should do it, and fail if the test says you fail | |
| 18:40:29 | mriedem | yeah it does feel like a lot of bending over backwards to hack around something | |
| 18:40:53 | dansmith | with the exception of anything that really isn't idempotent, but the only thing i can think of that fits that is cell creation, which should be obvious when to do | |
| 18:41:01 | melwitt | I was thinking the problem was, if they get a fail, then they don't know which of the things placement or cells they need to set up | |
| 18:41:14 | dansmith | melwitt: they have to set up both things | |
| 18:41:26 | dansmith | I'm not sure why there's a question of when or if to do those | |
| 18:41:43 | andymccr | dansmith: the issue is in upgrades you have to make sure you do a cell create before a api_db sync - in greenfield you have to do the reverse order | |
| 18:41:48 | andymccr | that means the upgrade vs new install path is different | |
| 18:42:06 | dansmith | andymccr: how is it opposite? | |
| 18:42:06 | andymccr | so having a mechanism to determine what needs to happen is important unless we want to carry an "upgrade" flag which is really not a great way to do deploys | |
| 18:42:19 | andymccr | dansmith: if you run api_db sync when cell_v2 create hasnt run it fails saying "cell doesnt exist" on an upgrade | |
| 18:42:29 | openstackgerrit | Feodor Tersin proposed openstack/nova-specs master: Add ScaleIO ephemeral storage backend https://review.openstack.org/304715 | |
| 18:42:35 | andymccr | but on a greenfield deploy if you run "cell_create" without api_db sync having happened it says "cant write to nova_api database" | |
| 18:42:55 | dansmith | ah, okay, I see | |
| 18:43:22 | dansmith | that's a very atypical example, so maybe we work around that one specifically instead of building in a lot of "check this before doing a thing" for other stuff? | |
| 18:43:52 | dansmith | andymccr: how do you know if flavors need to be created? | |
| 18:44:00 | dansmith | (i.e. upgrade or new install) | |
| 18:44:48 | andymccr | we don't create flavors as part of the deploy - that's up to the deployer afterwards if they want (or in testing the tempest role sets up the bits) | |
| 18:45:00 | dansmith | oh okay I thought you did | |
| 18:45:00 | andymccr | but the create bits are idempotent so if they exist it doesnt bomb or anything | |
| 18:45:21 | dansmith | okay, so, the migration uses flavors to determine if it should fail | |
| 18:45:34 | dansmith | depending on where this runs you could probably do the same | |
| 18:45:39 | mriedem | sdague: sfinucan: i've figured out that https://review.openstack.org/#/c/434339/ introduces a bug | |
| 18:45:48 | mriedem | in the 204 case | |
| 18:46:01 | dansmith | andymccr: also you could sync to before the blocker, do the create, then finish the sync and that would work for both cases | |
| 18:46:36 | andymccr | dansmith: how can i specify to sync to before the blocker? | |
| 18:46:54 | mriedem | fixing | |
| 18:46:59 | andymccr | dansmith: if i understand what you mean you do the first part of the sync, then create the cell if it doesnt exist, then finish the sync | |
| 18:47:13 | dansmith | andymccr: sync --version 123 | |
| 18:47:19 | dansmith | andymccr: yup | |
| 18:47:58 | mriedem | andymccr: there is also a nova-manage cell_v2 list_cells command | |
| 18:48:04 | dansmith | andymccr: that blocker was attempting to prevent people from screwing themselves by not reading the docs, but I see how it puts you in a tight spot | |
| 18:48:11 | mriedem | you could parse that for the nova_cell0 entry | |