| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-01 | |||
| 18:29:29 | mriedem | andymccr: have you hit issues like this in any other project? or do other projects just fail at runtime? | |
| 18:29:59 | melwitt | mriedem: owalsh mentioned being able to run individual checks with nova-status would help. so when you're automating, you can tell which things to check and respond to. currently I think the nova-status just checks all the things at the same time and they can't tell which thing failed to know which thing to do next? | |
| 18:29:59 | mriedem | +2 | |
| 18:30:24 | andymccr | melwitt: yeah that would work too and may be cleaner - e.g. you can do a specific check for the one thing you need | |
| 18:30:30 | mriedem | melwitt: yes it runs through a series of checks today and dumps a summary table with results and the most severe rc | |
| 18:30:48 | andymccr | mriedem: we do some logic around keystone rolling upgrades and rc responses afaik | |
| 18:32:30 | andymccr | so i think keystone-manage db_sync has rc codes for 2 3 4 with 1 being a flat out fail | |
| 18:33:34 | dansmith | I'm not really in favor of making db_sync indicate levels of fail | |
| 18:34:01 | dansmith | if you run db_sync, look for rc=2 and if so, do something and try to run it again, you're doing it wrong | |
| 18:34:17 | andymccr | dansmith: agree - in keystone its a --check | |
| 18:34:27 | andymccr | which basically tells you what is needed - you do those things, if they fail its a fail | |
| 18:34:43 | dansmith | ah, okay | |
| 18:34:46 | andymccr | keystone-manage db_sync --check - can return 2,3,4 (or 1 if it fails i guess) | |
| 18:34:47 | mriedem | andymccr: that's basically the same idea with nova-status upgrade check | |
| 18:34:52 | dansmith | right | |
| 18:35:19 | andymccr | mriedem: with rc's included? or just a table output - if so would it be a reasonable idea to have rc's that indicate what the table says? | |
| 18:35:24 | melwitt | yeah, I think we should make it possible to check individual checks via subcommand or something | |
| 18:35:27 | mriedem | andymccr: rc's | |
| 18:35:31 | andymccr | mriedem: ahh interesting | |
| 18:35:32 | mriedem | 0 is good, 1 is warning, 2 is fail | |
| 18:35:33 | andymccr | in that case | |
| 18:35:50 | andymccr | mriedem: ok i'll play around and see if i can get a spot that basically says "if 2 on status do these, otherwise do these" | |
| 18:35:58 | mriedem | melwitt: "check" is the subcommand, i don't think it goes lower than that, but we could build in an option | |
| 18:36:00 | andymccr | (for example) | |
| 18:36:03 | dansmith | mriedem: if they want to be able to run single checks from nova-status, that's probably okay too, but they have to run it once without that restriction otherwise they won't notice when we add things | |
| 18:36:06 | mriedem | nova-status upgrade check --cells | |
| 18:36:08 | mriedem | --placement | |
| 18:36:09 | mriedem | etc | |
| 18:36:17 | melwitt | I see | |
| 18:36:22 | mriedem | dansmith: yup | |
| 18:36:54 | mriedem | andymccr: v | |
| 18:36:55 | mriedem | https://docs.openstack.org/developer/nova/man/nova-status.html | |
| 18:36:57 | mriedem | before you ask | |
| 18:37:00 | dansmith | heh | |
| 18:37:15 | mriedem | god isn't it fucking awesome everything is documented | |
| 18:37:19 | andymccr | ah ok so status 2 is just "anything happened" | |
| 18:37:38 | andymccr | and 1 is "random warning" so there is no real way to tell what that means unless you inspect the output | |
| 18:37:40 | mriedem | andymccr: yeah it's not granular per check right now | |
| 18:37:43 | andymccr | ok | |
| 18:37:50 | andymccr | maybe i can work around that somehow | |
| 18:37:51 | mriedem | 2 means, stop, shit won't start if you proceed | |
| 18:38:03 | melwitt | is the only return item a table? maybe we need a json option for parsing by automations? | |
| 18:38:09 | mriedem | 1 means, you should investigate, but things will start and shouldn't cause huge issues | |
| 18:38:10 | dansmith | mriedem: well, to be fair "shit" will start [happening] if you proceed | |
| 18:38:29 | mriedem | well, e.g. n-cpu won't start | |
| 18:38:40 | dansmith | mriedem: right.. the "shit" will start [hitting the fan] | |
| 18:38:41 | mriedem | in ocata if placement isn't configured | |
| 18:38:42 | dansmith | nevermind. | |
| 18:38:44 | mriedem | yeah | |
| 18:39:10 | mriedem | melwitt: it's a table in stdout and a return code right now yeah | |
| 18:39:34 | mriedem | we could build in a json output option if needed | |
| 18:39:53 | dansmith | if you know what to do with rich per-test output from that, | |
| 18:40:02 | dansmith | then it means you already know what to do, you should do it, and fail if the test says you fail | |
| 18:40:29 | mriedem | yeah it does feel like a lot of bending over backwards to hack around something | |
| 18:40:53 | dansmith | with the exception of anything that really isn't idempotent, but the only thing i can think of that fits that is cell creation, which should be obvious when to do | |
| 18:41:01 | melwitt | I was thinking the problem was, if they get a fail, then they don't know which of the things placement or cells they need to set up | |
| 18:41:14 | dansmith | melwitt: they have to set up both things | |
| 18:41:26 | dansmith | I'm not sure why there's a question of when or if to do those | |
| 18:41:43 | andymccr | dansmith: the issue is in upgrades you have to make sure you do a cell create before a api_db sync - in greenfield you have to do the reverse order | |
| 18:41:48 | andymccr | that means the upgrade vs new install path is different | |
| 18:42:06 | andymccr | so having a mechanism to determine what needs to happen is important unless we want to carry an "upgrade" flag which is really not a great way to do deploys | |
| 18:42:06 | dansmith | andymccr: how is it opposite? | |
| 18:42:19 | andymccr | dansmith: if you run api_db sync when cell_v2 create hasnt run it fails saying "cell doesnt exist" on an upgrade | |
| 18:42:29 | openstackgerrit | Feodor Tersin proposed openstack/nova-specs master: Add ScaleIO ephemeral storage backend https://review.openstack.org/304715 | |
| 18:42:35 | andymccr | but on a greenfield deploy if you run "cell_create" without api_db sync having happened it says "cant write to nova_api database" | |
| 18:42:55 | dansmith | ah, okay, I see | |
| 18:43:22 | dansmith | that's a very atypical example, so maybe we work around that one specifically instead of building in a lot of "check this before doing a thing" for other stuff? | |
| 18:43:52 | dansmith | andymccr: how do you know if flavors need to be created? | |
| 18:44:00 | dansmith | (i.e. upgrade or new install) | |
| 18:44:48 | andymccr | we don't create flavors as part of the deploy - that's up to the deployer afterwards if they want (or in testing the tempest role sets up the bits) | |
| 18:45:00 | andymccr | but the create bits are idempotent so if they exist it doesnt bomb or anything | |
| 18:45:00 | dansmith | oh okay I thought you did | |
| 18:45:21 | dansmith | okay, so, the migration uses flavors to determine if it should fail | |
| 18:45:34 | dansmith | depending on where this runs you could probably do the same | |
| 18:45:39 | mriedem | sdague: sfinucan: i've figured out that https://review.openstack.org/#/c/434339/ introduces a bug | |
| 18:45:48 | mriedem | in the 204 case | |
| 18:46:01 | dansmith | andymccr: also you could sync to before the blocker, do the create, then finish the sync and that would work for both cases | |
| 18:46:36 | andymccr | dansmith: how can i specify to sync to before the blocker? | |
| 18:46:54 | mriedem | fixing | |
| 18:46:59 | andymccr | dansmith: if i understand what you mean you do the first part of the sync, then create the cell if it doesnt exist, then finish the sync | |
| 18:47:13 | dansmith | andymccr: sync --version 123 | |
| 18:47:19 | dansmith | andymccr: yup | |
| 18:47:58 | mriedem | andymccr: there is also a nova-manage cell_v2 list_cells command | |
| 18:48:04 | dansmith | andymccr: that blocker was attempting to prevent people from screwing themselves by not reading the docs, but I see how it puts you in a tight spot | |
| 18:48:11 | mriedem | you could parse that for the nova_cell0 entry | |
| 18:48:12 | dansmith | mriedem: that too | |
| 18:48:32 | dansmith | andymccr: with nova-status we may be able to get away from doing that as much in the future | |
| 18:48:51 | andymccr | mriedem: that would tell if cells exist, so in the case of greenfield i still need an api_db sync first - in upgrades i wouldn't | |
| 18:48:54 | mriedem | yeah, we did this blocker migration before we had nova-status | |
| 18:49:04 | andymccr | mriedem: the key one would be a "has an api_db sync ever run successfully" command | |
| 18:49:16 | dansmith | andymccr: db version | |
| 18:49:33 | mriedem | yeah | |
| 18:49:38 | mriedem | nova-manage api_db version | |
| 18:49:39 | dansmith | andymccr: 'nova-manage api_db version' to see if you're at 0, before the blocker, or after, then do the right thing | |
| 18:49:45 | mriedem | that will tell you if you've synced | |
| 18:50:30 | openstackgerrit | Sujitha proposed openstack/nova master: Remove mox from nova.tests.unit.virt.xenapi.test_vmops[2] https://review.openstack.org/439770 | |
| 18:50:58 | andymccr | mriedem: dansmith thanks a bunch i'll play around with those and try see if i can get something working. that sounds like it may work | |
| 18:51:06 | dansmith | andymccr: cool | |
| 18:51:13 | andymccr | owalsh-afk: i'll let you know what i come up with | |
| 18:51:20 | andymccr | it might be helpful for tripleo too | |