| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-10 | |||
| 19:58:01 | dansmith | imacdonn: well, if this is wrong it's possible grenade is wrong | |
| 19:58:19 | jaypipes | mriedem: yesterday, which scheduler filter did you say already looked at instance metadata by querying the BuildRequest? | |
| 19:58:41 | mriedem | jaypipes: the one you're writing | |
| 19:58:42 | dansmith | oh wait, | |
| 19:58:45 | dansmith | maybe I'm remembering this now | |
| 19:58:54 | imacdonn | dansmith: it's possible, yes ... although it seems like if you run the command without --max-count, an outcome of exit status 0 (generally interpreted as success), is what would be expected | |
| 19:59:05 | dansmith | right | |
| 19:59:05 | dansmith | in the unlimited case, you really need the exit code to be zero | |
| 19:59:30 | dansmith | oof, this should be commented in here for sure | |
| 19:59:58 | imacdonn | so that's how I came up with the term "migrations may still be pending" .... when ran is not zero, which can only happen if you use --max-count | |
| 19:59:58 | dansmith | imacdonn: so maybe just keep the full count separate from ran and use that for the gate on exit 2 | |
| 20:00:08 | sean-k-mooney | dansmith: yes if you dont pass --max-count the exit 0 should mean all migration ran sucessfuly | |
| 20:00:14 | dansmith | imacdonn: like I say, you don't know what is pending or not really | |
| 20:00:26 | imacdonn | dansmith: that's why I used "may" :) | |
| 20:00:32 | dansmith | imacdonn: don't. | |
| 20:00:36 | sean-k-mooney | actully if you do pass --max-could exit 0 should still mean the same thing | |
| 20:00:44 | dansmith | imacdonn: so if you keep total_ran and use that then you're good right? | |
| 20:01:01 | dansmith | sean-k-mooney: no it shouldn't | |
| 20:01:16 | dansmith | sean-k-mooney: because you ran --max-count=50 and you get zero, you expect you're done | |
| 20:01:31 | dansmith | sean-k-mooney: if you don't pass --max-count, zero means all of them were completed, no errors so you're done | |
| 20:01:32 | sean-k-mooney | if there are 50 migrations and i pass --max-count=100 it should return 0 if all 50 ran successfully | |
| 20:01:39 | dansmith | sean-k-mooney: no it shouldn't | |
| 20:01:49 | dansmith | sean-k-mooney: read the man page | |
| 20:01:50 | imacdonn | dansmith: are you saying exit 2 if exceptions and total_ran>0, and exit 1 if ran (not total_ran) > 0 ? | |
| 20:02:20 | sean-k-mooney | so if max is greater then total you dont want the same behavior as if there was no max | |
| 20:02:27 | dansmith | imacdonn: I'm saying leave the 0 and 1 the way they are, and exit 2 if total_ran=0 and exceptions | |
| 20:02:36 | dansmith | sean-k-mooney: it's about definition of done-ness | |
| 20:02:58 | imacdonn | dansmith: that's what I had in PS2, and it broke grenade, because it exits with 1 if any migrations were done | |
| 20:03:21 | imacdonn | dansmith: or maybe I didn't interpret your last statement right | |
| 20:03:22 | dansmith | imacdonn: I'm not talking about changing the definition of 1 | |
| 20:03:22 | sean-k-mooney | yes which is why im saying if i set a max larger then the amount to do and they all ran sucsessfully im done hence 0 | |
| 20:03:24 | dansmith | imacdonn: right | |
| 20:03:40 | dansmith | imacdonn: if you don't change the logic for 1 then grenade can't break, unless we're actually hiding exceptions | |
| 20:05:02 | imacdonn | dansmith: OK, I think that might work ... I'll have to try it ... sean-k-mooney, do you still have concerns ? | |
| 20:05:06 | sean-k-mooney | dansmith: we used to hide exceptions but my understand of 1 is there are still more migration to run i would prefer it to be "there are more migration to run and all the ones i just ran did not have error" | |
| 20:05:19 | dansmith | sean-k-mooney: that's not what 1 means | |
| 20:05:27 | dansmith | we can't really know if there are more unless we try and hit zero | |
| 20:05:36 | dansmith | without doing more database stuff to count separately from doing | |
| 20:05:41 | dansmith | which is not worth it, IMHO | |
| 20:06:04 | dansmith | so, while (rc==1) { doit } | |
| 20:06:22 | imacdonn | personally, I kinda wish there was a way to find out how many migrations are needed, before attempting any | |
| 20:06:33 | imacdonn | I might want to do this when planning an upgrade | |
| 20:06:58 | sean-k-mooney | ok so 1 just used to mean there was a partial update so we need to loop that i think makes sense | |
| 20:07:05 | sean-k-mooney | and 0 means we are done | |
| 20:07:13 | dansmith | sean-k-mooney: no, 1 means we did things, that's ALL it means :) | |
| 20:07:15 | sean-k-mooney | dansmith: what is the smantics of 2 in your case | |
| 20:07:16 | dansmith | might be partial, might be full | |
| 20:07:18 | mriedem | sorrison: if you wanted to policy something, i'd think having policy on the ability to create multiple servers in a single request would be a good one so you can avoid tenants killing your scheduler with large burst multi-create requests | |
| 20:07:25 | mriedem | especially if those users have higher than normal quota | |
| 20:07:40 | dansmith | sean-k-mooney: hopefully this can be the last time I say this, but 2 means we couldn't make any more progress, and exceptions were raised | |
| 20:07:41 | mriedem | we probably had a policy on the multi-create API extension at some point | |
| 20:08:08 | mriedem | make it 2.5.... | |
| 20:08:09 | mriedem | and i'm sold | |
| 20:08:14 | dansmith | sean-k-mooney: so you can distinguish between "can't make any more progress" and "can't make any more progress, but things seem unhappy" | |
| 20:08:19 | sean-k-mooney | dansmith: ok but we dont stop trying migration on the first exception | |
| 20:08:32 | dansmith | sean-k-mooney: exactly | |
| 20:09:18 | sean-k-mooney | my piont it 2 would only ever be retruned if all the migrations in the first iteration failed | |
| 20:09:25 | dansmith | no | |
| 20:09:39 | mriedem | sorrison: maybe that's just never been a problem b/c of quota restrictions, the multi-create thing i mean | |
| 20:10:05 | dansmith | sean-k-mooney: did you mean the last iteration? my answer is still no, but... | |
| 20:10:29 | dansmith | sean-k-mooney: with the suggestion I just made to imacdonn, we keep a total_run count and use that for 2 instead of ran, which is just "the last iteration" | |
| 20:10:29 | imacdonn | no, 2 gets returned if the only remaining possible migrations barfed for some reason | |
| 20:10:38 | dansmith | imacdonn: right | |
| 20:10:38 | openstack | Launchpad bug 1795982 in OpenStack Compute (nova) "/os-console-auth-tokens/{console_token} API doesn't handle the database backend" [High,In progress] - Assigned to melanie witt (melwitt) | |
| 20:10:38 | mriedem | melwitt: isn't this more than just the db backend for console auth? https://bugs.launchpad.net/nova/+bug/1795982 - it's been regressed since multi-cell support in pike | |
| 20:10:56 | dansmith | imacdonn: maybe just tweak and push that up and we can argue about it after there's something to see? | |
| 20:11:12 | imacdonn | dansmith: roger. will do after lunch | |
| 20:11:12 | dansmith | I'm getting kinda frustrated with this overly minute detail and really want to get on to other stuff before I end my day | |
| 20:11:16 | dansmith | imacdonn: thanks | |
| 20:11:30 | melwitt | mriedem: I don't think so because nova-consoleauth is global, you don't need to know anything about cells to query it | |
| 20:11:34 | sean-k-mooney | dansmith: yes i was referring to "total_run==0 and exceptions" for retrun 2 | |
| 20:12:10 | efried | dansmith, imacdonn: I don't intend to get super involved in this, but have we considered allowing behavior change(s) (e.g. where exceptions cause failure) based on an env var or CLI switch? (Sorry, this thought has been bouncing around in my head for a week, had to get it out.) | |
| 20:12:10 | melwitt | mriedem: that is, it's storage of console token auths is global across all cells | |
| 20:12:40 | dansmith | efried: I don't think we need a behavior change here | |
| 20:12:48 | mriedem | hmm, ok https://docs.openstack.org/nova/queens/user/cellsv2-layout.html#consoleauth-service-and-console-proxies | |
| 20:12:52 | efried | ...so that existing automations aren't affected, but new users can have the benefit of potentially improved UX | |
| 20:12:54 | mriedem | got confused about what's global or not | |
| 20:12:57 | dansmith | efried: if we did, then sure | |
| 20:13:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Handle volume API failure in post_live_migration https://review.openstack.org/609517 | |
| 20:13:39 | sean-k-mooney | dansmith: imacdonn you know what im fine with what ever you implement and ill read teh code after to understand what the exit code actully mean. | |
| 20:13:51 | efried | Okay. When I stopped looking, the patch was suggesting a behavior change that seemed fair and just to me, but was nacked for that reason. | |
| 20:14:57 | imacdonn | efried: we've introduced a new exit status, so the logic for the existing ones doesn't have to change .. I think it'll only "break" when there are exceptions and no more possible migrations, and we *want* it to break in that case | |
| 20:15:02 | dansmith | efried: the only thing that could make this more dreadful would be two behaviors in the same set of code :) | |
| 20:15:13 | dansmith | imacdonn: ++ | |
| 20:15:40 | efried | dansmith: Hm, we should introduce a microversioning system for that. | |
| 20:15:54 | dansmith | efried: we should microversion your butt. | |
| 20:16:22 | dansmith | if we ever have nova-manage cellv2 thingy --cli-version=2.123 | |
| 20:16:25 | dansmith | then just shoot me | |
| 20:16:54 | melwitt | nova-consoleauth service | |
| 20:16:54 | melwitt | mriedem: yeah, so the /os-console-auth-tokens/{console_token} API calls nova-consoleauth over RPC, and nova-consoleauth was made cell-aware sometime in the past, so all was working fine with multi-cell (unless there's a bug we don't know about). but when we moved to the database backend, that's what made it so that the /os-console-auth-tokens API would need to be able to talk to cell databases directly instead of going through the | |
| 20:17:37 | efried | what we really need is nested microversions | |
| 20:17:59 | sean-k-mooney | imacdonn: retruaning any new code will still requrie autoation scirpt to handel the new error case or manual intervention so the fact there is a behavior change or not is slight less important that said in the sucess case the codes should not change | |
| 20:18:01 | mriedem | god i bet my cross-cell resize stuff needs to recreate console auth tokens for the moved instance in the target cell db too... | |
| 20:18:03 | mriedem | like bdms and tags | |
| 20:18:13 | mriedem | and virtual_interfaces... | |
| 20:19:18 | melwitt | you could probably just punt that though, let them have to get a fresh console token after a cross-cell move | |
| 20:19:31 | imacdonn | sean-k-mooney: that's true, and that's why there's a release note for this, and it may not be backportable ... but the most common case will be running the command without --max-count and expecting a 0, or rerunning it until you don't get 1 | |
| 20:19:55 | melwitt | default TTL for console token auth is 10 minutes so they aren't designed to live long. operators can configure longer TTL but I'm not sure they'd expect you to solve for that | |
| 20:20:44 | imacdonn | sean-k-mooney: I guess the possible case where it could break automation is if the command is being rerun infinitely until it gets a zero, which would never happen if it's returning 2 every time | |
| 20:20:47 | sean-k-mooney | imacdonn: if i was writingin this in ansiable and i called it without --max-count i would have interpereted a non 0 result as an error just fyi | |