| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-19 | |||
| 18:20:46 | aspiers | efried: nice work :) | |
| 18:20:54 | efried | WIP incoming, lemme finish writing up the commit message. | |
| 18:20:56 | aspiers | I bet whatever you changed would have taken me a week | |
| 18:21:26 | aspiers | mriedem: OK thanks, I'll do that | |
| 18:21:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: De-shim PlacementFixture https://review.openstack.org/637959 | |
| 18:21:41 | efried | aspiers: ^ | |
| 18:21:43 | aspiers | mriedem, melwitt: is it OK to reference 2 blueprints in a single commit message? | |
| 18:21:46 | aspiers | efried: ack | |
| 18:22:05 | efried | I originally wrote it on top of your patch, to make sure your test case worked, and it did. | |
| 18:22:15 | efried | But this version will be lacking your delete helper. | |
| 18:22:27 | efried | Once I've got it polished I'll rebase it under yours. | |
| 18:22:34 | efried | or you can | |
| 18:22:40 | aspiers | Sure, I can do that | |
| 18:22:48 | aspiers | I have some other tweaks to go in mine anyway | |
| 18:23:09 | melwitt | aspiers: I don't see why not. doing that would just put the link on both lp whiteboard AFAIK | |
| 18:23:19 | aspiers | not to mention fixes for the glaring errors you and mriedem are about to uncover ;-) | |
| 18:23:31 | aspiers | melwitt: cool thanks, that's what I expected | |
| 18:25:19 | aspiers | woohoo, my version switcher patch landed on https://docs.openstack.org/ | |
| 18:25:36 | aspiers | hopefully it doesn't break anything | |
| 18:25:43 | aspiers | s/version/release/ | |
| 18:40:24 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Fallback to the first element if no nodes match https://review.openstack.org/637965 | |
| 19:19:54 | openstackgerrit | Merged openstack/nova master: api-ref: warn about changing/unsetting AZ name with instances https://review.openstack.org/635315 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | |
| 19:29:41 | openstackgerrit | melanie witt proposed openstack/nova master: Add user_id field to InstanceMapping https://review.openstack.org/633350 | |
| 19:35:24 | openstackgerrit | Merged openstack/nova master: api-ref: mention policy defaults for aggregates https://review.openstack.org/635317 | |
| 19:37:10 | efried | mriedem: easy +A if you please https://review.openstack.org/#/c/637657/ | |
| 19:45:43 | mriedem | +W | |
| 19:50:11 | efried | thanks mriedem | |
| 19:57:42 | efried | aspiers: I had to give up. Your fix is gateward. | |
| 20:28:34 | openstackgerrit | Merged openstack/nova master: Fix fake DELETE in PlacementFixture https://review.openstack.org/637657 | |
| 20:42:25 | openstackgerrit | Merged openstack/os-vif master: Change python3.5 job to python3.7 job on Stein+ https://review.openstack.org/610068 | |
| 21:07:56 | mriedem | rom error) | |
| 21:07:56 | mriedem | i wonder how many operators are trying to recover an instance from a failed resize confirm/revert where the instance is in ERROR status and they fixed something in the compute and just want to retry the confirm/revert operation, but can't b/c the API doesn't allow those operations on an instance in ERROR status. seems weird that the os-resetStatus server action API doesn't allow you to put an instance in 'resized' status agai | |
| 21:08:10 | mriedem | dansmith: ever thunk on that?^ | |
| 21:08:48 | dansmith | well, it would be pretty bad if you put it into resized status when it wasn't, | |
| 21:08:58 | dansmith | and it'd be hard to tell when and when not to do something like that | |
| 21:09:29 | mriedem | if you got through revert_resize on the dest but finish_revert_resize fails on the source, you can maybe hard reboot your way out of it | |
| 21:09:41 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Don't reference unassigned variable in xml matchers in test https://review.openstack.org/637965 | |
| 21:09:42 | mriedem | if revert_resize on the dest fails, you're kind of stuck there now | |
| 21:09:48 | mriedem | anywho | |
| 21:10:06 | dansmith | right but is it realistic for an operator to know that it failed on finish_revert_resize? | |
| 21:10:09 | dansmith | even if they fix something, | |
| 21:10:28 | dansmith | they could reset into resized state from another place where they think it's likely to help and further break stuff | |
| 21:10:42 | dansmith | I mean, I know what you're saying, but.. it's a dangerous tool | |
| 21:10:56 | mriedem | sure, and i wouldn't expect most operators to know that unless they were pretty knowledgeable on low level nova | |
| 21:11:06 | mriedem | which their time is probably just better spent on rebuilding the vm | |
| 21:11:16 | artom | And, is it needed? I mean, if they have data they want to keep on a volume, they can just attach that to a new instance with the correct flavor? | |
| 21:11:18 | mriedem | and writing an apology to the user | |
| 21:11:51 | mriedem | artom: i'm thinking more about local storage, no volumes | |
| 21:12:08 | artom | So actually, is a resize destructive in that respect? | |
| 21:12:09 | mriedem | granted your data should be stored on a persistent volume | |
| 21:12:31 | mriedem | no, the local disks are left intact on the source host when you resize | |
| 21:12:46 | mriedem | they are destroyed when you confirm the resize to keep the instance on the dest host | |
| 21:12:53 | mriedem | and re-used on the source if you revert | |
| 21:13:48 | artom | And your idea is - if the resize went to ERROR, rather than just reverting back to the source, allow them to force completion, so to speak? | |
| 21:14:00 | mriedem | not force completion, retry | |
| 21:14:12 | mriedem | but, | |
| 21:14:20 | mriedem | that retry could fail if we got part way the first time | |
| 21:14:25 | mriedem | so retrying isn't without risk either | |
| 21:14:42 | mriedem | i.e. revert_resize on dest fails the 2nd time b/c we already deleted some stuff the first time | |
| 21:15:01 | artom | Yeah... | |
| 21:15:31 | artom | In that situation, what's wrong with going through the resize again, after reverting the failed attempt? | |
| 21:15:33 | mriedem | i've just got some sticky logic in the cross-cell revert code where if something fails part way through it's not so good | |
| 21:15:54 | mriedem | "reverting the failed attempt"? | |
| 21:16:02 | mriedem | s/reverting/fixing/? | |
| 21:16:23 | mnaser | maybe in an ideal world we can collect more of the cases where finish_revert_resize fails and test those out *before* starting to revert the resize | |
| 21:16:24 | artom | Oh. You can't revert out of an ERROR, can you? | |
| 21:16:25 | mriedem | you could reset status to active and try another resize | |
| 21:16:41 | mriedem | no you can only confirm/revert a server in 'resized' status | |
| 21:16:43 | mnaser | no you cant revert out of ERROR, only RESIZE_CONFIRM i think | |
| 21:16:49 | mriedem | you can hard reboot / rebuild a server in ERROR | |
| 21:17:03 | mnaser | tbh if you're going to hand a tool this dangerous, you might as well as have people login to their dbs and edit stuff manually | |
| 21:17:13 | artom | Which *they do*. | |
| 21:17:13 | mnaser | but thats my drive-by 2 cents | |
| 21:17:19 | artom | Unfortunately. | |
| 21:17:37 | artom | And not only that, but we help them do it. | |
| 21:17:54 | mriedem | that kind of goes back to my original question - allowing only setting active or error in os-resetState - i wonder how many operators are manually changing status to something else to retry an operation | |
| 21:18:26 | mnaser | wait os-resetState lets you do more than active and error? | |
| 21:18:29 | mriedem | no | |
| 21:18:42 | mnaser | oo | |
| 21:18:47 | mriedem | in cinder it lets you reset volume status to just about any known volume status i think | |
| 21:18:53 | mnaser | yeah | |
| 21:19:05 | mnaser | tbh my workflow has always been "bring something back to as consistent of a state as it is" | |
| 21:19:16 | mnaser | and then use api afterwards | |
| 21:21:01 | artom | Whoa, "shards cells by hardware generation" is a thing? | |
| 21:22:54 | mriedem | you don't need to read the spec, i'm going to go back in my cave | |
| 21:23:45 | artom | mriedem, just curious, and procrastinating this downstream thing :) | |
| 21:23:57 | mriedem | feel free to read it if you want | |
| 21:24:08 | artom | Yes. I'm a free man in a free country. | |
| 21:24:08 | mriedem | it is quite masterful in it's word count | |
| 21:24:27 | mriedem | sorry *its i know efried is watching | |
| 21:24:43 | artom | Uh huh. "Masterful count" is exactly what I think of you ;) | |
| 21:29:37 | mriedem | i guess in the case of normal resize failures, there is stuff in the compute on restart that tries to clean that up, | |
| 21:29:51 | mriedem | which isn't as possible with cross-cell since it's orchestrated across 2 cells from the top | |
| 21:30:30 | mriedem | but on revert i think the idea is basically, best effort cleanup from the target and make sure the instance is mapped back to the source so if there is an error the user can work with it there on the source with the original flavor, since they were trying to revert in the first place | |
| 21:31:17 | artom | (This isn't really related or asked for, but I feel that if you're using cells to segregate hardware you're doing it wrong? Wouldn't that what hosts aggregates are for?) | |
| 21:32:22 | mriedem | what if you have 10K old hosts and 10K new hosts? | |
| 21:32:31 | mriedem | you want them all in the same cell with the same mq/db cluster? | |
| 21:32:36 | mriedem | but 2 aggregates? | |
| 21:32:46 | mriedem | you can certainly do aggregates, but cells is also for the scaling issue | |