| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-19 | |||
| 21:07:56 | mriedem | i wonder how many operators are trying to recover an instance from a failed resize confirm/revert where the instance is in ERROR status and they fixed something in the compute and just want to retry the confirm/revert operation, but can't b/c the API doesn't allow those operations on an instance in ERROR status. seems weird that the os-resetStatus server action API doesn't allow you to put an instance in 'resized' status agai | |
| 21:08:10 | mriedem | dansmith: ever thunk on that?^ | |
| 21:08:48 | dansmith | well, it would be pretty bad if you put it into resized status when it wasn't, | |
| 21:08:58 | dansmith | and it'd be hard to tell when and when not to do something like that | |
| 21:09:29 | mriedem | if you got through revert_resize on the dest but finish_revert_resize fails on the source, you can maybe hard reboot your way out of it | |
| 21:09:41 | openstackgerrit | Boris Bobrov proposed openstack/nova master: Don't reference unassigned variable in xml matchers in test https://review.openstack.org/637965 | |
| 21:09:42 | mriedem | if revert_resize on the dest fails, you're kind of stuck there now | |
| 21:09:48 | mriedem | anywho | |
| 21:10:06 | dansmith | right but is it realistic for an operator to know that it failed on finish_revert_resize? | |
| 21:10:09 | dansmith | even if they fix something, | |
| 21:10:28 | dansmith | they could reset into resized state from another place where they think it's likely to help and further break stuff | |
| 21:10:42 | dansmith | I mean, I know what you're saying, but.. it's a dangerous tool | |
| 21:10:56 | mriedem | sure, and i wouldn't expect most operators to know that unless they were pretty knowledgeable on low level nova | |
| 21:11:06 | mriedem | which their time is probably just better spent on rebuilding the vm | |
| 21:11:16 | artom | And, is it needed? I mean, if they have data they want to keep on a volume, they can just attach that to a new instance with the correct flavor? | |
| 21:11:18 | mriedem | and writing an apology to the user | |
| 21:11:51 | mriedem | artom: i'm thinking more about local storage, no volumes | |
| 21:12:08 | artom | So actually, is a resize destructive in that respect? | |
| 21:12:09 | mriedem | granted your data should be stored on a persistent volume | |
| 21:12:31 | mriedem | no, the local disks are left intact on the source host when you resize | |
| 21:12:46 | mriedem | they are destroyed when you confirm the resize to keep the instance on the dest host | |
| 21:12:53 | mriedem | and re-used on the source if you revert | |
| 21:13:48 | artom | And your idea is - if the resize went to ERROR, rather than just reverting back to the source, allow them to force completion, so to speak? | |
| 21:14:00 | mriedem | not force completion, retry | |
| 21:14:12 | mriedem | but, | |
| 21:14:20 | mriedem | that retry could fail if we got part way the first time | |
| 21:14:25 | mriedem | so retrying isn't without risk either | |
| 21:14:42 | mriedem | i.e. revert_resize on dest fails the 2nd time b/c we already deleted some stuff the first time | |
| 21:15:01 | artom | Yeah... | |
| 21:15:31 | artom | In that situation, what's wrong with going through the resize again, after reverting the failed attempt? | |
| 21:15:33 | mriedem | i've just got some sticky logic in the cross-cell revert code where if something fails part way through it's not so good | |
| 21:15:54 | mriedem | "reverting the failed attempt"? | |
| 21:16:02 | mriedem | s/reverting/fixing/? | |
| 21:16:23 | mnaser | maybe in an ideal world we can collect more of the cases where finish_revert_resize fails and test those out *before* starting to revert the resize | |
| 21:16:24 | artom | Oh. You can't revert out of an ERROR, can you? | |
| 21:16:25 | mriedem | you could reset status to active and try another resize | |
| 21:16:41 | mriedem | no you can only confirm/revert a server in 'resized' status | |
| 21:16:43 | mnaser | no you cant revert out of ERROR, only RESIZE_CONFIRM i think | |
| 21:16:49 | mriedem | you can hard reboot / rebuild a server in ERROR | |
| 21:17:03 | mnaser | tbh if you're going to hand a tool this dangerous, you might as well as have people login to their dbs and edit stuff manually | |
| 21:17:13 | artom | Which *they do*. | |
| 21:17:13 | mnaser | but thats my drive-by 2 cents | |
| 21:17:19 | artom | Unfortunately. | |
| 21:17:37 | artom | And not only that, but we help them do it. | |
| 21:17:54 | mriedem | that kind of goes back to my original question - allowing only setting active or error in os-resetState - i wonder how many operators are manually changing status to something else to retry an operation | |
| 21:18:26 | mnaser | wait os-resetState lets you do more than active and error? | |
| 21:18:29 | mriedem | no | |
| 21:18:42 | mnaser | oo | |
| 21:18:47 | mriedem | in cinder it lets you reset volume status to just about any known volume status i think | |
| 21:18:53 | mnaser | yeah | |
| 21:19:05 | mnaser | tbh my workflow has always been "bring something back to as consistent of a state as it is" | |
| 21:19:16 | mnaser | and then use api afterwards | |
| 21:21:01 | artom | Whoa, "shards cells by hardware generation" is a thing? | |
| 21:22:54 | mriedem | you don't need to read the spec, i'm going to go back in my cave | |
| 21:23:45 | artom | mriedem, just curious, and procrastinating this downstream thing :) | |
| 21:23:57 | mriedem | feel free to read it if you want | |
| 21:24:08 | artom | Yes. I'm a free man in a free country. | |
| 21:24:08 | mriedem | it is quite masterful in it's word count | |
| 21:24:27 | mriedem | sorry *its i know efried is watching | |
| 21:24:43 | artom | Uh huh. "Masterful count" is exactly what I think of you ;) | |
| 21:29:37 | mriedem | i guess in the case of normal resize failures, there is stuff in the compute on restart that tries to clean that up, | |
| 21:29:51 | mriedem | which isn't as possible with cross-cell since it's orchestrated across 2 cells from the top | |
| 21:30:30 | mriedem | but on revert i think the idea is basically, best effort cleanup from the target and make sure the instance is mapped back to the source so if there is an error the user can work with it there on the source with the original flavor, since they were trying to revert in the first place | |
| 21:31:17 | artom | (This isn't really related or asked for, but I feel that if you're using cells to segregate hardware you're doing it wrong? Wouldn't that what hosts aggregates are for?) | |
| 21:32:22 | mriedem | what if you have 10K old hosts and 10K new hosts? | |
| 21:32:31 | mriedem | you want them all in the same cell with the same mq/db cluster? | |
| 21:32:36 | mriedem | but 2 aggregates? | |
| 21:32:46 | mriedem | you can certainly do aggregates, but cells is also for the scaling issue | |
| 21:33:19 | mriedem | cern also segregates by hardware type i believe, | |
| 21:33:23 | mriedem | and by vm/bm flavors | |
| 21:33:33 | mriedem | lots of ways to slice it up | |
| 21:33:48 | artom | Yeah, I'd understood that cells sharding was for scale | |
| 21:34:57 | artom | So if you want to scale different hardware generations/types, it'd be cells for scale, and then in each cell aggregates of hardware | |
| 21:36:00 | artom | So in your 10K example, it'd be 1K or whatever per cell, and 500/500 old/new hardware | |
| 21:36:19 | artom | But I guess that ignore the flow of time | |
| 21:36:49 | artom | You'll have set up your cells with old hardware, not necessarily leaving wiggling room to add new hardware | |
| 21:37:13 | artom | Man, I'm really good at this procrastinating thing | |
| 22:30:12 | openstackgerrit | sean mooney proposed openstack/os-vif master: remove brctl from vif_plug_ovs https://review.openstack.org/636821 | |
| 22:30:13 | openstackgerrit | sean mooney proposed openstack/os-vif master: remove use of brctl from vif_plug_linux_bridge https://review.openstack.org/636822 | |
| 22:41:56 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: De-shim PlacementFixture https://review.openstack.org/637959 | |
| 23:38:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | |
| 23:38:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | |
| 23:38:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | |
| 23:38:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | |
| 23:38:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | |
| 23:38:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | |
| 23:38:45 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add PrepResizeAtDestTask https://review.openstack.org/627890 | |
| 23:38:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | |
| 23:38:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | |
| 23:38:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.openstack.org/635646 | |
| 23:38:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add finish_snapshot_based_resize_at_dest compute method https://review.openstack.org/635080 | |
| 23:38:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.openstack.org/635684 | |
| 23:38:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.openstack.org/635668 | |
| 23:38:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.openstack.org/636253 | |
| 23:38:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Filter duplicates from compute API get_migrations_sorted() https://review.openstack.org/636224 | |
| 23:38:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.openstack.org/637070 | |
| 23:38:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source https://review.openstack.org/637058 | |
| 23:38:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize conductor RPC method https://review.openstack.org/637075 | |
| #openstack-nova - 2019-02-20 | |||
| 00:58:50 | openstackgerrit | sean mooney proposed openstack/os-vif master: make functional tests run on python 3 https://review.openstack.org/638053 | |
| 01:01:34 | openstackgerrit | sean mooney proposed openstack/os-vif master: make functional tests run on python 3 https://review.openstack.org/638053 | |