| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 20:56:16 | dansmith | but we love pets around here, so.. | |
| 20:56:45 | melwitt | fwiw, from a user perspective I would have expected a graceful delete by default. if anything in there was mounted | |
| 20:57:02 | cfriesen | heh...honestly if we're not going to default to soft, then I don't see much gain since we can just stop it first | |
| 20:57:24 | mriedem | until you forget to do soft first, | |
| 20:57:29 | mriedem | or didn't know you needed to, | |
| 20:57:35 | mriedem | and now your attached volume is corrupted | |
| 20:57:39 | mriedem | and you curse openstack and use aws | |
| 20:57:49 | cfriesen | yeah, it's a bit gross that attached cinder volumes get corrupted | |
| 20:58:00 | mriedem | s/soft/stop/ | |
| 20:58:13 | dansmith | to be clear, I think it's only application-level state, | |
| 20:58:20 | melwitt | I guess that means it doesn't detach them first before destroying | |
| 20:58:37 | dansmith | libvirt will still honor the write semantics of the block device I think | |
| 20:58:42 | dansmith | so the filesystem should be okay | |
| 20:58:53 | cfriesen | anything cached in the guest may get lost | |
| 20:58:56 | smcginnis | Depends on what IOs are in flight I think. | |
| 20:59:02 | dansmith | cfriesen: just like a power outage | |
| 20:59:03 | cburgess | dansmith That depends on how you mount the FS inside the VM really. | |
| 20:59:04 | cfriesen | but that should be okay assuming proper locking | |
| 20:59:12 | dansmith | cburgess: you mean "safely" or not? :P | |
| 20:59:17 | smcginnis | Unless the OS knows to flush its IO cache, filesystem or anything could get messed up. | |
| 20:59:23 | cburgess | dansmith Yes. Its stupid.. but it does happen | |
| 20:59:45 | cburgess | I feel like a best effort to gracefully stop first is a good idea. But I agree its probably a blueprint. | |
| 20:59:48 | dansmith | smcginnis: the only way to safely do it at all is with libvirt honoring disk-like cache semantics and using a proper filesystem config so that works anyway | |
| 20:59:53 | dansmith | smcginnis: otherwise you're running raid with memory cache and no battery | |
| 21:00:22 | mriedem | melwitt: we destroy the guest before we detach the volumes in _shutdown_instance | |
| 21:00:26 | dansmith | cburgess: it's just a change in behavior from right now, whereas we could add it for people that know, is all | |
| 21:00:37 | melwitt | mriedem: a-ha, okay | |
| 21:00:52 | dansmith | cburgess: anyone wondering why the volume isn't becoming available for a minute will have to realize we changed | |
| 21:00:55 | cburgess | dansmith Well I agree its a change which is why it should be a blueprint for P+? | |
| 21:01:02 | dansmith | cburgess: but agree it probably should have been this way from the beginning | |
| 21:01:14 | cburgess | dansmith Can't we just handle that with release notes? | |
| 21:01:14 | dansmith | cburgess: it should be a blueprint regardless of the default, IMHO | |
| 21:01:22 | dansmith | cburgess: lol | |
| 21:01:28 | cburgess | I mean we can add a flag I guess. | |
| 21:01:35 | cburgess | I know sdague will kill me for suggesting a flag. | |
| 21:01:37 | mriedem | no config options | |
| 21:01:44 | cburgess | Apparently mriedem too | |
| 21:01:47 | cburgess | :P | |
| 21:01:59 | dansmith | I'm not -2 on changing the default, I'm just not as up in arms about the current default | |
| 21:02:05 | dansmith | but I treat my data nicely, so | |
| 21:02:35 | melwitt | graceful destroy by default seems reasonable to me. I don't think we need a config option | |
| 21:03:19 | sdague | I think we've been back and forth across that line a bunch in the past | |
| 21:03:21 | cburgess | I agree it *should* be the default but if there is complaint I feel a config option is reasonable. Then again config options are so out of fashion now. | |
| 21:03:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: check if we can quiesce before volume-backed snapshot https://review.openstack.org/380433 | |
| 21:03:57 | dansmith | cburgess: were definitely not doing a config option :) | |
| 21:04:07 | sdague | one of the challenges with graceful shutdown by default is that you end up with a lot of cloud guests that don't have acpi actually working | |
| 21:04:15 | dansmith | right | |
| 21:04:20 | sdague | so it turns into add a 2 minute timeout to all deletes | |
| 21:04:54 | dansmith | well, four :) | |
| 21:04:55 | sdague | dansmith: oh, sorry, I only caught from cburgess yelling at me | |
| 21:05:09 | cburgess | sdague I didn't yell at you.. said you would yell at me. | |
| 21:06:18 | dansmith | with bfv, soft delete, and graceful shutdown before delete, we've really watered down the meaning of "destroy" | |
| 21:06:46 | cburgess | dansmith New call? New option to the call? destroy vs hard destroy? destroy + graceful? | |
| 21:07:12 | sdague | dansmith: well, we've given it a distinctly british flavour. please sir, may you stop now. | |
| 21:07:25 | dansmith | cburgess: well, my preference (after just telling people to shutdown their instances first) would be a "delete with flag for shut-me-down-first" | |
| 21:07:27 | cburgess | LOL | |
| 21:07:47 | dansmith | sdague: we should start apologizing to the users | |
| 21:07:49 | dansmith | 499: Sooory | |
| 21:07:54 | sdague | dansmith: ++ | |
| 21:07:56 | melwitt | the api doesn't call it "destroy" though. it's called "delete" which doesn't seem to imply anything to me | |
| 21:07:58 | cburgess | dansmith Thats not unreasonable either. New micro-version that adds an option graceful flag? | |
| 21:08:14 | dansmith | cburgess: right | |
| 21:08:14 | cburgess | optional graceful flag even | |
| 21:08:19 | melwitt | but, if the timeout is really 2 minutes then I could see the --graceful option being an option | |
| 21:08:45 | cburgess | OK well sorry to leave in the middle of this but wife just called to say the hot water heater is making a very loud banging noise continually.. so... yeah... thats fun. | |
| 21:09:08 | dansmith | melwitt: to me "delete" does not mean "don't really delete", but.. we lost the actually-deleting-things battle a while ago :P | |
| 21:09:36 | sdague | the main reason I think a lot of people want this is windows shutting down, and you get the "I'm applying patches don't turn me off" screen | |
| 21:09:51 | sdague | melwitt: so the timeout has to be at least 2 minutes so you don't kill that | |
| 21:09:58 | dansmith | oh forgot about that, that's a good one | |
| 21:10:00 | melwitt | dansmith: heh, I guess. as a layperson, I didn't think of "delete" as meaning much other than "get rid of this instance" and not thinking about what happens underneath | |
| 21:10:45 | mriedem | i think of delete as 'get rid of this instance' but also, don't mess up all my crap that's attached to it unless you're also deleting that, like volumes and ports | |
| 21:10:47 | cfriesen | isn't the default shutdown timeout 60 sec? | |
| 21:10:53 | melwitt | 2 minutes for windows update, that's generous (to think it would be that short) | |
| 21:11:06 | dansmith | melwitt: I know, 2 minutes is laughable | |
| 21:11:08 | sdague | melwitt: once upon a time there was 120 timer here | |
| 21:11:27 | dansmith | mriedem: "mess up my crap" is relative though. it won't corrupt the filesystem | |
| 21:11:30 | sdague | and it did do all sorts of oddity in the gate, because cirros totally ignores acpi | |
| 21:11:38 | sdague | so we were hanging around forever | |
| 21:11:49 | dansmith | mriedem: if your app corrupts your data on this event, then it also corrupts your data if the power goes out in the datacenter, so.. meh | |
| 21:11:52 | mriedem | i vaguely remember that, | |
| 21:12:00 | mriedem | didn't we revert Phil's like 1K LOC patch or somethign? | |
| 21:12:03 | melwitt | if it were called 'nova pull-the-power-cord' that would be clear to me :P | |
| 21:12:11 | mriedem | dansmith: true | |
| 21:12:22 | sdague | mriedem: maybe, I don't remember exactly how long it was around | |
| 21:12:32 | dansmith | sdague: ah, that's what I was thinking of | |
| 21:12:34 | melwitt | yeah, I can see both arguments | |
| 21:12:59 | mriedem | so at what point so volume multi-attach fix all of this? :) | |
| 21:13:05 | mriedem | s/so/does/ | |
| 21:13:35 | sdague | so, if it was a thing... (which I'm still not convinced it should be) it definitely doesn't seem like a system wide answer is the right answer | |
| 21:13:48 | dansmith | mriedem: volume multiattach asumes you don't have an app that would corrupt things like this, so that's a good argument for not caring IMHO :) | |
| 21:13:54 | sdague | it would be image based or instance based metadata for shutdown time | |
| 21:14:15 | sdague | so windows people could tune it up to 20 minutes for snowflakes | |
| 21:14:17 | dansmith | image.I_suck=True | |
| 21:14:17 | cfriesen | sdague: you mean like os_shutdown_timeout ? | |
| 21:14:33 | sdague | cfriesen: maybe, that already in there? | |
| 21:14:40 | cfriesen | sdague: yep | |
| 21:14:41 | sdague | that vaguely sounds familiar | |
| 21:14:43 | dansmith | lol | |
| 21:14:48 | sdague | ok, probablem solved! | |
| 21:14:58 | dansmith | unfuckingbelievable | |