| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-08 | |||
| 20:10:18 | mriedem | *they're | |
| 20:10:24 | tbachman | baby got (diarrhea) black | |
| 20:10:34 | mriedem | Meconium | |
| 20:10:52 | artom | mriedem, that's technically meconium, not shit </pedant> | |
| 20:10:59 | mriedem | i'd like to start a new openstack big tent project and call it Meconium | |
| 20:11:06 | mriedem | artom: meh | |
| 20:11:10 | artom | Bah, and there you go jinxing me while I look up the damn spelling | |
| 20:11:47 | dansmith | that is .. not a thing I needed to know | |
| 20:11:57 | artom | mriedem, the first release will change the world, but after that it's all shit? | |
| 20:12:06 | mriedem | artom: heh | |
| 20:12:08 | cdent | artom++ | |
| 20:14:13 | mriedem | dansmith: jaypipes: also, fyi, i emailed the foundation people back about this boston ocata recap session asking if it was required or "optional" - so far no reply | |
| 20:14:27 | mriedem | probably trying to figure out the best way to say yes it's optional but not really | |
| 20:14:30 | jaypipes | mriedem: kk | |
| 20:15:05 | dansmith | it's probably "optional" but not optional | |
| 20:15:38 | jaypipes | dansmith: you know, like MySQL in OpenStack. | |
| 20:15:50 | dansmith | jaypipes: and support for bfv instances in nova | |
| 20:19:29 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: api-ref: Add tags to server detail response https://review.openstack.org/431149 | |
| 20:31:36 | cfriesen | if I do a "nova delete" on an instance with a mounted cinder volume, is it guaranteed to do a clean shutdown or do I need to do a "nova stop" first to avoid potential disk corruption? | |
| 20:32:05 | mriedem | cfriesen: you shouldn't need to stop a vm before deleting it | |
| 20:32:48 | sdague | mriedem: clean shutdown I think was the critical bit | |
| 20:33:02 | mriedem | hey, i didn't say it "worked" | |
| 20:40:05 | cfriesen | mriedem: sdague: looking at the code, the "nova delete" path calls guest.poweroff(), which calls self._domain.destroy(). The libvirt docs say it sends a SIGTERM then after a "reasonable timeout" a SIGKILL, but that timeout appears to be really small. | |
| 20:41:55 | cfriesen | okay, the libvirt docs describe a virDomainDestroy as the equivalent of having the virtual power cord pulled, so not exactly safe | |
| 20:42:27 | melwitt | cfriesen: I just got to the same conclusion. apparently the VIR_DOMAIN_DESTROY_GRACEFUL flag should be used with destroyFlags to avoid possible corruption. but in that case, it could return an error instead of doing the destroy | |
| 20:42:46 | cburgess | cfriesen Yeah I would say if the concern is data integrity on the volume, you should do a stop first. | |
| 20:44:30 | mriedem | https://www.redhat.com/archives/libvir-list/2012-February/msg00124.html | |
| 20:44:54 | openstackgerrit | Maciej Szankin proposed openstack/nova master: WIP conf: remove invalid TODOs https://review.openstack.org/428811 | |
| 20:45:00 | dansmith | I really thought we did a graceful shutdown even on delete | |
| 20:45:04 | mriedem | "Once this patch is in, the recommended method for applications to call virDomainDestroyFlags will be with VIR_DOMAIN_DESTROY_GRACEFUL included. If that fails, then the application can decide if and when to call virDomainDestroyFlags again without VIR_DOMAIN_DESTROY_GRACEFUL (to force the issue with SIGKILL)." | |
| 20:45:32 | mriedem | so, if we have attached volumes, do a graceful destroy, else we don't care? | |
| 20:45:38 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: api-ref: Add tags to server(s) detail response https://review.openstack.org/431149 | |
| 20:45:49 | cburgess | dansmith Well we do... but its just not very graceful | |
| 20:45:56 | gatuus | virsh stop vs virsh destroy | |
| 20:46:08 | dansmith | cburgess: I'm talking about an acpi event | |
| 20:46:10 | cburgess | dansmith ahh oh no sorry according to cfriesen we don't my bad. | |
| 20:46:35 | cburgess | dansmith Oh... no I think thats only on stop. I suppose we could call stop first from destroy if the power state is not already stopped or errored. | |
| 20:46:41 | cburgess | mriedem I think ^^ is a better idea. | |
| 20:47:03 | dansmith | I know it was part of phil's original blueprint, but maybe it never got implemented | |
| 20:47:05 | dansmith | I know we do on stop | |
| 20:47:08 | cburgess | The stop logic today as I recall (and reboot) is to send the ACPI event, wait some period of time, and then go hard kill. | |
| 20:47:10 | dansmith | and didn't originally | |
| 20:47:18 | cfriesen | mriedem: yeah, I think that'd make sense to use VIR_DOMAIN_DESTROY_GRACEFUL if we have attached volumes. | |
| 20:47:44 | cburgess | cfriesen Why not call stop internall? That already has logic to attempt to gracefully stop and if that fails hard kill. Then do the cleanup in destroy. | |
| 20:47:54 | cfriesen | dansmith: on stop we currently use virDomainShutdown first, then virDomainDestroy | |
| 20:48:23 | dansmith | cfriesen: I know | |
| 20:48:31 | cfriesen | cburgess: there's no need for a graceful stop if there are no attached volumes | |
| 20:48:41 | gatuus | even virsh will not allow such a corruption | |
| 20:48:53 | cfriesen | cburgess: but yeah, we could make it do a shutdown the same way we do for "nova stop" | |
| 20:48:57 | cburgess | cfriesen I agree but again rather then having different logic for being graceful, why not simple say if volumes, call stop, then destroy. | |
| 20:49:19 | cburgess | cfriesen Right. Then we don't have to implement a 2nd 'be graceful' routine. | |
| 20:50:04 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: api-ref: Add tags to server(s) detail response https://review.openstack.org/431149 | |
| 20:50:26 | cfriesen | okay....does this need a spec, just a blueprint, or is it a bugfix? | |
| 20:50:48 | dansmith | it's not just a bugfix, IMHO | |
| 20:51:00 | dansmith | and I'm not sure we can call stop without some real refactoring | |
| 20:51:13 | cburgess | That might true | |
| 20:51:24 | cburgess | Some of the logic might need to come out into its own function that can be shared with destroy | |
| 20:51:50 | melwitt | does mounting a volume in the vm mean it has to be attached from nova's perspective? like mounting of a shared drive? | |
| 20:52:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: api-ref: Add tags to server(s) detail response https://review.openstack.org/431149 | |
| 20:52:31 | cfriesen | melwitt: you're thinking some non-cinder volume? | |
| 20:52:44 | melwitt | cfriesen: yeah | |
| 20:53:02 | cfriesen | melwitt: right, for something like that you'd want a clean shutdown even if there aren't any cinder volumes attached. | |
| 20:53:10 | melwitt | I was thinking, is there a case where checking nova for "if volumes" isn't enough and if we should just do the graceful destroy regardless | |
| 20:53:31 | dansmith | if a cinder volume is attached from nova's perspective, we have to assume it's mounted or dirty | |
| 20:54:09 | cfriesen | melwitt: at the same time I could see a useful purpose for a "just kill it dead instantly" command | |
| 20:54:36 | dansmith | right, I mean if we care about the data, we have to assume the above | |
| 20:54:46 | dansmith | if we're told not to, then we could kill it outright | |
| 20:55:07 | cfriesen | do we need a "nova delete --hard" option? (or a --soft and default to hard)? | |
| 20:55:37 | dansmith | well, delete is kinda always documented as hard | |
| 20:55:56 | dansmith | the most compatible thing would be a delete --soft, IMHO | |
| 20:56:01 | dansmith | or just stopping the instance yourself first | |
| 20:56:16 | dansmith | but we love pets around here, so.. | |
| 20:56:45 | melwitt | fwiw, from a user perspective I would have expected a graceful delete by default. if anything in there was mounted | |
| 20:57:02 | cfriesen | heh...honestly if we're not going to default to soft, then I don't see much gain since we can just stop it first | |
| 20:57:24 | mriedem | until you forget to do soft first, | |
| 20:57:29 | mriedem | or didn't know you needed to, | |
| 20:57:35 | mriedem | and now your attached volume is corrupted | |
| 20:57:39 | mriedem | and you curse openstack and use aws | |
| 20:57:49 | cfriesen | yeah, it's a bit gross that attached cinder volumes get corrupted | |
| 20:58:00 | mriedem | s/soft/stop/ | |
| 20:58:13 | dansmith | to be clear, I think it's only application-level state, | |
| 20:58:20 | melwitt | I guess that means it doesn't detach them first before destroying | |
| 20:58:37 | dansmith | libvirt will still honor the write semantics of the block device I think | |
| 20:58:42 | dansmith | so the filesystem should be okay | |
| 20:58:53 | cfriesen | anything cached in the guest may get lost | |
| 20:58:56 | smcginnis | Depends on what IOs are in flight I think. | |
| 20:59:02 | dansmith | cfriesen: just like a power outage | |
| 20:59:03 | cburgess | dansmith That depends on how you mount the FS inside the VM really. | |
| 20:59:04 | cfriesen | but that should be okay assuming proper locking | |
| 20:59:12 | dansmith | cburgess: you mean "safely" or not? :P | |
| 20:59:17 | smcginnis | Unless the OS knows to flush its IO cache, filesystem or anything could get messed up. | |
| 20:59:23 | cburgess | dansmith Yes. Its stupid.. but it does happen | |
| 20:59:45 | cburgess | I feel like a best effort to gracefully stop first is a good idea. But I agree its probably a blueprint. | |
| 20:59:48 | dansmith | smcginnis: the only way to safely do it at all is with libvirt honoring disk-like cache semantics and using a proper filesystem config so that works anyway | |
| 20:59:53 | dansmith | smcginnis: otherwise you're running raid with memory cache and no battery | |
| 21:00:22 | mriedem | melwitt: we destroy the guest before we detach the volumes in _shutdown_instance | |
| 21:00:26 | dansmith | cburgess: it's just a change in behavior from right now, whereas we could add it for people that know, is all | |
| 21:00:37 | melwitt | mriedem: a-ha, okay | |
| 21:00:52 | dansmith | cburgess: anyone wondering why the volume isn't becoming available for a minute will have to realize we changed | |
| 21:00:55 | cburgess | dansmith Well I agree its a change which is why it should be a blueprint for P+? | |