| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 19:43:09 | dansmith | efried: or nova migrate | |
| 19:43:35 | efried | dansmith thx | |
| 19:43:55 | mriedem | ildikov: easy fixes in https://review.openstack.org/#/c/499777/ - you want to update quick? otherwise i can | |
| 19:44:18 | efried | how do you tell it to resize to another host? | |
| 19:44:44 | melwitt | resize automatically goes to another host | |
| 19:45:00 | dansmith | well | |
| 19:45:05 | mriedem | efried: only way it can go back is the config option | |
| 19:45:07 | dansmith | unless configured | |
| 19:45:09 | mriedem | allow_resize_to_same_host | |
| 19:45:09 | dansmith | yeah | |
| 19:45:14 | melwitt | unless you have configured "allow_resize_to_same_host" it won't consider the same host | |
| 19:45:23 | efried | whoah, okay. | |
| 19:45:34 | melwitt | note that means it CAN go to the same host if you config that, not that it will ONLY go there | |
| 19:45:50 | efried | And if you only have one host, presumably, you don't need to switch the setting on? | |
| 19:45:52 | melwitt | some ppl get confused by it | |
| 19:45:53 | mriedem | efried: remember that cold migrate is called 'resize' in the code, | |
| 19:45:57 | mriedem | just like evacuate is called rebuild | |
| 19:46:02 | mriedem | totally not confusing | |
| 19:46:08 | cdent | efried: force resize to same has been discussed at various times and I’m sure patched in by vairous people | |
| 19:46:10 | mriedem | and migrate != live migration | |
| 19:46:11 | efried | "remember", sure. | |
| 19:46:14 | mriedem | because if it's not live, it's old | |
| 19:46:15 | mriedem | *cold | |
| 19:46:34 | mriedem | takashi has a spec for specifying a host for cold migratoin | |
| 19:46:41 | mriedem | it was only approved after i got him to remove the 'force' flag option | |
| 19:46:47 | mriedem | specifying a host is fine as long as the scheduler validates it first, | |
| 19:46:56 | efried | Oh, not the same spec as 'abort cold', the reviewing of which led me to this series of questions? | |
| 19:46:57 | mriedem | force flag bypasses the scheduler which gets all sorts of crazy | |
| 19:47:05 | mriedem | abort cold is different | |
| 19:47:19 | dansmith | efried: you still need the config option even if you only have one host | |
| 19:47:34 | mriedem | devstack defaults to set allow_resize_to_same_host=True | |
| 19:47:37 | mriedem | b/c it's single node | |
| 19:47:41 | efried | dansmith otherwise it'll just fail? | |
| 19:47:44 | mriedem | efried: yes | |
| 19:47:50 | efried | whee | |
| 19:48:49 | mriedem | because cold migrating to yourself doesn't make sense, | |
| 19:48:57 | mriedem | unless you're resizing the flavor | |
| 19:49:13 | mriedem | and if you're resizing to the same host, we quintuple your cpu allocations during scheduling just to make you pay for it | |
| 19:49:31 | efried | Well, to me "resize" means "change the size of". Not "move to another host". But what do I know. | |
| 19:49:50 | mriedem | hence the 'remember cold migrate is called resize internally | |
| 19:50:04 | efried | perfect. | |
| 19:50:07 | efried | Well, also in the docs. | |
| 19:50:08 | mriedem | and evacuate == rebuild | |
| 19:50:12 | efried | Which is why /me was confused. | |
| 19:50:17 | penick_ | dansmith You're probably right about the thundering herd being the problem. I think the SE team missed that these machines have CPU and memory utilization spiking. The memory spiked enough to hit swap. Also I see at least one panic in the log. | |
| 19:50:43 | dansmith | efried: have you read my blog post? | |
| 19:50:56 | efried | dansmith Which? | |
| 19:51:08 | dansmith | penick_: ack | |
| 19:51:09 | dansmith | efried: the secret decoder ring for all these move operations | |
| 19:51:14 | efried | dansmith I hope it's not too long. | |
| 19:51:32 | efried | dansmith But send me the link and I'll see if I can plough my way through it. | |
| 19:51:57 | efried | (In all seriousness, dansmith, I would love a decoder ring for the move ops) | |
| 19:52:07 | dansmith | oh it's long, real long.. | |
| 19:52:14 | dansmith | er, the post.. yeah, not too long | |
| 19:52:23 | dansmith | efried: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 19:52:30 | efried | IBM forbids me even responding to that. | |
| 19:53:05 | dansmith | ack | |
| 19:53:22 | dansmith | my corporate compliance training requirement is coming up soon | |
| 19:53:24 | dansmith | I better re-read | |
| 19:54:15 | openstackgerrit | Merged openstack/nova master: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/510324 | |
| 19:56:34 | ildikov | mriedem: not at the laptop now | |
| 19:56:54 | mriedem | ildikov: i'll update it | |
| 19:57:44 | ildikov | mriedem: thank you! I owe you, I guess way more than one!!! | |
| 19:58:08 | mriedem | get me out of work meetings and customer dinners in sydney and the debt i spaid | |
| 19:58:42 | efried | dansmith Super helpful article, thank you. | |
| 20:01:05 | penick_ | dansmith and melwitt thanks for the help, btw | |
| 20:01:32 | dansmith | penick_: no problem. Now I have to go shower the juno off me | |
| 20:02:10 | penick_ | Sorry about that. On the upside you don't need to bathe in the juno-ness all day every day. | |
| 20:04:19 | dansmith | sdague: we kinda lost momentum on mikal's privsepapalooza: https://review.openstack.org/#/c/494423/30 | |
| 20:08:08 | openstackgerrit | Merged openstack/nova master: Pick ironic nodes without VCPU set https://review.openstack.org/511844 | |
| 20:08:15 | openstackgerrit | Merged openstack/nova master: Add attachment_get to refresh_connection_info https://review.openstack.org/512626 | |
| 20:08:24 | openstackgerrit | Merged openstack/nova master: Update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 20:12:58 | mriedem | stvnoyes: ^ woot | |
| 20:14:20 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 20:14:42 | mriedem | johnthetubaguy: ^ updated the multiattach spec - i'll proxy your +2 if you're not around to handle it tonight | |
| 20:16:48 | mriedem | oh yeah today is the q-1 tag, suppose i should tag that | |
| 20:20:31 | cdent | mriedem, dansmith: since it is spec freezey day, and this spec is required by a spec already merged, you wanna drop some eyes on it? It was done as a seperate spec from the POST allocation (which is what requires it) because it’s a different microversion: https://review.openstack.org/#/c/508164/ please and thank you. | |
| 20:20:49 | cdent | It’s probably not strictly necessary that it merges as the work is going to be done anyway. It is mostly a bookkeeping thing I guess. | |
| 20:22:48 | mriedem | probably just gets a freeze exception given it's a dependency | |
| 20:23:24 | cdent | cool, thanks | |
| 20:58:37 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:10:04 | openstackgerrit | Merged openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 21:12:19 | openstackgerrit | Merged openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:12:29 | efried | mriedem I think you added me to https://review.openstack.org/#/c/334732/ to render opinion as a driver owner; did that. | |
| 21:13:27 | mriedem | efried: ok, takashi said it was for NFS in his use case too, and it takes "several minutes" for the disk to migrate, | |
| 21:13:40 | mriedem | which to me isn't a good use case for needing to add plumbing to abort the thing, | |
| 21:13:42 | efried | mriedem wha? How could it take minutes to "migrate" NFS? | |
| 21:13:47 | mriedem | if the admin is cold migrating the instance, it's for good reason | |
| 21:14:48 | mriedem | i think the live migrate cancel stuff was needed because live migrate could stall out and we needed a hook in to abort the job in the hypervisor | |
| 21:15:04 | efried | Okay, that's not what the spec says. | |
| 21:15:08 | mriedem | this is just like, whiny user says their cold migrating vm is down too long and tells the admin to stop doing that thing | |
| 21:15:23 | mriedem | right, this doesn't say anything about cold migrate stalling out and we need a way to kill it | |
| 21:15:28 | efried | The spec implies that we're copying gigabytes of local (*actual* local) disk over the wire. | |
| 21:15:36 | efried | ...and want to be able to bail on that. | |
| 21:15:38 | mriedem | this is saying, "admin is cold migrating, user doesn't like that it's taking several minutes, and wants the admin to stop' | |
| 21:16:23 | mriedem | idk maybe if there were actual numbers in here, | |
| 21:16:29 | mriedem | like, i've got a 200GB disk windows vm | |
| 21:16:36 | mriedem | and it's taking me 3 hours to cold migrate that thing | |
| 21:17:15 | mriedem | this is why i asked cburgess this morning b/c i know he uses ceph and i know he loves to migrate stuff | |
| 21:17:25 | cburgess | Oh sorry I missed something... | |