| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 19:46:15 | mriedem | *cold | |
| 19:46:34 | mriedem | takashi has a spec for specifying a host for cold migratoin | |
| 19:46:41 | mriedem | it was only approved after i got him to remove the 'force' flag option | |
| 19:46:47 | mriedem | specifying a host is fine as long as the scheduler validates it first, | |
| 19:46:56 | efried | Oh, not the same spec as 'abort cold', the reviewing of which led me to this series of questions? | |
| 19:46:57 | mriedem | force flag bypasses the scheduler which gets all sorts of crazy | |
| 19:47:05 | mriedem | abort cold is different | |
| 19:47:19 | dansmith | efried: you still need the config option even if you only have one host | |
| 19:47:34 | mriedem | devstack defaults to set allow_resize_to_same_host=True | |
| 19:47:37 | mriedem | b/c it's single node | |
| 19:47:41 | efried | dansmith otherwise it'll just fail? | |
| 19:47:44 | mriedem | efried: yes | |
| 19:47:50 | efried | whee | |
| 19:48:49 | mriedem | because cold migrating to yourself doesn't make sense, | |
| 19:48:57 | mriedem | unless you're resizing the flavor | |
| 19:49:13 | mriedem | and if you're resizing to the same host, we quintuple your cpu allocations during scheduling just to make you pay for it | |
| 19:49:31 | efried | Well, to me "resize" means "change the size of". Not "move to another host". But what do I know. | |
| 19:49:50 | mriedem | hence the 'remember cold migrate is called resize internally | |
| 19:50:04 | efried | perfect. | |
| 19:50:07 | efried | Well, also in the docs. | |
| 19:50:08 | mriedem | and evacuate == rebuild | |
| 19:50:12 | efried | Which is why /me was confused. | |
| 19:50:17 | penick_ | dansmith You're probably right about the thundering herd being the problem. I think the SE team missed that these machines have CPU and memory utilization spiking. The memory spiked enough to hit swap. Also I see at least one panic in the log. | |
| 19:50:43 | dansmith | efried: have you read my blog post? | |
| 19:50:56 | efried | dansmith Which? | |
| 19:51:08 | dansmith | penick_: ack | |
| 19:51:09 | dansmith | efried: the secret decoder ring for all these move operations | |
| 19:51:14 | efried | dansmith I hope it's not too long. | |
| 19:51:32 | efried | dansmith But send me the link and I'll see if I can plough my way through it. | |
| 19:51:57 | efried | (In all seriousness, dansmith, I would love a decoder ring for the move ops) | |
| 19:52:07 | dansmith | oh it's long, real long.. | |
| 19:52:14 | dansmith | er, the post.. yeah, not too long | |
| 19:52:23 | dansmith | efried: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 19:52:30 | efried | IBM forbids me even responding to that. | |
| 19:53:05 | dansmith | ack | |
| 19:53:22 | dansmith | my corporate compliance training requirement is coming up soon | |
| 19:53:24 | dansmith | I better re-read | |
| 19:54:15 | openstackgerrit | Merged openstack/nova master: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/510324 | |
| 19:56:34 | ildikov | mriedem: not at the laptop now | |
| 19:56:54 | mriedem | ildikov: i'll update it | |
| 19:57:44 | ildikov | mriedem: thank you! I owe you, I guess way more than one!!! | |
| 19:58:08 | mriedem | get me out of work meetings and customer dinners in sydney and the debt i spaid | |
| 19:58:42 | efried | dansmith Super helpful article, thank you. | |
| 20:01:05 | penick_ | dansmith and melwitt thanks for the help, btw | |
| 20:01:32 | dansmith | penick_: no problem. Now I have to go shower the juno off me | |
| 20:02:10 | penick_ | Sorry about that. On the upside you don't need to bathe in the juno-ness all day every day. | |
| 20:04:19 | dansmith | sdague: we kinda lost momentum on mikal's privsepapalooza: https://review.openstack.org/#/c/494423/30 | |
| 20:08:08 | openstackgerrit | Merged openstack/nova master: Pick ironic nodes without VCPU set https://review.openstack.org/511844 | |
| 20:08:15 | openstackgerrit | Merged openstack/nova master: Add attachment_get to refresh_connection_info https://review.openstack.org/512626 | |
| 20:08:24 | openstackgerrit | Merged openstack/nova master: Update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 20:12:58 | mriedem | stvnoyes: ^ woot | |
| 20:14:20 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 20:14:42 | mriedem | johnthetubaguy: ^ updated the multiattach spec - i'll proxy your +2 if you're not around to handle it tonight | |
| 20:16:48 | mriedem | oh yeah today is the q-1 tag, suppose i should tag that | |
| 20:20:31 | cdent | mriedem, dansmith: since it is spec freezey day, and this spec is required by a spec already merged, you wanna drop some eyes on it? It was done as a seperate spec from the POST allocation (which is what requires it) because it’s a different microversion: https://review.openstack.org/#/c/508164/ please and thank you. | |
| 20:20:49 | cdent | It’s probably not strictly necessary that it merges as the work is going to be done anyway. It is mostly a bookkeeping thing I guess. | |
| 20:22:48 | mriedem | probably just gets a freeze exception given it's a dependency | |
| 20:23:24 | cdent | cool, thanks | |
| 20:58:37 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:10:04 | openstackgerrit | Merged openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 21:12:19 | openstackgerrit | Merged openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:12:29 | efried | mriedem I think you added me to https://review.openstack.org/#/c/334732/ to render opinion as a driver owner; did that. | |
| 21:13:27 | mriedem | efried: ok, takashi said it was for NFS in his use case too, and it takes "several minutes" for the disk to migrate, | |
| 21:13:40 | mriedem | which to me isn't a good use case for needing to add plumbing to abort the thing, | |
| 21:13:42 | efried | mriedem wha? How could it take minutes to "migrate" NFS? | |
| 21:13:47 | mriedem | if the admin is cold migrating the instance, it's for good reason | |
| 21:14:48 | mriedem | i think the live migrate cancel stuff was needed because live migrate could stall out and we needed a hook in to abort the job in the hypervisor | |
| 21:15:04 | efried | Okay, that's not what the spec says. | |
| 21:15:08 | mriedem | this is just like, whiny user says their cold migrating vm is down too long and tells the admin to stop doing that thing | |
| 21:15:23 | mriedem | right, this doesn't say anything about cold migrate stalling out and we need a way to kill it | |
| 21:15:28 | efried | The spec implies that we're copying gigabytes of local (*actual* local) disk over the wire. | |
| 21:15:36 | efried | ...and want to be able to bail on that. | |
| 21:15:38 | mriedem | this is saying, "admin is cold migrating, user doesn't like that it's taking several minutes, and wants the admin to stop' | |
| 21:16:23 | mriedem | idk maybe if there were actual numbers in here, | |
| 21:16:29 | mriedem | like, i've got a 200GB disk windows vm | |
| 21:16:36 | mriedem | and it's taking me 3 hours to cold migrate that thing | |
| 21:17:15 | mriedem | this is why i asked cburgess this morning b/c i know he uses ceph and i know he loves to migrate stuff | |
| 21:17:25 | cburgess | Oh sorry I missed something... | |
| 21:17:29 | cburgess | Do what now? | |
| 21:17:33 | stvnoyes | mridem. fantastic! :-) thanks for all the help from both you and johnthetubaguy! | |
| 21:18:05 | mriedem | cburgess: do you think https://review.openstack.org/#/c/334732/ is useful? | |
| 21:18:06 | mriedem | cburgess: do you have users with 100+ gb disks and if so, do you cold migrate those and if so, how long does that take? | |
| 21:18:53 | cburgess | mriedem Funny you should ask about 100+ GB disk... I'm just now starting at a 15TB one. | |
| 21:19:03 | cburgess | mriedem Give me a few to read this spec and I'll let you know. | |
| 21:19:09 | mriedem | thanks | |
| 21:21:01 | ildikov | stvnoyes: congrats!!! :) | |
| 21:37:15 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526 | |
| 21:37:31 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API https://review.openstack.org/507762 | |
| 21:41:12 | mriedem | jaypipes: addressed your comments and sylvain's in https://review.openstack.org/#/c/507762/ - left a todo for alex's thing about sort order which i think we could hash out in review later | |
| 21:42:45 | cdent | night all | |
| 21:50:40 | openstackgerrit | Merged openstack/nova master: [Trivial] Fix up a docstring https://review.openstack.org/513447 | |
| 22:03:11 | cburgess | mriedem So yeah thats useful. We tend to not recommend cold migrations with large amounts of data. Clients that use a lot of space we push towards a volume solution for the large storage and to either use BFV or to have small roots locally and put the bulk storage in volumes. | |
| 22:03:15 | cburgess | But its still a useful feature. | |
| 22:03:37 | mikal | Random fact for the day. Name the commit which added metadata to nova. | |
| 22:04:02 | cburgess | mikal: Which part of metadata? The service? | |
| 22:04:18 | mikal | There has always been a web service | |
| 22:04:23 | mriedem | cburgess: want to comment on the spec? operator input has weight on something like this. | |
| 22:04:28 | mikal | So, the commit which added any instance accessible metadata functionality | |
| 22:04:46 | cburgess | mriedem Sure | |
| 22:04:49 | mriedem | "copy what ec2 / aws does"? | |