| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-19 | |||
| 19:50:56 | efried | dansmith Which? | |
| 19:51:08 | dansmith | penick_: ack | |
| 19:51:09 | dansmith | efried: the secret decoder ring for all these move operations | |
| 19:51:14 | efried | dansmith I hope it's not too long. | |
| 19:51:32 | efried | dansmith But send me the link and I'll see if I can plough my way through it. | |
| 19:51:57 | efried | (In all seriousness, dansmith, I would love a decoder ring for the move ops) | |
| 19:52:07 | dansmith | oh it's long, real long.. | |
| 19:52:14 | dansmith | er, the post.. yeah, not too long | |
| 19:52:23 | dansmith | efried: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 19:52:30 | efried | IBM forbids me even responding to that. | |
| 19:53:05 | dansmith | ack | |
| 19:53:22 | dansmith | my corporate compliance training requirement is coming up soon | |
| 19:53:24 | dansmith | I better re-read | |
| 19:54:15 | openstackgerrit | Merged openstack/nova master: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/510324 | |
| 19:56:34 | ildikov | mriedem: not at the laptop now | |
| 19:56:54 | mriedem | ildikov: i'll update it | |
| 19:57:44 | ildikov | mriedem: thank you! I owe you, I guess way more than one!!! | |
| 19:58:08 | mriedem | get me out of work meetings and customer dinners in sydney and the debt i spaid | |
| 19:58:42 | efried | dansmith Super helpful article, thank you. | |
| 20:01:05 | penick_ | dansmith and melwitt thanks for the help, btw | |
| 20:01:32 | dansmith | penick_: no problem. Now I have to go shower the juno off me | |
| 20:02:10 | penick_ | Sorry about that. On the upside you don't need to bathe in the juno-ness all day every day. | |
| 20:04:19 | dansmith | sdague: we kinda lost momentum on mikal's privsepapalooza: https://review.openstack.org/#/c/494423/30 | |
| 20:08:08 | openstackgerrit | Merged openstack/nova master: Pick ironic nodes without VCPU set https://review.openstack.org/511844 | |
| 20:08:15 | openstackgerrit | Merged openstack/nova master: Add attachment_get to refresh_connection_info https://review.openstack.org/512626 | |
| 20:08:24 | openstackgerrit | Merged openstack/nova master: Update live migration to use v3 cinder api https://review.openstack.org/463987 | |
| 20:12:58 | mriedem | stvnoyes: ^ woot | |
| 20:14:20 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 20:14:42 | mriedem | johnthetubaguy: ^ updated the multiattach spec - i'll proxy your +2 if you're not around to handle it tonight | |
| 20:16:48 | mriedem | oh yeah today is the q-1 tag, suppose i should tag that | |
| 20:20:31 | cdent | mriedem, dansmith: since it is spec freezey day, and this spec is required by a spec already merged, you wanna drop some eyes on it? It was done as a seperate spec from the POST allocation (which is what requires it) because it’s a different microversion: https://review.openstack.org/#/c/508164/ please and thank you. | |
| 20:20:49 | cdent | It’s probably not strictly necessary that it merges as the work is going to be done anyway. It is mostly a bookkeeping thing I guess. | |
| 20:22:48 | mriedem | probably just gets a freeze exception given it's a dependency | |
| 20:23:24 | cdent | cool, thanks | |
| 20:58:37 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:10:04 | openstackgerrit | Merged openstack/nova-specs master: Add multiattach support to Nova https://review.openstack.org/499777 | |
| 21:12:19 | openstackgerrit | Merged openstack/nova-specs master: Add support for certificate validation https://review.openstack.org/488541 | |
| 21:12:29 | efried | mriedem I think you added me to https://review.openstack.org/#/c/334732/ to render opinion as a driver owner; did that. | |
| 21:13:27 | mriedem | efried: ok, takashi said it was for NFS in his use case too, and it takes "several minutes" for the disk to migrate, | |
| 21:13:40 | mriedem | which to me isn't a good use case for needing to add plumbing to abort the thing, | |
| 21:13:42 | efried | mriedem wha? How could it take minutes to "migrate" NFS? | |
| 21:13:47 | mriedem | if the admin is cold migrating the instance, it's for good reason | |
| 21:14:48 | mriedem | i think the live migrate cancel stuff was needed because live migrate could stall out and we needed a hook in to abort the job in the hypervisor | |
| 21:15:04 | efried | Okay, that's not what the spec says. | |
| 21:15:08 | mriedem | this is just like, whiny user says their cold migrating vm is down too long and tells the admin to stop doing that thing | |
| 21:15:23 | mriedem | right, this doesn't say anything about cold migrate stalling out and we need a way to kill it | |
| 21:15:28 | efried | The spec implies that we're copying gigabytes of local (*actual* local) disk over the wire. | |
| 21:15:36 | efried | ...and want to be able to bail on that. | |
| 21:15:38 | mriedem | this is saying, "admin is cold migrating, user doesn't like that it's taking several minutes, and wants the admin to stop' | |
| 21:16:23 | mriedem | idk maybe if there were actual numbers in here, | |
| 21:16:29 | mriedem | like, i've got a 200GB disk windows vm | |
| 21:16:36 | mriedem | and it's taking me 3 hours to cold migrate that thing | |
| 21:17:15 | mriedem | this is why i asked cburgess this morning b/c i know he uses ceph and i know he loves to migrate stuff | |
| 21:17:25 | cburgess | Oh sorry I missed something... | |
| 21:17:29 | cburgess | Do what now? | |
| 21:17:33 | stvnoyes | mridem. fantastic! :-) thanks for all the help from both you and johnthetubaguy! | |
| 21:18:05 | mriedem | cburgess: do you think https://review.openstack.org/#/c/334732/ is useful? | |
| 21:18:06 | mriedem | cburgess: do you have users with 100+ gb disks and if so, do you cold migrate those and if so, how long does that take? | |
| 21:18:53 | cburgess | mriedem Funny you should ask about 100+ GB disk... I'm just now starting at a 15TB one. | |
| 21:19:03 | cburgess | mriedem Give me a few to read this spec and I'll let you know. | |
| 21:19:09 | mriedem | thanks | |
| 21:21:01 | ildikov | stvnoyes: congrats!!! :) | |
| 21:37:15 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526 | |
| 21:37:31 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Add pagination and changes since filter support for os-instance-action API https://review.openstack.org/507762 | |
| 21:41:12 | mriedem | jaypipes: addressed your comments and sylvain's in https://review.openstack.org/#/c/507762/ - left a todo for alex's thing about sort order which i think we could hash out in review later | |
| 21:42:45 | cdent | night all | |
| 21:50:40 | openstackgerrit | Merged openstack/nova master: [Trivial] Fix up a docstring https://review.openstack.org/513447 | |
| 22:03:11 | cburgess | mriedem So yeah thats useful. We tend to not recommend cold migrations with large amounts of data. Clients that use a lot of space we push towards a volume solution for the large storage and to either use BFV or to have small roots locally and put the bulk storage in volumes. | |
| 22:03:15 | cburgess | But its still a useful feature. | |
| 22:03:37 | mikal | Random fact for the day. Name the commit which added metadata to nova. | |
| 22:04:02 | cburgess | mikal: Which part of metadata? The service? | |
| 22:04:18 | mikal | There has always been a web service | |
| 22:04:23 | mriedem | cburgess: want to comment on the spec? operator input has weight on something like this. | |
| 22:04:28 | mikal | So, the commit which added any instance accessible metadata functionality | |
| 22:04:46 | cburgess | mriedem Sure | |
| 22:04:49 | mriedem | "copy what ec2 / aws does"? | |
| 22:04:54 | cfriesen | mriedem: for what it's worth, the largest EC2 instances have 48TB of disk | |
| 22:05:02 | cburgess | mikal Um... probably somewhere in in the first few commits if not the initial import. | |
| 22:05:39 | cfriesen | mriedem: for the "abort cold migration" thing, what about the case where the dest node crashes mid-transfer? | |
| 22:05:54 | mriedem | cfriesen: wouldn't that auto-kill the cold migrate? | |
| 22:05:56 | cburgess | cfriesen I think thats covered already isn't it? Don't we detect that? | |
| 22:06:15 | cburgess | mriedem cfriesen Oh maybe not now that we can't monitor disk transfer progress. | |
| 22:06:17 | cfriesen | I'm thinking the connection could stay open until the TCP connection times out | |
| 22:06:31 | cfriesen | in the scp case, at least | |
| 22:06:32 | cburgess | Oh right... libvirt will report the error back. | |
| 22:06:39 | cburgess | At some point | |
| 22:06:40 | cburgess | In theory | |
| 22:06:54 | cburgess | mikal So whats the answer? | |
| 22:07:10 | mikal | cburgess: commit bf6e6e718cdc7488e2da87b21e258ccc065fe499 | |
| 22:07:17 | mikal | cburgess: which has the commit message "initial import" | |
| 22:07:32 | mikal | cburgess: at that point, nova was 5,083 lines of code | |
| 22:07:52 | cburgess | mikal Called it! | |
| 22:08:45 | mikal | In other words, its always been weird | |
| 22:08:54 | cburgess | wow 5K lines of code. I have scripts to fix horked up volumes that are longer then that... | |
| 22:09:40 | cburgess | OK thats a lie.. its only 1349 lines of code. | |
| 22:19:20 | cburgess | mriedem Is the existing abort admin only? Or is that policy controlable? | |
| 22:19:46 | mriedem | cburgess: anytihng related to live migration is admin-only by default | |
| 22:19:55 | mriedem | and cold migration | |
| 22:20:16 | cburgess | mriedem OK | |
| 22:20:48 | mikal | cburgess: I'm not sure volumes existed in the first commit, I'd have to check... | |