| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-28 | |||
| 12:34:59 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Ensure pre-migrating instances are destroyed during init_host https://review.openstack.org/562284 | |
| 12:37:29 | openstackgerrit | wu.chunyang proposed openstack/python-novaclient master: Add release note link in README https://review.openstack.org/578664 | |
| 12:41:20 | andrewbogott | This isn't exactly a dev question, but it's one that only a nova-developer can answer: what records in the nova database constitute the complete state of a VM? I need to copy VMs between regions and I expected that a simple copy of the 'instance' record would do the job, but that's getting me exceptions about my new VM being an orphan. Are there other records in other tables I need to be copying? | |
| 12:54:48 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add rules column to instance_group_policy table. https://review.openstack.org/560832 | |
| 12:54:49 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add InstanceGroupPolicy object https://review.openstack.org/573628 | |
| 13:16:08 | efried | andrewbogott: I'm not an expert in this space, but attempting to copy an instance by copying just the database info is not likely to work, even if you were able to identify all the various tendrils in all the various tables. | |
| 13:17:07 | andrewbogott | efried: 'not likely to work' is my prediction as well, but I'm a bit desperate :) | |
| 13:17:22 | efried | andrewbogott: The data isn't the instance - it's just stuff nova uses to keep track of the real thing, which is an artifact of the hypervisor. | |
| 13:18:00 | andrewbogott | efried: oh yeah, I'm also copying the VM files as well. I just need to tell nova 'congrats, you have a new VM, here it is' | |
| 13:18:11 | andrewbogott | which I don't think there's an API call for | |
| 13:18:22 | efried | andrewbogott: Have you tried snapshot? | |
| 13:18:57 | efried | Depending how deeply you want to "copy" the VM... | |
| 13:19:23 | andrewbogott | there are a few problems with snapshot — one issue is that as far as I can tell that will lose any copy-on-write compactness that my existing VMs have | |
| 13:19:49 | efried | andrewbogott: Can I assume you're using libvirt? | |
| 13:20:30 | andrewbogott | but, the big picture here is that I'm doing a nova-network -> neutron migration. If I can insert one VM into my new region, then I can swap an entire hypervisor from the nova-network region to the neutron region just by copying database records. That's my ultimate goal. | |
| 13:20:38 | andrewbogott | yep, libvirt, kvm | |
| 13:22:46 | efried | andrewbogott: I wonder if the CERN folks might be able to help you here. I want to say they are in the process of transitioning from nova-network, so they may have trodden this ground already. tssurya, you around? | |
| 13:23:05 | tssurya | efried: yes | |
| 13:23:08 | tssurya | reading the convo | |
| 13:23:12 | efried | tssurya: thanks | |
| 13:23:16 | andrewbogott | thanks efried | |
| 13:23:38 | efried | andrewbogott: I suspect the CoW thing is going to be a separate headache, though. How did you plan to clone the disk across? | |
| 13:24:36 | andrewbogott | just rsync | |
| 13:24:45 | andrewbogott | (plus rsyincing the base image so the backing files are there) | |
| 13:25:12 | efried | andrewbogott: BTW, the obvious answer is to live migrate your instances; I assume the "different region" thing is what's stopping you there? | |
| 13:25:17 | andrewbogott | Cloning the files is a solved problem on my end — I already do that to cold migrate instances between hypervisors (within a single region), have a working script for that | |
| 13:25:30 | andrewbogott | right, different regions | |
| 13:25:30 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 13:25:31 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 13:25:32 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | |
| 13:25:33 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | |
| 13:25:34 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | |
| 13:25:35 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | |
| 13:25:36 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | |
| 13:25:38 | andrewbogott | and I don't have live migration, since, no shared storage | |
| 13:26:38 | efried | andrewbogott: It sounds like a) you don't mind a bit of downtime, and b) you're not opposed to doing the storage migration manually. | |
| 13:27:13 | efried | andrewbogott: So can't you just create a new instance at the target, using the same flavor as the original, and then replace its disk? | |
| 13:27:48 | andrewbogott | yeah — downtime was unavoidable, and doing a cross-region migration of VMs seemed safer than trying to do a big 'upgrade our whole cloud to neutron in one go' in place upgrade because there was no guarantee of what we'd end up with. | |
| 13:28:00 | andrewbogott | efried: that has certainly crossed my mind :) | |
| 13:28:39 | tssurya | efried: not in my scope, belmoreira is the guy who did this at CERN :) | |
| 13:28:48 | tssurya | sorry | |
| 13:28:49 | efried | tssurya: Okay, thank you for looking. | |
| 13:29:54 | efried | andrewbogott: I'm a bit rusty on this - do you remember whether you need to specify a flavor when you spawn an instance from a snapshot? | |
| 13:30:36 | andrewbogott | efried: this is a bit hard to explain… my current test case involves copying a VM. but my future game plan won't actually involve copying any files. Rather, I'll shut down a single hypervisor, change that hypervisor's region, insert all the VMs from that hypervisor into the new region's DB, and then bring up the hypervisor. | |
| 13:30:53 | andrewbogott | The create-new-VM and then clobber with old-vm will work for the one-off case but it's less obvious how to adapt that to the 'whole hypervisor at once' case | |
| 13:31:13 | andrewbogott | whereas the transition from the one-VM to batch-VM is obvious in the 'create from scratch' approach | |
| 13:31:19 | andrewbogott | (hope I'm making sense here) | |
| 13:32:25 | efried | andrewbogott: You mean the physical host is going to be the same before & after, just brought up in a different region? | |
| 13:32:33 | andrewbogott | right, that's the idea | |
| 13:33:52 | andrewbogott | hence, wanting a way to duplicate nova's knowledge of a VM from one region to another | |
| 13:34:06 | andrewbogott | (and then I'll have to plumb them up with Neutron) | |
| 13:34:29 | efried | andrewbogott: Then it sounds like what you actually want to do is bring down the service and simply change the region associated with the existing instances in the database... | |
| 13:34:42 | efried | as opposed to actually moving data. | |
| 13:35:06 | andrewbogott | 'region' = separate nova deploy with a separate DB | |
| 13:35:10 | efried | I'm not familiar with the structure of the database - is the region represented as a field, or are regions whole groups of tables? | |
| 13:35:13 | efried | okay, that answers that. | |
| 13:35:23 | efried | So I gotta ask... why are you needing to change regions? | |
| 13:35:36 | andrewbogott | (Except for the API which has its own multi-region db) | |
| 13:35:45 | andrewbogott | the old region uses nova-network, the new one neutron | |
| 13:36:55 | andrewbogott | the two can't coexist in a single region as far as I can tell | |
| 13:38:03 | efried | Maybe I still don't get it. Instead of migrating your existing region from nova-network to neutron, you want to create a new region using neutron and then move your instances from the old to the new. | |
| 13:38:47 | andrewbogott | correct | |
| 13:39:05 | andrewbogott | And the reason to not 'migrate existing region' is wanting a safe, understandable transition. | |
| 13:39:52 | andrewbogott | Migrating an existing region means basically 1) shut down a working cloud 2) do a bunch of stuff 3) <unknown time elapses> 4) <hope that the new neutronified deploy ever works again> | |
| 13:40:16 | andrewbogott | Whereas we prefer to move our VMs to a cloud that we already know works :) | |
| 13:42:29 | mnaser | "To expand your deployment with a new cell, first follow the usual steps for standing up a new Cells V1 cell. After that is finished, follow step 4 in Upgrade with Cells V1 to create a new Cells V2 cell for it." | |
| 13:42:32 | mnaser | is this a doc type or :( | |
| 13:42:35 | mnaser | typo* | |
| 13:42:56 | andrewbogott | efried: If there existed an actual supported/documented upgrade path to Neutron then I wouldn't be engaged in such obscure and desperate measures — but that ship has sailed :( | |
| 13:42:58 | efried | andrewbogott: Okay, except that steps 2-4 represent a documented process that at least somebody has actually tried and (presumably) completed successfully. Whereas what you're talking about... | |
| 13:43:11 | efried | ahjeez, there's no documented process? | |
| 13:43:11 | mnaser | there's not even any more docs on how to 'stand up a cells v1 cell'.. and im sure hoping that i dont have to end up doing that :X | |
| 13:43:27 | andrewbogott | efried: there are 100 different documented processes, each bespoke for a single cloud | |
| 13:43:34 | efried | oy vay. | |
| 13:43:46 | andrewbogott | I've been waiting since 2013 for the promised upgrade tool to arrive but can wait no longer | |
| 13:44:10 | mnaser | nova-network to neutron? | |
| 13:44:24 | andrewbogott | yeah, with flat dhcp and floating IPs | |
| 13:45:00 | mnaser | that's a rough one. we did it a really long time ago but it involved having 2 clouds running at the same time, one with old setup and one with neutron | |
| 13:45:09 | mnaser | and a lot of very bad ideas™ | |
| 13:45:19 | mnaser | i dunno, i dont think we're at a point that someone is going to write a tool for this migration | |
| 13:45:46 | efried | Save me, mnaser! I've been like a physics professor trying to explain a real spaceship here (i.e. my scope of knowledge is barely overlapping and purely theoretical). | |
| 13:45:59 | andrewbogott | ahah! 'two clouds running at the same time' <- exactly what I'm doing | |
| 13:46:15 | andrewbogott | mnaser: how did you move your VMs between the clouds? | |
| 13:46:45 | mnaser | efried: aha | |
| 13:46:52 | mnaser | andrewbogott: well, it involved downtime, the idea was: | |
| 13:47:02 | mnaser | 1) create matching neutron network in new region | |
| 13:47:17 | mnaser | 2) create port with matching fixed ip, mac address, etc | |
| 13:47:36 | mnaser | 3) boot vm attached to that port under the same tenant or so | |
| 13:47:49 | mnaser | 4) shutdown vm, copy vm image from old region to new one | |
| 13:47:55 | mnaser | 5) start up vm again | |
| 13:47:59 | mnaser | 6) ??? | |
| 13:48:00 | mnaser | 7) profit | |
| 13:48:17 | andrewbogott | ok, so step 3 and 4 is basically that you create a new VM and then clobber it with the files from the old VM? | |
| 13:49:04 | andrewbogott | mnaser: it's nice to hear you having used a similar approach to the one we're converging on | |
| 13:49:17 | jangutter | andrewbogott: in summary, you'd like to "abandon" a VM, and "adopt" it without disturbing the hypervisor or networking? | |
| 13:49:18 | mnaser | essentially yes | |
| 13:49:39 | andrewbogott | jangutter: that's pretty much right. | |
| 13:49:44 | mnaser | that could be a possibility too actually, which avoids the move | |
| 13:49:52 | mnaser | the only thing is you have to stop access to your APIs | |
| 13:49:59 | mnaser | and do this per hypervisor | |