| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-28 | |||
| 13:23:38 | efried | andrewbogott: I suspect the CoW thing is going to be a separate headache, though. How did you plan to clone the disk across? | |
| 13:24:36 | andrewbogott | just rsync | |
| 13:24:45 | andrewbogott | (plus rsyincing the base image so the backing files are there) | |
| 13:25:12 | efried | andrewbogott: BTW, the obvious answer is to live migrate your instances; I assume the "different region" thing is what's stopping you there? | |
| 13:25:17 | andrewbogott | Cloning the files is a solved problem on my end — I already do that to cold migrate instances between hypervisors (within a single region), have a working script for that | |
| 13:25:30 | andrewbogott | right, different regions | |
| 13:25:30 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 13:25:31 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 13:25:32 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: New dict format of allocations (v1.11, v1.12) https://review.openstack.org/542819 | |
| 13:25:33 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Transactionally update allocations (v1.13) https://review.openstack.org/546674 | |
| 13:25:34 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Add nested resource providers (v1.14) https://review.openstack.org/546675 | |
| 13:25:35 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Limit allocation candidates (v1.15, v1.16) https://review.openstack.org/548043 | |
| 13:25:36 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Allocation candidates parameter: required (v1.17) https://review.openstack.org/548326 | |
| 13:25:38 | andrewbogott | and I don't have live migration, since, no shared storage | |
| 13:26:38 | efried | andrewbogott: It sounds like a) you don't mind a bit of downtime, and b) you're not opposed to doing the storage migration manually. | |
| 13:27:13 | efried | andrewbogott: So can't you just create a new instance at the target, using the same flavor as the original, and then replace its disk? | |
| 13:27:48 | andrewbogott | yeah — downtime was unavoidable, and doing a cross-region migration of VMs seemed safer than trying to do a big 'upgrade our whole cloud to neutron in one go' in place upgrade because there was no guarantee of what we'd end up with. | |
| 13:28:00 | andrewbogott | efried: that has certainly crossed my mind :) | |
| 13:28:39 | tssurya | efried: not in my scope, belmoreira is the guy who did this at CERN :) | |
| 13:28:48 | tssurya | sorry | |
| 13:28:49 | efried | tssurya: Okay, thank you for looking. | |
| 13:29:54 | efried | andrewbogott: I'm a bit rusty on this - do you remember whether you need to specify a flavor when you spawn an instance from a snapshot? | |
| 13:30:36 | andrewbogott | efried: this is a bit hard to explain… my current test case involves copying a VM. but my future game plan won't actually involve copying any files. Rather, I'll shut down a single hypervisor, change that hypervisor's region, insert all the VMs from that hypervisor into the new region's DB, and then bring up the hypervisor. | |
| 13:30:53 | andrewbogott | The create-new-VM and then clobber with old-vm will work for the one-off case but it's less obvious how to adapt that to the 'whole hypervisor at once' case | |
| 13:31:13 | andrewbogott | whereas the transition from the one-VM to batch-VM is obvious in the 'create from scratch' approach | |
| 13:31:19 | andrewbogott | (hope I'm making sense here) | |
| 13:32:25 | efried | andrewbogott: You mean the physical host is going to be the same before & after, just brought up in a different region? | |
| 13:32:33 | andrewbogott | right, that's the idea | |
| 13:33:52 | andrewbogott | hence, wanting a way to duplicate nova's knowledge of a VM from one region to another | |
| 13:34:06 | andrewbogott | (and then I'll have to plumb them up with Neutron) | |
| 13:34:29 | efried | andrewbogott: Then it sounds like what you actually want to do is bring down the service and simply change the region associated with the existing instances in the database... | |
| 13:34:42 | efried | as opposed to actually moving data. | |
| 13:35:06 | andrewbogott | 'region' = separate nova deploy with a separate DB | |
| 13:35:10 | efried | I'm not familiar with the structure of the database - is the region represented as a field, or are regions whole groups of tables? | |
| 13:35:13 | efried | okay, that answers that. | |
| 13:35:23 | efried | So I gotta ask... why are you needing to change regions? | |
| 13:35:36 | andrewbogott | (Except for the API which has its own multi-region db) | |
| 13:35:45 | andrewbogott | the old region uses nova-network, the new one neutron | |
| 13:36:55 | andrewbogott | the two can't coexist in a single region as far as I can tell | |
| 13:38:03 | efried | Maybe I still don't get it. Instead of migrating your existing region from nova-network to neutron, you want to create a new region using neutron and then move your instances from the old to the new. | |
| 13:38:47 | andrewbogott | correct | |
| 13:39:05 | andrewbogott | And the reason to not 'migrate existing region' is wanting a safe, understandable transition. | |
| 13:39:52 | andrewbogott | Migrating an existing region means basically 1) shut down a working cloud 2) do a bunch of stuff 3) <unknown time elapses> 4) <hope that the new neutronified deploy ever works again> | |
| 13:40:16 | andrewbogott | Whereas we prefer to move our VMs to a cloud that we already know works :) | |
| 13:42:29 | mnaser | "To expand your deployment with a new cell, first follow the usual steps for standing up a new Cells V1 cell. After that is finished, follow step 4 in Upgrade with Cells V1 to create a new Cells V2 cell for it." | |
| 13:42:32 | mnaser | is this a doc type or :( | |
| 13:42:35 | mnaser | typo* | |
| 13:42:56 | andrewbogott | efried: If there existed an actual supported/documented upgrade path to Neutron then I wouldn't be engaged in such obscure and desperate measures — but that ship has sailed :( | |
| 13:42:58 | efried | andrewbogott: Okay, except that steps 2-4 represent a documented process that at least somebody has actually tried and (presumably) completed successfully. Whereas what you're talking about... | |
| 13:43:11 | efried | ahjeez, there's no documented process? | |
| 13:43:11 | mnaser | there's not even any more docs on how to 'stand up a cells v1 cell'.. and im sure hoping that i dont have to end up doing that :X | |
| 13:43:27 | andrewbogott | efried: there are 100 different documented processes, each bespoke for a single cloud | |
| 13:43:34 | efried | oy vay. | |
| 13:43:46 | andrewbogott | I've been waiting since 2013 for the promised upgrade tool to arrive but can wait no longer | |
| 13:44:10 | mnaser | nova-network to neutron? | |
| 13:44:24 | andrewbogott | yeah, with flat dhcp and floating IPs | |
| 13:45:00 | mnaser | that's a rough one. we did it a really long time ago but it involved having 2 clouds running at the same time, one with old setup and one with neutron | |
| 13:45:09 | mnaser | and a lot of very bad ideas™ | |
| 13:45:19 | mnaser | i dunno, i dont think we're at a point that someone is going to write a tool for this migration | |
| 13:45:46 | efried | Save me, mnaser! I've been like a physics professor trying to explain a real spaceship here (i.e. my scope of knowledge is barely overlapping and purely theoretical). | |
| 13:45:59 | andrewbogott | ahah! 'two clouds running at the same time' <- exactly what I'm doing | |
| 13:46:15 | andrewbogott | mnaser: how did you move your VMs between the clouds? | |
| 13:46:45 | mnaser | efried: aha | |
| 13:46:52 | mnaser | andrewbogott: well, it involved downtime, the idea was: | |
| 13:47:02 | mnaser | 1) create matching neutron network in new region | |
| 13:47:17 | mnaser | 2) create port with matching fixed ip, mac address, etc | |
| 13:47:36 | mnaser | 3) boot vm attached to that port under the same tenant or so | |
| 13:47:49 | mnaser | 4) shutdown vm, copy vm image from old region to new one | |
| 13:47:55 | mnaser | 5) start up vm again | |
| 13:47:59 | mnaser | 6) ??? | |
| 13:48:00 | mnaser | 7) profit | |
| 13:48:17 | andrewbogott | ok, so step 3 and 4 is basically that you create a new VM and then clobber it with the files from the old VM? | |
| 13:49:04 | andrewbogott | mnaser: it's nice to hear you having used a similar approach to the one we're converging on | |
| 13:49:17 | jangutter | andrewbogott: in summary, you'd like to "abandon" a VM, and "adopt" it without disturbing the hypervisor or networking? | |
| 13:49:18 | mnaser | essentially yes | |
| 13:49:39 | andrewbogott | jangutter: that's pretty much right. | |
| 13:49:44 | mnaser | that could be a possibility too actually, which avoids the move | |
| 13:49:52 | mnaser | the only thing is you have to stop access to your APIs | |
| 13:49:59 | mnaser | and do this per hypervisor | |
| 13:50:09 | mnaser | but that was too wild for us | |
| 13:50:10 | andrewbogott | Which I continue to believe/hope is just a matter of duplicating database rows between the oldnova DB and the new one... | |
| 13:50:16 | andrewbogott | Oh yeah, this is going to involve epic downtime | |
| 13:50:19 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle CannotDeleteParentResourceProvider to 409 Conflict https://review.openstack.org/578804 | |
| 13:53:07 | yikun | mriedem: and we need keep _policies and policy in API model? and joinload _policies and policy? | |
| 13:53:25 | yikun | https://review.openstack.org/#/c/563375/17/nova/objects/instance_group.py@41 | |
| 13:54:45 | melwitt | nova meeting in 6 minutes | |
| 13:55:18 | mriedem | yikun: honestly i'm not fully following dansmith's comments there either, but unfortunately he's on vacation for the rest of this week. | |
| 13:55:27 | andrewbogott | mnaser, jangutter, my original question was: which tables in the nova db (besides 'instances') express the full state of a VM? | |
| 13:55:38 | mnaser | a lot | |
| 13:55:41 | andrewbogott | which, probably I just need to keep digging in the code/schema for the answer to this | |
| 13:55:45 | yikun | - -! | |
| 13:55:47 | mnaser | i think thats hard to like nail down | |
| 13:55:50 | mnaser | in a message :p | |
| 13:56:03 | andrewbogott | ok, fair enough :) | |
| 13:56:07 | mnaser | yes, you'll have to do research, my suggestion: dump your database | |
| 13:56:13 | mnaser | grep for a really old running instance and see what it's touching | |
| 13:56:19 | mriedem | yikun: oh i think i get it, | |
| 13:56:44 | mriedem | he's saying if we check db_policy._policies anywhere, we'll lazy-load it from sqlalchemy, in addition to the new db_policy.policy join you've added | |
| 13:56:44 | yikun | em? | |
| 13:57:01 | andrewbogott | mnaser: you'd expect that to be keyed using the uuid? Or the 'id' from the instances table? | |