| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-28 | |||
| 13:35:23 | efried | So I gotta ask... why are you needing to change regions? | |
| 13:35:36 | andrewbogott | (Except for the API which has its own multi-region db) | |
| 13:35:45 | andrewbogott | the old region uses nova-network, the new one neutron | |
| 13:36:55 | andrewbogott | the two can't coexist in a single region as far as I can tell | |
| 13:38:03 | efried | Maybe I still don't get it. Instead of migrating your existing region from nova-network to neutron, you want to create a new region using neutron and then move your instances from the old to the new. | |
| 13:38:47 | andrewbogott | correct | |
| 13:39:05 | andrewbogott | And the reason to not 'migrate existing region' is wanting a safe, understandable transition. | |
| 13:39:52 | andrewbogott | Migrating an existing region means basically 1) shut down a working cloud 2) do a bunch of stuff 3) <unknown time elapses> 4) <hope that the new neutronified deploy ever works again> | |
| 13:40:16 | andrewbogott | Whereas we prefer to move our VMs to a cloud that we already know works :) | |
| 13:42:29 | mnaser | "To expand your deployment with a new cell, first follow the usual steps for standing up a new Cells V1 cell. After that is finished, follow step 4 in Upgrade with Cells V1 to create a new Cells V2 cell for it." | |
| 13:42:32 | mnaser | is this a doc type or :( | |
| 13:42:35 | mnaser | typo* | |
| 13:42:56 | andrewbogott | efried: If there existed an actual supported/documented upgrade path to Neutron then I wouldn't be engaged in such obscure and desperate measures — but that ship has sailed :( | |
| 13:42:58 | efried | andrewbogott: Okay, except that steps 2-4 represent a documented process that at least somebody has actually tried and (presumably) completed successfully. Whereas what you're talking about... | |
| 13:43:11 | mnaser | there's not even any more docs on how to 'stand up a cells v1 cell'.. and im sure hoping that i dont have to end up doing that :X | |
| 13:43:11 | efried | ahjeez, there's no documented process? | |
| 13:43:27 | andrewbogott | efried: there are 100 different documented processes, each bespoke for a single cloud | |
| 13:43:34 | efried | oy vay. | |
| 13:43:46 | andrewbogott | I've been waiting since 2013 for the promised upgrade tool to arrive but can wait no longer | |
| 13:44:10 | mnaser | nova-network to neutron? | |
| 13:44:24 | andrewbogott | yeah, with flat dhcp and floating IPs | |
| 13:45:00 | mnaser | that's a rough one. we did it a really long time ago but it involved having 2 clouds running at the same time, one with old setup and one with neutron | |
| 13:45:09 | mnaser | and a lot of very bad ideas™ | |
| 13:45:19 | mnaser | i dunno, i dont think we're at a point that someone is going to write a tool for this migration | |
| 13:45:46 | efried | Save me, mnaser! I've been like a physics professor trying to explain a real spaceship here (i.e. my scope of knowledge is barely overlapping and purely theoretical). | |
| 13:45:59 | andrewbogott | ahah! 'two clouds running at the same time' <- exactly what I'm doing | |
| 13:46:15 | andrewbogott | mnaser: how did you move your VMs between the clouds? | |
| 13:46:45 | mnaser | efried: aha | |
| 13:46:52 | mnaser | andrewbogott: well, it involved downtime, the idea was: | |
| 13:47:02 | mnaser | 1) create matching neutron network in new region | |
| 13:47:17 | mnaser | 2) create port with matching fixed ip, mac address, etc | |
| 13:47:36 | mnaser | 3) boot vm attached to that port under the same tenant or so | |
| 13:47:49 | mnaser | 4) shutdown vm, copy vm image from old region to new one | |
| 13:47:55 | mnaser | 5) start up vm again | |
| 13:47:59 | mnaser | 6) ??? | |
| 13:48:00 | mnaser | 7) profit | |
| 13:48:17 | andrewbogott | ok, so step 3 and 4 is basically that you create a new VM and then clobber it with the files from the old VM? | |
| 13:49:04 | andrewbogott | mnaser: it's nice to hear you having used a similar approach to the one we're converging on | |
| 13:49:17 | jangutter | andrewbogott: in summary, you'd like to "abandon" a VM, and "adopt" it without disturbing the hypervisor or networking? | |
| 13:49:18 | mnaser | essentially yes | |
| 13:49:39 | andrewbogott | jangutter: that's pretty much right. | |
| 13:49:44 | mnaser | that could be a possibility too actually, which avoids the move | |
| 13:49:52 | mnaser | the only thing is you have to stop access to your APIs | |
| 13:49:59 | mnaser | and do this per hypervisor | |
| 13:50:09 | mnaser | but that was too wild for us | |
| 13:50:10 | andrewbogott | Which I continue to believe/hope is just a matter of duplicating database rows between the oldnova DB and the new one... | |
| 13:50:16 | andrewbogott | Oh yeah, this is going to involve epic downtime | |
| 13:50:19 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Handle CannotDeleteParentResourceProvider to 409 Conflict https://review.openstack.org/578804 | |
| 13:53:07 | yikun | mriedem: and we need keep _policies and policy in API model? and joinload _policies and policy? | |
| 13:53:25 | yikun | https://review.openstack.org/#/c/563375/17/nova/objects/instance_group.py@41 | |
| 13:54:45 | melwitt | nova meeting in 6 minutes | |
| 13:55:18 | mriedem | yikun: honestly i'm not fully following dansmith's comments there either, but unfortunately he's on vacation for the rest of this week. | |
| 13:55:27 | andrewbogott | mnaser, jangutter, my original question was: which tables in the nova db (besides 'instances') express the full state of a VM? | |
| 13:55:38 | mnaser | a lot | |
| 13:55:41 | andrewbogott | which, probably I just need to keep digging in the code/schema for the answer to this | |
| 13:55:45 | yikun | - -! | |
| 13:55:47 | mnaser | i think thats hard to like nail down | |
| 13:55:50 | mnaser | in a message :p | |
| 13:56:03 | andrewbogott | ok, fair enough :) | |
| 13:56:07 | mnaser | yes, you'll have to do research, my suggestion: dump your database | |
| 13:56:13 | mnaser | grep for a really old running instance and see what it's touching | |
| 13:56:19 | mriedem | yikun: oh i think i get it, | |
| 13:56:44 | yikun | em? | |
| 13:56:44 | mriedem | he's saying if we check db_policy._policies anywhere, we'll lazy-load it from sqlalchemy, in addition to the new db_policy.policy join you've added | |
| 13:57:01 | andrewbogott | mnaser: you'd expect that to be keyed using the uuid? Or the 'id' from the instances table? | |
| 13:57:18 | mnaser | both | |
| 13:57:18 | mnaser | :) | |
| 13:57:44 | mnaser | nova is complex and in my experience you really dont want to muck around the db manually | |
| 13:57:55 | jangutter | andrewbogott: the scary part of this is not accidentally re-inventing all the instance creation code... | |
| 13:58:05 | melwitt | gmann: I put it on the agenda for the nova meeting today | |
| 13:58:09 | mnaser | it might work today but break down in terrible ways in your next upgrade | |
| 13:58:34 | andrewbogott | ok, I'll see if I can do use the instance creation code instead, then. | |
| 13:58:40 | mnaser | melwitt: i know this is pretty last minute but if we have time to discuss https://review.openstack.org/#/c/577933/ that'd be nice | |
| 13:58:42 | andrewbogott | thank you for your thoughts, all | |
| 13:58:45 | mnaser | though i am not attached or that is not a priority | |
| 13:58:50 | yikun | mriedem: "if we check db_policy._policies anywhere", why we check this? | |
| 13:59:17 | mriedem | yikun: replied | |
| 13:59:22 | mriedem | yikun: yeah i don't get it, | |
| 13:59:38 | melwitt | mnaser: you mean during open discussion in the nova meeting? that's fine, I'm sure we'll have time | |
| 13:59:53 | mriedem | yikun: i'd say leave that part as you have it now, address the other comments and then we can come back to it when dan is back | |
| 13:59:55 | mnaser | melwitt: ok cools | |
| 14:00:22 | mriedem | yikun: i can maybe get jaypipes to take a look later too | |
| 14:00:36 | mriedem | he knows sqla db objects and versioned object mysteries | |
| 14:00:58 | Guest88823 | mriedem: nothing to do with ovo, just sqla | |
| 14:01:08 | mriedem | TheLurker | |
| 14:01:17 | yikun | hah | |
| 14:01:19 | dansmith | mriedem: yikun: You always need to check the old field until we're fully migrated yeah? | |
| 14:01:45 | mriedem | but it doesn't exist in the model https://review.openstack.org/#/c/563375/17/nova/db/sqlalchemy/api_models.py | |
| 14:01:55 | mriedem | _policies was changed to policy in the model | |
| 14:01:57 | mriedem | for the backref | |
| 14:02:12 | mriedem | crap meeting time | |
| 14:02:29 | dansmith | ohh | |
| 14:02:33 | dansmith | I never got that far | |
| 14:02:36 | dansmith | why is that changing? | |
| 14:02:48 | mriedem | i asked that as well, | |
| 14:03:00 | mriedem | i assume it's for model sanity since we don't have many policies per group, we have one | |
| 14:03:10 | mriedem | it's not a column, just a backref attribute | |
| 14:03:16 | dansmith | I thought we were adding a new backref | |
| 14:03:23 | dansmith | not a column obviously, but a new backreg | |
| 14:03:29 | dansmith | alright, well, nevermind me then | |