| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-08 | |||
| 15:21:14 | dansmith | yeah, I'm trying to roll back my brain to think about this | |
| 15:21:34 | dansmith | mriedem: do you remember what belmiro said he was doing? | |
| 15:22:45 | dansmith | they have an api db in each cellsv1 cell, IIRC, and they migrated the flavors to those api databases? | |
| 15:22:50 | mgagne | I'm just trying to figure out something: I run online migration with newton. Flavors get moved to api database. But I also need to run online migration in compute cells right? (still cellsv1) Otherwise ocata complains. So I run that migration but it fails because flavors found in compute cells already exist in api database and migration fails silently with exit code 0. (and don't mark them as deleted in compute cell database) | |
| 15:23:05 | mriedem | context? | |
| 15:23:10 | mriedem | i've been in osc land all day | |
| 15:23:19 | mgagne | dansmith: they had an api db per cell due to per cell placement | |
| 15:23:20 | dansmith | mgagne: are you sharing the api database amongst all the cellsv1 cells? | |
| 15:23:28 | mgagne | dansmith: yes | |
| 15:23:28 | dansmith | ah, right, | |
| 15:23:40 | dansmith | so that's the difference between you and belmiro's approach | |
| 15:23:48 | mgagne | which I assumed was the recommended way unless you wanted per cell placement service | |
| 15:23:51 | mgagne | yes | |
| 15:23:58 | dansmith | mgagne: so you'll probably have to hack that check out | |
| 15:24:18 | dansmith | which I think actually sam mentioned in YVR in passing | |
| 15:24:31 | mgagne | ok, I don't mind hacking it. I just want to make sure that cellsv1 will read from api db now. | |
| 15:24:36 | mgagne | ok, I will rewatch the video | |
| 15:25:29 | dansmith | mgagne: it will read from the api db for things like flavor-list and nova boot... I'm not positive that there won't be other cellsv1-specific paths that aren't updated though | |
| 15:26:43 | mgagne | hehe, I understand the risk. I will rewatch the video and do some tests then. | |
| 15:27:27 | mriedem | i believe in our nova-cells v1 job the child cell doesn't have access to the api db | |
| 15:27:33 | mriedem | since we have single placement we didn't need it | |
| 15:27:37 | mgagne | right, belmiro mentioned that case in his talk | |
| 15:27:58 | dansmith | mriedem: okay I'm surprised it doesn't need it somewhere, but cool if so | |
| 15:28:07 | mgagne | I think it needs it? | |
| 15:28:22 | mriedem | or i guess they do http://logs.openstack.org/68/569268/4/gate/nova-cells-v1/545ba44/logs/etc/nova/nova-cells.conf.txt.gz | |
| 15:28:24 | mgagne | because I have it configured already. I think something complained about it. | |
| 15:28:59 | mgagne | so my bad, just a matter of paying more attention to belmiro's talk =) | |
| 15:29:26 | mriedem | 345 must be the migration that fails? | |
| 15:29:49 | mriedem | yeah.... | |
| 15:30:03 | mgagne | yes | |
| 15:30:36 | mriedem | 345 should probably have a CONF.cells.enable condition | |
| 15:30:40 | mgagne | he had a slide about those migrations | |
| 15:30:50 | mgagne | mriedem: was thinking the same ++ | |
| 15:31:10 | mriedem | but if we did that, you might not migrate your flavors to the api db and then you have other problems, i think? | |
| 15:31:24 | mriedem | we could log a warning, but likely no one would notice if db sync didn't fail | |
| 15:31:30 | mgagne | yea, that's what I'm trying to figure out | |
| 15:31:32 | mriedem | nova-status upgrade check would be good here | |
| 15:31:36 | dansmith | mriedem: yeah and looking at conf in migrations got people into trouble before, so... | |
| 15:31:58 | mriedem | if cells v1 and no flavors in api db, fail (nova-status check) | |
| 15:32:22 | mriedem | at the very least, we need something in the 'upgrade from cells v1' docs | |
| 15:32:38 | mriedem | https://docs.openstack.org/nova/latest/user/cells.html#upgrade-with-cells-v1 | |
| 15:33:34 | mgagne | mriedem: I think the special case that isn't documented yet is: if you wish to upgrade but wish to keep cellsv1 for reasons. what to do. | |
| 15:34:00 | mgagne | but I think it would create more confusion if it was in the "main" documentation. | |
| 15:39:34 | mriedem | we havea cells faqs section... | |
| 15:39:55 | mriedem | "DB migration 345 fails saying I haven't migrated all flavors to the API DB but I'm using cells v1 still..." | |
| 15:40:06 | mriedem | https://docs.openstack.org/nova/latest/user/cells.html#faqs | |
| 15:50:59 | mriedem | mgagne: i guess let us know what you think would be most helpful for documenting this once you get past it - we could also put a comment in the db migration itself in case people are hitting it and then go there to see what's going on | |
| 15:52:35 | mgagne | nice :D | |
| 15:53:16 | mgagne | I managed to create a lightweight setup to test the migration so I can iterate faster. shouldn't be long before I get some interesting results | |
| 15:54:53 | melwitt | mriedem: are you okay with a minor release bump on this based on the last comment? sounds like it might be more intuitive to people https://review.openstack.org/573454 | |
| 15:57:10 | mriedem | s/people/doug/ | |
| 15:57:18 | mriedem | i'm a big meh | |
| 15:57:30 | mriedem | if we do a minor release for every os-traits updates, we'll just have a ton of minor updates | |
| 15:57:41 | mriedem | s/updates/release/ on that first one | |
| 15:57:49 | melwitt | yeah | |
| 16:00:37 | melwitt | initially I had done a minor version bump locally before I saw that a past release did a patch bump. but I think the release prior to that one was a minor bump. it's been inconsistent | |
| 16:02:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/573768 | |
| 16:04:29 | cfriesen | arguably a new trait is a new "feature", which would make it a minor update if you follow semver. but so much of openstack doesn't anyways... | |
| 16:06:44 | melwitt | I shall change it to a minor bump | |
| 16:17:01 | cfriesen | mriedem: thanks for starting that OSC etherpad. you mentioned that aborting an in-progress migration doesn't really fit on the server resource. From a user perspective it seems to me that "openstack server migration list" and "openstack server migration abort" would be sensible commands | |
| 16:46:04 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 16:46:05 | openstackgerrit | Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435 | |
| 16:46:06 | openstackgerrit | Chris Dent proposed openstack/nova master: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857 | |
| 17:02:28 | openstackgerrit | Chris Dent proposed openstack/nova master: Add PLACEMENT_DB_ENABLED=True to the nova-next job https://review.openstack.org/564067 | |
| 17:02:42 | jmlowe | cdent: You've got some migration path for getting existing placement db's out of nova_api db? | |
| 17:04:28 | cdent | jmlowe: we discussed that at summit (see https://etherpad.openstack.org/p/YVR-placement-extraction ) and the general consensus in the room was allowing people to do it manually according to whatever worked for them (there are a variety of options on the etherpad) | |
| 17:05:09 | jmlowe | ok, I was probably harassing Jeremy at the time and missed it during that session | |
| 17:07:36 | jmlowe | I'm probably good with stop, dump, restore to new db, start, make schemas diverge | |
| 17:08:31 | cdent | jmlowe: if you feel inclined to add that "I'm good with" to the etherpad, that would be a good datapoint | |
| 17:09:48 | jmlowe | done | |
| 17:12:24 | cdent | thanks | |
| 17:41:39 | openstackgerrit | Jan Gutter proposed openstack/os-vif master: [WIP] Add support for generic offload representors https://review.openstack.org/572081 | |
| 18:21:24 | mgagne | mriedem: with newton, if flavors are fully converted to object, wouldn't it mean that all code path use the API database to read the information since it goes through object? If running CellsV1, how could it still read from local cell database? Isn't the code reading from local gone in newton? But I found lot of notes by alaski telling models in non-api database is deprecated since they got moved to API database. So I'm a bit confused. | |
| 18:22:42 | mriedem | i believe the flavor object looks for the flavor in the api db and if not there, falls back to the cell db | |
| 18:22:44 | mgagne | I'm trying to see if I just need to skip the migration and still continue creating flavor in compute cell OR if I can delete them since they got all moved to API database and everything should be reading from it. | |
| 18:23:17 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/objects/flavor.py#L403 | |
| 18:23:32 | mriedem | that checks the api first then falls back to the cell db | |
| 18:23:33 | mgagne | yes, that place. | |
| 18:23:38 | mriedem | in ocata at least | |
| 18:23:45 | mriedem | i know we dropped that compat code.... | |
| 18:24:01 | mriedem | gone in queens https://github.com/openstack/nova/blob/stable/queens/nova/objects/flavor.py#L377 | |
| 18:24:45 | mgagne | but lets say I query an instance and join to flavor. Would the join use the flavor object or would db.api load the data first and use Flavor object later? | |
| 18:25:32 | mriedem | the instance.flavor is not in the flavors table, | |
| 18:25:33 | mgagne | because it wouldn't make much sense to join instance to flavor cross databases right? I'm not familiar with that part tbh. | |
| 18:25:36 | mriedem | it's in the instance_extras table | |
| 18:25:42 | dansmith | mgagne: the instance has the flavor stashed with it in the cell db now | |
| 18:25:43 | dansmith | yeah that | |
| 18:25:48 | mriedem | the instance.flavor is stored separately keyed per the the instance | |
| 18:25:50 | mgagne | oh, you get a copy of the flavor right | |
| 18:25:51 | dansmith | so you only need it in the api db for build and list | |
| 18:25:51 | mriedem | in case the flavor is deleted | |
| 18:27:08 | mgagne | so this would mean I could delete content of instance_types in compute cell db just like the original migration tried to do. | |
| 18:27:32 | mriedem | yeah, the api isn't going to read flavors out of the cell dbs | |
| 18:27:43 | mgagne | awesome news then | |
| 18:27:45 | dansmith | probably should to make sure nothing falls back to that when it shouldn't, since it'll go away | |
| 18:28:00 | mgagne | yes, that's my main concern | |
| 18:28:38 | mgagne | so if I skip flavor migration in cell, does it mean entries are now orphan and can be deleted (and not migrated). I guess that's my question. | |
| 18:28:49 | mgagne | I suppose I can test a bit. | |
| 18:28:54 | mriedem | i believe so | |
| 18:30:02 | mriedem | when migrating flavors from child cells to the api db, it handles duplicates (per the flavorid) so should be fine https://github.com/openstack/nova/blob/stable/ocata/nova/objects/flavor.py#L755 | |