Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-06-08
15:17:36 mgagne when migrating flavor in newton, this moves flavor to api database. Will CellsV1 read from that database from now on?
15:19:17 dansmith mgagne: you mean cellsv2?
15:19:23 mgagne cellsv1
15:19:48 mgagne because you can still run cellsv1 in newton right?
15:20:06 dansmith mgagne: the api will read it from the api database yeah, but nothing else changes
15:20:23 dansmith mgagne: so you likely still need all the flavor replication bits or whatever you're doing I think
15:20:31 mgagne so will I need to create them in both api database and api/compute cells database?
15:20:35 mgagne right
15:20:42 mgagne but
15:20:51 mgagne db sync complains if you don't migrate them all to api db
15:21:01 mgagne in ocata I think
15:21:14 dansmith yeah, I'm trying to roll back my brain to think about this
15:21:34 dansmith mriedem: do you remember what belmiro said he was doing?
15:22:45 dansmith they have an api db in each cellsv1 cell, IIRC, and they migrated the flavors to those api databases?
15:22:50 mgagne I'm just trying to figure out something: I run online migration with newton. Flavors get moved to api database. But I also need to run online migration in compute cells right? (still cellsv1) Otherwise ocata complains. So I run that migration but it fails because flavors found in compute cells already exist in api database and migration fails silently with exit code 0. (and don't mark them as deleted in compute cell database)
15:23:05 mriedem context?
15:23:10 mriedem i've been in osc land all day
15:23:19 mgagne dansmith: they had an api db per cell due to per cell placement
15:23:20 dansmith mgagne: are you sharing the api database amongst all the cellsv1 cells?
15:23:28 dansmith ah, right,
15:23:28 mgagne dansmith: yes
15:23:40 dansmith so that's the difference between you and belmiro's approach
15:23:48 mgagne which I assumed was the recommended way unless you wanted per cell placement service
15:23:51 mgagne yes
15:23:58 dansmith mgagne: so you'll probably have to hack that check out
15:24:18 dansmith which I think actually sam mentioned in YVR in passing
15:24:31 mgagne ok, I don't mind hacking it. I just want to make sure that cellsv1 will read from api db now.
15:24:36 mgagne ok, I will rewatch the video
15:25:29 dansmith mgagne: it will read from the api db for things like flavor-list and nova boot... I'm not positive that there won't be other cellsv1-specific paths that aren't updated though
15:26:43 mgagne hehe, I understand the risk. I will rewatch the video and do some tests then.
15:27:27 mriedem i believe in our nova-cells v1 job the child cell doesn't have access to the api db
15:27:33 mriedem since we have single placement we didn't need it
15:27:37 mgagne right, belmiro mentioned that case in his talk
15:27:58 dansmith mriedem: okay I'm surprised it doesn't need it somewhere, but cool if so
15:28:07 mgagne I think it needs it?
15:28:22 mriedem or i guess they do http://logs.openstack.org/68/569268/4/gate/nova-cells-v1/545ba44/logs/etc/nova/nova-cells.conf.txt.gz
15:28:24 mgagne because I have it configured already. I think something complained about it.
15:28:59 mgagne so my bad, just a matter of paying more attention to belmiro's talk =)
15:29:26 mriedem 345 must be the migration that fails?
15:29:49 mriedem yeah....
15:30:03 mgagne yes
15:30:36 mriedem 345 should probably have a CONF.cells.enable condition
15:30:40 mgagne he had a slide about those migrations
15:30:50 mgagne mriedem: was thinking the same ++
15:31:10 mriedem but if we did that, you might not migrate your flavors to the api db and then you have other problems, i think?
15:31:24 mriedem we could log a warning, but likely no one would notice if db sync didn't fail
15:31:30 mgagne yea, that's what I'm trying to figure out
15:31:32 mriedem nova-status upgrade check would be good here
15:31:36 dansmith mriedem: yeah and looking at conf in migrations got people into trouble before, so...
15:31:58 mriedem if cells v1 and no flavors in api db, fail (nova-status check)
15:32:22 mriedem at the very least, we need something in the 'upgrade from cells v1' docs
15:32:38 mriedem https://docs.openstack.org/nova/latest/user/cells.html#upgrade-with-cells-v1
15:33:34 mgagne mriedem: I think the special case that isn't documented yet is: if you wish to upgrade but wish to keep cellsv1 for reasons. what to do.
15:34:00 mgagne but I think it would create more confusion if it was in the "main" documentation.
15:39:34 mriedem we havea cells faqs section...
15:39:55 mriedem "DB migration 345 fails saying I haven't migrated all flavors to the API DB but I'm using cells v1 still..."
15:40:06 mriedem https://docs.openstack.org/nova/latest/user/cells.html#faqs
15:50:59 mriedem mgagne: i guess let us know what you think would be most helpful for documenting this once you get past it - we could also put a comment in the db migration itself in case people are hitting it and then go there to see what's going on
15:52:35 mgagne nice :D
15:53:16 mgagne I managed to create a lightweight setup to test the migration so I can iterate faster. shouldn't be long before I get some interesting results
15:54:53 melwitt mriedem: are you okay with a minor release bump on this based on the last comment? sounds like it might be more intuitive to people https://review.openstack.org/573454
15:57:10 mriedem s/people/doug/
15:57:18 mriedem i'm a big meh
15:57:30 mriedem if we do a minor release for every os-traits updates, we'll just have a ton of minor updates
15:57:41 mriedem s/updates/release/ on that first one
15:57:49 melwitt yeah
16:00:37 melwitt initially I had done a minor version bump locally before I saw that a past release did a patch bump. but I think the release prior to that one was a minor bump. it's been inconsistent
16:02:36 openstackgerrit Balazs Gibizer proposed openstack/nova master: Fix unbound local when saving an unchanged RequestSpec https://review.openstack.org/573768
16:04:29 cfriesen arguably a new trait is a new "feature", which would make it a minor update if you follow semver. but so much of openstack doesn't anyways...
16:06:44 melwitt I shall change it to a minor bump
16:17:01 cfriesen mriedem: thanks for starting that OSC etherpad. you mentioned that aborting an in-progress migration doesn't really fit on the server resource. From a user perspective it seems to me that "openstack server migration list" and "openstack server migration abort" would be sensible commands
16:46:04 openstackgerrit Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766
16:46:05 openstackgerrit Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435
16:46:06 openstackgerrit Chris Dent proposed openstack/nova master: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857
17:02:28 openstackgerrit Chris Dent proposed openstack/nova master: Add PLACEMENT_DB_ENABLED=True to the nova-next job https://review.openstack.org/564067
17:02:42 jmlowe cdent: You've got some migration path for getting existing placement db's out of nova_api db?
17:04:28 cdent jmlowe: we discussed that at summit (see https://etherpad.openstack.org/p/YVR-placement-extraction ) and the general consensus in the room was allowing people to do it manually according to whatever worked for them (there are a variety of options on the etherpad)
17:05:09 jmlowe ok, I was probably harassing Jeremy at the time and missed it during that session
17:07:36 jmlowe I'm probably good with stop, dump, restore to new db, start, make schemas diverge
17:08:31 cdent jmlowe: if you feel inclined to add that "I'm good with" to the etherpad, that would be a good datapoint
17:09:48 jmlowe done
17:12:24 cdent thanks
17:41:39 openstackgerrit Jan Gutter proposed openstack/os-vif master: [WIP] Add support for generic offload representors https://review.openstack.org/572081
18:21:24 mgagne mriedem: with newton, if flavors are fully converted to object, wouldn't it mean that all code path use the API database to read the information since it goes through object? If running CellsV1, how could it still read from local cell database? Isn't the code reading from local gone in newton? But I found lot of notes by alaski telling models in non-api database is deprecated since they got moved to API database. So I'm a bit confused.
18:22:42 mriedem i believe the flavor object looks for the flavor in the api db and if not there, falls back to the cell db
18:22:44 mgagne I'm trying to see if I just need to skip the migration and still continue creating flavor in compute cell OR if I can delete them since they got all moved to API database and everything should be reading from it.
18:23:17 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/objects/flavor.py#L403
18:23:32 mriedem that checks the api first then falls back to the cell db
18:23:33 mgagne yes, that place.
18:23:38 mriedem in ocata at least
18:23:45 mriedem i know we dropped that compat code....
18:24:01 mriedem gone in queens https://github.com/openstack/nova/blob/stable/queens/nova/objects/flavor.py#L377
18:24:45 mgagne but lets say I query an instance and join to flavor. Would the join use the flavor object or would db.api load the data first and use Flavor object later?
18:25:32 mriedem the instance.flavor is not in the flavors table,
18:25:33 mgagne because it wouldn't make much sense to join instance to flavor cross databases right? I'm not familiar with that part tbh.
18:25:36 mriedem it's in the instance_extras table
18:25:42 dansmith mgagne: the instance has the flavor stashed with it in the cell db now
18:25:43 dansmith yeah that
18:25:48 mriedem the instance.flavor is stored separately keyed per the the instance
18:25:50 mgagne oh, you get a copy of the flavor right

Earlier   Later