Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-11-29
23:38:02 mgagne so I can't find the exact cause but I think it was looking for service entries in the API cell database and didn't exclude deleted rows.
23:38:39 dansmith mgagne: I have no doubt that none of this works the way it should for cellsv1
23:39:17 dansmith mixed-version computes I mean
23:39:17 mgagne or could be with the online migration
23:39:36 mriedem services entries are in the cell dbs
23:39:38 mriedem not the api db
23:40:11 mgagne here https://github.com/openstack/nova/blob/mitaka-eol/nova/objects/pci_device.py#L118-L127
23:40:24 mgagne if no conductor service is found, the value returned fails the check
23:40:53 mgagne well, maybe I shouldn't have run the online migration in the API cell? ¯\_(ツ)_/¯
23:41:44 mriedem hm
23:42:37 mriedem dansmith: we create the api/scheduler/conductor services in the cell0 db don't we?
23:42:47 dansmith yup
23:42:50 dansmith I mean, in devstack
23:42:54 mriedem right
23:43:26 mriedem yeah http://logs.openstack.org/87/523187/2/check/legacy-tempest-dsvm-neutron-full/6b78222/logs/etc/nova/nova.conf.txt.gz
23:43:29 mriedem [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8
23:43:40 dansmith dude, passwords!
23:46:09 mriedem mgagne: i wonder if you had this fix before you upgraded https://review.openstack.org/#/q/Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b
23:46:16 mriedem https://review.openstack.org/#/c/438632/
23:46:48 mgagne yes
23:47:07 mgagne but if no entry is found, I think it returns 0 ou None and it fails
23:47:59 mriedem yeah you're right https://github.com/openstack/nova/blob/master/nova/objects/service.py#L431-L434
23:48:03 mgagne but I don't know if I had to run the migration in api cell or not
23:48:54 mgagne if i shouldn't, well I think the migration script should have told me: hey, this is an api cell, you shouldn't do that. But I understand that cellsv1 isn't fully tested so yea, what can you do =)
23:49:26 mriedem https://github.com/openstack/nova/commit/50355c4595e08f293f610da32247e405b20c1c5b
23:49:44 mriedem yeah my guess is there was no consideration for cells v1 when that was written in mitaka
23:49:51 mriedem and we don't have grenade (upgrade) ci jobs for cellsv1
23:50:18 mriedem cellsv1 upgrade testing was usually literally alaski or johnthetubaguy saying something broke at rax
23:50:31 mgagne ok, that's fine, mitaka migration is behind us. but I guess same migration will fail again with newton if I try to run it in api cell.
23:50:42 mriedem that code was dropped in newton
23:50:50 mriedem because we have that schema migration blocker in newton
23:50:56 mriedem https://github.com/openstack/nova/blob/stable/newton/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py
23:51:11 mriedem but ^ assumes the child cell db
23:51:33 mriedem so nova-manage db sync
23:51:38 mriedem not nova-manage api_db sync
23:51:43 mgagne that's a fun one: "until all records have been migrated". Ok, let's run that migration then! oh way, code is gone. what now? /sad panda
23:51:47 mriedem so you should be fine
23:52:17 mgagne but then, you will say there is an upgrade readiness check now you can run
23:52:27 mgagne which I'm fine with. ;)
23:52:30 mriedem nova-status was added in ocata
23:53:20 mriedem and checked for things like making sure placement was deployed and cellsv2 mappings existed
23:53:22 mgagne yea, I had a lot of "fun" when it complained about the online migration and old code was gone. had to find a copy in a different environment and rsync that thing.
23:54:16 mriedem i thought people stood up a separate env to run the db sync on the new code before upgrading the old code that's actually running?
23:54:52 mgagne I guess I'm not in that ideal world yet :P
23:55:33 mriedem you can also run the online data migrations from the old mitaka code before upgrading to newton, and run them after upgrading to newton if yo uwant
23:55:36 mgagne usually I stop all services, upgrade package, run db sync, start service.
23:56:19 mgagne but if I forgot to run online migration and db sync fails, I'm screwed because I already upgraded the packages. but my bad for not checking if all online migration ran properly before the upgrade.
23:56:54 mgagne sure but I wasn't prepared for that maneuver
23:58:02 mriedem mgagne: even with that blocker migration script in newton, we didn't delete the cold that allows you to run the online data migrations from mitaka https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L787
23:58:08 mriedem so, we didn't hose you there
23:58:18 mriedem we just said, you can't continue until you do your homework from mitaka
23:58:21 mgagne very much appreciated =)
23:59:31 mriedem pretty sure that's standard operating procedure
23:59:41 mriedem in queens we still have online data migration code from newton
23:59:57 mriedem we only remove it if we're at least n+1 and someone gets around to caring
#openstack-nova - 2017-11-30
00:00:18 mriedem e.g. https://review.openstack.org/#/c/517158/
00:01:07 mriedem mgagne: btw, that's another reason people can't/shoudn't literally skip through releases for upgrades
00:02:06 mgagne mriedem: I can't afford to not skip versions ;)
00:02:19 mriedem fast forwarding through versions is fine
00:02:23 mriedem but you have to run the data migratoins
00:02:26 mriedem per release
00:02:45 mriedem same issues with dropping config options after n+1
00:03:09 mgagne yes, but with cells, I'm not sure if I will be able to skip anymore, too many unknown for now
00:03:37 mgagne mriedem: configs are fine, we have funky stuff in puppet to support multiple versions
00:03:54 mriedem like aliases?
00:06:01 mgagne very funky stuff: https://gist.github.com/mgagne/7146424416eda597563c4018ce50cf97
00:06:08 mgagne copied as-is so you can see our mess
00:06:33 mgagne this class is included in our main nova.pp which does the main configuration
00:08:14 mgagne so I just make it so puppet-nova for newton works with mitaka. and I do the same with other services/modules
00:08:49 tssurya_ mriedem : actually belmiro is currently on newton, moving to ocata (which would be only for a short duration), but main goal is pike.
00:09:44 tssurya_ mridem, dansmith : http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-10-24.log.html#t2017-10-24T13:12:39 , the conversation you guys had regarding placement,
00:10:06 mriedem tssurya_: thanks, put that into https://etherpad.openstack.org/p/cellsv1-to-v2-migration
00:10:17 tssurya_ mriedem : sure !
00:14:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964
00:14:17 openstackgerrit Matt Riedemann proposed openstack/nova master: Add multi-cell negative test for cold migration with target host https://review.openstack.org/524027
00:16:09 mgagne mriedem: ok so if you run Cellsv1, you should run placement per cell otherwise nova-scheduler in cell *could* pickup hosts from a different cell?
00:16:56 dansmith it will
00:17:01 dansmith and will have to filter them out
00:17:14 mgagne "and will have to filter them out" how?
00:17:28 mriedem the scheduler does a db query to the compute_nodes table per cell
00:17:28 dansmith by the ones it has host state for
00:17:54 mriedem 1. scheduler asks placement for resource providers (compute nodes) for a given request (flavor)
00:18:05 mriedem 2. scheduler queries the cell db for the compute nodes by the list of uuids from placement
00:18:17 mriedem 3. scheduler converts those compute nodes to HostState objects and those go through the enabled filters
00:18:57 mgagne does it mean UUID returned by 2) from placement would be filtered by the ones found in compute nodes in cell db?
00:19:35 mriedem yeah https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L628
00:19:53 mriedem starts here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L645
00:20:50 mriedem although, that's pike+
00:20:51 mriedem https://github.com/openstack/nova/commit/d1de5385233ce4379b17a7404557c6724dc37cd4
00:23:23 mgagne I'm more concerned about cellsv1, I feel like linked commit is for cellsv2 where scheduling happens in api/top cell? Or am I mistaken?
00:23:49 mriedem with cells v2 yeah there is just a flat scheduler
00:23:55 mriedem which is multi-cell aware starting in pike
00:25:11 mriedem my brain is getting fried and i can already hear my wife getting mad because i'm in my office after 6pm,
00:25:30 mriedem but i'm not sure if https://review.openstack.org/#/c/436648/ makes a significant difference in the decision to run per-cell placement or not in ocata
00:25:50 mriedem because the nova-scheduler in each cell in ocata would only query computes from the cell db it's configured for
00:26:06 mriedem so placement might say there are 1000 computes that could satisfy a request, but only 100 of those might be in the cell db that that cell knows about
00:26:16 mriedem which is this https://github.com/openstack/nova/blob/stable/ocata/nova/scheduler/host_manager.py#L579
00:26:46 mriedem so i think global placement is still ok in ocata (still not sure why belmiro wanted per-cell placement, would have to read the notes again)
00:28:03 mgagne alright, I guess it addresses my concern. would need to test for sure but I get the idea
00:29:01 mriedem i put more notes about this in https://etherpad.openstack.org/p/cellsv1-to-v2-migration

Earlier   Later