| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-11-29 | |||
| 23:31:41 | melwitt | I see | |
| 23:31:44 | mriedem | get_minimum_version_all_cells is | |
| 23:32:29 | mriedem | dansmith: looks like we only use that in one place, server create for device tagging, | |
| 23:32:45 | mriedem | but, | |
| 23:32:56 | dansmith | hmm, actually | |
| 23:33:00 | mriedem | anything else acting on an instance would be scoped to the cell that instance is in, so min service version checks scoped to that cell would be fine | |
| 23:33:01 | dansmith | now that you mention it, | |
| 23:33:18 | mriedem | i'm thinking about your conductor migration allocation min service versoin check | |
| 23:34:00 | mriedem | when we do that, the context should be targeted to the cell that the instance was pulled from | |
| 23:34:14 | dansmith | the problem is we cache it in places | |
| 23:34:19 | mriedem | https://github.com/openstack/nova/blob/master/nova/conductor/tasks/migrate.py#L148 | |
| 23:35:01 | mriedem | oh hmm | |
| 23:35:06 | dansmith | https://github.com/openstack/nova/blob/master/nova/objects/service.py#L396 | |
| 23:35:14 | mriedem | so min compute in cell1 might be 22 and 23 in cell2 | |
| 23:35:21 | mriedem | and if we cached 23, we could f up cell1 | |
| 23:35:25 | mriedem | ? | |
| 23:35:31 | dansmith | I was going to say we iterate it where we need it and otherwise it's scoped per instance, but.... | |
| 23:36:02 | mriedem | yeah we cache in the api, compute, conductor and scheduler services | |
| 23:36:16 | mriedem | compute is probably fine right? it's scoped to a cell anyway | |
| 23:36:25 | mriedem | but api/scheduler/conductor could be a problem | |
| 23:36:46 | dansmith | right "inside the cell things" are fine | |
| 23:37:59 | dansmith | man I wish we had the CellMapping in the context so we could make the cache just use that to key | |
| 23:38:02 | mgagne | so I can't find the exact cause but I think it was looking for service entries in the API cell database and didn't exclude deleted rows. | |
| 23:38:39 | dansmith | mgagne: I have no doubt that none of this works the way it should for cellsv1 | |
| 23:39:17 | mgagne | or could be with the online migration | |
| 23:39:17 | dansmith | mixed-version computes I mean | |
| 23:39:36 | mriedem | services entries are in the cell dbs | |
| 23:39:38 | mriedem | not the api db | |
| 23:40:11 | mgagne | here https://github.com/openstack/nova/blob/mitaka-eol/nova/objects/pci_device.py#L118-L127 | |
| 23:40:24 | mgagne | if no conductor service is found, the value returned fails the check | |
| 23:40:53 | mgagne | well, maybe I shouldn't have run the online migration in the API cell? ¯\_(ツ)_/¯ | |
| 23:41:44 | mriedem | hm | |
| 23:42:37 | mriedem | dansmith: we create the api/scheduler/conductor services in the cell0 db don't we? | |
| 23:42:47 | dansmith | yup | |
| 23:42:50 | dansmith | I mean, in devstack | |
| 23:42:54 | mriedem | right | |
| 23:43:26 | mriedem | yeah http://logs.openstack.org/87/523187/2/check/legacy-tempest-dsvm-neutron-full/6b78222/logs/etc/nova/nova.conf.txt.gz | |
| 23:43:29 | mriedem | [database] connection = mysql+pymysql://root:secretmysql@127.0.0.1/nova_cell0?charset=utf8 | |
| 23:43:40 | dansmith | dude, passwords! | |
| 23:46:09 | mriedem | mgagne: i wonder if you had this fix before you upgraded https://review.openstack.org/#/q/Ic96a5eb3728f97a3c35d2c5121e6fdcd4fd1c70b | |
| 23:46:16 | mriedem | https://review.openstack.org/#/c/438632/ | |
| 23:46:48 | mgagne | yes | |
| 23:47:07 | mgagne | but if no entry is found, I think it returns 0 ou None and it fails | |
| 23:47:59 | mriedem | yeah you're right https://github.com/openstack/nova/blob/master/nova/objects/service.py#L431-L434 | |
| 23:48:03 | mgagne | but I don't know if I had to run the migration in api cell or not | |
| 23:48:54 | mgagne | if i shouldn't, well I think the migration script should have told me: hey, this is an api cell, you shouldn't do that. But I understand that cellsv1 isn't fully tested so yea, what can you do =) | |
| 23:49:26 | mriedem | https://github.com/openstack/nova/commit/50355c4595e08f293f610da32247e405b20c1c5b | |
| 23:49:44 | mriedem | yeah my guess is there was no consideration for cells v1 when that was written in mitaka | |
| 23:49:51 | mriedem | and we don't have grenade (upgrade) ci jobs for cellsv1 | |
| 23:50:18 | mriedem | cellsv1 upgrade testing was usually literally alaski or johnthetubaguy saying something broke at rax | |
| 23:50:31 | mgagne | ok, that's fine, mitaka migration is behind us. but I guess same migration will fail again with newton if I try to run it in api cell. | |
| 23:50:42 | mriedem | that code was dropped in newton | |
| 23:50:50 | mriedem | because we have that schema migration blocker in newton | |
| 23:50:56 | mriedem | https://github.com/openstack/nova/blob/stable/newton/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py | |
| 23:51:11 | mriedem | but ^ assumes the child cell db | |
| 23:51:33 | mriedem | so nova-manage db sync | |
| 23:51:38 | mriedem | not nova-manage api_db sync | |
| 23:51:43 | mgagne | that's a fun one: "until all records have been migrated". Ok, let's run that migration then! oh way, code is gone. what now? /sad panda | |
| 23:51:47 | mriedem | so you should be fine | |
| 23:52:17 | mgagne | but then, you will say there is an upgrade readiness check now you can run | |
| 23:52:27 | mgagne | which I'm fine with. ;) | |
| 23:52:30 | mriedem | nova-status was added in ocata | |
| 23:53:20 | mriedem | and checked for things like making sure placement was deployed and cellsv2 mappings existed | |
| 23:53:22 | mgagne | yea, I had a lot of "fun" when it complained about the online migration and old code was gone. had to find a copy in a different environment and rsync that thing. | |
| 23:54:16 | mriedem | i thought people stood up a separate env to run the db sync on the new code before upgrading the old code that's actually running? | |
| 23:54:52 | mgagne | I guess I'm not in that ideal world yet :P | |
| 23:55:33 | mriedem | you can also run the online data migrations from the old mitaka code before upgrading to newton, and run them after upgrading to newton if yo uwant | |
| 23:55:36 | mgagne | usually I stop all services, upgrade package, run db sync, start service. | |
| 23:56:19 | mgagne | but if I forgot to run online migration and db sync fails, I'm screwed because I already upgraded the packages. but my bad for not checking if all online migration ran properly before the upgrade. | |
| 23:56:54 | mgagne | sure but I wasn't prepared for that maneuver | |
| 23:58:02 | mriedem | mgagne: even with that blocker migration script in newton, we didn't delete the cold that allows you to run the online data migrations from mitaka https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L787 | |
| 23:58:08 | mriedem | so, we didn't hose you there | |
| 23:58:18 | mriedem | we just said, you can't continue until you do your homework from mitaka | |
| 23:58:21 | mgagne | very much appreciated =) | |
| 23:59:31 | mriedem | pretty sure that's standard operating procedure | |
| 23:59:41 | mriedem | in queens we still have online data migration code from newton | |
| 23:59:57 | mriedem | we only remove it if we're at least n+1 and someone gets around to caring | |
| #openstack-nova - 2017-11-30 | |||
| 00:00:18 | mriedem | e.g. https://review.openstack.org/#/c/517158/ | |
| 00:01:07 | mriedem | mgagne: btw, that's another reason people can't/shoudn't literally skip through releases for upgrades | |
| 00:02:06 | mgagne | mriedem: I can't afford to not skip versions ;) | |
| 00:02:19 | mriedem | fast forwarding through versions is fine | |
| 00:02:23 | mriedem | but you have to run the data migratoins | |
| 00:02:26 | mriedem | per release | |
| 00:02:45 | mriedem | same issues with dropping config options after n+1 | |
| 00:03:09 | mgagne | yes, but with cells, I'm not sure if I will be able to skip anymore, too many unknown for now | |
| 00:03:37 | mgagne | mriedem: configs are fine, we have funky stuff in puppet to support multiple versions | |
| 00:03:54 | mriedem | like aliases? | |
| 00:06:01 | mgagne | very funky stuff: https://gist.github.com/mgagne/7146424416eda597563c4018ce50cf97 | |
| 00:06:08 | mgagne | copied as-is so you can see our mess | |
| 00:06:33 | mgagne | this class is included in our main nova.pp which does the main configuration | |
| 00:08:14 | mgagne | so I just make it so puppet-nova for newton works with mitaka. and I do the same with other services/modules | |
| 00:08:49 | tssurya_ | mriedem : actually belmiro is currently on newton, moving to ocata (which would be only for a short duration), but main goal is pike. | |
| 00:09:44 | tssurya_ | mridem, dansmith : http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2017-10-24.log.html#t2017-10-24T13:12:39 , the conversation you guys had regarding placement, | |
| 00:10:06 | mriedem | tssurya_: thanks, put that into https://etherpad.openstack.org/p/cellsv1-to-v2-migration | |
| 00:10:17 | tssurya_ | mriedem : sure ! | |
| 00:14:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Enable cold migration with target host(2/2) https://review.openstack.org/408964 | |
| 00:14:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add multi-cell negative test for cold migration with target host https://review.openstack.org/524027 | |
| 00:16:09 | mgagne | mriedem: ok so if you run Cellsv1, you should run placement per cell otherwise nova-scheduler in cell *could* pickup hosts from a different cell? | |
| 00:16:56 | dansmith | it will | |
| 00:17:01 | dansmith | and will have to filter them out | |