| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:03:49 | zioproto | in the release notes | |
| 14:03:50 | johnthetubaguy | zioproto: the details should be here: https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal | |
| 14:03:51 | zioproto | there is written | |
| 14:03:54 | zioproto | configuring Cells V2 is still fully optional for this cycle | |
| 14:04:07 | zioproto | https://docs.openstack.org/releasenotes/nova/newton.html -> configuring Cells V2 is still fully optional for this cycle | |
| 14:05:04 | zioproto | oh so the transport-url is something to access the db | |
| 14:05:06 | johnthetubaguy | ah, yes, I am out by one | |
| 14:05:23 | johnthetubaguy | its optional in newton, required in ocata | |
| 14:05:27 | johnthetubaguy | what errors are you seeing? | |
| 14:05:35 | zioproto | wait, I pastebin | |
| 14:05:51 | johnthetubaguy | it sounds like you ran the cell mapping, cmd, which would make it required | |
| 14:06:03 | johnthetubaguy | if there is no cell mapping done, its not needed, I think... | |
| 14:06:19 | zioproto | http://paste.openstack.org/show/600616/ | |
| 14:06:24 | johnthetubaguy | we might want to take this onto #openstack, given its an older release | |
| 14:07:00 | johnthetubaguy | zioproto: do you have the API database created, that is not optional in newton | |
| 14:07:25 | zioproto | yes I have that | |
| 14:07:30 | zioproto | I have it since mitaka | |
| 14:07:33 | johnthetubaguy | has it been updated with api_db sync? | |
| 14:07:36 | zioproto | yes | |
| 14:07:38 | zioproto | that worked | |
| 14:07:50 | bauzas | johnthetubaguy: instance_extra is not in the API DB | |
| 14:08:09 | bauzas | zioproto: looks like a migration badly happened when db sync | |
| 14:08:29 | zioproto | I have the backup of the db | |
| 14:08:33 | zioproto | at mitaka version | |
| 14:08:36 | zioproto | I can reimport it | |
| 14:08:39 | bauzas | zioproto: do you still have the logs of the 'nova-manage db sync' command ? | |
| 14:08:47 | bauzas | I mean the output | |
| 14:09:27 | johnthetubaguy | bauzas: ah, yeah, I forgot that was "new" | |
| 14:10:04 | zioproto | bauzas, http://paste.openstack.org/show/600617/ | |
| 14:10:08 | johnthetubaguy | oh wait, its doing migrate keypairs on an old DB somehow | |
| 14:10:33 | johnthetubaguy | ah, so that did fail | |
| 14:11:04 | johnthetubaguy | https://github.com/openstack/nova/blob/6d64b7274410ae45b95bd7ac7f702c16daaa0fcd/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py#L19 | |
| 14:11:17 | johnthetubaguy | zioproto: it seems the online db migrations on mitaka were not run | |
| 14:11:26 | johnthetubaguy | zioproto: you need to run those using the mitaka code base | |
| 14:11:34 | zioproto | what did I do wrong ? | |
| 14:11:42 | bauzas | sec, grabbing coffee for a jet-lag problem | |
| 14:11:43 | bauzas | https://www.internetmarketingninjas.com/blog/wp-content/uploads/2012/07/6-cool-productivity-charts-06.jpg | |
| 14:11:49 | zioproto | I cant rollback the api_db to mitaka, is this bad ? | |
| 14:11:57 | zioproto | I can only rollback the main db | |
| 14:12:37 | bauzas | zioproto: http://paste.openstack.org/show/600617/ means it didn't went all the way down | |
| 14:12:42 | zioproto | I see | |
| 14:12:43 | zioproto | but | |
| 14:12:46 | zioproto | if I try | |
| 14:12:48 | zioproto | in mitaka | |
| 14:12:48 | bauzas | zioproto: hence the db online migration failing | |
| 14:12:50 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity in vmops._spawn method https://review.openstack.org/438502 | |
| 14:12:57 | johnthetubaguy | no rollback needed here | |
| 14:13:02 | johnthetubaguy | you just need to run mitaka code | |
| 14:13:08 | johnthetubaguy | then do the online migrations | |
| 14:13:09 | bauzas | just run a new db sync | |
| 14:13:17 | mriedem | o/ | |
| 14:13:18 | johnthetubaguy | then do the new db sync | |
| 14:13:29 | bauzas | yeah what johnthetubaguy said | |
| 14:13:55 | bauzas | for mitaka, you need a full code+sync+online_migrations | |
| 14:13:59 | johnthetubaguy | bauzas: do we have docs for this somewhere? like what to do if the migrations tell me I forgot to do online db migrations after upgrading the last release? | |
| 14:14:16 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/438309 | |
| 14:14:39 | johnthetubaguy | mriedem: so I +Wed a thing this morning | |
| 14:14:58 | zioproto | nova-manage: error: argument category: invalid choice: 'help' (choose from 'version', 'bash-completion', 'project', 'account', 'shell', 'logs', 'service', 'cell_v2', 'db', 'vm', 'agent', 'cell', 'network', 'host', 'floating', 'fixed', 'vpn', 'api_db') | |
| 14:14:59 | bauzas | johnthetubaguy: well, I remember some cases where we preventively asked to have the online migrations be done before upgdrading | |
| 14:15:01 | mriedem | this thing? https://review.openstack.org/#/c/335358/ | |
| 14:15:19 | zioproto | what is the command for the onlinemigrations in mitaka ? | |
| 14:15:26 | bauzas | yup | |
| 14:15:37 | zioproto | nova-manage 'what?' | |
| 14:16:12 | mriedem | nova-manage db online_data_migrations | |
| 14:16:12 | zioproto | nova-manage db online_data_migrations | |
| 14:16:17 | zioproto | I try | |
| 14:16:22 | bauzas | nova-manage db online_data_migrations | |
| 14:16:24 | bauzas | graaah | |
| 14:16:30 | bauzas | but I have the jetlag excuse | |
| 14:16:46 | johnthetubaguy | yeah, its not changed between the releases | |
| 14:17:30 | bauzas | johnthetubaguy: https://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process last bullet kinda covers your ask | |
| 14:17:37 | zioproto | this fails | |
| 14:17:38 | zioproto | ERROR nova.db.sqlalchemy.api [req-cc2969ea-476a-4c4c-afe8-5012ad4f2a04 - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 14:18:12 | zioproto | ( I am on a phone call, away from the keyboard for 10 minutes) | |
| 14:19:03 | edleafe | alex_xu: you didn't miss much :) | |
| 14:19:30 | alex_xu | edleafe: yea, i found it ends super early :) | |
| 14:20:06 | edleafe | alex_xu: we | |
| 14:20:10 | edleafe | ugh | |
| 14:20:25 | edleafe | alex_xu: we're all talked out and jetlagged it seems :) | |
| 14:21:01 | bauzas | oh heh https://www.flickr.com/photos/152419717@N06/32267855063/in/album-72157680602754246/lightbox/ | |
| 14:21:13 | alex_xu | cool, nothing I missed | |
| 14:32:41 | openstackgerrit | Monty Taylor proposed openstack/nova master: Use adapter instead of ks_filter https://review.openstack.org/438512 | |
| 14:33:54 | zioproto | I am back | |
| 14:34:06 | zioproto | any idea what is this 'Data migrations for PciDevice are not safe' | |
| 14:34:28 | zioproto | ? | |
| 14:34:45 | dansmith | zioproto: that means you have to start the compute nodes first and let them settle before you can continue to run that migration | |
| 14:34:46 | bauzas | zioproto: the error seems because you haven't upgraded all your controlplane | |
| 14:34:47 | johnthetubaguy | zioproto: was that from mitaka migrations? | |
| 14:34:55 | johnthetubaguy | ah, there we go | |
| 14:34:57 | dansmith | bauzas: computes not control place, iirc | |
| 14:35:15 | bauzas | semantics good point | |
| 14:35:25 | zioproto | what ? | |
| 14:35:30 | zioproto | wait this is a working mitaka setup | |
| 14:35:33 | zioproto | all mitaka now | |
| 14:35:37 | zioproto | and I just run | |
| 14:35:44 | zioproto | nova-manage db online_data_migrations | |
| 14:35:54 | zioproto | all compute nodes are also running mitaka | |
| 14:36:42 | zioproto | johnthetubaguy, yes that was from mitaka migrations | |
| 14:37:53 | bauzas | just pulling the reasoning | |
| 14:37:54 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6487 | |