| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 13:44:04 | akijak | rpodolyaka: I've found a bug in nova concerning rebuilding and 'hw_disk_bus' image property (https://bugs.launchpad.net/nova/+bug/1667667), I'm looking for someone who could give me some directions how to fix that :) can you help? | |
| 13:44:30 | satyar | i tried but not able to get it | |
| 13:44:43 | satyar | it seems its a blocker for ocata release | |
| 13:44:44 | openstack | Launchpad bug 1668267 in OpenStack Compute (nova) "Nova ignores to update the quota for VMs in error" [Undecided,New] | |
| 13:44:44 | satyar | https://bugs.launchpad.net/nova/+bug/1668267 | |
| 13:50:57 | zioproto | lyarwood, can anyone confirm that to run newton I have to deploy a third mysql database for nova calling it 'cell0' | |
| 13:50:59 | zioproto | > | |
| 13:51:00 | zioproto | ?? | |
| 13:51:46 | edleafe | Nova scheduler subteam meeting in 8 minutes in #openstack-meeting-alt | |
| 13:59:03 | lyarwood | zioproto: I believe so but that isn't really my area, have you checked devstack stable/newton out? | |
| 14:00:09 | johnthetubaguy | satyar: instances in error have always consumed quota I believe, you have to delete them to release the quota | |
| 14:01:57 | zioproto | lyarwood, I am upgrading a real staging cluster from mitaka to newton | |
| 14:02:07 | zioproto | in the newton release notes it does not say anything about this db | |
| 14:02:50 | johnthetubaguy | its required for newton, it is in the release notes, but its maybe not that obvious | |
| 14:02:53 | johnthetubaguy | its an ordering thing | |
| 14:03:09 | johnthetubaguy | the db sync will create the DB once the cell0 mapping is in place | |
| 14:03:29 | zioproto | so nova.conf should have cell0 db config ? | |
| 14:03:36 | zioproto | before running the db sync | |
| 14:03:40 | zioproto | of course | |
| 14:03:42 | zioproto | but wht | |
| 14:03:44 | zioproto | why | |
| 14:03:49 | zioproto | in the release notes | |
| 14:03:50 | johnthetubaguy | zioproto: the details should be here: https://docs.openstack.org/developer/nova/cells.html#upgrade-minimal | |
| 14:03:51 | zioproto | there is written | |
| 14:03:54 | zioproto | configuring Cells V2 is still fully optional for this cycle | |
| 14:04:07 | zioproto | https://docs.openstack.org/releasenotes/nova/newton.html -> configuring Cells V2 is still fully optional for this cycle | |
| 14:05:04 | zioproto | oh so the transport-url is something to access the db | |
| 14:05:06 | johnthetubaguy | ah, yes, I am out by one | |
| 14:05:23 | johnthetubaguy | its optional in newton, required in ocata | |
| 14:05:27 | johnthetubaguy | what errors are you seeing? | |
| 14:05:35 | zioproto | wait, I pastebin | |
| 14:05:51 | johnthetubaguy | it sounds like you ran the cell mapping, cmd, which would make it required | |
| 14:06:03 | johnthetubaguy | if there is no cell mapping done, its not needed, I think... | |
| 14:06:19 | zioproto | http://paste.openstack.org/show/600616/ | |
| 14:06:24 | johnthetubaguy | we might want to take this onto #openstack, given its an older release | |
| 14:07:00 | johnthetubaguy | zioproto: do you have the API database created, that is not optional in newton | |
| 14:07:25 | zioproto | yes I have that | |
| 14:07:30 | zioproto | I have it since mitaka | |
| 14:07:33 | johnthetubaguy | has it been updated with api_db sync? | |
| 14:07:36 | zioproto | yes | |
| 14:07:38 | zioproto | that worked | |
| 14:07:50 | bauzas | johnthetubaguy: instance_extra is not in the API DB | |
| 14:08:09 | bauzas | zioproto: looks like a migration badly happened when db sync | |
| 14:08:29 | zioproto | I have the backup of the db | |
| 14:08:33 | zioproto | at mitaka version | |
| 14:08:36 | zioproto | I can reimport it | |
| 14:08:39 | bauzas | zioproto: do you still have the logs of the 'nova-manage db sync' command ? | |
| 14:08:47 | bauzas | I mean the output | |
| 14:09:27 | johnthetubaguy | bauzas: ah, yeah, I forgot that was "new" | |
| 14:10:04 | zioproto | bauzas, http://paste.openstack.org/show/600617/ | |
| 14:10:08 | johnthetubaguy | oh wait, its doing migrate keypairs on an old DB somehow | |
| 14:10:33 | johnthetubaguy | ah, so that did fail | |
| 14:11:04 | johnthetubaguy | https://github.com/openstack/nova/blob/6d64b7274410ae45b95bd7ac7f702c16daaa0fcd/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py#L19 | |
| 14:11:17 | johnthetubaguy | zioproto: it seems the online db migrations on mitaka were not run | |
| 14:11:26 | johnthetubaguy | zioproto: you need to run those using the mitaka code base | |
| 14:11:34 | zioproto | what did I do wrong ? | |
| 14:11:42 | bauzas | sec, grabbing coffee for a jet-lag problem | |
| 14:11:43 | bauzas | https://www.internetmarketingninjas.com/blog/wp-content/uploads/2012/07/6-cool-productivity-charts-06.jpg | |
| 14:11:49 | zioproto | I cant rollback the api_db to mitaka, is this bad ? | |
| 14:11:57 | zioproto | I can only rollback the main db | |
| 14:12:37 | bauzas | zioproto: http://paste.openstack.org/show/600617/ means it didn't went all the way down | |
| 14:12:42 | zioproto | I see | |
| 14:12:43 | zioproto | but | |
| 14:12:46 | zioproto | if I try | |
| 14:12:48 | zioproto | in mitaka | |
| 14:12:48 | bauzas | zioproto: hence the db online migration failing | |
| 14:12:50 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity in vmops._spawn method https://review.openstack.org/438502 | |
| 14:12:57 | johnthetubaguy | no rollback needed here | |
| 14:13:02 | johnthetubaguy | you just need to run mitaka code | |
| 14:13:08 | johnthetubaguy | then do the online migrations | |
| 14:13:09 | bauzas | just run a new db sync | |
| 14:13:17 | mriedem | o/ | |
| 14:13:18 | johnthetubaguy | then do the new db sync | |
| 14:13:29 | bauzas | yeah what johnthetubaguy said | |
| 14:13:55 | bauzas | for mitaka, you need a full code+sync+online_migrations | |
| 14:13:59 | johnthetubaguy | bauzas: do we have docs for this somewhere? like what to do if the migrations tell me I forgot to do online db migrations after upgrading the last release? | |
| 14:14:16 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/438309 | |
| 14:14:39 | johnthetubaguy | mriedem: so I +Wed a thing this morning | |
| 14:14:58 | zioproto | nova-manage: error: argument category: invalid choice: 'help' (choose from 'version', 'bash-completion', 'project', 'account', 'shell', 'logs', 'service', 'cell_v2', 'db', 'vm', 'agent', 'cell', 'network', 'host', 'floating', 'fixed', 'vpn', 'api_db') | |
| 14:14:59 | bauzas | johnthetubaguy: well, I remember some cases where we preventively asked to have the online migrations be done before upgdrading | |
| 14:15:01 | mriedem | this thing? https://review.openstack.org/#/c/335358/ | |
| 14:15:19 | zioproto | what is the command for the onlinemigrations in mitaka ? | |
| 14:15:26 | bauzas | yup | |
| 14:15:37 | zioproto | nova-manage 'what?' | |
| 14:16:12 | mriedem | nova-manage db online_data_migrations | |
| 14:16:12 | zioproto | nova-manage db online_data_migrations | |
| 14:16:17 | zioproto | I try | |
| 14:16:22 | bauzas | nova-manage db online_data_migrations | |
| 14:16:24 | bauzas | graaah | |
| 14:16:30 | bauzas | but I have the jetlag excuse | |
| 14:16:46 | johnthetubaguy | yeah, its not changed between the releases | |
| 14:17:30 | bauzas | johnthetubaguy: https://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process last bullet kinda covers your ask | |
| 14:17:37 | zioproto | this fails | |
| 14:17:38 | zioproto | ERROR nova.db.sqlalchemy.api [req-cc2969ea-476a-4c4c-afe8-5012ad4f2a04 - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 14:18:12 | zioproto | ( I am on a phone call, away from the keyboard for 10 minutes) | |
| 14:19:03 | edleafe | alex_xu: you didn't miss much :) | |
| 14:19:30 | alex_xu | edleafe: yea, i found it ends super early :) | |
| 14:20:06 | edleafe | alex_xu: we | |
| 14:20:10 | edleafe | ugh | |
| 14:20:25 | edleafe | alex_xu: we're all talked out and jetlagged it seems :) | |