| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:09:27 | johnthetubaguy | bauzas: ah, yeah, I forgot that was "new" | |
| 14:10:04 | zioproto | bauzas, http://paste.openstack.org/show/600617/ | |
| 14:10:08 | johnthetubaguy | oh wait, its doing migrate keypairs on an old DB somehow | |
| 14:10:33 | johnthetubaguy | ah, so that did fail | |
| 14:11:04 | johnthetubaguy | https://github.com/openstack/nova/blob/6d64b7274410ae45b95bd7ac7f702c16daaa0fcd/nova/db/sqlalchemy/migrate_repo/versions/330_enforce_mitaka_online_migrations.py#L19 | |
| 14:11:17 | johnthetubaguy | zioproto: it seems the online db migrations on mitaka were not run | |
| 14:11:26 | johnthetubaguy | zioproto: you need to run those using the mitaka code base | |
| 14:11:34 | zioproto | what did I do wrong ? | |
| 14:11:42 | bauzas | sec, grabbing coffee for a jet-lag problem | |
| 14:11:43 | bauzas | https://www.internetmarketingninjas.com/blog/wp-content/uploads/2012/07/6-cool-productivity-charts-06.jpg | |
| 14:11:49 | zioproto | I cant rollback the api_db to mitaka, is this bad ? | |
| 14:11:57 | zioproto | I can only rollback the main db | |
| 14:12:37 | bauzas | zioproto: http://paste.openstack.org/show/600617/ means it didn't went all the way down | |
| 14:12:42 | zioproto | I see | |
| 14:12:43 | zioproto | but | |
| 14:12:46 | zioproto | if I try | |
| 14:12:48 | bauzas | zioproto: hence the db online migration failing | |
| 14:12:48 | zioproto | in mitaka | |
| 14:12:50 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity in vmops._spawn method https://review.openstack.org/438502 | |
| 14:12:57 | johnthetubaguy | no rollback needed here | |
| 14:13:02 | johnthetubaguy | you just need to run mitaka code | |
| 14:13:08 | johnthetubaguy | then do the online migrations | |
| 14:13:09 | bauzas | just run a new db sync | |
| 14:13:17 | mriedem | o/ | |
| 14:13:18 | johnthetubaguy | then do the new db sync | |
| 14:13:29 | bauzas | yeah what johnthetubaguy said | |
| 14:13:55 | bauzas | for mitaka, you need a full code+sync+online_migrations | |
| 14:13:59 | johnthetubaguy | bauzas: do we have docs for this somewhere? like what to do if the migrations tell me I forgot to do online db migrations after upgrading the last release? | |
| 14:14:16 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/438309 | |
| 14:14:39 | johnthetubaguy | mriedem: so I +Wed a thing this morning | |
| 14:14:58 | zioproto | nova-manage: error: argument category: invalid choice: 'help' (choose from 'version', 'bash-completion', 'project', 'account', 'shell', 'logs', 'service', 'cell_v2', 'db', 'vm', 'agent', 'cell', 'network', 'host', 'floating', 'fixed', 'vpn', 'api_db') | |
| 14:14:59 | bauzas | johnthetubaguy: well, I remember some cases where we preventively asked to have the online migrations be done before upgdrading | |
| 14:15:01 | mriedem | this thing? https://review.openstack.org/#/c/335358/ | |
| 14:15:19 | zioproto | what is the command for the onlinemigrations in mitaka ? | |
| 14:15:26 | bauzas | yup | |
| 14:15:37 | zioproto | nova-manage 'what?' | |
| 14:16:12 | zioproto | nova-manage db online_data_migrations | |
| 14:16:12 | mriedem | nova-manage db online_data_migrations | |
| 14:16:17 | zioproto | I try | |
| 14:16:22 | bauzas | nova-manage db online_data_migrations | |
| 14:16:24 | bauzas | graaah | |
| 14:16:30 | bauzas | but I have the jetlag excuse | |
| 14:16:46 | johnthetubaguy | yeah, its not changed between the releases | |
| 14:17:30 | bauzas | johnthetubaguy: https://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process last bullet kinda covers your ask | |
| 14:17:37 | zioproto | this fails | |
| 14:17:38 | zioproto | ERROR nova.db.sqlalchemy.api [req-cc2969ea-476a-4c4c-afe8-5012ad4f2a04 - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 14:18:12 | zioproto | ( I am on a phone call, away from the keyboard for 10 minutes) | |
| 14:19:03 | edleafe | alex_xu: you didn't miss much :) | |
| 14:19:30 | alex_xu | edleafe: yea, i found it ends super early :) | |
| 14:20:06 | edleafe | alex_xu: we | |
| 14:20:10 | edleafe | ugh | |
| 14:20:25 | edleafe | alex_xu: we're all talked out and jetlagged it seems :) | |
| 14:21:01 | bauzas | oh heh https://www.flickr.com/photos/152419717@N06/32267855063/in/album-72157680602754246/lightbox/ | |
| 14:21:13 | alex_xu | cool, nothing I missed | |
| 14:32:41 | openstackgerrit | Monty Taylor proposed openstack/nova master: Use adapter instead of ks_filter https://review.openstack.org/438512 | |
| 14:33:54 | zioproto | I am back | |
| 14:34:06 | zioproto | any idea what is this 'Data migrations for PciDevice are not safe' | |
| 14:34:28 | zioproto | ? | |
| 14:34:45 | dansmith | zioproto: that means you have to start the compute nodes first and let them settle before you can continue to run that migration | |
| 14:34:46 | bauzas | zioproto: the error seems because you haven't upgraded all your controlplane | |
| 14:34:47 | johnthetubaguy | zioproto: was that from mitaka migrations? | |
| 14:34:55 | johnthetubaguy | ah, there we go | |
| 14:34:57 | dansmith | bauzas: computes not control place, iirc | |
| 14:35:15 | bauzas | semantics good point | |
| 14:35:25 | zioproto | what ? | |
| 14:35:30 | zioproto | wait this is a working mitaka setup | |
| 14:35:33 | zioproto | all mitaka now | |
| 14:35:37 | zioproto | and I just run | |
| 14:35:44 | zioproto | nova-manage db online_data_migrations | |
| 14:35:54 | zioproto | all compute nodes are also running mitaka | |
| 14:36:42 | zioproto | johnthetubaguy, yes that was from mitaka migrations | |
| 14:37:53 | bauzas | just pulling the reasoning | |
| 14:37:54 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6487 | |
| 14:38:18 | johnthetubaguy | what is the compute RPC pin in your configuration file? | |
| 14:38:28 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/objects/pci_device.py#L121-L127 | |
| 14:38:47 | johnthetubaguy | oh... | |
| 14:38:53 | bauzas | so, you need to have all your conductors and computes being Mitaka | |
| 14:38:58 | johnthetubaguy | zioproto: do you have two stale service records in your DB? | |
| 14:39:06 | dansmith | conductors would have to have gone first anyway | |
| 14:39:08 | bauzas | well, at least running service version #4 | |
| 14:39:12 | johnthetubaguy | zioproto: like old compute nodes that are no longer running? | |
| 14:39:19 | dansmith | johnthetubaguy: yeah, might be that, although it should be ignoring things that are aged out | |
| 14:39:21 | zioproto | let me check | |
| 14:39:28 | johnthetubaguy | zioproto: I think the other error said you had two nodes that were not migrated | |
| 14:40:10 | dansmith | johnthetubaguy: ah, it ignored forced-down but not aged-down | |
| 14:40:21 | zioproto | this is the staging setup, I have exactly two compute nodes | |
| 14:40:21 | johnthetubaguy | dansmith: ah, was just going to say that | |
| 14:40:22 | dansmith | so yeah, likely an orphaned record or something | |
| 14:41:01 | satyar | Hi johnthethubaguy | |
| 14:41:18 | satyar | are you johngarbutt!!! | |
| 14:42:16 | zioproto | so, I dont undertstand, you want the output of 'openstack hypervisor list' ? | |
| 14:42:27 | dansmith | zioproto: service-list | |
| 14:42:57 | zioproto | http://paste.openstack.org/show/600624/ | |
| 14:43:08 | kashyap | satyar: No need for so many exclamations. If you have a question, just write to the channel, unless you are already interacting with a specific reviewer. | |
| 14:43:09 | dansmith | there ya go | |
| 14:43:23 | dansmith | zioproto: nova-0 used to have a compute on it apparently? | |
| 14:43:47 | dansmith | zioproto: you probably want to just delete that from the db, but you may also be able to force-down it to get past this point | |
| 14:43:49 | zioproto | should not | |
| 14:44:04 | zioproto | nova-0 is a vm where I just run the nova-api services | |
| 14:44:10 | zioproto | it is not a compute node | |