| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:11:49 | zioproto | I cant rollback the api_db to mitaka, is this bad ? | |
| 14:11:57 | zioproto | I can only rollback the main db | |
| 14:12:37 | bauzas | zioproto: http://paste.openstack.org/show/600617/ means it didn't went all the way down | |
| 14:12:42 | zioproto | I see | |
| 14:12:43 | zioproto | but | |
| 14:12:46 | zioproto | if I try | |
| 14:12:48 | zioproto | in mitaka | |
| 14:12:48 | bauzas | zioproto: hence the db online migration failing | |
| 14:12:50 | openstackgerrit | Alex Szarka proposed openstack/nova master: Reduce code complexity in vmops._spawn method https://review.openstack.org/438502 | |
| 14:12:57 | johnthetubaguy | no rollback needed here | |
| 14:13:02 | johnthetubaguy | you just need to run mitaka code | |
| 14:13:08 | johnthetubaguy | then do the online migrations | |
| 14:13:09 | bauzas | just run a new db sync | |
| 14:13:17 | mriedem | o/ | |
| 14:13:18 | johnthetubaguy | then do the new db sync | |
| 14:13:29 | bauzas | yeah what johnthetubaguy said | |
| 14:13:55 | bauzas | for mitaka, you need a full code+sync+online_migrations | |
| 14:13:59 | johnthetubaguy | bauzas: do we have docs for this somewhere? like what to do if the migrations tell me I forgot to do online db migrations after upgrading the last release? | |
| 14:14:16 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/438309 | |
| 14:14:39 | johnthetubaguy | mriedem: so I +Wed a thing this morning | |
| 14:14:58 | zioproto | nova-manage: error: argument category: invalid choice: 'help' (choose from 'version', 'bash-completion', 'project', 'account', 'shell', 'logs', 'service', 'cell_v2', 'db', 'vm', 'agent', 'cell', 'network', 'host', 'floating', 'fixed', 'vpn', 'api_db') | |
| 14:14:59 | bauzas | johnthetubaguy: well, I remember some cases where we preventively asked to have the online migrations be done before upgdrading | |
| 14:15:01 | mriedem | this thing? https://review.openstack.org/#/c/335358/ | |
| 14:15:19 | zioproto | what is the command for the onlinemigrations in mitaka ? | |
| 14:15:26 | bauzas | yup | |
| 14:15:37 | zioproto | nova-manage 'what?' | |
| 14:16:12 | mriedem | nova-manage db online_data_migrations | |
| 14:16:12 | zioproto | nova-manage db online_data_migrations | |
| 14:16:17 | zioproto | I try | |
| 14:16:22 | bauzas | nova-manage db online_data_migrations | |
| 14:16:24 | bauzas | graaah | |
| 14:16:30 | bauzas | but I have the jetlag excuse | |
| 14:16:46 | johnthetubaguy | yeah, its not changed between the releases | |
| 14:17:30 | bauzas | johnthetubaguy: https://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process last bullet kinda covers your ask | |
| 14:17:37 | zioproto | this fails | |
| 14:17:38 | zioproto | ERROR nova.db.sqlalchemy.api [req-cc2969ea-476a-4c4c-afe8-5012ad4f2a04 - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 14:18:12 | zioproto | ( I am on a phone call, away from the keyboard for 10 minutes) | |
| 14:19:03 | edleafe | alex_xu: you didn't miss much :) | |
| 14:19:30 | alex_xu | edleafe: yea, i found it ends super early :) | |
| 14:20:06 | edleafe | alex_xu: we | |
| 14:20:10 | edleafe | ugh | |
| 14:20:25 | edleafe | alex_xu: we're all talked out and jetlagged it seems :) | |
| 14:21:01 | bauzas | oh heh https://www.flickr.com/photos/152419717@N06/32267855063/in/album-72157680602754246/lightbox/ | |
| 14:21:13 | alex_xu | cool, nothing I missed | |
| 14:32:41 | openstackgerrit | Monty Taylor proposed openstack/nova master: Use adapter instead of ks_filter https://review.openstack.org/438512 | |
| 14:33:54 | zioproto | I am back | |
| 14:34:06 | zioproto | any idea what is this 'Data migrations for PciDevice are not safe' | |
| 14:34:28 | zioproto | ? | |
| 14:34:45 | dansmith | zioproto: that means you have to start the compute nodes first and let them settle before you can continue to run that migration | |
| 14:34:46 | bauzas | zioproto: the error seems because you haven't upgraded all your controlplane | |
| 14:34:47 | johnthetubaguy | zioproto: was that from mitaka migrations? | |
| 14:34:55 | johnthetubaguy | ah, there we go | |
| 14:34:57 | dansmith | bauzas: computes not control place, iirc | |
| 14:35:15 | bauzas | semantics good point | |
| 14:35:25 | zioproto | what ? | |
| 14:35:30 | zioproto | wait this is a working mitaka setup | |
| 14:35:33 | zioproto | all mitaka now | |
| 14:35:37 | zioproto | and I just run | |
| 14:35:44 | zioproto | nova-manage db online_data_migrations | |
| 14:35:54 | zioproto | all compute nodes are also running mitaka | |
| 14:36:42 | zioproto | johnthetubaguy, yes that was from mitaka migrations | |
| 14:37:53 | bauzas | just pulling the reasoning | |
| 14:37:54 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6487 | |
| 14:38:18 | johnthetubaguy | what is the compute RPC pin in your configuration file? | |
| 14:38:28 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/objects/pci_device.py#L121-L127 | |
| 14:38:47 | johnthetubaguy | oh... | |
| 14:38:53 | bauzas | so, you need to have all your conductors and computes being Mitaka | |
| 14:38:58 | johnthetubaguy | zioproto: do you have two stale service records in your DB? | |
| 14:39:06 | dansmith | conductors would have to have gone first anyway | |
| 14:39:08 | bauzas | well, at least running service version #4 | |
| 14:39:12 | johnthetubaguy | zioproto: like old compute nodes that are no longer running? | |
| 14:39:19 | dansmith | johnthetubaguy: yeah, might be that, although it should be ignoring things that are aged out | |
| 14:39:21 | zioproto | let me check | |
| 14:39:28 | johnthetubaguy | zioproto: I think the other error said you had two nodes that were not migrated | |
| 14:40:10 | dansmith | johnthetubaguy: ah, it ignored forced-down but not aged-down | |
| 14:40:21 | johnthetubaguy | dansmith: ah, was just going to say that | |
| 14:40:21 | zioproto | this is the staging setup, I have exactly two compute nodes | |
| 14:40:22 | dansmith | so yeah, likely an orphaned record or something | |
| 14:41:01 | satyar | Hi johnthethubaguy | |
| 14:41:18 | satyar | are you johngarbutt!!! | |
| 14:42:16 | zioproto | so, I dont undertstand, you want the output of 'openstack hypervisor list' ? | |
| 14:42:27 | dansmith | zioproto: service-list | |
| 14:42:57 | zioproto | http://paste.openstack.org/show/600624/ | |
| 14:43:08 | kashyap | satyar: No need for so many exclamations. If you have a question, just write to the channel, unless you are already interacting with a specific reviewer. | |
| 14:43:09 | dansmith | there ya go | |
| 14:43:23 | dansmith | zioproto: nova-0 used to have a compute on it apparently? | |
| 14:43:47 | dansmith | zioproto: you probably want to just delete that from the db, but you may also be able to force-down it to get past this point | |
| 14:43:49 | zioproto | should not | |
| 14:44:04 | zioproto | nova-0 is a vm where I just run the nova-api services | |
| 14:44:10 | zioproto | it is not a compute node | |
| 14:44:15 | dansmith | well, it reported nova-compute three days ago | |
| 14:44:26 | zioproto | ok good, but what about the second one ? | |
| 14:44:44 | dansmith | zioproto: what second one? | |
| 14:44:46 | johnthetubaguy | satyar: yeah, thats right :) | |
| 14:45:42 | openstack | Launchpad bug 1668267 in OpenStack Compute (nova) "Nova ignores to update the quota for VMs in error" [Undecided,Confirmed] | |
| 14:45:42 | satyar | Thanks johnthetubaguy i just updated the bug(https://bugs.launchpad.net/nova/+bug/1668267) its in the step 3 delete all Vms | |
| 14:46:21 | openstack | Launchpad bug 1667667 in OpenStack Compute (nova) "Rebuilding instance ignores image's property hw_disk_bus" [Low,Confirmed] | |
| 14:46:21 | akijak | hi, I've found a bug in nova concerning rebuilding and 'hw_disk_bus' image property (https://bugs.launchpad.net/nova/+bug/1667667), I'm looking for someone who could give me some directions how to fix that :) can you help? | |
| 14:46:39 | zioproto | dansmith, http://paste.openstack.org/show/600617/ There are still 2 unmigrated records in the compute_nodes table. | |
| 14:47:11 | satyar | kashyap its a usual practice if you are not sure about asking someone is thats right what you are asking putting 3 exclamation marks | |