| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:16:17 | zioproto | I try | |
| 14:16:22 | bauzas | nova-manage db online_data_migrations | |
| 14:16:24 | bauzas | graaah | |
| 14:16:30 | bauzas | but I have the jetlag excuse | |
| 14:16:46 | johnthetubaguy | yeah, its not changed between the releases | |
| 14:17:30 | bauzas | johnthetubaguy: https://docs.openstack.org/developer/nova/upgrade.html#rolling-upgrade-process last bullet kinda covers your ask | |
| 14:17:37 | zioproto | this fails | |
| 14:17:38 | zioproto | ERROR nova.db.sqlalchemy.api [req-cc2969ea-476a-4c4c-afe8-5012ad4f2a04 - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 14:18:12 | zioproto | ( I am on a phone call, away from the keyboard for 10 minutes) | |
| 14:19:03 | edleafe | alex_xu: you didn't miss much :) | |
| 14:19:30 | alex_xu | edleafe: yea, i found it ends super early :) | |
| 14:20:06 | edleafe | alex_xu: we | |
| 14:20:10 | edleafe | ugh | |
| 14:20:25 | edleafe | alex_xu: we're all talked out and jetlagged it seems :) | |
| 14:21:01 | bauzas | oh heh https://www.flickr.com/photos/152419717@N06/32267855063/in/album-72157680602754246/lightbox/ | |
| 14:21:13 | alex_xu | cool, nothing I missed | |
| 14:32:41 | openstackgerrit | Monty Taylor proposed openstack/nova master: Use adapter instead of ks_filter https://review.openstack.org/438512 | |
| 14:33:54 | zioproto | I am back | |
| 14:34:06 | zioproto | any idea what is this 'Data migrations for PciDevice are not safe' | |
| 14:34:28 | zioproto | ? | |
| 14:34:45 | dansmith | zioproto: that means you have to start the compute nodes first and let them settle before you can continue to run that migration | |
| 14:34:46 | bauzas | zioproto: the error seems because you haven't upgraded all your controlplane | |
| 14:34:47 | johnthetubaguy | zioproto: was that from mitaka migrations? | |
| 14:34:55 | johnthetubaguy | ah, there we go | |
| 14:34:57 | dansmith | bauzas: computes not control place, iirc | |
| 14:35:15 | bauzas | semantics good point | |
| 14:35:25 | zioproto | what ? | |
| 14:35:30 | zioproto | wait this is a working mitaka setup | |
| 14:35:33 | zioproto | all mitaka now | |
| 14:35:37 | zioproto | and I just run | |
| 14:35:44 | zioproto | nova-manage db online_data_migrations | |
| 14:35:54 | zioproto | all compute nodes are also running mitaka | |
| 14:36:42 | zioproto | johnthetubaguy, yes that was from mitaka migrations | |
| 14:37:53 | bauzas | just pulling the reasoning | |
| 14:37:54 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6487 | |
| 14:38:18 | johnthetubaguy | what is the compute RPC pin in your configuration file? | |
| 14:38:28 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/objects/pci_device.py#L121-L127 | |
| 14:38:47 | johnthetubaguy | oh... | |
| 14:38:53 | bauzas | so, you need to have all your conductors and computes being Mitaka | |
| 14:38:58 | johnthetubaguy | zioproto: do you have two stale service records in your DB? | |
| 14:39:06 | dansmith | conductors would have to have gone first anyway | |
| 14:39:08 | bauzas | well, at least running service version #4 | |
| 14:39:12 | johnthetubaguy | zioproto: like old compute nodes that are no longer running? | |
| 14:39:19 | dansmith | johnthetubaguy: yeah, might be that, although it should be ignoring things that are aged out | |
| 14:39:21 | zioproto | let me check | |
| 14:39:28 | johnthetubaguy | zioproto: I think the other error said you had two nodes that were not migrated | |
| 14:40:10 | dansmith | johnthetubaguy: ah, it ignored forced-down but not aged-down | |
| 14:40:21 | zioproto | this is the staging setup, I have exactly two compute nodes | |
| 14:40:21 | johnthetubaguy | dansmith: ah, was just going to say that | |
| 14:40:22 | dansmith | so yeah, likely an orphaned record or something | |
| 14:41:01 | satyar | Hi johnthethubaguy | |
| 14:41:18 | satyar | are you johngarbutt!!! | |
| 14:42:16 | zioproto | so, I dont undertstand, you want the output of 'openstack hypervisor list' ? | |
| 14:42:27 | dansmith | zioproto: service-list | |
| 14:42:57 | zioproto | http://paste.openstack.org/show/600624/ | |
| 14:43:08 | kashyap | satyar: No need for so many exclamations. If you have a question, just write to the channel, unless you are already interacting with a specific reviewer. | |
| 14:43:09 | dansmith | there ya go | |
| 14:43:23 | dansmith | zioproto: nova-0 used to have a compute on it apparently? | |
| 14:43:47 | dansmith | zioproto: you probably want to just delete that from the db, but you may also be able to force-down it to get past this point | |
| 14:43:49 | zioproto | should not | |
| 14:44:04 | zioproto | nova-0 is a vm where I just run the nova-api services | |
| 14:44:10 | zioproto | it is not a compute node | |
| 14:44:15 | dansmith | well, it reported nova-compute three days ago | |
| 14:44:26 | zioproto | ok good, but what about the second one ? | |
| 14:44:44 | dansmith | zioproto: what second one? | |
| 14:44:46 | johnthetubaguy | satyar: yeah, thats right :) | |
| 14:45:42 | satyar | Thanks johnthetubaguy i just updated the bug(https://bugs.launchpad.net/nova/+bug/1668267) its in the step 3 delete all Vms | |
| 14:45:42 | openstack | Launchpad bug 1668267 in OpenStack Compute (nova) "Nova ignores to update the quota for VMs in error" [Undecided,Confirmed] | |
| 14:46:21 | akijak | hi, I've found a bug in nova concerning rebuilding and 'hw_disk_bus' image property (https://bugs.launchpad.net/nova/+bug/1667667), I'm looking for someone who could give me some directions how to fix that :) can you help? | |
| 14:46:21 | openstack | Launchpad bug 1667667 in OpenStack Compute (nova) "Rebuilding instance ignores image's property hw_disk_bus" [Low,Confirmed] | |
| 14:46:39 | zioproto | dansmith, http://paste.openstack.org/show/600617/ There are still 2 unmigrated records in the compute_nodes table. | |
| 14:47:11 | satyar | kashyap its a usual practice if you are not sure about asking someone is thats right what you are asking putting 3 exclamation marks | |
| 14:47:40 | johnthetubaguy | satyar: asked you a follow up on the bug, I am curious about what list instances shows after deleting the VMs | |
| 14:47:47 | dansmith | zioproto: I would resolve the pci thing and see if you make more online migration progress | |
| 14:47:50 | dansmith | zioproto: that message is from the newton db sync right? | |
| 14:48:25 | zioproto | right | |
| 14:48:28 | satyar | no VMs present | |
| 14:48:51 | johnthetubaguy | satyar: do you have a setup where you can test the same thing with master? | |
| 14:49:04 | johnthetubaguy | its basically the same thing right now | |
| 14:49:19 | satyar | i am running with master also | |
| 14:49:28 | satyar | same issue | |
| 14:50:17 | johnthetubaguy | satyar: sounds crazy to ask, but how did you delete all the VMs, just a quick loop that called the delete API call? | |
| 14:50:30 | dansmith | zioproto: just to be clear, you're running online migrations with the mitaka code, correct? | |
| 14:50:47 | satyar | no | |
| 14:50:47 | zioproto | correct | |
| 14:50:58 | zioproto | I deleted that service | |
| 14:51:05 | zioproto | now I will try again the online migrations | |
| 14:51:09 | dansmith | zioproto: cool, yep | |
| 14:51:18 | johnthetubaguy | satyar: how did you delete them all? | |
| 14:51:49 | zioproto | dansmith, is it a problem that I have already the db_api in the newton version ? I cant downgrade that db :( | |
| 14:51:51 | satyar | johnthetubagu: i waited for VMs to go to error state for which block device mapping failed then after 5 mins delete the VMs | |
| 14:51:57 | satyar | from horizon | |
| 14:52:03 | satyar | select all and delete | |
| 14:52:21 | johnthetubaguy | satyar: thats cool, can you add that detail into the bug please? | |
| 14:52:41 | dansmith | zioproto: not sure what you mean.. the db sync is stopping on the newton side because it's detecting you haven't done your work in mitaka | |
| 14:52:51 | satyar | sure | |
| 14:52:52 | zioproto | dansmith, same problem http://paste.openstack.org/show/600626/ | |
| 14:53:13 | johnthetubaguy | satyar: I know the current quota system is racey on delete, but this seems a bit odd | |
| 14:53:17 | mriedem | edleafe: if you're aware of specs needing re-approvals for any of the resource provider stuff just ping me if they aren't getting attention | |
| 14:53:19 | dansmith | zioproto: what does service-list show now? and what did you do to delete the old service? | |