| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:35:30 | zioproto | wait this is a working mitaka setup | |
| 14:35:33 | zioproto | all mitaka now | |
| 14:35:37 | zioproto | and I just run | |
| 14:35:44 | zioproto | nova-manage db online_data_migrations | |
| 14:35:54 | zioproto | all compute nodes are also running mitaka | |
| 14:36:42 | zioproto | johnthetubaguy, yes that was from mitaka migrations | |
| 14:37:53 | bauzas | just pulling the reasoning | |
| 14:37:54 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6487 | |
| 14:38:18 | johnthetubaguy | what is the compute RPC pin in your configuration file? | |
| 14:38:28 | bauzas | https://github.com/openstack/nova/blob/stable/mitaka/nova/objects/pci_device.py#L121-L127 | |
| 14:38:47 | johnthetubaguy | oh... | |
| 14:38:53 | bauzas | so, you need to have all your conductors and computes being Mitaka | |
| 14:38:58 | johnthetubaguy | zioproto: do you have two stale service records in your DB? | |
| 14:39:06 | dansmith | conductors would have to have gone first anyway | |
| 14:39:08 | bauzas | well, at least running service version #4 | |
| 14:39:12 | johnthetubaguy | zioproto: like old compute nodes that are no longer running? | |
| 14:39:19 | dansmith | johnthetubaguy: yeah, might be that, although it should be ignoring things that are aged out | |
| 14:39:21 | zioproto | let me check | |
| 14:39:28 | johnthetubaguy | zioproto: I think the other error said you had two nodes that were not migrated | |
| 14:40:10 | dansmith | johnthetubaguy: ah, it ignored forced-down but not aged-down | |
| 14:40:21 | zioproto | this is the staging setup, I have exactly two compute nodes | |
| 14:40:21 | johnthetubaguy | dansmith: ah, was just going to say that | |
| 14:40:22 | dansmith | so yeah, likely an orphaned record or something | |
| 14:41:01 | satyar | Hi johnthethubaguy | |
| 14:41:18 | satyar | are you johngarbutt!!! | |
| 14:42:16 | zioproto | so, I dont undertstand, you want the output of 'openstack hypervisor list' ? | |
| 14:42:27 | dansmith | zioproto: service-list | |
| 14:42:57 | zioproto | http://paste.openstack.org/show/600624/ | |
| 14:43:08 | kashyap | satyar: No need for so many exclamations. If you have a question, just write to the channel, unless you are already interacting with a specific reviewer. | |
| 14:43:09 | dansmith | there ya go | |
| 14:43:23 | dansmith | zioproto: nova-0 used to have a compute on it apparently? | |
| 14:43:47 | dansmith | zioproto: you probably want to just delete that from the db, but you may also be able to force-down it to get past this point | |
| 14:43:49 | zioproto | should not | |
| 14:44:04 | zioproto | nova-0 is a vm where I just run the nova-api services | |
| 14:44:10 | zioproto | it is not a compute node | |
| 14:44:15 | dansmith | well, it reported nova-compute three days ago | |
| 14:44:26 | zioproto | ok good, but what about the second one ? | |
| 14:44:44 | dansmith | zioproto: what second one? | |
| 14:44:46 | johnthetubaguy | satyar: yeah, thats right :) | |
| 14:45:42 | satyar | Thanks johnthetubaguy i just updated the bug(https://bugs.launchpad.net/nova/+bug/1668267) its in the step 3 delete all Vms | |
| 14:45:42 | openstack | Launchpad bug 1668267 in OpenStack Compute (nova) "Nova ignores to update the quota for VMs in error" [Undecided,Confirmed] | |
| 14:46:21 | akijak | hi, I've found a bug in nova concerning rebuilding and 'hw_disk_bus' image property (https://bugs.launchpad.net/nova/+bug/1667667), I'm looking for someone who could give me some directions how to fix that :) can you help? | |
| 14:46:21 | openstack | Launchpad bug 1667667 in OpenStack Compute (nova) "Rebuilding instance ignores image's property hw_disk_bus" [Low,Confirmed] | |
| 14:46:39 | zioproto | dansmith, http://paste.openstack.org/show/600617/ There are still 2 unmigrated records in the compute_nodes table. | |
| 14:47:11 | satyar | kashyap its a usual practice if you are not sure about asking someone is thats right what you are asking putting 3 exclamation marks | |
| 14:47:40 | johnthetubaguy | satyar: asked you a follow up on the bug, I am curious about what list instances shows after deleting the VMs | |
| 14:47:47 | dansmith | zioproto: I would resolve the pci thing and see if you make more online migration progress | |
| 14:47:50 | dansmith | zioproto: that message is from the newton db sync right? | |
| 14:48:25 | zioproto | right | |
| 14:48:28 | satyar | no VMs present | |
| 14:48:51 | johnthetubaguy | satyar: do you have a setup where you can test the same thing with master? | |
| 14:49:04 | johnthetubaguy | its basically the same thing right now | |
| 14:49:19 | satyar | i am running with master also | |
| 14:49:28 | satyar | same issue | |
| 14:50:17 | johnthetubaguy | satyar: sounds crazy to ask, but how did you delete all the VMs, just a quick loop that called the delete API call? | |
| 14:50:30 | dansmith | zioproto: just to be clear, you're running online migrations with the mitaka code, correct? | |
| 14:50:47 | satyar | no | |
| 14:50:47 | zioproto | correct | |
| 14:50:58 | zioproto | I deleted that service | |
| 14:51:05 | zioproto | now I will try again the online migrations | |
| 14:51:09 | dansmith | zioproto: cool, yep | |
| 14:51:18 | johnthetubaguy | satyar: how did you delete them all? | |
| 14:51:49 | zioproto | dansmith, is it a problem that I have already the db_api in the newton version ? I cant downgrade that db :( | |
| 14:51:51 | satyar | johnthetubagu: i waited for VMs to go to error state for which block device mapping failed then after 5 mins delete the VMs | |
| 14:51:57 | satyar | from horizon | |
| 14:52:03 | satyar | select all and delete | |
| 14:52:21 | johnthetubaguy | satyar: thats cool, can you add that detail into the bug please? | |
| 14:52:41 | dansmith | zioproto: not sure what you mean.. the db sync is stopping on the newton side because it's detecting you haven't done your work in mitaka | |
| 14:52:51 | satyar | sure | |
| 14:52:52 | zioproto | dansmith, same problem http://paste.openstack.org/show/600626/ | |
| 14:53:13 | johnthetubaguy | satyar: I know the current quota system is racey on delete, but this seems a bit odd | |
| 14:53:17 | mriedem | edleafe: if you're aware of specs needing re-approvals for any of the resource provider stuff just ping me if they aren't getting attention | |
| 14:53:19 | dansmith | zioproto: what does service-list show now? and what did you do to delete the old service? | |
| 14:53:25 | mriedem | i'm procrastinating on ptg recap emails | |
| 14:53:26 | johnthetubaguy | satyar: if you only create 15, what do you see? left with 5 quota, or still left with 10 quota in use? | |
| 14:53:37 | satyar | yes | |
| 14:53:58 | johnthetubaguy | mriedem: I think the policy specs include the PTG updates now, incase that helps | |
| 14:54:49 | zioproto | dansmith, http://paste.openstack.org/show/600627/ | |
| 14:54:53 | mriedem | johnthetubaguy: but re-approvals are mindless reviews :) | |
| 14:54:57 | mriedem | i need mindless right now | |
| 14:55:23 | zioproto | dansmith, I also tried to apply this fix to my mitaka installation but it did not help: https://bugs.launchpad.net/nova/+bug/1630732 | |
| 14:55:23 | openstack | Launchpad bug 1630732 in OpenStack Compute (nova) newton "online_data_migrations does not ignore deleted services" [High,Fix committed] - Assigned to Dan Smith (danms) | |
| 14:56:15 | satyar | johnthetubaguy: to reproduce the issue 1. set the volume quota to 10 and VM quota to 20 or higher 2. create 15 VMs with boot from volume option from horizon | |
| 14:56:23 | dansmith | zioproto: yeah, that's not related, but I think it might be soft-delete related | |
| 14:56:31 | dansmith | zioproto: willing to execute a sql query to confirm? | |
| 14:57:14 | satyar | wait for 5 VMs to go to error state and 10 to come to active | |
| 14:57:19 | edleafe | mriedem: not atm, but will do | |
| 14:57:21 | satyar | then delete all VMs | |
| 14:58:06 | zioproto | dansmith, sure, is the staging env | |
| 14:58:17 | zioproto | dansmith, I have to make it work here before uprgading production | |
| 14:58:21 | zioproto | dansmith, I can access the db | |
| 14:58:42 | dansmith | zioproto: so in the nova database, services table, look for the record that matches that service you deleted.. should have the deleted column as nonzero | |
| 14:58:53 | dansmith | zioproto: then delete that record | |
| 15:00:15 | zioproto | dansmith, http://paste.openstack.org/show/600630/ | |
| 15:00:23 | zioproto | it is not many records, which one should I delete ? | |
| 15:00:32 | satyar | johnthetubaguy: i attached the screenshot how to create boot from volume from horizon | |
| 15:00:58 | zioproto | dansmith, 11 and 12 look weird, no ? | |
| 15:01:10 | dansmith | zioproto: ah, so you have a deleted conductor record too, which will do it as well.. so really I think you want "DELETE FROM services WHERE deleted != 0" | |
| 15:02:58 | zioproto | dansmith, http://paste.openstack.org/show/600632/ | |
| 15:03:16 | zioproto | but if I run nova-manage db online_data_migrations this still fails | |