| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:40:21 | zioproto | this is the staging setup, I have exactly two compute nodes | |
| 14:40:21 | johnthetubaguy | dansmith: ah, was just going to say that | |
| 14:40:22 | dansmith | so yeah, likely an orphaned record or something | |
| 14:41:01 | satyar | Hi johnthethubaguy | |
| 14:41:18 | satyar | are you johngarbutt!!! | |
| 14:42:16 | zioproto | so, I dont undertstand, you want the output of 'openstack hypervisor list' ? | |
| 14:42:27 | dansmith | zioproto: service-list | |
| 14:42:57 | zioproto | http://paste.openstack.org/show/600624/ | |
| 14:43:08 | kashyap | satyar: No need for so many exclamations. If you have a question, just write to the channel, unless you are already interacting with a specific reviewer. | |
| 14:43:09 | dansmith | there ya go | |
| 14:43:23 | dansmith | zioproto: nova-0 used to have a compute on it apparently? | |
| 14:43:47 | dansmith | zioproto: you probably want to just delete that from the db, but you may also be able to force-down it to get past this point | |
| 14:43:49 | zioproto | should not | |
| 14:44:04 | zioproto | nova-0 is a vm where I just run the nova-api services | |
| 14:44:10 | zioproto | it is not a compute node | |
| 14:44:15 | dansmith | well, it reported nova-compute three days ago | |
| 14:44:26 | zioproto | ok good, but what about the second one ? | |
| 14:44:44 | dansmith | zioproto: what second one? | |
| 14:44:46 | johnthetubaguy | satyar: yeah, thats right :) | |
| 14:45:42 | satyar | Thanks johnthetubaguy i just updated the bug(https://bugs.launchpad.net/nova/+bug/1668267) its in the step 3 delete all Vms | |
| 14:45:42 | openstack | Launchpad bug 1668267 in OpenStack Compute (nova) "Nova ignores to update the quota for VMs in error" [Undecided,Confirmed] | |
| 14:46:21 | akijak | hi, I've found a bug in nova concerning rebuilding and 'hw_disk_bus' image property (https://bugs.launchpad.net/nova/+bug/1667667), I'm looking for someone who could give me some directions how to fix that :) can you help? | |
| 14:46:21 | openstack | Launchpad bug 1667667 in OpenStack Compute (nova) "Rebuilding instance ignores image's property hw_disk_bus" [Low,Confirmed] | |
| 14:46:39 | zioproto | dansmith, http://paste.openstack.org/show/600617/ There are still 2 unmigrated records in the compute_nodes table. | |
| 14:47:11 | satyar | kashyap its a usual practice if you are not sure about asking someone is thats right what you are asking putting 3 exclamation marks | |
| 14:47:40 | johnthetubaguy | satyar: asked you a follow up on the bug, I am curious about what list instances shows after deleting the VMs | |
| 14:47:47 | dansmith | zioproto: I would resolve the pci thing and see if you make more online migration progress | |
| 14:47:50 | dansmith | zioproto: that message is from the newton db sync right? | |
| 14:48:25 | zioproto | right | |
| 14:48:28 | satyar | no VMs present | |
| 14:48:51 | johnthetubaguy | satyar: do you have a setup where you can test the same thing with master? | |
| 14:49:04 | johnthetubaguy | its basically the same thing right now | |
| 14:49:19 | satyar | i am running with master also | |
| 14:49:28 | satyar | same issue | |
| 14:50:17 | johnthetubaguy | satyar: sounds crazy to ask, but how did you delete all the VMs, just a quick loop that called the delete API call? | |
| 14:50:30 | dansmith | zioproto: just to be clear, you're running online migrations with the mitaka code, correct? | |
| 14:50:47 | satyar | no | |
| 14:50:47 | zioproto | correct | |
| 14:50:58 | zioproto | I deleted that service | |
| 14:51:05 | zioproto | now I will try again the online migrations | |
| 14:51:09 | dansmith | zioproto: cool, yep | |
| 14:51:18 | johnthetubaguy | satyar: how did you delete them all? | |
| 14:51:49 | zioproto | dansmith, is it a problem that I have already the db_api in the newton version ? I cant downgrade that db :( | |
| 14:51:51 | satyar | johnthetubagu: i waited for VMs to go to error state for which block device mapping failed then after 5 mins delete the VMs | |
| 14:51:57 | satyar | from horizon | |
| 14:52:03 | satyar | select all and delete | |
| 14:52:21 | johnthetubaguy | satyar: thats cool, can you add that detail into the bug please? | |
| 14:52:41 | dansmith | zioproto: not sure what you mean.. the db sync is stopping on the newton side because it's detecting you haven't done your work in mitaka | |
| 14:52:51 | satyar | sure | |
| 14:52:52 | zioproto | dansmith, same problem http://paste.openstack.org/show/600626/ | |
| 14:53:13 | johnthetubaguy | satyar: I know the current quota system is racey on delete, but this seems a bit odd | |
| 14:53:17 | mriedem | edleafe: if you're aware of specs needing re-approvals for any of the resource provider stuff just ping me if they aren't getting attention | |
| 14:53:19 | dansmith | zioproto: what does service-list show now? and what did you do to delete the old service? | |
| 14:53:25 | mriedem | i'm procrastinating on ptg recap emails | |
| 14:53:26 | johnthetubaguy | satyar: if you only create 15, what do you see? left with 5 quota, or still left with 10 quota in use? | |
| 14:53:37 | satyar | yes | |
| 14:53:58 | johnthetubaguy | mriedem: I think the policy specs include the PTG updates now, incase that helps | |
| 14:54:49 | zioproto | dansmith, http://paste.openstack.org/show/600627/ | |
| 14:54:53 | mriedem | johnthetubaguy: but re-approvals are mindless reviews :) | |
| 14:54:57 | mriedem | i need mindless right now | |
| 14:55:23 | zioproto | dansmith, I also tried to apply this fix to my mitaka installation but it did not help: https://bugs.launchpad.net/nova/+bug/1630732 | |
| 14:55:23 | openstack | Launchpad bug 1630732 in OpenStack Compute (nova) newton "online_data_migrations does not ignore deleted services" [High,Fix committed] - Assigned to Dan Smith (danms) | |
| 14:56:15 | satyar | johnthetubaguy: to reproduce the issue 1. set the volume quota to 10 and VM quota to 20 or higher 2. create 15 VMs with boot from volume option from horizon | |
| 14:56:23 | dansmith | zioproto: yeah, that's not related, but I think it might be soft-delete related | |
| 14:56:31 | dansmith | zioproto: willing to execute a sql query to confirm? | |
| 14:57:14 | satyar | wait for 5 VMs to go to error state and 10 to come to active | |
| 14:57:19 | edleafe | mriedem: not atm, but will do | |
| 14:57:21 | satyar | then delete all VMs | |
| 14:58:06 | zioproto | dansmith, sure, is the staging env | |
| 14:58:17 | zioproto | dansmith, I have to make it work here before uprgading production | |
| 14:58:21 | zioproto | dansmith, I can access the db | |
| 14:58:42 | dansmith | zioproto: so in the nova database, services table, look for the record that matches that service you deleted.. should have the deleted column as nonzero | |
| 14:58:53 | dansmith | zioproto: then delete that record | |
| 15:00:15 | zioproto | dansmith, http://paste.openstack.org/show/600630/ | |
| 15:00:23 | zioproto | it is not many records, which one should I delete ? | |
| 15:00:32 | satyar | johnthetubaguy: i attached the screenshot how to create boot from volume from horizon | |
| 15:00:58 | zioproto | dansmith, 11 and 12 look weird, no ? | |
| 15:01:10 | dansmith | zioproto: ah, so you have a deleted conductor record too, which will do it as well.. so really I think you want "DELETE FROM services WHERE deleted != 0" | |
| 15:02:58 | zioproto | dansmith, http://paste.openstack.org/show/600632/ | |
| 15:03:16 | zioproto | but if I run nova-manage db online_data_migrations this still fails | |
| 15:03:35 | zioproto | 2017-02-27 16:02:14.167 16580 ERROR nova.db.sqlalchemy.api [req-cbc733c3-8c99-4f93-8b51-07d61d77102b - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 15:03:35 | mriedem | cached? | |
| 15:03:43 | mriedem | the service version is cached right? | |
| 15:03:55 | zioproto | cached where ? | |
| 15:03:56 | dansmith | mriedem: not in nova-manage | |
| 15:04:24 | dansmith | zioproto: 11 and 12 do look very weird yeah | |
| 15:04:33 | zioproto | should I delete those too ? | |
| 15:04:47 | zioproto | I will | |
| 15:05:06 | dansmith | actually, the api one might be okay | |
| 15:05:12 | dansmith | it's definitely weird because it doesn't get updated periodically | |
| 15:05:21 | dansmith | neither of these really should be in effect though, which is strange | |
| 15:05:36 | zioproto | same error | |
| 15:05:47 | dansmith | ah, and the metadata one is probably weird for the same reason | |
| 15:05:57 | dansmith | zioproto: are you deploying from git or distro packages or what? | |
| 15:06:02 | zioproto | ubuntu packages | |
| 15:06:05 | zioproto | mitaka | |
| 15:06:33 | dansmith | zioproto: can you give me an exact version string so I can try to hunt down a git sha? | |
| 15:06:47 | zioproto | 2:13.1.2-0ubuntu2 | |
| 15:07:15 | zioproto | the repo is http://ch.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages | |
| 15:09:08 | zioproto | should be the nova tag 13.1.2 | |