| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:46:39 | zioproto | dansmith, http://paste.openstack.org/show/600617/ There are still 2 unmigrated records in the compute_nodes table. | |
| 14:47:11 | satyar | kashyap its a usual practice if you are not sure about asking someone is thats right what you are asking putting 3 exclamation marks | |
| 14:47:40 | johnthetubaguy | satyar: asked you a follow up on the bug, I am curious about what list instances shows after deleting the VMs | |
| 14:47:47 | dansmith | zioproto: I would resolve the pci thing and see if you make more online migration progress | |
| 14:47:50 | dansmith | zioproto: that message is from the newton db sync right? | |
| 14:48:25 | zioproto | right | |
| 14:48:28 | satyar | no VMs present | |
| 14:48:51 | johnthetubaguy | satyar: do you have a setup where you can test the same thing with master? | |
| 14:49:04 | johnthetubaguy | its basically the same thing right now | |
| 14:49:19 | satyar | i am running with master also | |
| 14:49:28 | satyar | same issue | |
| 14:50:17 | johnthetubaguy | satyar: sounds crazy to ask, but how did you delete all the VMs, just a quick loop that called the delete API call? | |
| 14:50:30 | dansmith | zioproto: just to be clear, you're running online migrations with the mitaka code, correct? | |
| 14:50:47 | zioproto | correct | |
| 14:50:47 | satyar | no | |
| 14:50:58 | zioproto | I deleted that service | |
| 14:51:05 | zioproto | now I will try again the online migrations | |
| 14:51:09 | dansmith | zioproto: cool, yep | |
| 14:51:18 | johnthetubaguy | satyar: how did you delete them all? | |
| 14:51:49 | zioproto | dansmith, is it a problem that I have already the db_api in the newton version ? I cant downgrade that db :( | |
| 14:51:51 | satyar | johnthetubagu: i waited for VMs to go to error state for which block device mapping failed then after 5 mins delete the VMs | |
| 14:51:57 | satyar | from horizon | |
| 14:52:03 | satyar | select all and delete | |
| 14:52:21 | johnthetubaguy | satyar: thats cool, can you add that detail into the bug please? | |
| 14:52:41 | dansmith | zioproto: not sure what you mean.. the db sync is stopping on the newton side because it's detecting you haven't done your work in mitaka | |
| 14:52:51 | satyar | sure | |
| 14:52:52 | zioproto | dansmith, same problem http://paste.openstack.org/show/600626/ | |
| 14:53:13 | johnthetubaguy | satyar: I know the current quota system is racey on delete, but this seems a bit odd | |
| 14:53:17 | mriedem | edleafe: if you're aware of specs needing re-approvals for any of the resource provider stuff just ping me if they aren't getting attention | |
| 14:53:19 | dansmith | zioproto: what does service-list show now? and what did you do to delete the old service? | |
| 14:53:25 | mriedem | i'm procrastinating on ptg recap emails | |
| 14:53:26 | johnthetubaguy | satyar: if you only create 15, what do you see? left with 5 quota, or still left with 10 quota in use? | |
| 14:53:37 | satyar | yes | |
| 14:53:58 | johnthetubaguy | mriedem: I think the policy specs include the PTG updates now, incase that helps | |
| 14:54:49 | zioproto | dansmith, http://paste.openstack.org/show/600627/ | |
| 14:54:53 | mriedem | johnthetubaguy: but re-approvals are mindless reviews :) | |
| 14:54:57 | mriedem | i need mindless right now | |
| 14:55:23 | openstack | Launchpad bug 1630732 in OpenStack Compute (nova) newton "online_data_migrations does not ignore deleted services" [High,Fix committed] - Assigned to Dan Smith (danms) | |
| 14:55:23 | zioproto | dansmith, I also tried to apply this fix to my mitaka installation but it did not help: https://bugs.launchpad.net/nova/+bug/1630732 | |
| 14:56:15 | satyar | johnthetubaguy: to reproduce the issue 1. set the volume quota to 10 and VM quota to 20 or higher 2. create 15 VMs with boot from volume option from horizon | |
| 14:56:23 | dansmith | zioproto: yeah, that's not related, but I think it might be soft-delete related | |
| 14:56:31 | dansmith | zioproto: willing to execute a sql query to confirm? | |
| 14:57:14 | satyar | wait for 5 VMs to go to error state and 10 to come to active | |
| 14:57:19 | edleafe | mriedem: not atm, but will do | |
| 14:57:21 | satyar | then delete all VMs | |
| 14:58:06 | zioproto | dansmith, sure, is the staging env | |
| 14:58:17 | zioproto | dansmith, I have to make it work here before uprgading production | |
| 14:58:21 | zioproto | dansmith, I can access the db | |
| 14:58:42 | dansmith | zioproto: so in the nova database, services table, look for the record that matches that service you deleted.. should have the deleted column as nonzero | |
| 14:58:53 | dansmith | zioproto: then delete that record | |
| 15:00:15 | zioproto | dansmith, http://paste.openstack.org/show/600630/ | |
| 15:00:23 | zioproto | it is not many records, which one should I delete ? | |
| 15:00:32 | satyar | johnthetubaguy: i attached the screenshot how to create boot from volume from horizon | |
| 15:00:58 | zioproto | dansmith, 11 and 12 look weird, no ? | |
| 15:01:10 | dansmith | zioproto: ah, so you have a deleted conductor record too, which will do it as well.. so really I think you want "DELETE FROM services WHERE deleted != 0" | |
| 15:02:58 | zioproto | dansmith, http://paste.openstack.org/show/600632/ | |
| 15:03:16 | zioproto | but if I run nova-manage db online_data_migrations this still fails | |
| 15:03:35 | mriedem | cached? | |
| 15:03:35 | zioproto | 2017-02-27 16:02:14.167 16580 ERROR nova.db.sqlalchemy.api [req-cbc733c3-8c99-4f93-8b51-07d61d77102b - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 15:03:43 | mriedem | the service version is cached right? | |
| 15:03:55 | zioproto | cached where ? | |
| 15:03:56 | dansmith | mriedem: not in nova-manage | |
| 15:04:24 | dansmith | zioproto: 11 and 12 do look very weird yeah | |
| 15:04:33 | zioproto | should I delete those too ? | |
| 15:04:47 | zioproto | I will | |
| 15:05:06 | dansmith | actually, the api one might be okay | |
| 15:05:12 | dansmith | it's definitely weird because it doesn't get updated periodically | |
| 15:05:21 | dansmith | neither of these really should be in effect though, which is strange | |
| 15:05:36 | zioproto | same error | |
| 15:05:47 | dansmith | ah, and the metadata one is probably weird for the same reason | |
| 15:05:57 | dansmith | zioproto: are you deploying from git or distro packages or what? | |
| 15:06:02 | zioproto | ubuntu packages | |
| 15:06:05 | zioproto | mitaka | |
| 15:06:33 | dansmith | zioproto: can you give me an exact version string so I can try to hunt down a git sha? | |
| 15:06:47 | zioproto | 2:13.1.2-0ubuntu2 | |
| 15:07:15 | zioproto | the repo is http://ch.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages | |
| 15:09:08 | zioproto | should be the nova tag 13.1.2 | |
| 15:09:47 | dansmith | should be, but... | |
| 15:11:22 | zioproto | I am checking the ubuntu patches in the deb packages on top of 13.1.2, nothing related to this thing AFAIK | |
| 15:11:57 | zioproto | dansmith, git://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/nova and see branch stable/mitaka | |
| 15:13:51 | belmoreira | testing "db sync" in a copy of our production DBs from liberty to mitaka and getting an error in the migration 317->318: error: Index 'allocations_resource_provider_class_used_idx' already exists! | |
| 15:14:09 | belmoreira | Is this a known issue? google is not being my friend | |
| 15:14:40 | dansmith | zioproto: just to be clear, you have this applied to the code you're running nova-manage from? https://review.openstack.org/#/c/385195/1/nova/objects/pci_device.py | |
| 15:15:10 | mriedem | oh wow that's not released yet.. | |
| 15:15:52 | dansmith | mriedem: which? | |
| 15:15:55 | mriedem | ^ | |
| 15:16:00 | mriedem | that's in stable/mitaka but not released | |
| 15:16:05 | mriedem | in a 13.x point release | |
| 15:16:08 | mriedem | so might not be in distros | |
| 15:16:10 | dansmith | mriedem: the index thing or my link? | |
| 15:16:14 | zioproto | dansmith, let me check | |
| 15:16:14 | mriedem | your link | |
| 15:16:17 | dansmith | okay | |
| 15:16:42 | dansmith | zioproto: that could definitely be your problem if you don't.. now that my brain re-caffeinates, it would cause us to always get zero as the minimum version regardless of what is in the db | |
| 15:17:35 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Replace diagnostics objects with Nova diagnostics objects https://review.openstack.org/394480 | |
| 15:17:36 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Added nova objects for intance diagnostics https://review.openstack.org/434745 | |
| 15:17:37 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Standardization of VM diagnostics info API. https://review.openstack.org/355540 | |
| 15:17:38 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Completed implementation of instance diagnostics for Xen https://review.openstack.org/399613 | |
| 15:17:39 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Add RPC version aliases for Ocata https://review.openstack.org/425958 | |
| 15:17:46 | zioproto | dansmith, yes, I applied that patch, and also regenrated the pyc file. And still the problem is there | |