| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-27 | |||
| 14:50:47 | satyar | no | |
| 14:50:47 | zioproto | correct | |
| 14:50:58 | zioproto | I deleted that service | |
| 14:51:05 | zioproto | now I will try again the online migrations | |
| 14:51:09 | dansmith | zioproto: cool, yep | |
| 14:51:18 | johnthetubaguy | satyar: how did you delete them all? | |
| 14:51:49 | zioproto | dansmith, is it a problem that I have already the db_api in the newton version ? I cant downgrade that db :( | |
| 14:51:51 | satyar | johnthetubagu: i waited for VMs to go to error state for which block device mapping failed then after 5 mins delete the VMs | |
| 14:51:57 | satyar | from horizon | |
| 14:52:03 | satyar | select all and delete | |
| 14:52:21 | johnthetubaguy | satyar: thats cool, can you add that detail into the bug please? | |
| 14:52:41 | dansmith | zioproto: not sure what you mean.. the db sync is stopping on the newton side because it's detecting you haven't done your work in mitaka | |
| 14:52:51 | satyar | sure | |
| 14:52:52 | zioproto | dansmith, same problem http://paste.openstack.org/show/600626/ | |
| 14:53:13 | johnthetubaguy | satyar: I know the current quota system is racey on delete, but this seems a bit odd | |
| 14:53:17 | mriedem | edleafe: if you're aware of specs needing re-approvals for any of the resource provider stuff just ping me if they aren't getting attention | |
| 14:53:19 | dansmith | zioproto: what does service-list show now? and what did you do to delete the old service? | |
| 14:53:25 | mriedem | i'm procrastinating on ptg recap emails | |
| 14:53:26 | johnthetubaguy | satyar: if you only create 15, what do you see? left with 5 quota, or still left with 10 quota in use? | |
| 14:53:37 | satyar | yes | |
| 14:53:58 | johnthetubaguy | mriedem: I think the policy specs include the PTG updates now, incase that helps | |
| 14:54:49 | zioproto | dansmith, http://paste.openstack.org/show/600627/ | |
| 14:54:53 | mriedem | johnthetubaguy: but re-approvals are mindless reviews :) | |
| 14:54:57 | mriedem | i need mindless right now | |
| 14:55:23 | zioproto | dansmith, I also tried to apply this fix to my mitaka installation but it did not help: https://bugs.launchpad.net/nova/+bug/1630732 | |
| 14:55:23 | openstack | Launchpad bug 1630732 in OpenStack Compute (nova) newton "online_data_migrations does not ignore deleted services" [High,Fix committed] - Assigned to Dan Smith (danms) | |
| 14:56:15 | satyar | johnthetubaguy: to reproduce the issue 1. set the volume quota to 10 and VM quota to 20 or higher 2. create 15 VMs with boot from volume option from horizon | |
| 14:56:23 | dansmith | zioproto: yeah, that's not related, but I think it might be soft-delete related | |
| 14:56:31 | dansmith | zioproto: willing to execute a sql query to confirm? | |
| 14:57:14 | satyar | wait for 5 VMs to go to error state and 10 to come to active | |
| 14:57:19 | edleafe | mriedem: not atm, but will do | |
| 14:57:21 | satyar | then delete all VMs | |
| 14:58:06 | zioproto | dansmith, sure, is the staging env | |
| 14:58:17 | zioproto | dansmith, I have to make it work here before uprgading production | |
| 14:58:21 | zioproto | dansmith, I can access the db | |
| 14:58:42 | dansmith | zioproto: so in the nova database, services table, look for the record that matches that service you deleted.. should have the deleted column as nonzero | |
| 14:58:53 | dansmith | zioproto: then delete that record | |
| 15:00:15 | zioproto | dansmith, http://paste.openstack.org/show/600630/ | |
| 15:00:23 | zioproto | it is not many records, which one should I delete ? | |
| 15:00:32 | satyar | johnthetubaguy: i attached the screenshot how to create boot from volume from horizon | |
| 15:00:58 | zioproto | dansmith, 11 and 12 look weird, no ? | |
| 15:01:10 | dansmith | zioproto: ah, so you have a deleted conductor record too, which will do it as well.. so really I think you want "DELETE FROM services WHERE deleted != 0" | |
| 15:02:58 | zioproto | dansmith, http://paste.openstack.org/show/600632/ | |
| 15:03:16 | zioproto | but if I run nova-manage db online_data_migrations this still fails | |
| 15:03:35 | zioproto | 2017-02-27 16:02:14.167 16580 ERROR nova.db.sqlalchemy.api [req-cbc733c3-8c99-4f93-8b51-07d61d77102b - - - - -] Data migrations for PciDevice are not safe, likely because not all services that access the DB directly are updated to the latest version | |
| 15:03:35 | mriedem | cached? | |
| 15:03:43 | mriedem | the service version is cached right? | |
| 15:03:55 | zioproto | cached where ? | |
| 15:03:56 | dansmith | mriedem: not in nova-manage | |
| 15:04:24 | dansmith | zioproto: 11 and 12 do look very weird yeah | |
| 15:04:33 | zioproto | should I delete those too ? | |
| 15:04:47 | zioproto | I will | |
| 15:05:06 | dansmith | actually, the api one might be okay | |
| 15:05:12 | dansmith | it's definitely weird because it doesn't get updated periodically | |
| 15:05:21 | dansmith | neither of these really should be in effect though, which is strange | |
| 15:05:36 | zioproto | same error | |
| 15:05:47 | dansmith | ah, and the metadata one is probably weird for the same reason | |
| 15:05:57 | dansmith | zioproto: are you deploying from git or distro packages or what? | |
| 15:06:02 | zioproto | ubuntu packages | |
| 15:06:05 | zioproto | mitaka | |
| 15:06:33 | dansmith | zioproto: can you give me an exact version string so I can try to hunt down a git sha? | |
| 15:06:47 | zioproto | 2:13.1.2-0ubuntu2 | |
| 15:07:15 | zioproto | the repo is http://ch.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages | |
| 15:09:08 | zioproto | should be the nova tag 13.1.2 | |
| 15:09:47 | dansmith | should be, but... | |
| 15:11:22 | zioproto | I am checking the ubuntu patches in the deb packages on top of 13.1.2, nothing related to this thing AFAIK | |
| 15:11:57 | zioproto | dansmith, git://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/nova and see branch stable/mitaka | |
| 15:13:51 | belmoreira | testing "db sync" in a copy of our production DBs from liberty to mitaka and getting an error in the migration 317->318: error: Index 'allocations_resource_provider_class_used_idx' already exists! | |
| 15:14:09 | belmoreira | Is this a known issue? google is not being my friend | |
| 15:14:40 | dansmith | zioproto: just to be clear, you have this applied to the code you're running nova-manage from? https://review.openstack.org/#/c/385195/1/nova/objects/pci_device.py | |
| 15:15:10 | mriedem | oh wow that's not released yet.. | |
| 15:15:52 | dansmith | mriedem: which? | |
| 15:15:55 | mriedem | ^ | |
| 15:16:00 | mriedem | that's in stable/mitaka but not released | |
| 15:16:05 | mriedem | in a 13.x point release | |
| 15:16:08 | mriedem | so might not be in distros | |
| 15:16:10 | dansmith | mriedem: the index thing or my link? | |
| 15:16:14 | mriedem | your link | |
| 15:16:14 | zioproto | dansmith, let me check | |
| 15:16:17 | dansmith | okay | |
| 15:16:42 | dansmith | zioproto: that could definitely be your problem if you don't.. now that my brain re-caffeinates, it would cause us to always get zero as the minimum version regardless of what is in the db | |
| 15:17:35 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Replace diagnostics objects with Nova diagnostics objects https://review.openstack.org/394480 | |
| 15:17:36 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Added nova objects for intance diagnostics https://review.openstack.org/434745 | |
| 15:17:37 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Standardization of VM diagnostics info API. https://review.openstack.org/355540 | |
| 15:17:38 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Completed implementation of instance diagnostics for Xen https://review.openstack.org/399613 | |
| 15:17:39 | openstackgerrit | Sergey Nikitin proposed openstack/nova master: Add RPC version aliases for Ocata https://review.openstack.org/425958 | |
| 15:17:46 | zioproto | dansmith, yes, I applied that patch, and also regenrated the pyc file. And still the problem is there | |
| 15:18:10 | dansmith | zioproto: since this is pre-prod can you share with me a dump of the db by chance? | |
| 15:18:50 | zioproto | sure, as long as I share it just with you and not will the all world in public it is okay :)( | |
| 15:18:53 | zioproto | :) | |
| 15:18:58 | dansmith | zioproto: sure, will pm | |
| 15:20:35 | dansmith | mriedem: that definitely should get out to people, but it also shouldn't be blocking zioproto from getting to the migration he needs, so it really should only affect pci-using people | |
| 15:20:40 | dansmith | but still important of course | |
| 15:26:37 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged nic attachment https://review.openstack.org/415622 | |
| 15:26:37 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete nic metadata when detaching interface https://review.openstack.org/435387 | |
| 15:26:38 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged volume attachment https://review.openstack.org/418100 | |
| 15:26:38 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Delete disk metadata when detaching volume https://review.openstack.org/435082 | |
| 15:26:39 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Plumbing for tagged volume attachment https://review.openstack.org/391941 | |
| 15:26:40 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: API support for tagged device attachment https://review.openstack.org/376021 | |
| 15:26:41 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Libvirt support for tagged nic attachment https://review.openstack.org/415520 | |