| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-02-17 | |||
| 19:07:34 | mriedem | *reviews | |
| 19:07:56 | mriedem | superdan: hmm cells v1 yeah | |
| 19:08:06 | mriedem | superdan: it is a negative test... | |
| 19:08:13 | cburgess | mriedem Yeah there are a few in ATL I've been to one twice and its not bad. The BBQ is better but Ted's is closer and I can try and see if we can get a reso. I'll call them today. | |
| 19:08:24 | superdan | mriedem: yeah, | |
| 19:08:40 | openstackgerrit | Sean Dague proposed openstack/nova master: Verify project id for flavor access calls https://review.openstack.org/435432 | |
| 19:08:40 | openstackgerrit | Sean Dague proposed openstack/nova master: Verify project_id when quotas are checked https://review.openstack.org/435010 | |
| 19:08:48 | mriedem | cburgess: it doesn't look like ted's is vegan/vegetarian/gluten free friendly either | |
| 19:08:59 | mriedem | but maybe they have an eraser people can eat | |
| 19:09:11 | superdan | mriedem: so maybe that trace is incidental to the negative test and these others are silently failing I guess | |
| 19:09:43 | mriedem | yeah it looks like http://logs.openstack.org/62/433362/5/check/gate-tempest-dsvm-cells-ubuntu-xenial/ac9584a/console.html#_2017-02-16_22_58_39_396399 is the test failure | |
| 19:09:46 | mriedem | the snapshot test | |
| 19:09:51 | mriedem | i've seen that one fail plenty in the cells v1 job | |
| 19:10:17 | mriedem | although i'm seeing lots of this in the console log | |
| 19:10:17 | superdan | yeah | |
| 19:10:17 | mriedem | Details: {u'code': 404, u'message': u'Cell is not known for instance 518086f2-4bf8-412e-9758-8d66947b8668'} | |
| 19:10:23 | superdan | right | |
| 19:10:25 | mriedem | i guess that's what you're debugging | |
| 19:10:28 | superdan | that's a cellsv1 error | |
| 19:10:28 | cburgess | mriedem THey have a gluten free menu https://www.tedsmontanagrill.com/menu/download/TMGGlutenFreeMenu.pdf | |
| 19:10:29 | superdan | yeah | |
| 19:11:10 | cburgess | mriedem They have a lot of fish and salads. Vegans though will probably have a hard time but I'm not sure of a vegan place in downtown (the BBQ places surely wouldn't work either). | |
| 19:11:45 | mriedem | there may be some random roots growing between buildings downtown | |
| 19:12:07 | cburgess | mriedem Indeed | |
| 19:12:10 | superdan | mriedem: that patch is only changing the rest api which really shouldn't know or care or influence any of the cells stuff below | |
| 19:12:21 | superdan | so maybe I'll recheck it and see if it's the same set of failures or not | |
| 19:32:05 | openstackgerrit | John L. Villalovos proposed openstack/nova master: Use flake8-import-order https://review.openstack.org/435575 | |
| 19:37:12 | openstackgerrit | Sean Dague proposed openstack/nova master: Verify project id for flavor access calls https://review.openstack.org/435432 | |
| 19:37:12 | openstackgerrit | Sean Dague proposed openstack/nova master: Verify project_id when quotas are checked https://review.openstack.org/435010 | |
| 19:55:11 | melwitt | mriedem, superdan: I have a patch up for a blocker migration that operators can get stuck on upgrading from mitaka -> newton if they have soft-deleted records https://review.openstack.org/435546 | |
| 19:55:44 | superdan | oh yeah | |
| 19:55:56 | superdan | mriedem: that one needs to get in | |
| 20:00:14 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Only create vendordata_dynamic ksa session if needed https://review.openstack.org/435563 | |
| 20:00:20 | mriedem | i don't want to actually think today | |
| 20:00:34 | mriedem | that needs to go back to stable/newton? | |
| 20:00:44 | mriedem | because we haven't released https://review.openstack.org/#/c/434928/ yet | |
| 20:01:24 | superdan | it does yeah | |
| 20:01:33 | mriedem | btw, this just again goes to show how much no one is event close to trunk | |
| 20:01:49 | superdan | mriedem: well, it's actually probably not very common | |
| 20:01:59 | superdan | mriedem: but definitely hit-able | |
| 20:02:11 | mriedem | let me read up on the bug report and such | |
| 20:02:33 | superdan | it would only happen if you completely decommissioned compute nodes and never replaced them | |
| 20:03:23 | melwitt | it can also happen if you've ever deleted aggregates or pci devices. it's just that wasn't the case in our real world example | |
| 20:04:09 | superdan | melwitt: yeah, true I guess I forgot about those since the CN was the issue in the bug | |
| 20:05:04 | melwitt | superdan: yeah. I was debating whether to fix it only for compute_nodes or for the other tables too. I figured it's the same problem, and that assumes no apis ever read those soft-deleted records for anything later. I don't think they do | |
| 20:05:20 | mriedem | in the red hat bugzilla i guess | |
| 20:05:31 | mriedem | so i guess, | |
| 20:05:38 | superdan | was the aggregate move to api in ocata? | |
| 20:06:00 | melwitt | I thought it was earlier | |
| 20:06:08 | melwitt | I mean, it's definitely before ocata | |
| 20:06:09 | superdan | melwitt: if so we can't have deleted aggregates | |
| 20:06:11 | mriedem | newton | |
| 20:06:23 | mriedem | https://github.com/openstack/nova/commit/35748e7a9b3445a60522e088e38bf25e6bfef25e | |
| 20:06:30 | mriedem | i was thinking it's odd to update the migration itself, | |
| 20:06:34 | mriedem | but i guess if you're stuck on 330, | |
| 20:06:40 | mriedem | then you can't get past that anyway | |
| 20:06:41 | superdan | I guess that move happens after the sync though, if they both happened in newton | |
| 20:06:52 | superdan | mriedem: yeah | |
| 20:07:05 | superdan | mriedem: the other option is to convert all the deleted records, but that's kinda silly | |
| 20:07:18 | melwitt | there's a REST API for deleting an aggregate https://developer.openstack.org/api-ref/compute/?expanded=delete-aggregate-detail#delete-aggregate | |
| 20:07:22 | superdan | especially if we're about to delete them in the case of aggregates | |
| 20:07:31 | melwitt | so if that connects to an Aggregate.destroy(), that would do it | |
| 20:07:34 | superdan | melwitt: right, but once they're in the api db they can;t be soft deleted | |
| 20:07:43 | melwitt | ah, that's what you meant. yeah | |
| 20:07:55 | mriedem | so in mitaka i deleted a compute node, | |
| 20:08:14 | mriedem | actually i need to see the online migration code first | |
| 20:08:25 | melwitt | yeah, I have links up still I think | |
| 20:08:49 | melwitt | the online migration only migrates non-soft-deleted things and the blocker doesn't filter on deleted=0 so it just gets everything | |
| 20:09:00 | mriedem | what does https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L595 have to do with compute nodes? | |
| 20:09:36 | mriedem | you guys got some out of tree online migrations? | |
| 20:09:58 | mriedem | or is it just this? https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L597 | |
| 20:10:03 | mriedem | db.aggregate_uuids_online_data_migration, | |
| 20:10:07 | mriedem | that's the only mitaka one in that list | |
| 20:10:29 | mriedem | i should be looking at stable/newton i guess https://github.com/openstack/nova/blob/stable/newton/nova/cmd/manage.py#L786 | |
| 20:10:32 | melwitt | no, it looks like we removed the compute nodes one | |
| 20:10:33 | mriedem | which includes pci devices | |
| 20:10:37 | melwitt | after | |
| 20:10:40 | melwitt | yeah | |
| 20:11:40 | mriedem | oh nvm i see https://github.com/openstack/nova/blob/stable/mitaka/nova/cmd/manage.py#L930 | |
| 20:11:48 | mriedem | in mitaka we had an online migration for compute nodes | |
| 20:11:49 | mriedem | for the uuid | |
| 20:12:27 | mriedem | https://github.com/openstack/nova/blob/stable/mitaka/nova/db/sqlalchemy/api.py#L6531 | |
| 20:12:50 | mriedem | so that online migration can get soft deleted compute nodes with no uuid | |
| 20:12:52 | mriedem | right? | |
| 20:13:31 | melwitt | the online migration does not migrate soft-deleted compute nodes with no uuid | |
| 20:13:56 | melwitt | so I thought our choices are: migrate even soft-deleted things or skip soft-deleted things in the blocker | |
| 20:14:04 | mriedem | because of model_query | |
| 20:14:05 | mriedem | i suppose | |
| 20:14:07 | melwitt | yeah | |
| 20:14:16 | melwitt | model_query defaults to read_deleted='no' | |
| 20:14:18 | mriedem | and the context doesn't have read_deleted='yes' or whatever | |
| 20:14:19 | mriedem | ok | |
| 20:14:20 | mriedem | yup | |
| 20:14:55 | mriedem | ok so in mitaka we have soft deleted compute_nodes, online data migration doesn't get them b/c of model_query, but 330 blocked migration pulls them in b/c it's not using the orm and it explodes | |
| 20:15:01 | mriedem | because we don't filter on deleted==0 | |
| 20:15:07 | melwitt | the only gotcha would be if it were valid for any of those objects: compute nodes, aggregates, pci devices to be read in a deleted state | |
| 20:15:14 | mriedem | which is your fix | |
| 20:15:19 | melwitt | right | |
| 20:15:46 | mriedem | geez melwitt your commit message could have been several more paragraphs of explanation :) | |
| 20:16:11 | melwitt | because this fix would leave the soft-deleted unmigrated records around. the alternate fix would migrate potentially a lot of soft-deleted things for no reason | |