Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-19
22:16:04 openstack Launchpad bug 1701129 in OpenStack Compute (nova) "Functional tests fail intermittently with 400 Bad Request from placement" [Low,Fix released] - Assigned to melanie witt (melwitt)
22:21:24 mnaser i guess for some reason RequestSpec is getting an empty project_id
22:21:47 openstackgerrit Lance Bragstad proposed openstack/nova master: Add scope_types to server policies https://review.openstack.org/525772
22:23:12 openstackgerrit Merged openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
22:23:18 openstackgerrit Merged openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
22:23:25 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611
22:23:49 openstackgerrit Merged openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505
22:23:57 openstackgerrit Merged openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227
22:24:04 openstackgerrit Merged openstack/nova master: Fix possible TypeError in VIF.fixed_ips https://review.openstack.org/527920
22:49:34 mnaser instance = common.get_instance(self.compute_api, context, id) <== would anyone know if this supplies project_id by default?
22:50:28 mnaser because that's the instance which is passed down to conductor and by the time its at the scheduler, instance.project_id == None which then in turn makes it fail the request to the placement api
22:55:12 mnaser further investigation - {"project_id": null, "user_id": "695d5f386eed440cb0e38455e1afdc9e", "allocations": [{"resource_provider": {"uuid": "5d5c5177-29bb-484f-9cc6-928360afa195"}, "resources": {"MEMORY_MB": 512, "VCPU": 2, "DISK_GB": 20}}, {"resource_provider": {"uuid": "4e43861e-ee36-40b7-ba7b-2239b46a1609"}, "resources": {"VCPU": 2, "MEMORY_MB": 512, "DISK_GB": 20}}]} .. for some reason, user_id comes in but
22:55:13 mnaser project_id doesn't. fwiw, this is a server created in 2015.
22:57:59 mnaser the user_id is the user of the one executing the live migration, not the owner of the instance oddly enough
23:06:45 mnaser ok.. request_spec record has project_id set to null for that vm
23:06:54 mnaser in the database
23:06:56 mnaser why and how.. :(
23:06:57 mriedem hmm, not sure why the project_id would be null
23:07:02 mriedem should come off the context
23:07:07 mriedem sorry, was on a call for the last hour
23:07:32 mnaser mriedem: no problem, its null because .. its null in the request_specs table too..
23:07:39 mnaser i wonder why
23:07:52 mriedem you said it's a really old instance right?
23:07:59 mnaser yes mriedem
23:08:25 mriedem ok reqspec is created here https://github.com/openstack/nova/blob/16.0.4/nova/compute/api.py#L899
23:08:30 mnaser the created_at for the requestspec is "2017-03-07 02:28:47"
23:08:34 mnaser but no updated_at
23:08:49 mriedem https://github.com/openstack/nova/blob/16.0.4/nova/objects/request_spec.py#L411
23:09:05 mriedem 2017-03-07 is ocata right?
23:09:25 mriedem i'm wondering if this was a request spec created for an older instance
23:09:29 mnaser it was
23:09:29 mriedem what's the created_at on the instance?
23:09:38 mnaser 2015 created_at, 2017 requestspec
23:09:59 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/objects/request_spec.py#L590
23:09:59 mriedem ok in ocata this is the routine for creating requestspecs for old instances
23:10:31 mriedem which https://github.com/openstack/nova/blob/stable/ocata/nova/objects/request_spec.py#L405
23:10:32 mriedem however,
23:10:42 mriedem if that's an admin context, from the online data migration, it won't have a project id...
23:11:03 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L776
23:11:05 mnaser which explains how we landed in this case
23:11:15 mriedem https://github.com/openstack/nova/blob/stable/ocata/nova/context.py#L313
23:11:17 mriedem yup
23:11:32 mnaser i guess its probably not the only one
23:11:51 mriedem probably not
23:12:12 mriedem ok so you're hitting this trying to live migrate that instance right?
23:12:33 mnaser mriedem: yes but i believe that any operations involving placement will likely fail
23:12:41 mriedem so that's this http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1141
23:12:51 mriedem the scheduler is trying to create allocations in placement on the target node for that instance
23:13:06 mnaser correct, and because im not forcing it, it goes through the scheduler
23:13:16 mnaser and the scheduler tacks on project_id from the request_spec
23:14:10 mriedem yup https://github.com/openstack/nova/blob/16.0.4/nova/scheduler/filter_scheduler.py#L287
23:14:43 mriedem and in this case, the instance project_id is likely != the context.project_id because the context is the admin user
23:14:46 mriedem doing the live migration
23:14:48 mriedem SOB
23:15:14 mnaser i looked at the number of request_specs
23:15:19 mnaser lol
23:15:19 mnaser and its pretty terrifying to have to update it all
23:15:30 mriedem the number of reqspecs that don't have a project_id set?
23:15:42 mnaser i didnt want to run that query because im pretty sure ill burn down the sql server
23:15:56 mnaser close to a million records and i probably would have to wildcard match it
23:16:32 mriedem ?
23:16:32 mriedem select count(*) from nova_api.request_specs where project_id is null and deleted == 0;
23:16:50 mnaser request_specs contains a json thingy called 'spec'
23:17:02 mnaser {"nova_object.version": "1.5", ...}
23:17:06 mriedem oh right
23:17:16 mriedem yeah the request_specs.spec is a serialized json blob of the object
23:17:19 mriedem so forget your db query
23:17:23 mriedem jaypipes: ^
23:17:49 mriedem mnaser: well, i could hack something up for you quickish
23:17:58 mriedem mnaser: have you reported a bug yet?
23:18:14 mnaser mriedem: i havent yet, i just kinda discovered how i ended up here with your information
23:18:23 mnaser (i got as far as .. request spec doesnt have project id) but the online migration confirms it
23:18:27 mriedem ok, i can start hacking up a workaround if you can report a bug
23:18:52 mnaser mriedem: just out of curiosity, is project_id/user_id actually used by the placement api ?
23:19:07 mriedem not yet
23:19:18 mnaser but i guess we dont want to make it from bad to worse
23:19:37 mriedem the long-term idea is we can leverage the allocations with the project/user information for doing things like counting quotas without iterating the cells
23:20:27 mnaser gotcha
23:20:31 mnaser alright let me write up a bug
23:20:41 mriedem this would be very wrong for that though https://github.com/openstack/nova/blob/16.0.4/nova/scheduler/filter_scheduler.py#L293
23:20:46 mriedem if we're live migrating or evacuating
23:21:16 mnaser i guess thats why it says todo :>
23:21:28 mriedem heh
23:21:48 mriedem melwitt: ^ a todo to keep in mind if we ever want to use placement allocations to mine data for counting quotas
23:21:54 mriedem we aren't storing the correct user_id for all allocations
23:23:34 melwitt so we should have one claim per allocation or?
23:24:00 mriedem when migrating or evacuating, by default the context is the admin
23:24:03 mriedem b/c those are admin apis
23:24:25 mriedem so the user_id we're storing in the allocation for the instance is from the admin, but the project_id should come from the instance, which is the user
23:24:35 melwitt yeah, I see. guh
23:28:57 melwitt does it maybe work out because allocations are updated by the compute host every update interval? would it auto heal the user/project once we fix it?
23:29:07 mriedem no
23:29:15 mriedem computes don't mess with allocations once you're upgraded to pike
23:30:20 mnaser mriedem: https://bugs.launchpad.net/nova/+bug/1739318
23:30:21 openstack Launchpad bug 1739318 in OpenStack Compute (nova) "Online data migration context does not contain project_id" [Undecided,New]
23:30:28 mriedem mnaser: thanks
23:30:31 melwitt hm, I thought that's what update_available_resource did
23:30:39 mriedem melwitt: used to did
23:30:42 mnaser also looks like the claim resources which did `project_id = spec_obj.project_id` was moved to scheduler utils
23:30:44 melwitt damn

Earlier   Later