| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 21:51:14 | ludovic | until the source node repaired | |
| 21:51:47 | ludovic | so the goal is to failback and so free the evacuate aggregat | |
| 21:52:45 | mriedem | ludovic: sure, that's why it's not something built natively into nova | |
| 21:52:58 | mriedem | nova provides the API so a higher level service can orchestrate whatever you need here | |
| 21:53:57 | mriedem | edleafe: ack, will run the ironic patch on that | |
| 21:55:50 | jose-phillips | hi any idea | |
| 21:56:00 | jose-phillips | why im im getting this error on devstack | |
| 21:59:05 | jose-phillips | using qemu | |
| 21:59:05 | jose-phillips | can't apply process capabilities -1 | |
| 22:00:51 | rybridges | for the record mriedem, i tried recompiling qemu like you said... took the latest version i could find here http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/ | |
| 22:00:52 | wind | Hi, I'm trying to make a rest-api call from ironic to nova, was hoping if someone could gimme a hint how to do so ... Ironic.conf doesn't have anything in there for nova, so should i build a keystoneclient session, and then use that to retrieve the token, and compute-api endpoint and then send the GET & PUT Commands | |
| 22:00:56 | rybridges | still get the exact same erroro | |
| 22:01:21 | wind | Any suggestions would be really helpful.... I'm new to openstack | |
| 22:05:47 | mnaser | DEBUG? | |
| 22:05:47 | mnaser | 2017-12-19 22:04:53.593 44047 DEBUG nova.compute.resource_tracker [req-2858b7b9-d273-4347-aff3-dfa3fa10f4cd - - - - -] We're on a Pike compute host in a deployment with Ocata compute hosts. Auto-correcting allocations to handle Ocata-style assumptions. _update_usage_from_instance /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1042 <=== wouldnt this be nice if this was a warning instead of | |
| 22:06:17 | mnaser | given that I just found out Nova thinks there are Ocata hosts in my all-pike installation (i guess something is wrong somewhere) | |
| 22:08:02 | ludovic | mriedem: Just to be sure, in order to influence the host-evacuate process, i don't have other options than using host aggregate with flavor and extra_specs ? | |
| 22:08:36 | mnaser | ludovic: yes, but afaik migrations use the flavor and extra_specs that it had at the moment of provisioning | |
| 22:09:12 | mnaser | ex: if your flavor A had aggregate "foo" when a VM is booted, and you changed the aggregate to "bar", and then do a migration/evacuate, it will still try to look for "foo" | |
| 22:09:21 | mnaser | afaik that has been my experience but i might be wrong? | |
| 22:10:54 | ludovic | oh yes so it's necessary to provide two aggregates at the moemet of provisioning "prod" and "spare" | |
| 22:11:03 | ludovic | moment | |
| 22:11:15 | mnaser | well ideally if you launched an instance in 'prod', you dont want it to be evacuated into 'staging' | |
| 22:11:26 | mnaser | because maybe your host aggregate hardware in staging gets turned off at 5pm | |
| 22:13:30 | mnaser | oh would you look at that | |
| 22:13:54 | mnaser | "Unable to submit allocation for instance 42e2e0cd-0dd2-48c0-b873-ed9cd08a451a" .. placement returning 400, JSON does not validate: None is not of type 'string' ... instance['project_id'] == None somehow?! | |
| 22:14:12 | mnaser | does this ring any bells to anyone or should i start diving in | |
| 22:14:47 | penick | Working hard to get live migrations functional is like cranking on the handle for a jack-in-the-box. Except the thing in the box is a fist. And it punches you in the face. | |
| 22:15:21 | ludovic | mnaser: that sound JSON Filter not working no ? | |
| 22:16:03 | mnaser | penick: its always worked, but i guess this is a weird pike corner case, i see this - https://bugs.launchpad.net/nova/+bug/1701129 but it should be in pike which we're running | |
| 22:16:04 | openstack | Launchpad bug 1701129 in OpenStack Compute (nova) "Functional tests fail intermittently with 400 Bad Request from placement" [Low,Fix released] - Assigned to melanie witt (melwitt) | |
| 22:21:24 | mnaser | i guess for some reason RequestSpec is getting an empty project_id | |
| 22:21:47 | openstackgerrit | Lance Bragstad proposed openstack/nova master: Add scope_types to server policies https://review.openstack.org/525772 | |
| 22:23:12 | openstackgerrit | Merged openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516 | |
| 22:23:18 | openstackgerrit | Merged openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010 | |
| 22:23:25 | openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611 | |
| 22:23:49 | openstackgerrit | Merged openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505 | |
| 22:23:57 | openstackgerrit | Merged openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227 | |
| 22:24:04 | openstackgerrit | Merged openstack/nova master: Fix possible TypeError in VIF.fixed_ips https://review.openstack.org/527920 | |
| 22:49:34 | mnaser | instance = common.get_instance(self.compute_api, context, id) <== would anyone know if this supplies project_id by default? | |
| 22:50:28 | mnaser | because that's the instance which is passed down to conductor and by the time its at the scheduler, instance.project_id == None which then in turn makes it fail the request to the placement api | |
| 22:55:12 | mnaser | further investigation - {"project_id": null, "user_id": "695d5f386eed440cb0e38455e1afdc9e", "allocations": [{"resource_provider": {"uuid": "5d5c5177-29bb-484f-9cc6-928360afa195"}, "resources": {"MEMORY_MB": 512, "VCPU": 2, "DISK_GB": 20}}, {"resource_provider": {"uuid": "4e43861e-ee36-40b7-ba7b-2239b46a1609"}, "resources": {"VCPU": 2, "MEMORY_MB": 512, "DISK_GB": 20}}]} .. for some reason, user_id comes in but | |
| 22:55:13 | mnaser | project_id doesn't. fwiw, this is a server created in 2015. | |
| 22:57:59 | mnaser | the user_id is the user of the one executing the live migration, not the owner of the instance oddly enough | |
| 23:06:45 | mnaser | ok.. request_spec record has project_id set to null for that vm | |
| 23:06:54 | mnaser | in the database | |
| 23:06:56 | mnaser | why and how.. :( | |
| 23:06:57 | mriedem | hmm, not sure why the project_id would be null | |
| 23:07:02 | mriedem | should come off the context | |
| 23:07:07 | mriedem | sorry, was on a call for the last hour | |
| 23:07:32 | mnaser | mriedem: no problem, its null because .. its null in the request_specs table too.. | |
| 23:07:39 | mnaser | i wonder why | |
| 23:07:52 | mriedem | you said it's a really old instance right? | |
| 23:07:59 | mnaser | yes mriedem | |
| 23:08:25 | mriedem | ok reqspec is created here https://github.com/openstack/nova/blob/16.0.4/nova/compute/api.py#L899 | |
| 23:08:30 | mnaser | the created_at for the requestspec is "2017-03-07 02:28:47" | |
| 23:08:34 | mnaser | but no updated_at | |
| 23:08:49 | mriedem | https://github.com/openstack/nova/blob/16.0.4/nova/objects/request_spec.py#L411 | |
| 23:09:05 | mriedem | 2017-03-07 is ocata right? | |
| 23:09:25 | mriedem | i'm wondering if this was a request spec created for an older instance | |
| 23:09:29 | mnaser | it was | |
| 23:09:29 | mriedem | what's the created_at on the instance? | |
| 23:09:38 | mnaser | 2015 created_at, 2017 requestspec | |
| 23:09:59 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/objects/request_spec.py#L590 | |
| 23:09:59 | mriedem | ok in ocata this is the routine for creating requestspecs for old instances | |
| 23:10:31 | mriedem | which https://github.com/openstack/nova/blob/stable/ocata/nova/objects/request_spec.py#L405 | |
| 23:10:32 | mriedem | however, | |
| 23:10:42 | mriedem | if that's an admin context, from the online data migration, it won't have a project id... | |
| 23:11:03 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/cmd/manage.py#L776 | |
| 23:11:05 | mnaser | which explains how we landed in this case | |
| 23:11:15 | mriedem | https://github.com/openstack/nova/blob/stable/ocata/nova/context.py#L313 | |
| 23:11:17 | mriedem | yup | |
| 23:11:32 | mnaser | i guess its probably not the only one | |
| 23:11:51 | mriedem | probably not | |
| 23:12:12 | mriedem | ok so you're hitting this trying to live migrate that instance right? | |
| 23:12:33 | mnaser | mriedem: yes but i believe that any operations involving placement will likely fail | |
| 23:12:41 | mriedem | so that's this http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/client/report.py#n1141 | |
| 23:12:51 | mriedem | the scheduler is trying to create allocations in placement on the target node for that instance | |
| 23:13:06 | mnaser | correct, and because im not forcing it, it goes through the scheduler | |
| 23:13:16 | mnaser | and the scheduler tacks on project_id from the request_spec | |
| 23:14:10 | mriedem | yup https://github.com/openstack/nova/blob/16.0.4/nova/scheduler/filter_scheduler.py#L287 | |
| 23:14:43 | mriedem | and in this case, the instance project_id is likely != the context.project_id because the context is the admin user | |
| 23:14:46 | mriedem | doing the live migration | |
| 23:14:48 | mriedem | SOB | |
| 23:15:14 | mnaser | i looked at the number of request_specs | |
| 23:15:19 | mnaser | lol | |
| 23:15:19 | mnaser | and its pretty terrifying to have to update it all | |
| 23:15:30 | mriedem | the number of reqspecs that don't have a project_id set? | |
| 23:15:42 | mnaser | i didnt want to run that query because im pretty sure ill burn down the sql server | |
| 23:15:56 | mnaser | close to a million records and i probably would have to wildcard match it | |
| 23:16:32 | mriedem | ? | |
| 23:16:32 | mriedem | select count(*) from nova_api.request_specs where project_id is null and deleted == 0; | |
| 23:16:50 | mnaser | request_specs contains a json thingy called 'spec' | |
| 23:17:02 | mnaser | {"nova_object.version": "1.5", ...} | |
| 23:17:06 | mriedem | oh right | |
| 23:17:16 | mriedem | yeah the request_specs.spec is a serialized json blob of the object | |
| 23:17:19 | mriedem | so forget your db query | |
| 23:17:23 | mriedem | jaypipes: ^ | |
| 23:17:49 | mriedem | mnaser: well, i could hack something up for you quickish | |
| 23:17:58 | mriedem | mnaser: have you reported a bug yet? | |