Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-12-19
21:29:37 ludovic i use ocata for the moment
21:29:53 mriedem and the ram usage doesn't change until one of the evacuations makes it to the compute and claims those resources, and updates the compute node record in the db, which the scheduler will read in the next scheduling attempt
21:30:21 mriedem so the problem is if you're sending all 31 evacuate requests in a for loop, for example, with no time in between for the scheduler to catch up to the changes in the computes,
21:30:47 mriedem the scheduler thinks the compute is fine and sends the instance there, but the claim on the compute might fail because another request claimed those resources in the meantime
21:31:01 ludovic yes exactly
21:31:18 mriedem this should be fixed in pike,
21:31:35 mriedem because in pike, the RamFilter can be removed and the FilterScheduler uses the Placement service to claim the resources during scheduling
21:32:05 mriedem any claim collisions on the compute node in pike due to concurrent requests will be retried up to 3 times
21:32:35 mriedem otherwise what you're hitting is latent behavior
21:32:36 ludovic ah ok so with ocata the process is not enough efficient ?
21:32:59 mriedem correct; the late claim on the compute has always been a known issue with scheduling
21:33:10 mriedem with server create, if you hit this, we would reschedule to another compute
21:33:16 mriedem we don't do reschedules with evacuate though
21:33:45 ludovic But is it possible to influence the evacuate with max_concurrent_build ?
21:33:49 ludovic in nova.conf ?
21:34:27 ludovic by default this parameter is 10 , if we reduce to one ?
21:36:34 mriedem no max_concurrent_build is for server create, not evacuate
21:36:56 ludovic ah ok i understand .
21:36:58 mriedem there is no option like that for limiting evacuates
21:37:34 mriedem that's not to say one couldn't be added, but adding that in queens or pike doesn't make sense when we've solved this part of the problem in the scheduler
21:38:58 mriedem still, it seems reasonable to allow limiting the number of concurrent evacuates on a given compute, since we do that for spawn and live migrate
21:39:03 mriedem i wouldn't be opposed to adding something like that
21:40:54 ludovic ok thank you for your explanations, it's precious for me because I'm testing openstack for a big French Company you know
21:41:40 ludovic And the goal is to see if we can in the future replace the massive usage of WMare with OpenStack you know
21:41:59 mriedem oolala
21:42:00 ludovic near future
21:42:21 mriedem ok. would be cool if you could test this out on a pike deployment.
21:42:32 mriedem to make sure the filter scheduler + placement is correctly handling this for you
21:42:42 ludovic That's why i asked if reservation and /or prioritizing exist under OpenStack
21:42:56 mriedem remember to remove the RamFilter in pike if you're using the FilterScheduler since (1) it's redundant and (2) it will remove the memory_mb claim in the compute
21:43:39 ludovic ok
21:44:40 ludovic don't you think it will be interesting to add the possibility to have spare Compute Nodes ?
21:45:02 ludovic with aggregate host spare for example
21:45:10 mriedem you mean build something into nova to mark specific computes as only used for evacuate?
21:45:37 ludovic or in a aggregat to propose the prioritinzing of important workload when evacuate
21:45:46 ludovic yes for example
21:46:35 mriedem yeah idk, maybe. i wouldn't want to change the evacuate api to pass through scheduler hints probably.
21:46:36 ludovic That will be ensure a very good SLA because the evacuate processus would be securized
21:46:54 mriedem i don't know how many deployments just have compute nodes lying around as spares for evacuate
21:47:20 mriedem you can also control which host is used client-side
21:47:21 mriedem as noted
21:48:00 mriedem so as a client, if you have a special "evacuate" host aggregate, you could round robin through those hosts and send it with the evacuate request
21:48:38 mriedem but, as your evacuate aggregate starts to fill it, it is no longer spare capacity
21:48:39 mriedem so...
21:48:46 mriedem *fill up
21:49:55 openstackgerrit Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358
21:49:56 openstackgerrit Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436
21:50:01 edleafe ^^ fixed pep8 booboo
21:50:33 ludovic it seems to not be easy to design ...
21:50:49 ludovic evacuate is temporary
21:51:14 ludovic until the source node repaired
21:51:47 ludovic so the goal is to failback and so free the evacuate aggregat
21:52:45 mriedem ludovic: sure, that's why it's not something built natively into nova
21:52:58 mriedem nova provides the API so a higher level service can orchestrate whatever you need here
21:53:57 mriedem edleafe: ack, will run the ironic patch on that
21:55:50 jose-phillips hi any idea
21:56:00 jose-phillips why im im getting this error on devstack
21:59:05 jose-phillips using qemu
21:59:05 jose-phillips can't apply process capabilities -1
22:00:51 rybridges for the record mriedem, i tried recompiling qemu like you said... took the latest version i could find here http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/
22:00:52 wind Hi, I'm trying to make a rest-api call from ironic to nova, was hoping if someone could gimme a hint how to do so ... Ironic.conf doesn't have anything in there for nova, so should i build a keystoneclient session, and then use that to retrieve the token, and compute-api endpoint and then send the GET & PUT Commands
22:00:56 rybridges still get the exact same erroro
22:01:21 wind Any suggestions would be really helpful.... I'm new to openstack
22:05:47 mnaser DEBUG?
22:05:47 mnaser 2017-12-19 22:04:53.593 44047 DEBUG nova.compute.resource_tracker [req-2858b7b9-d273-4347-aff3-dfa3fa10f4cd - - - - -] We're on a Pike compute host in a deployment with Ocata compute hosts. Auto-correcting allocations to handle Ocata-style assumptions. _update_usage_from_instance /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1042 <=== wouldnt this be nice if this was a warning instead of
22:06:17 mnaser given that I just found out Nova thinks there are Ocata hosts in my all-pike installation (i guess something is wrong somewhere)
22:08:02 ludovic mriedem: Just to be sure, in order to influence the host-evacuate process, i don't have other options than using host aggregate with flavor and extra_specs ?
22:08:36 mnaser ludovic: yes, but afaik migrations use the flavor and extra_specs that it had at the moment of provisioning
22:09:12 mnaser ex: if your flavor A had aggregate "foo" when a VM is booted, and you changed the aggregate to "bar", and then do a migration/evacuate, it will still try to look for "foo"
22:09:21 mnaser afaik that has been my experience but i might be wrong?
22:10:54 ludovic oh yes so it's necessary to provide two aggregates at the moemet of provisioning "prod" and "spare"
22:11:03 ludovic moment
22:11:15 mnaser well ideally if you launched an instance in 'prod', you dont want it to be evacuated into 'staging'
22:11:26 mnaser because maybe your host aggregate hardware in staging gets turned off at 5pm
22:13:30 mnaser oh would you look at that
22:13:54 mnaser "Unable to submit allocation for instance 42e2e0cd-0dd2-48c0-b873-ed9cd08a451a" .. placement returning 400, JSON does not validate: None is not of type 'string' ... instance['project_id'] == None somehow?!
22:14:12 mnaser does this ring any bells to anyone or should i start diving in
22:14:47 penick Working hard to get live migrations functional is like cranking on the handle for a jack-in-the-box. Except the thing in the box is a fist. And it punches you in the face.
22:15:21 ludovic mnaser: that sound JSON Filter not working no ?
22:16:03 mnaser penick: its always worked, but i guess this is a weird pike corner case, i see this - https://bugs.launchpad.net/nova/+bug/1701129 but it should be in pike which we're running
22:16:04 openstack Launchpad bug 1701129 in OpenStack Compute (nova) "Functional tests fail intermittently with 400 Bad Request from placement" [Low,Fix released] - Assigned to melanie witt (melwitt)
22:21:24 mnaser i guess for some reason RequestSpec is getting an empty project_id
22:21:47 openstackgerrit Lance Bragstad proposed openstack/nova master: Add scope_types to server policies https://review.openstack.org/525772
22:23:12 openstackgerrit Merged openstack/nova master: Convert ext filesystem resizes to privsep. https://review.openstack.org/517516
22:23:18 openstackgerrit Merged openstack/nova master: Move flushing block devices to privsep. https://review.openstack.org/519010
22:23:25 openstackgerrit Merged openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611
22:23:49 openstackgerrit Merged openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505
22:23:57 openstackgerrit Merged openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227
22:24:04 openstackgerrit Merged openstack/nova master: Fix possible TypeError in VIF.fixed_ips https://review.openstack.org/527920
22:49:34 mnaser instance = common.get_instance(self.compute_api, context, id) <== would anyone know if this supplies project_id by default?
22:50:28 mnaser because that's the instance which is passed down to conductor and by the time its at the scheduler, instance.project_id == None which then in turn makes it fail the request to the placement api
22:55:12 mnaser further investigation - {"project_id": null, "user_id": "695d5f386eed440cb0e38455e1afdc9e", "allocations": [{"resource_provider": {"uuid": "5d5c5177-29bb-484f-9cc6-928360afa195"}, "resources": {"MEMORY_MB": 512, "VCPU": 2, "DISK_GB": 20}}, {"resource_provider": {"uuid": "4e43861e-ee36-40b7-ba7b-2239b46a1609"}, "resources": {"VCPU": 2, "MEMORY_MB": 512, "DISK_GB": 20}}]} .. for some reason, user_id comes in but
22:55:13 mnaser project_id doesn't. fwiw, this is a server created in 2015.
22:57:59 mnaser the user_id is the user of the one executing the live migration, not the owner of the instance oddly enough
23:06:45 mnaser ok.. request_spec record has project_id set to null for that vm
23:06:54 mnaser in the database
23:06:56 mnaser why and how.. :(
23:06:57 mriedem hmm, not sure why the project_id would be null
23:07:02 mriedem should come off the context
23:07:07 mriedem sorry, was on a call for the last hour

Earlier   Later