| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 21:08:35 | mriedem | and you evacuate one instance at a time, | |
| 21:08:42 | mriedem | so if there is priority, the caller handles that | |
| 21:10:56 | ludovic | ok, so we can't ensure a good SLA | |
| 21:11:37 | ludovic | of we can't reserve resources to securely evacuate all workload | |
| 21:11:54 | ludovic | the SLA is impacted | |
| 21:15:59 | ludovic | mriedem: does the scheduler propose as a trick to overcome this? | |
| 21:19:07 | mriedem | ludovic: how is this any different than the scheduler correctly picking a host during the initial server create? | |
| 21:19:28 | mriedem | besides reschedules | |
| 21:20:14 | ludovic | excuse-me , maybe can we have a private discussion about the Filter scheduler VS host-evacuate process with HA compute instances ? | |
| 21:20:28 | mriedem | why private? | |
| 21:21:01 | ludovic | in order to not disturb the room | |
| 21:21:11 | ludovic | np | |
| 21:21:48 | ludovic | I'm testing the HA compute instances with tripleo deployment | |
| 21:22:13 | ludovic | so the nova-evacuate pacemaker process is working well | |
| 21:22:35 | ludovic | but i was suprised by the Ram Filter scheduler | |
| 21:24:37 | ludovic | to explain the case , I have two compute Nodes. I have 2 Computes Nodes. The first compute Node with 31 instances ( total RAM allocated = 177152 M) , the second one with 2 instances ( Total RAM allocated = 24576 ) | |
| 21:24:55 | ludovic | RAM of computes Nodes is 196483 (memory_mb) | |
| 21:26:07 | ludovic | The Host-evacuate worked but 5 VMs was on ERROR with insufficient memory (nova-compute log) and on was with NO STATE (No Host found by RAM Filter) | |
| 21:26:24 | ludovic | one | |
| 21:26:37 | ludovic | I expected the same ERROR on these 6 instances | |
| 21:27:04 | ludovic | HOST NOT FOUND or ERROR with insufficient memory | |
| 21:27:44 | mriedem | so you're evacuating the 31 instances on the one compute node to the other compute node with 2 instances? | |
| 21:27:48 | ludovic | So i don't understand the result | |
| 21:28:02 | ludovic | yes | |
| 21:28:08 | mriedem | and 5 of 31 fail | |
| 21:28:13 | ludovic | 6 | |
| 21:28:14 | mriedem | which release? | |
| 21:28:40 | ludovic | 5 with insufficient memory and 1 with no valid host found | |
| 21:28:44 | mriedem | if you're in pike+ and using the FilterScheduler, you should remove the RamFilter | |
| 21:28:46 | ludovic | ocata | |
| 21:29:06 | mriedem | so the problem is, | |
| 21:29:22 | mriedem | the scheduler has a point in time snapshot of the resources from the compute, per request, | |
| 21:29:37 | ludovic | i use ocata for the moment | |
| 21:29:53 | mriedem | and the ram usage doesn't change until one of the evacuations makes it to the compute and claims those resources, and updates the compute node record in the db, which the scheduler will read in the next scheduling attempt | |
| 21:30:21 | mriedem | so the problem is if you're sending all 31 evacuate requests in a for loop, for example, with no time in between for the scheduler to catch up to the changes in the computes, | |
| 21:30:47 | mriedem | the scheduler thinks the compute is fine and sends the instance there, but the claim on the compute might fail because another request claimed those resources in the meantime | |
| 21:31:01 | ludovic | yes exactly | |
| 21:31:18 | mriedem | this should be fixed in pike, | |
| 21:31:35 | mriedem | because in pike, the RamFilter can be removed and the FilterScheduler uses the Placement service to claim the resources during scheduling | |
| 21:32:05 | mriedem | any claim collisions on the compute node in pike due to concurrent requests will be retried up to 3 times | |
| 21:32:35 | mriedem | otherwise what you're hitting is latent behavior | |
| 21:32:36 | ludovic | ah ok so with ocata the process is not enough efficient ? | |
| 21:32:59 | mriedem | correct; the late claim on the compute has always been a known issue with scheduling | |
| 21:33:10 | mriedem | with server create, if you hit this, we would reschedule to another compute | |
| 21:33:16 | mriedem | we don't do reschedules with evacuate though | |
| 21:33:45 | ludovic | But is it possible to influence the evacuate with max_concurrent_build ? | |
| 21:33:49 | ludovic | in nova.conf ? | |
| 21:34:27 | ludovic | by default this parameter is 10 , if we reduce to one ? | |
| 21:36:34 | mriedem | no max_concurrent_build is for server create, not evacuate | |
| 21:36:56 | ludovic | ah ok i understand . | |
| 21:36:58 | mriedem | there is no option like that for limiting evacuates | |
| 21:37:34 | mriedem | that's not to say one couldn't be added, but adding that in queens or pike doesn't make sense when we've solved this part of the problem in the scheduler | |
| 21:38:58 | mriedem | still, it seems reasonable to allow limiting the number of concurrent evacuates on a given compute, since we do that for spawn and live migrate | |
| 21:39:03 | mriedem | i wouldn't be opposed to adding something like that | |
| 21:40:54 | ludovic | ok thank you for your explanations, it's precious for me because I'm testing openstack for a big French Company you know | |
| 21:41:40 | ludovic | And the goal is to see if we can in the future replace the massive usage of WMare with OpenStack you know | |
| 21:41:59 | mriedem | oolala | |
| 21:42:00 | ludovic | near future | |
| 21:42:21 | mriedem | ok. would be cool if you could test this out on a pike deployment. | |
| 21:42:32 | mriedem | to make sure the filter scheduler + placement is correctly handling this for you | |
| 21:42:42 | ludovic | That's why i asked if reservation and /or prioritizing exist under OpenStack | |
| 21:42:56 | mriedem | remember to remove the RamFilter in pike if you're using the FilterScheduler since (1) it's redundant and (2) it will remove the memory_mb claim in the compute | |
| 21:43:39 | ludovic | ok | |
| 21:44:40 | ludovic | don't you think it will be interesting to add the possibility to have spare Compute Nodes ? | |
| 21:45:02 | ludovic | with aggregate host spare for example | |
| 21:45:10 | mriedem | you mean build something into nova to mark specific computes as only used for evacuate? | |
| 21:45:37 | ludovic | or in a aggregat to propose the prioritinzing of important workload when evacuate | |
| 21:45:46 | ludovic | yes for example | |
| 21:46:35 | mriedem | yeah idk, maybe. i wouldn't want to change the evacuate api to pass through scheduler hints probably. | |
| 21:46:36 | ludovic | That will be ensure a very good SLA because the evacuate processus would be securized | |
| 21:46:54 | mriedem | i don't know how many deployments just have compute nodes lying around as spares for evacuate | |
| 21:47:20 | mriedem | you can also control which host is used client-side | |
| 21:47:21 | mriedem | as noted | |
| 21:48:00 | mriedem | so as a client, if you have a special "evacuate" host aggregate, you could round robin through those hosts and send it with the evacuate request | |
| 21:48:38 | mriedem | but, as your evacuate aggregate starts to fill it, it is no longer spare capacity | |
| 21:48:39 | mriedem | so... | |
| 21:48:46 | mriedem | *fill up | |
| 21:49:55 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 21:49:56 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 21:50:01 | edleafe | ^^ fixed pep8 booboo | |
| 21:50:33 | ludovic | it seems to not be easy to design ... | |
| 21:50:49 | ludovic | evacuate is temporary | |
| 21:51:14 | ludovic | until the source node repaired | |
| 21:51:47 | ludovic | so the goal is to failback and so free the evacuate aggregat | |
| 21:52:45 | mriedem | ludovic: sure, that's why it's not something built natively into nova | |
| 21:52:58 | mriedem | nova provides the API so a higher level service can orchestrate whatever you need here | |
| 21:53:57 | mriedem | edleafe: ack, will run the ironic patch on that | |
| 21:55:50 | jose-phillips | hi any idea | |
| 21:56:00 | jose-phillips | why im im getting this error on devstack | |
| 21:59:05 | jose-phillips | can't apply process capabilities -1 | |
| 21:59:05 | jose-phillips | using qemu | |
| 22:00:51 | rybridges | for the record mriedem, i tried recompiling qemu like you said... took the latest version i could find here http://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/ | |
| 22:00:52 | wind | Hi, I'm trying to make a rest-api call from ironic to nova, was hoping if someone could gimme a hint how to do so ... Ironic.conf doesn't have anything in there for nova, so should i build a keystoneclient session, and then use that to retrieve the token, and compute-api endpoint and then send the GET & PUT Commands | |
| 22:00:56 | rybridges | still get the exact same erroro | |
| 22:01:21 | wind | Any suggestions would be really helpful.... I'm new to openstack | |
| 22:05:47 | mnaser | 2017-12-19 22:04:53.593 44047 DEBUG nova.compute.resource_tracker [req-2858b7b9-d273-4347-aff3-dfa3fa10f4cd - - - - -] We're on a Pike compute host in a deployment with Ocata compute hosts. Auto-correcting allocations to handle Ocata-style assumptions. _update_usage_from_instance /usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py:1042 <=== wouldnt this be nice if this was a warning instead of | |
| 22:05:47 | mnaser | DEBUG? | |
| 22:06:17 | mnaser | given that I just found out Nova thinks there are Ocata hosts in my all-pike installation (i guess something is wrong somewhere) | |
| 22:08:02 | ludovic | mriedem: Just to be sure, in order to influence the host-evacuate process, i don't have other options than using host aggregate with flavor and extra_specs ? | |
| 22:08:36 | mnaser | ludovic: yes, but afaik migrations use the flavor and extra_specs that it had at the moment of provisioning | |