| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-19 | |||
| 17:57:48 | mriedem | jaypipes: yeah the version samples | |
| 17:58:11 | mriedem | jaypipes: https://github.com/openstack/nova/tree/master/doc/api_samples/versions | |
| 17:58:46 | jaypipes | mriedem: I looked there but all I see is an interpolation marker for max_api_version | |
| 17:59:09 | jaypipes | mriedem: gah, never mind. | |
| 17:59:15 | jaypipes | mriedem: sigh... | |
| 17:59:26 | jaypipes | mriedem: was looking in nova/tests/functional/api_samples/ | |
| 17:59:44 | jaypipes | mriedem: have I mentioned I hate these? :) | |
| 18:04:35 | cfriesen | is all quota information now going into the API DB? or will we still put some in the main DB? | |
| 18:05:32 | openstackgerrit | Merged openstack/nova master: Fix 4 doc typos https://review.openstack.org/529084 | |
| 18:06:18 | jaypipes | melwitt: see cfriesen's ? above... | |
| 18:40:25 | openstackgerrit | Merged openstack/python-novaclient master: CommandError is raised for invalid server fields https://review.openstack.org/525110 | |
| 18:51:24 | mriedem | cfriesen: api db | |
| 18:51:54 | mriedem | cfriesen: starting in pike we don't use the usages or reservations tables anymore | |
| 19:01:48 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Support aggregate affinity scheduler filters https://review.openstack.org/529135 | |
| 19:31:02 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Support aggregate affinity scheduler filters https://review.openstack.org/529135 | |
| 20:11:11 | rybridges | So I am seeing errors every time i make a snapshot | |
| 20:11:43 | rybridges | regarldess of whether i use disable_libvirt_livesnapshot = true or disable_libvirt_livesnapshot = false | |
| 20:11:49 | rybridges | the error is the same every time | |
| 20:11:53 | rybridges | it happens on the hypervisor | |
| 20:12:04 | rybridges | this is nova-compute.log https://pastebin.com/WiqYtZzF | |
| 20:13:34 | rybridges | this is the error for libvirt log: https://pastebin.com/4NeSuNvX | |
| 20:14:17 | rybridges | its impossible to take a snapshot on ocata from the horizon ui | |
| 20:14:20 | rybridges | as far as we can tell | |
| 20:18:42 | mriedem | https://www.jrssite.com/wordpress/?p=302 | |
| 20:18:49 | mriedem | https://bugs.launchpad.net/nova/+bug/1381153 | |
| 20:18:50 | openstack | Launchpad bug 1381153 in OpenStack Compute (nova) "Cannot create instance live snapshots in Centos7 (icehouse)" [Undecided,Invalid] | |
| 20:28:19 | cfriesen | mriedem: thanks for the quota answer earlier...in objects.Quotas we're still looking at both the api DB and the main DB. is that left over from Pike? Presumably now we could remove the main DB access? | |
| 20:30:07 | mriedem | cfriesen: that's for the online data migration | |
| 20:30:17 | alee | hi - does anyone know how to force nova to re-fetch images from glance? | |
| 20:30:22 | mriedem | so we can't remove that until people have run through the online data migrations, | |
| 20:30:35 | mriedem | and we have a blocker schema migration in place to enforce that there are no quota limits/classes in the main cell db | |
| 20:30:35 | cfriesen | mriedem: that should have happened in Pike though, right? so we could remove it in Q? | |
| 20:30:49 | mriedem | cfriesen: you'd have to add a blocker migration | |
| 20:31:05 | cfriesen | mriedem: okay | |
| 21:04:45 | openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | |
| 21:04:46 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 21:04:52 | edleafe | mriedem: ^^ should address the functional test failures | |
| 21:06:11 | ludovic | Hi, I 'd like to ask few questions to nova experts: is it exists tips to prioritize evacuation order for instances | |
| 21:06:53 | openstackgerrit | Merged openstack/nova master: Some nit fix in multi_cell_list https://review.openstack.org/527597 | |
| 21:07:15 | openstackgerrit | Merged openstack/nova master: doc: add note about fixing admin-only APIs without a microversion https://review.openstack.org/527421 | |
| 21:07:33 | ludovic | or is it possible to reserve spare nodes ti ensure host-evacuation | |
| 21:08:15 | mriedem | ludovic: no and no, | |
| 21:08:24 | mriedem | you can specify a target host during evacuate, but that doesn't reserve it | |
| 21:08:35 | mriedem | and you evacuate one instance at a time, | |
| 21:08:42 | mriedem | so if there is priority, the caller handles that | |
| 21:10:56 | ludovic | ok, so we can't ensure a good SLA | |
| 21:11:37 | ludovic | of we can't reserve resources to securely evacuate all workload | |
| 21:11:54 | ludovic | the SLA is impacted | |
| 21:15:59 | ludovic | mriedem: does the scheduler propose as a trick to overcome this? | |
| 21:19:07 | mriedem | ludovic: how is this any different than the scheduler correctly picking a host during the initial server create? | |
| 21:19:28 | mriedem | besides reschedules | |
| 21:20:14 | ludovic | excuse-me , maybe can we have a private discussion about the Filter scheduler VS host-evacuate process with HA compute instances ? | |
| 21:20:28 | mriedem | why private? | |
| 21:21:01 | ludovic | in order to not disturb the room | |
| 21:21:11 | ludovic | np | |
| 21:21:48 | ludovic | I'm testing the HA compute instances with tripleo deployment | |
| 21:22:13 | ludovic | so the nova-evacuate pacemaker process is working well | |
| 21:22:35 | ludovic | but i was suprised by the Ram Filter scheduler | |
| 21:24:37 | ludovic | to explain the case , I have two compute Nodes. I have 2 Computes Nodes. The first compute Node with 31 instances ( total RAM allocated = 177152 M) , the second one with 2 instances ( Total RAM allocated = 24576 ) | |
| 21:24:55 | ludovic | RAM of computes Nodes is 196483 (memory_mb) | |
| 21:26:07 | ludovic | The Host-evacuate worked but 5 VMs was on ERROR with insufficient memory (nova-compute log) and on was with NO STATE (No Host found by RAM Filter) | |
| 21:26:24 | ludovic | one | |
| 21:26:37 | ludovic | I expected the same ERROR on these 6 instances | |
| 21:27:04 | ludovic | HOST NOT FOUND or ERROR with insufficient memory | |
| 21:27:44 | mriedem | so you're evacuating the 31 instances on the one compute node to the other compute node with 2 instances? | |
| 21:27:48 | ludovic | So i don't understand the result | |
| 21:28:02 | ludovic | yes | |
| 21:28:08 | mriedem | and 5 of 31 fail | |
| 21:28:13 | ludovic | 6 | |
| 21:28:14 | mriedem | which release? | |
| 21:28:40 | ludovic | 5 with insufficient memory and 1 with no valid host found | |
| 21:28:44 | mriedem | if you're in pike+ and using the FilterScheduler, you should remove the RamFilter | |
| 21:28:46 | ludovic | ocata | |
| 21:29:06 | mriedem | so the problem is, | |
| 21:29:22 | mriedem | the scheduler has a point in time snapshot of the resources from the compute, per request, | |
| 21:29:37 | ludovic | i use ocata for the moment | |
| 21:29:53 | mriedem | and the ram usage doesn't change until one of the evacuations makes it to the compute and claims those resources, and updates the compute node record in the db, which the scheduler will read in the next scheduling attempt | |
| 21:30:21 | mriedem | so the problem is if you're sending all 31 evacuate requests in a for loop, for example, with no time in between for the scheduler to catch up to the changes in the computes, | |
| 21:30:47 | mriedem | the scheduler thinks the compute is fine and sends the instance there, but the claim on the compute might fail because another request claimed those resources in the meantime | |
| 21:31:01 | ludovic | yes exactly | |
| 21:31:18 | mriedem | this should be fixed in pike, | |
| 21:31:35 | mriedem | because in pike, the RamFilter can be removed and the FilterScheduler uses the Placement service to claim the resources during scheduling | |
| 21:32:05 | mriedem | any claim collisions on the compute node in pike due to concurrent requests will be retried up to 3 times | |
| 21:32:35 | mriedem | otherwise what you're hitting is latent behavior | |
| 21:32:36 | ludovic | ah ok so with ocata the process is not enough efficient ? | |
| 21:32:59 | mriedem | correct; the late claim on the compute has always been a known issue with scheduling | |
| 21:33:10 | mriedem | with server create, if you hit this, we would reschedule to another compute | |
| 21:33:16 | mriedem | we don't do reschedules with evacuate though | |
| 21:33:45 | ludovic | But is it possible to influence the evacuate with max_concurrent_build ? | |
| 21:33:49 | ludovic | in nova.conf ? | |
| 21:34:27 | ludovic | by default this parameter is 10 , if we reduce to one ? | |
| 21:36:34 | mriedem | no max_concurrent_build is for server create, not evacuate | |
| 21:36:56 | ludovic | ah ok i understand . | |
| 21:36:58 | mriedem | there is no option like that for limiting evacuates | |
| 21:37:34 | mriedem | that's not to say one couldn't be added, but adding that in queens or pike doesn't make sense when we've solved this part of the problem in the scheduler | |
| 21:38:58 | mriedem | still, it seems reasonable to allow limiting the number of concurrent evacuates on a given compute, since we do that for spawn and live migrate | |
| 21:39:03 | mriedem | i wouldn't be opposed to adding something like that | |
| 21:40:54 | ludovic | ok thank you for your explanations, it's precious for me because I'm testing openstack for a big French Company you know | |
| 21:41:40 | ludovic | And the goal is to see if we can in the future replace the massive usage of WMare with OpenStack you know | |
| 21:41:59 | mriedem | oolala | |