| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-08-10 | |||
| 14:23:21 | dtantsur | bauzas: I've never heard of people using it, but yeah. For every crazy feature there are people to try it in production.. | |
| 14:23:26 | ioggstream | does anybody knows if soft-anti-affinity may be enabled in newton ? | |
| 14:23:38 | bauzas | ioggstream: IIRC, yes | |
| 14:24:15 | bauzas | ioggstream: https://blueprints.launchpad.net/nova/+spec/soft-affinity-for-server-group is Mitaka complete | |
| 14:24:16 | ioggstream | bauzas: by default it doesn't work but I saw that mitaka has an ERRATA | |
| 14:24:17 | mriedem | gibi: because the periodic task doesn't cleanup allocations anymore | |
| 14:24:24 | mriedem | gibi: it assumes the scheduler has everything correct | |
| 14:24:34 | mriedem | and the source node is 'down' | |
| 14:24:35 | gibi | mriedem: but not even the source compute cleans up? | |
| 14:24:46 | gibi | mriedem: after started up again? | |
| 14:24:48 | mriedem | if it's down we probably don't care about it | |
| 14:24:55 | mriedem | oh, we'll talk after the meeting | |
| 14:24:59 | gibi | mriedem: sure | |
| 14:26:15 | mriedem | but yeah the update_available_resource code in pike now does'nt overwrite the allocatoins | |
| 14:26:17 | mriedem | per that change | |
| 14:26:25 | mriedem | so that's why the source compute won't cleanup once it comes back up | |
| 14:26:32 | mriedem | it should remove though.... | |
| 14:27:03 | mriedem | gibi: this one https://review.openstack.org/#/c/491850/ | |
| 14:28:49 | gibi | mriedem: my test https://review.openstack.org/#/c/492548/ is top of https://review.openstack.org/#/c/491850/ and I still see the allocation on the source host | |
| 14:35:52 | mriedem | gibi: i think that's probably due to https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1047 | |
| 14:36:19 | gibi | mriedem: checking the debug log... | |
| 14:36:38 | mriedem | gibi: but we should get into https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1145 | |
| 14:36:44 | mriedem | _remove_deleted_instances_allocations | |
| 14:37:31 | gibi | mriedem: I see the debug log you pointed at | |
| 14:38:01 | mriedem | oh it could be https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1187 | |
| 14:38:07 | mriedem | instance.node == cn.hypervisor_hostname): | |
| 14:38:07 | mriedem | if (instance.host == cn.host and | |
| 14:38:13 | mriedem | we continue there | |
| 14:38:19 | mriedem | or if instance.host != cn.host: | |
| 14:38:21 | mriedem | we also continue there | |
| 14:38:46 | mriedem | seems we should check to see if the instance is in self.tracked_migrations | |
| 14:38:47 | gibi | I can insert some extra log to confirm | |
| 14:40:49 | gibi | ahh there is logs already | |
| 14:41:12 | gibi | it is the instance.host == cn.host where we continue | |
| 14:43:17 | jaypipes | mriedem: ty | |
| 14:43:28 | jaypipes | mriedem: fyi, kinda vacationing today... | |
| 14:43:48 | jaypipes | mriedem: will work on my patches thouhg | |
| 14:45:36 | mriedem | jaypipes: don't think you have anything to work on | |
| 14:45:43 | mriedem | except follow ups for additional testing and whatnot | |
| 14:46:21 | mriedem | gibi: ok so self.tracked_migrations probably won't help after we restart the compute service since that dict will probably be empty | |
| 14:46:28 | mriedem | gibi: i left some comments in your test change, | |
| 14:46:34 | mriedem | we could maybe do some allocation cleanup in https://github.com/openstack/nova/blob/9a66d039a14afd591f4a3b6e655580aeeed17d29/nova/compute/manager.py#L649 | |
| 14:46:48 | jaypipes | mriedem: yeah | |
| 14:47:24 | mriedem | gibi: this is similar to https://bugs.launchpad.net/nova/+bug/1679750 where we don't delete the allocations for the instance on the compute host during a 'local delete' in the API | |
| 14:47:25 | openstack | Launchpad bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress] | |
| 14:47:34 | mriedem | when we can't get to the compute host b/c it's down | |
| 14:47:41 | sdague | mriedem: ok, stepping away from the computer for a bit... | |
| 14:48:24 | bauzas | mriedem: https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1134 isn't a cellv2-compatible call since it would be an upcall, no ? | |
| 14:48:56 | gibi | mriedem: should I file a bug or we will squeez the fix into https://review.openstack.org/#/c/491012 ? | |
| 14:49:15 | mriedem | gibi: i think it's a new bug | |
| 14:49:19 | mriedem | gibi: similar to bug 1679750 | |
| 14:49:21 | openstack | bug 1679750 in OpenStack Compute (nova) "Allocations are not cleaned up in placement for instance 'local delete' case" [Medium,In progress] https://launchpad.net/bugs/1679750 | |
| 14:49:22 | mriedem | but slightly different | |
| 14:49:28 | gibi | mriedem: OK, I will report it | |
| 14:49:37 | mriedem | bug 1679750 is about how we don't remove the allocations when we delete the instance and the compute is down | |
| 14:49:54 | mriedem | in your case, the instance isn't deleted, it's evacuated, and we just don't cleanup the allocations from the original host when it comes back up | |
| 14:50:26 | mriedem | bauzas: services are per-cell | |
| 14:50:28 | mriedem | so it's not an upcall | |
| 14:50:50 | gibi | mriedem: thank for the pointers | |
| 14:50:51 | mriedem | bauzas: this is why we needed to expose the service uuids out of the API in 2.53 | |
| 14:51:01 | mriedem | to identify nova-compute 1 from nova-compute 1 in cell A and cell B | |
| 14:51:58 | bauzas | mriedem: so it only verifies if all the local-cell computes are having that version, right? | |
| 14:52:19 | bauzas | mriedem: since we're in a local compute | |
| 14:52:42 | mriedem | yes | |
| 14:52:53 | mriedem | moves are also restricted to the cell that the instance already lives in | |
| 14:53:12 | ioggstream | bauzas: https://bugzilla.redhat.com/show_bug.cgi?id=1447798 thx++ | |
| 14:53:13 | ioggstream | fyi | |
| 14:53:13 | openstack | bugzilla.redhat.com bug 1447798 in python-openstackclient "soft-anti-affinity is not accepted by python-openstackclient but is with python-novaclient" [High,New] - Assigned to jruzicka | |
| 14:53:23 | mriedem | https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1166 | |
| 14:53:43 | mriedem | ioggstream: i don't think osc is handling nova microversions | |
| 14:53:46 | bauzas | mriedem: correct, so it's probably not a problem | |
| 14:53:50 | mriedem | and the soft affinity policy rules were in a microversion | |
| 14:54:06 | mriedem | bauzas: yes of all the problems we could have with this series of changes, that's one i'm not worried about :) | |
| 14:54:15 | bauzas | fortunately because we don't support multi-cell moves | |
| 14:54:20 | bauzas | okay | |
| 14:54:23 | bauzas | gtk :) | |
| 14:54:26 | mriedem | and likely won't for a long time | |
| 14:54:40 | ioggstream | mriedem: you mean osc challenges the server for the supported options? | |
| 14:55:13 | mriedem | ioggstream: no, i mean i believe osc only ever requests compute API things with 2.1 | |
| 14:55:39 | mriedem | and the soft affinity group policy was added in 2.15 https://docs.openstack.org/nova/latest/reference/api-microversion-history#id13 | |
| 14:56:04 | mriedem | so if you're using osc and try to create a server group with policy 'soft-anti-affinity' using microversion 2.1, it's going to fail with a 400 | |
| 14:56:08 | mriedem | saying it doesn't know what that policy is | |
| 14:56:30 | gibi | mriedem: I filed https://bugs.launchpad.net/nova/+bug/1709902 | |
| 14:56:32 | openstack | Launchpad bug 1709902 in OpenStack Compute (nova) "source host allocation not cleaned up in placement after evacuation" [Undecided,New] | |
| 14:56:50 | ioggstream | mriedem: ok, I didn't know the microversion stuff, let me read about it | |
| 14:56:54 | mriedem | ioggstream: you might be able to pass an --openstack-compute-version or something option to osc cli, idk | |
| 14:58:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests https://review.openstack.org/492247 | |
| 14:59:05 | ioggstream | mriedem: I'll check and let you know, thx++ | |
| 14:59:39 | openstackgerrit | Dmitry Tantsur proposed openstack/nova master: Deprecate bare metal filters https://review.openstack.org/492563 | |
| 14:59:41 | dtantsur | edleafe: ^^^ | |
| 15:00:16 | edleafe | dtantsur: get a faster machine! :) | |
| 15:03:57 | bauzas | mriedem: do you agree with my comment on https://review.openstack.org/#/c/491012/12/nova/compute/resource_tracker.py@1073 and if so, possibly me passing a new revision ? | |
| 15:05:06 | mriedem | dtantsur: heh yeah you need more VCPUs | |
| 15:05:27 | dtantsur | LOOL | |
| 15:05:50 | bauzas | t470s FTW | |
| 15:06:00 | openstackgerrit | Eric Fried proposed openstack/nova master: nova.utils.get_ksa_adapter() https://review.openstack.org/488137 | |
| 15:06:00 | openstackgerrit | Eric Fried proposed openstack/nova master: Use ksa adapter for placement conf & requests https://review.openstack.org/492247 | |
| 15:06:18 | mriedem | bauzas: given the amount of shit we log continually all the time, like lazy-loading pci request and pci devices for every claim, then i'd say leave that for a follow up patch | |
| 15:06:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Get auth from context for glance endpoint https://review.openstack.org/490057 | |
| 15:06:30 | bauzas | mriedem: ok, firing it up | |