| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-22 | |||
| 16:40:18 | kashyap | (They're all fixed, though. And it hasn't been updated in ages) | |
| 16:45:08 | mriedem | the more i dig into https://bugs.launchpad.net/nova/+bug/1790204 the more it's a spec at this point | |
| 16:45:09 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 16:47:14 | mriedem | i imagine this is all dead code since queens or at least rocky https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L140 | |
| 16:47:43 | mriedem | because we should never have to double up allocations on the same consumer (instance) during scheduling since we moved the allocations on the source node to the migration record, | |
| 16:47:43 | sean-k-mooney | kashyap: let me rephsase this is the first time i have ever personally it a nested vert bug in the 5-6 years i have used nested virt by defualt | |
| 16:47:47 | mriedem | except maybe evacuate... | |
| 16:48:00 | kashyap | sean-k-mooney: Ah, okay. :-) | |
| 16:49:00 | melwitt | o/ | |
| 16:59:56 | cdent | fried_rice: my turn to ask you for a reminder: | |
| 17:00:20 | cdent | resource provider A represents a compute node, currently idle | |
| 17:00:58 | cdent | third part system makes allocations against that compute node (consuming capacity in placement but not in _reality_) | |
| 17:01:31 | cdent | the compute node will or will not correct that? | |
| 17:01:57 | cdent | I feel like we've gone back and forth on that a bit and I forget where we are now | |
| 17:02:11 | cdent | I'd just test it, but I currently have no cloud | |
| 17:03:30 | fried_rice | cdent: If the allocation is made against a real instance UUID, I think nova will "heal" it at some point. Otherwise, we don't muck with allocations that aren't related to instances afaik. | |
| 17:03:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Dedupe compute install guides https://review.openstack.org/638716 | |
| 17:03:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Dedupe controller install guides https://review.openstack.org/638715 | |
| 17:04:17 | fried_rice | cdent: mriedem might have to weigh in on the circumstances under which healing occurs at all. It may be just at the behest of a nova-manage command. | |
| 17:04:41 | cdent | i've managed to make myself curious enough that I'm going to go spin up the devstack | |
| 17:05:04 | fried_rice | cdent: Now, oob inventory/trait/agg changes against providers owned by the compute node will definitely be "healed" next time upt runs. | |
| 17:05:32 | fried_rice | which actually may not be as soon as we would like - was having this conversation with aspiers the other day. | |
| 17:06:36 | cdent | yeah, to inventory/trait stuff | |
| 17:07:01 | cdent | what's the issue on the "soon"? | |
| 17:09:25 | mriedem | there is no auto-healing of allocations by nova-compute | |
| 17:09:30 | kashyap | mriedem: I think I found the sucker in my case; spent the last hour-ish duking aroud the system. This looks suspicious: | |
| 17:09:34 | kashyap | --- | |
| 17:09:35 | kashyap | Feb 21 17:14:13 ubuntu-bionic-inap-mtl01-0002851272 kernel: traps: qemu-system-x86[31240] general protection ip:5600cbedaf78 sp:7f2dba1ebf00 error:0 in qemu-system-x86_64[5600cb81c000+8d2000] | |
| 17:09:39 | kashyap | --- | |
| 17:10:01 | mriedem | cdent: fried_rice: coincidentally, read the commit message on https://review.openstack.org/#/c/638700/ for a history lesson | |
| 17:10:22 | aspiers | cdent: IIUC, the sync from provider tree to placement is essentially one way, so if something gets removed from placement then it won't get replaced until the provider cache is reset | |
| 17:10:44 | fried_rice | cdent: It came up when aspiers was fixing the "automatic traits from capabilities" patch https://review.openstack.org/#/c/538498/ | |
| 17:10:45 | cdent | mriedem: I read that a bit earlier today and the aforementioned lack of brain made all the characters move around | |
| 17:10:45 | fried_rice | But we enabled switching off periodic refresh. So that might never happen. | |
| 17:10:45 | fried_rice | because one of the test scenarios needs to be: "user" adds or removes a compute-owned capability trait; compute restores it. | |
| 17:11:15 | aspiers | cdent: https://review.openstack.org/#/c/538498/15/nova/tests/functional/test_servers.py@2294 | |
| 17:11:34 | mriedem | tl;dr while we had computes older than pike, the compute service would overwrite instance allocations created by the scheduler | |
| 17:12:00 | cdent | I'm confused by this notion of user removes compute-owned capability trait. I'm guessing that "don't do that" is considered insufficient? | |
| 17:12:35 | fried_rice | cdent: yeah :) | |
| 17:12:37 | mriedem | users most often probably don't know what they should and shouldn't do | |
| 17:12:52 | aspiers | cdent: it's due to this optimisation https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1047 | |
| 17:13:19 | cdent | i guess I was hoping for "most users probably don't know how to delete a trait" | |
| 17:13:31 | mriedem | cdent: if any code is going to mess with your externally mucked allocation stuff, it's like this https://github.com/openstack/nova/blob/27617ee1931b3240dbd0ad4c7d8ffd64cc202bc9/nova/compute/resource_tracker.py#L1241 | |
| 17:13:31 | aspiers | cdent: see also https://pasteboard.co/I1Tbzp2.png (which is linked from the commit message) | |
| 17:14:05 | mriedem | cdent: which would get you an INFO message https://github.com/openstack/nova/blob/27617ee1931b3240dbd0ad4c7d8ffd64cc202bc9/nova/compute/resource_tracker.py#L1293 | |
| 17:14:11 | cdent | mriedem: yeah, was just looking at that method | |
| 17:14:47 | cdent | aspiers: yeah, that's why I was trying to suggest that the os-traits should grow bigger, faster | |
| 17:14:58 | fried_rice | mriedem: we still haven't solved the resize-to-same-host doubling though, right? | |
| 17:15:15 | aspiers | cdent: makes sense | |
| 17:15:55 | fried_rice | mriedem: which is what your last paragraph is talking about | |
| 17:16:11 | fried_rice | cdent: It doesn't matter if they're in os-traits or not. | |
| 17:16:58 | cdent | they can't be deleted if they are in os-traits | |
| 17:17:13 | cdent | (that was all I was getting at, nothing more) | |
| 17:17:16 | fried_rice | not deleted from placement; deleted from the compute node. | |
| 17:17:17 | aspiers | I think they can be deleted from a resource provider | |
| 17:17:21 | fried_rice | yeah | |
| 17:17:42 | fried_rice | aspiers worded it better, "delete from placement" is too vague. | |
| 17:17:44 | aspiers | just not from global traits list | |
| 17:17:56 | cdent | oh, from the rp, I see | |
| 17:18:03 | cdent | yeah, just don't do that :) | |
| 17:18:05 | aspiers | fried_rice: the ambiguity only just became clear in my head ;-/ | |
| 17:18:29 | aspiers | hence why it's still present in that Venn diagram I just linked | |
| 17:18:39 | cdent | I honestly think that we need to make the system break when people break it | |
| 17:18:46 | cdent | not try to fix it for them | |
| 17:18:53 | aspiers | shit, actually that diagram is wrong isn't it :-/ | |
| 17:19:25 | cdent | we want failures to show as soon as possible | |
| 17:19:40 | aspiers | yeah I'm totally OK with the "if you do that you get to keep all the broken pieces" approach | |
| 17:19:40 | cdent | not be in a situation where every 5 minutes system a removes a trait and every 7th minute something puts it back | |
| 17:19:49 | aspiers | good point | |
| 17:20:35 | mriedem | fried_rice: yes, i look at this bug every couple of weeks and then fall into a pit of depression | |
| 17:20:41 | mriedem | because all solutions out of this bug are bad | |
| 17:21:13 | fried_rice | mriedem: Seems like if you're removing that piece of the comment, you should add a TODO to fix that bit. Assume there's a bug associated with it, yah? | |
| 17:21:28 | mriedem | fried_rice: see the related bug at the bottom of the commit message | |
| 17:22:06 | mriedem | i thought about adding something like, NOTE(mriedem): This actually causes bug 1790204. - but the fix for that bug might not actually involve changing that code | |
| 17:22:07 | openstack | bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] https://launchpad.net/bugs/1790204 | |
| 17:22:22 | mriedem | i can add it if it's helpful though | |
| 17:22:48 | mriedem | also, having said that, i don't think we hit that code on resize anymore | |
| 17:23:06 | mriedem | see, "(10:47:20 AM) mriedem: i imagine this is all dead code since queens or at least rocky https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L140" | |
| 17:23:39 | mriedem | that's only called if the instance consumer has allocations prior to scheduling, which it shouldn't for resize https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1595 | |
| 17:23:47 | mriedem | because we swapped those to the migration record in conductor prior to calling the scheduler | |
| 17:24:06 | mriedem | i think _move_operation_alloc_request only happens during evacuate now | |
| 17:24:21 | mriedem | and you thankfully can't evacuate to the same host :) | |
| 17:24:50 | sean-k-mooney | we will proably get a bug request for that at some point | |
| 17:24:59 | sean-k-mooney | but its true | |
| 17:25:06 | sean-k-mooney | and it should never be possible | |
| 17:25:13 | fried_rice | can we prove it and just remove the code path? | |
| 17:25:15 | mriedem | you can't evacuate to a host that is down | |
| 17:25:31 | mriedem | and you can't evacuate while the service is up | |
| 17:25:50 | mriedem | fried_rice: we can't remove it because of evacuate | |
| 17:26:03 | fried_rice | not the method, just that condition, right? | |
| 17:26:09 | fried_rice | maybe I misunderstood what you said. | |
| 17:27:12 | mriedem | current design ideas on the resize to same host bug is in https://bugs.launchpad.net/nova/+bug/1790204/comments/14 | |
| 17:27:13 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 17:27:15 | fried_rice | I thought you said: you can only hit this method during evacuate, and you can't evacuate to same host, so `elif not new_rp_uuids` should never happen | |
| 17:27:18 | mriedem | which is built on an earlier comment from leakypipes | |
| 17:27:50 | mriedem | oh yeah this https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L172 | |
| 17:27:57 | mriedem | umm, i think that's probably true | |
| 17:28:10 | mriedem | so rather than me remove that comment, just remove the code | |
| 17:28:24 | fried_rice | meaning L169-179 can be reduced to L170-171 | |
| 17:28:43 | fried_rice | but only if we can prove that we only call this method for evacuate. | |