| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-15 | |||
| 15:05:49 | mriedem | "Custom resource classes and standard resource classes to override" | |
| 15:05:56 | mriedem | yeah | |
| 15:05:57 | mriedem | resources:CUSTOM_BAREMETAL_SMALL=1 resources:VCPU=0 | |
| 15:06:03 | edleafe | That was how it was supposed to be done. Keep the values in the flavor, and zero out in extra_specs | |
| 15:06:40 | mriedem | ok i don't know why that standard resource class 0 override wasn't done at the same time as the embedded instance.flavor migration once the node was reporting a custom resource class | |
| 15:07:16 | mriedem | plus that's what the ironic docs tell operators to do https://docs.openstack.org/ironic/latest/install/configure-nova-flavors | |
| 15:07:41 | edleafe | mriedem: https://docs.openstack.org/ironic/latest/install/configure-nova-flavors describes it pretty well | |
| 15:07:55 | edleafe | jinx | |
| 15:08:45 | dansmith | mriedem: yeah, for display and to avoid triggering any of our other magic behaviors with a value is zero, like root size | |
| 15:09:21 | mriedem | sure but we can override via the extra spec | |
| 15:09:24 | mriedem | that seems to be the missing piece | |
| 15:09:43 | mriedem | now whether or not the code that's reporting allocations actually looks at that override... | |
| 15:09:50 | dansmith | right, the point being.. leave the real values for display, override what we actually ask for in the resource overrides | |
| 15:11:23 | mriedem | ok i think this is where the override happens https://github.com/openstack/nova/blob/stable/rocky/nova/scheduler/utils.py#L366 | |
| 15:22:06 | mriedem | belmoreira: ok i'm working on a patch which you can take a look at - knowing this isn't all totally fubar will require a functional test though given how tightly coupled all of this is between the virt driver, resource tracker and scheduler | |
| 15:37:07 | belmoreira | mriedem: thanks | |
| 15:38:03 | mriedem | the commit message on this is going to be fun | |
| 15:38:37 | mriedem | "first belmiro lost a finger here, and then he blew off a toe here" | |
| 15:39:52 | belmoreira | as you may noticed I upgraded nova-compute for ironic only today :) as been an interesting day | |
| 15:40:50 | belmoreira | the next issue is: https://bugs.launchpad.net/nova/+bug/1816086 | |
| 15:40:51 | openstack | Launchpad bug 1816086 in OpenStack Compute (nova) "Resource Tracker performance with Ironic driver" [Undecided,New] | |
| 15:41:01 | fried_rice | uh oh | |
| 15:41:50 | fried_rice | I thought I fixed that. belmoreira are you using latest master? | |
| 15:42:09 | mriedem | he's upgrading ironic queens to rocky i assume | |
| 15:42:11 | mriedem | for compute services | |
| 15:42:22 | mriedem | so they would have had to backport your patches | |
| 15:42:22 | fried_rice | oh, that'd do it | |
| 15:42:25 | fried_rice | yeah | |
| 15:42:55 | belmoreira | I'm using rocky (back ported the patches that we discussed in the past) | |
| 15:43:23 | melwitt | ||
| 15:45:13 | belmoreira | fried_rice: so what I'm saying in the bug doesn't make sense anymore? | |
| 15:45:32 | fried_rice | belmoreira: Backported which patches, though? | |
| 15:45:51 | belmoreira | we couldn't cherrypick the patches in a clean way, we may have missed something | |
| 15:45:53 | fried_rice | belmoreira: Latest master has changes that made sure to only update the ironic node being touched. | |
| 15:46:05 | fried_rice | let me find the patch that did that... | |
| 15:47:20 | fried_rice | belmoreira: https://review.openstack.org/#/c/615677/ | |
| 15:49:01 | fried_rice | belmoreira: I think if you picked that up prior to about PS17, you would have the ironic explosion behavior you're seeing. But after that it *should* be fixed. | |
| 15:49:06 | jaypipes | gibi: hmm, couldn't my algorithm work if we just sorted the request groups in descending order of amount of resources being requested in the group? | |
| 15:49:24 | jaypipes | gibi: then the "greedy" approach would work fine, no? | |
| 15:49:40 | fried_rice | belmoreira: and I know you guys were picking up that series at various points along the way | |
| 15:49:44 | gibi | jaypipes: if only one resource class would be used then yes, but if you have two resource classes requested then which one would sorted first? | |
| 15:50:23 | fried_rice | jaypipes: I didn't feel strongly enough to say so in the review, but fwiw I liked gibi's algorithm as is. | |
| 15:50:24 | jaypipes | gibi: do we have tests that show the behaviour or your algorithm is correct for such situations? | |
| 15:50:24 | belmoreira | fried_rice: thanks, I'm checking | |
| 15:50:35 | tssurya | fried_rice: porting was done after that patch got merged | |
| 15:50:41 | jaypipes | fried_rice: it's difficult to read and understand, IMHO. | |
| 15:50:44 | fried_rice | tssurya: roger that, thanks. | |
| 15:51:03 | fried_rice | jaypipes: I don't disagree, but I'm not sure yours is more readable/understandable. (It is to you because you formulated it.) | |
| 15:51:15 | jaypipes | fried_rice: as is most things around multiple use_single_provider request groups, FWIW | |
| 15:51:22 | gibi | jaypipes: it worth to step through https://review.openstack.org/#/c/616239/26/nova/tests/unit/objects/test_request_spec.py@1305 | |
| 15:52:05 | fried_rice | tssurya: How far up the series did you port? | |
| 15:52:41 | gibi | jaypipes: there group1 could be fit first to 3 RPs but finally only the RP #3 leads to solution | |
| 15:54:18 | jaypipes | fried_rice, gibi: this all just leaves an odd taste in my mouth. | |
| 15:54:44 | jaypipes | fried_rice, gibi: a taste that says "oh, we're totally leaking the placement selection details out of the placement API" | |
| 15:54:51 | gibi | jaypipes: totally agree | |
| 15:54:53 | jaypipes | and yes, I understand this is a workaround | |
| 15:54:57 | fried_rice | jaypipes: Yeah, it sucks, but is hopefully temporary, to be whacked when placement lets ... just so | |
| 15:55:20 | jaypipes | until such point that the result of allocation_candidates contains an indicator of which request group "belongs" to which allocation provider | |
| 15:55:34 | gibi | jaypipes: yes | |
| 15:55:36 | fried_rice | So let's get that spec pushed through so we can implement it early in Train and get rid of this spaghetti. | |
| 15:55:44 | jaypipes | ack, ok | |
| 15:56:06 | gibi | fried_rice: unfortunately I cannot focus both on that spec and the bandwidth series at the same time (I'm limited) | |
| 15:56:14 | gibi | fried_rice: but after feature freeze I can update the spec | |
| 15:56:15 | fried_rice | gibi: Would you like me to take over that spec? | |
| 15:56:34 | jaypipes | gibi, fried_rice: question for you... are different request groups containing same resource class requests guaranteed to land on different resource providers? | |
| 15:56:40 | gibi | fried_rice: if it is OK to push it after the feature freeze then I'd like to keep the spec | |
| 15:56:46 | jaypipes | (I can never remember...) | |
| 15:56:49 | fried_rice | jaypipes: if group_policy=none, no | |
| 15:57:02 | jaypipes | ah, yes, group_policy=isolate.. | |
| 15:57:28 | jaypipes | gibi, fried_rice: is group_policy=isolate taken account of in this mapping patch? | |
| 15:57:29 | fried_rice | gibi: okay. Yes, that should be fine. Just lmk if you want help (other than review) | |
| 15:57:40 | fried_rice | jaypipes: oo, good question. | |
| 15:57:44 | jaypipes | or does it even need to be? /shurg | |
| 15:57:46 | gibi | fried_rice: thanks, I will need you review! :) | |
| 15:58:06 | fried_rice | jaypipes: I think it might need to be, yes. | |
| 15:58:07 | gibi | jaypipes, fried_rice: bahh, good question | |
| 15:58:13 | belmoreira | fried_rice: what is taking time is the last for in the update_from_provider_tree | |
| 15:58:47 | fried_rice | as is, gibi's algorithm takes the first of many possible matchups; but it assumes group_policy=none | |
| 15:58:58 | jaypipes | yeah | |
| 15:59:01 | belmoreira | https://github.com/openstack/nova/blob/d231a420d0d4865bb19da513af416cb8bc89010f/nova/scheduler/client/report.py#L1439 | |
| 15:59:03 | gibi | fried_rice, jaypipes: if the request was made with isolate but the algo founds an overlapping solution then we are screwed | |
| 15:59:17 | gibi | fried_rice: correct | |
| 15:59:28 | gibi | fried_rice: so I have to enhance this algo | |
| 15:59:32 | gibi | fried_rice: to handle isolate | |
| 15:59:48 | gibi | fried_rice: and in that case throw away overlapping solutions | |
| 16:00:04 | fried_rice | gibi: correct | |
| 16:00:08 | jaypipes | gibi, fried_rice: well, maybe... maybe not. :) after all, we *KNOW* that the returned allocations properly meet the group_policy=isolate constraint because the placement service is shown to be correct in that. | |
| 16:00:58 | fried_rice | mm, nooo, I still think there's a hole | |
| 16:01:05 | jaypipes | so we can assume that the provider that is assigned a particular amount of allocated resources does not have any "doubled up" allocations from multiple request groups, right? | |
| 16:01:47 | gibi | in case of isolate, yes | |
| 16:01:50 | fried_rice | belmoreira: noted | |
| 16:01:56 | jaypipes | this is what's "eww" about this... :) it's essentially needing to re-implement the entire request group selection algorithm that's in the placement service already. :) | |
| 16:02:06 | fried_rice | belmoreira: because new_uuids == {every ironic node} | |
| 16:02:12 | fried_rice | jaypipes: Exactly. | |
| 16:02:19 | gibi | jaypipes: you are totatlly right | |
| 16:02:21 | belmoreira | fried_rice: correct | |
| 16:02:23 | fried_rice | jaypipes: and we haven't even done forbidden traits and aggregates yet | |
| 16:02:40 | gibi | fried_rice: and I hope we never have to in this algo | |
| 16:03:19 | fried_rice | belmoreira: set_inventory_for_provider et al should be shorting out, not talking to the placement API, for any but the changed node. Are you seeing those hit placement? | |
| 16:03:55 | gibi | so I need to found an example where there is two possible valid mapping one with overlap between groups and one without. If I can make such an example then I need to change the algo | |