| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-03 | |||
| 16:16:58 | bauzas | mriedem: in case something goes wrong | |
| 16:17:03 | mriedem | what does that mean? | |
| 16:17:06 | jaypipes | mriedem: if the compute sees the allocations in the request spec, it wouldn't need to do the claims.Claim() build. | |
| 16:17:06 | bauzas | mriedem: the compute would DELETE those allocations | |
| 16:17:35 | bauzas | and it would trigger the reschedule by passing back the Spec | |
| 16:17:54 | mriedem | i thought the scheduler was going to delete the allocations if something failed on a given compute host? | |
| 16:18:03 | bauzas | jaypipes: yeah, the Spec object would signal if we support new-style of claiming | |
| 16:18:24 | bauzas | mriedem: the scheduler would POST allocations | |
| 16:18:30 | bauzas | mriedem: and get the UUIDs | |
| 16:18:30 | jaypipes | bauzas: yeah, the presence of allocation objects in the RequestSpec would be that signal. | |
| 16:18:35 | mriedem | jaypipes: so we bump the service version when computes start checking for claims in the scheduler (via the allocations in the request spec?) and then use that service version to determine when the scheduler can start doing this? once all computes are upgraded or what? | |
| 16:19:07 | bauzas | mriedem: I was considering that approach | |
| 16:19:21 | bauzas | mriedem: because we suppose a fully-upgraded cloud | |
| 16:19:24 | jaypipes | mriedem: ya | |
| 16:19:32 | mriedem | i'm not sure i like the compute deleting the allocations | |
| 16:19:36 | mriedem | on a retry | |
| 16:19:49 | dansmith | definitely not | |
| 16:19:55 | mriedem | wouldn't it be better to have the allocation create and cleanup in the same service? | |
| 16:20:01 | bauzas | I dunno | |
| 16:20:02 | mriedem | if we want to just pass a flag down to the compute, sure | |
| 16:20:10 | bauzas | I'm not opiniated | |
| 16:20:24 | bauzas | so | |
| 16:20:24 | dansmith | compute shouldn't be doing that once we're claiming in the scheduler, methinks | |
| 16:20:30 | mriedem | dansmith: agree | |
| 16:20:31 | bauzas | if we say no to that idea | |
| 16:20:39 | bauzas | then, we would just send a signal | |
| 16:20:42 | mriedem | at a high level, we want the compute out of the business of doing this stuff | |
| 16:20:52 | mriedem | yeah send a flag in the reqspec object i guess? | |
| 16:20:52 | bauzas | but the compute would pass down the Spec by signaling against the signal :) | |
| 16:21:04 | bauzas | I mean | |
| 16:21:07 | mriedem | do we pass the reqspec today to the compute | |
| 16:21:08 | mriedem | ? | |
| 16:21:17 | bauzas | mriedem: not yet, but I was planning to | |
| 16:21:22 | mriedem | oy | |
| 16:21:26 | bauzas | it's a specless bp | |
| 16:21:30 | mriedem | why not just pass a simple boolean on the rpc method? | |
| 16:21:35 | dansmith | mriedem: it's leftovers from that whole process still :/ | |
| 16:21:38 | mriedem | do_claims=False | |
| 16:21:54 | dansmith | mriedem: +1.. I think encoding the state in the reqspec is kinda bad | |
| 16:22:00 | bauzas | mriedem: I'd rather prefer to pass a Spec object because we could need it for rescheduling needs | |
| 16:22:01 | dansmith | it's not guaranteed to be right | |
| 16:22:11 | mriedem | bauzas: what rescheduling needs? | |
| 16:22:17 | mriedem | conductor / scheduler handles that | |
| 16:22:24 | bauzas | mriedem: the Retry nested object at least | |
| 16:22:42 | mriedem | in filter_properties? | |
| 16:22:48 | dansmith | bauzas: a call from compute to scheduler will be an upcall, you know | |
| 16:22:58 | bauzas | mriedem: at the moment, we have ugly object-to-dict and dict-to-objects handlers | |
| 16:23:09 | mriedem | let's step back, are you talking about build_and_run_instance? | |
| 16:23:15 | bauzas | dansmith: compute does upcall conductor when rescheduling | |
| 16:23:26 | bauzas | and conductor calls scheduler | |
| 16:23:32 | mriedem | because build_and_run_instance already gets the reqspec | |
| 16:23:48 | bauzas | so when talking compute > scheduler, I'm just summarizing rather compute > conductor > scheduler | |
| 16:24:14 | mriedem | the reqspec is already passed to the compute, but not used at all | |
| 16:24:22 | mriedem | it's a dead parameter in _do_build_and_run_instance in the compute manager | |
| 16:24:26 | dansmith | bauzas: if we're claiming in the scheduler, do we ever reschedule once it's made it to the compute? | |
| 16:24:27 | bauzas | it's passed as a dict | |
| 16:24:35 | bauzas | dansmith: we could | |
| 16:24:48 | bauzas | dansmith: we still need some way to assume something went bad | |
| 16:25:01 | mriedem | yeah we can fail for non-resource related reasons | |
| 16:25:05 | bauzas | libvirt could fail for some foolish BDM things | |
| 16:25:14 | bauzas | or some perms issue | |
| 16:25:18 | mriedem | vif plug can fail, etc | |
| 16:25:23 | bauzas | zactly | |
| 16:25:26 | dansmith | bauzas: so, there's literally no mechanism for an upcall from that layer | |
| 16:26:01 | mriedem | compute does an rpc cast to build_instances in conductor for the reschedule right? | |
| 16:26:11 | bauzas | dansmith: remind me at the moment, the super-conductor calls what ? local conductor or compute ? | |
| 16:26:18 | dansmith | mriedem: yeah, but it's cell conductor | |
| 16:26:37 | bauzas | dansmith: I think we agreed long time ago that a reschedule would not be multi-cell | |
| 16:26:46 | bauzas | at least for initial versions | |
| 16:26:53 | dansmith | bauzas: sure, but cell conductor can't talk to scheduler | |
| 16:26:59 | bauzas | oh snap indeed | |
| 16:27:00 | dansmith | because they're on different MQs | |
| 16:27:04 | bauzas | yeah | |
| 16:27:06 | bauzas | f*** | |
| 16:27:10 | bauzas | I forgot that | |
| 16:27:17 | bauzas | so, we're stuck | |
| 16:27:30 | bauzas | because we *need* to schedule again if we failed | |
| 16:27:58 | bauzas | dansmith: have you tested reschedules after all your cells-aware series ? | |
| 16:28:16 | dansmith | bauzas: not sure how we'd do that reliably from tempest | |
| 16:28:26 | bauzas | fair point | |
| 16:28:47 | bauzas | but anyway, even if not covered, the issue still remains | |
| 16:28:56 | dansmith | I guess I thought part of the point of the claiming in the scheduler was to get rid of the need to retry | |
| 16:29:16 | bauzas | dansmith: it would reduce races, and gives us better decisions | |
| 16:29:19 | mriedem | i had assumed we could still retry for other failures | |
| 16:29:21 | mriedem | specific to the ocmpute | |
| 16:29:22 | dansmith | if we fail to setup BDMs or ports, I would kinda expect that to affect all computes in a similar way, except maybe for misconfiguration | |
| 16:29:23 | mriedem | *compute | |
| 16:29:34 | bauzas | dansmith: but like mriedem said, compute-related failures can still exist | |
| 16:29:44 | bauzas | dansmith: I know | |
| 16:29:44 | dansmith | sure, but what kinds of things would not affect all computes? | |
| 16:29:53 | dansmith | talking to glance or neutron or cinder would likely affect them all | |
| 16:29:56 | bauzas | dansmith: but atm, we allow misconfigurations | |
| 16:30:06 | dansmith | we allow misconfigurations? :) | |
| 16:30:11 | bauzas | it just triggers reschedules | |
| 16:30:13 | dansmith | you mean we retry in those cases, yes | |
| 16:30:15 | mriedem | "tolerate" | |
| 16:30:22 | bauzas | we /tolerate/ if you prefer | |
| 16:30:39 | bauzas | but I'm pretty sure it would be drama if we stop providing reschedules for compute failures | |
| 16:30:40 | mriedem | if cinder/glance/neutron are goofed up, then yeah those are probably global failures | |
| 16:31:04 | bauzas | mriedem: well, I wouldn't be that strong | |