| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-03 | |||
| 16:22:42 | mriedem | in filter_properties? | |
| 16:22:48 | dansmith | bauzas: a call from compute to scheduler will be an upcall, you know | |
| 16:22:58 | bauzas | mriedem: at the moment, we have ugly object-to-dict and dict-to-objects handlers | |
| 16:23:09 | mriedem | let's step back, are you talking about build_and_run_instance? | |
| 16:23:15 | bauzas | dansmith: compute does upcall conductor when rescheduling | |
| 16:23:26 | bauzas | and conductor calls scheduler | |
| 16:23:32 | mriedem | because build_and_run_instance already gets the reqspec | |
| 16:23:48 | bauzas | so when talking compute > scheduler, I'm just summarizing rather compute > conductor > scheduler | |
| 16:24:14 | mriedem | the reqspec is already passed to the compute, but not used at all | |
| 16:24:22 | mriedem | it's a dead parameter in _do_build_and_run_instance in the compute manager | |
| 16:24:26 | dansmith | bauzas: if we're claiming in the scheduler, do we ever reschedule once it's made it to the compute? | |
| 16:24:27 | bauzas | it's passed as a dict | |
| 16:24:35 | bauzas | dansmith: we could | |
| 16:24:48 | bauzas | dansmith: we still need some way to assume something went bad | |
| 16:25:01 | mriedem | yeah we can fail for non-resource related reasons | |
| 16:25:05 | bauzas | libvirt could fail for some foolish BDM things | |
| 16:25:14 | bauzas | or some perms issue | |
| 16:25:18 | mriedem | vif plug can fail, etc | |
| 16:25:23 | bauzas | zactly | |
| 16:25:26 | dansmith | bauzas: so, there's literally no mechanism for an upcall from that layer | |
| 16:26:01 | mriedem | compute does an rpc cast to build_instances in conductor for the reschedule right? | |
| 16:26:11 | bauzas | dansmith: remind me at the moment, the super-conductor calls what ? local conductor or compute ? | |
| 16:26:18 | dansmith | mriedem: yeah, but it's cell conductor | |
| 16:26:37 | bauzas | dansmith: I think we agreed long time ago that a reschedule would not be multi-cell | |
| 16:26:46 | bauzas | at least for initial versions | |
| 16:26:53 | dansmith | bauzas: sure, but cell conductor can't talk to scheduler | |
| 16:26:59 | bauzas | oh snap indeed | |
| 16:27:00 | dansmith | because they're on different MQs | |
| 16:27:04 | bauzas | yeah | |
| 16:27:06 | bauzas | f*** | |
| 16:27:10 | bauzas | I forgot that | |
| 16:27:17 | bauzas | so, we're stuck | |
| 16:27:30 | bauzas | because we *need* to schedule again if we failed | |
| 16:27:58 | bauzas | dansmith: have you tested reschedules after all your cells-aware series ? | |
| 16:28:16 | dansmith | bauzas: not sure how we'd do that reliably from tempest | |
| 16:28:26 | bauzas | fair point | |
| 16:28:47 | bauzas | but anyway, even if not covered, the issue still remains | |
| 16:28:56 | dansmith | I guess I thought part of the point of the claiming in the scheduler was to get rid of the need to retry | |
| 16:29:16 | bauzas | dansmith: it would reduce races, and gives us better decisions | |
| 16:29:19 | mriedem | i had assumed we could still retry for other failures | |
| 16:29:21 | mriedem | specific to the ocmpute | |
| 16:29:22 | dansmith | if we fail to setup BDMs or ports, I would kinda expect that to affect all computes in a similar way, except maybe for misconfiguration | |
| 16:29:23 | mriedem | *compute | |
| 16:29:34 | bauzas | dansmith: but like mriedem said, compute-related failures can still exist | |
| 16:29:44 | dansmith | sure, but what kinds of things would not affect all computes? | |
| 16:29:44 | bauzas | dansmith: I know | |
| 16:29:53 | dansmith | talking to glance or neutron or cinder would likely affect them all | |
| 16:29:56 | bauzas | dansmith: but atm, we allow misconfigurations | |
| 16:30:06 | dansmith | we allow misconfigurations? :) | |
| 16:30:11 | bauzas | it just triggers reschedules | |
| 16:30:13 | dansmith | you mean we retry in those cases, yes | |
| 16:30:15 | mriedem | "tolerate" | |
| 16:30:22 | bauzas | we /tolerate/ if you prefer | |
| 16:30:39 | bauzas | but I'm pretty sure it would be drama if we stop providing reschedules for compute failures | |
| 16:30:40 | mriedem | if cinder/glance/neutron are goofed up, then yeah those are probably global failures | |
| 16:31:04 | bauzas | mriedem: well, I wouldn't be that strong | |
| 16:31:07 | mriedem | i'm trying to think of things that we fail on a compute and don't know about before hand | |
| 16:31:23 | mriedem | i think file injection was one of them | |
| 16:31:32 | bauzas | a vif plug event is the most known failure | |
| 16:31:43 | dansmith | bauzas: for what actual reason? | |
| 16:32:09 | bauzas | honestly ? I can google search vif_binding failed and find a ton of different stacktraces | |
| 16:32:23 | mriedem | do we reschedule on all of those? | |
| 16:32:30 | bauzas | yup | |
| 16:32:34 | mriedem | we don't reschedule on boot from volume failure, for example | |
| 16:32:38 | jaypipes | mriedem: no, we ERROR the instance, IIRC. | |
| 16:32:38 | dansmith | I'm seriously asking.. what kind of those failures aren't global? | |
| 16:32:42 | bauzas | *all* is maybe presumptuous | |
| 16:32:59 | bauzas | we *reschedule* on exceptions that are raised by the driver | |
| 16:33:01 | mriedem | i'm thinking of the vif plug timeout case | |
| 16:33:05 | mriedem | bauzas: not for all | |
| 16:33:05 | dansmith | I don't really understand why it's not okay for us to ERROR a build if we're far into the process and fail | |
| 16:33:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: doc: Separate the releasenotes guide from the code-review section https://review.openstack.org/451870 | |
| 16:33:34 | bauzas | mriedem: well, I'm mostly sure it's a catch-all exception handling | |
| 16:33:37 | jaypipes | bauzas: only in the instance_claim() context manager. once the claim is completed (and the "allocations" written to the compute_nodes table), we don't reschedule. | |
| 16:33:40 | dansmith | because if it is global, we bounce to three compute nodes, generate a bunch of glance download traffic, and then ultimately fail, having wasted a bunch of time and resources | |
| 16:33:55 | mriedem | bauzas: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1971 | |
| 16:33:55 | dansmith | jaypipes: orly, well, that helps | |
| 16:33:56 | bauzas | sec, the code will tell us the truth | |
| 16:34:00 | mriedem | raise exception.BuildAbortException | |
| 16:34:10 | dansmith | because if we claim in the scheduler, then after that we're committed to ACTIVE or ERROR | |
| 16:34:17 | jaypipes | right. | |
| 16:34:27 | bauzas | mriedem: jaypipes: nope, nope, general exception https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003-L2010 | |
| 16:34:58 | mriedem | bauzas: but you go through the specific errors before that, | |
| 16:34:59 | mriedem | like VirtualInterfaceCreateException | |
| 16:35:00 | dansmith | bauzas: that's only for we-haveno-idea | |
| 16:35:03 | mriedem | which is translated to an abort | |
| 16:35:12 | mriedem | VirtualInterfaceCreateException is raised for vif plug timeout | |
| 16:35:14 | mriedem | or error | |
| 16:35:40 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5090-L5106 | |
| 16:35:41 | bauzas | dansmith: correct me if I'm wrong, but for other cases, we raise exceptions and enter that section ? | |
| 16:35:44 | mriedem | anyway, that's one class of error | |
| 16:35:51 | mriedem | i know we don't reschedule on boot from volume failures | |
| 16:36:08 | dansmith | bauzas: I dunno | |
| 16:36:09 | mriedem | bauzas: we get into that general block for shit we aren't expecting | |
| 16:36:30 | mriedem | like i bet ProcessExecutionError gets into the generic retry block | |
| 16:36:42 | dansmith | which might be misconfiguration things, but more likely it's crazy shit failing that will affect the reschedule too I bet | |
| 16:36:55 | jaypipes | well, bauzas was correct that I was wrong about the instance_claim() not wrapping the entire spawn() operation. | |
| 16:37:56 | mriedem | yeah i would think most of these are localized misconfiguration, or global | |
| 16:38:12 | mriedem | like the one about not reserving enough resources on a given compute for ovs to run | |
| 16:38:16 | mriedem | remember that one? | |