| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-03 | |||
| 16:26:46 | bauzas | at least for initial versions | |
| 16:26:53 | dansmith | bauzas: sure, but cell conductor can't talk to scheduler | |
| 16:26:59 | bauzas | oh snap indeed | |
| 16:27:00 | dansmith | because they're on different MQs | |
| 16:27:04 | bauzas | yeah | |
| 16:27:06 | bauzas | f*** | |
| 16:27:10 | bauzas | I forgot that | |
| 16:27:17 | bauzas | so, we're stuck | |
| 16:27:30 | bauzas | because we *need* to schedule again if we failed | |
| 16:27:58 | bauzas | dansmith: have you tested reschedules after all your cells-aware series ? | |
| 16:28:16 | dansmith | bauzas: not sure how we'd do that reliably from tempest | |
| 16:28:26 | bauzas | fair point | |
| 16:28:47 | bauzas | but anyway, even if not covered, the issue still remains | |
| 16:28:56 | dansmith | I guess I thought part of the point of the claiming in the scheduler was to get rid of the need to retry | |
| 16:29:16 | bauzas | dansmith: it would reduce races, and gives us better decisions | |
| 16:29:19 | mriedem | i had assumed we could still retry for other failures | |
| 16:29:21 | mriedem | specific to the ocmpute | |
| 16:29:22 | dansmith | if we fail to setup BDMs or ports, I would kinda expect that to affect all computes in a similar way, except maybe for misconfiguration | |
| 16:29:23 | mriedem | *compute | |
| 16:29:34 | bauzas | dansmith: but like mriedem said, compute-related failures can still exist | |
| 16:29:44 | bauzas | dansmith: I know | |
| 16:29:44 | dansmith | sure, but what kinds of things would not affect all computes? | |
| 16:29:53 | dansmith | talking to glance or neutron or cinder would likely affect them all | |
| 16:29:56 | bauzas | dansmith: but atm, we allow misconfigurations | |
| 16:30:06 | dansmith | we allow misconfigurations? :) | |
| 16:30:11 | bauzas | it just triggers reschedules | |
| 16:30:13 | dansmith | you mean we retry in those cases, yes | |
| 16:30:15 | mriedem | "tolerate" | |
| 16:30:22 | bauzas | we /tolerate/ if you prefer | |
| 16:30:39 | bauzas | but I'm pretty sure it would be drama if we stop providing reschedules for compute failures | |
| 16:30:40 | mriedem | if cinder/glance/neutron are goofed up, then yeah those are probably global failures | |
| 16:31:04 | bauzas | mriedem: well, I wouldn't be that strong | |
| 16:31:07 | mriedem | i'm trying to think of things that we fail on a compute and don't know about before hand | |
| 16:31:23 | mriedem | i think file injection was one of them | |
| 16:31:32 | bauzas | a vif plug event is the most known failure | |
| 16:31:43 | dansmith | bauzas: for what actual reason? | |
| 16:32:09 | bauzas | honestly ? I can google search vif_binding failed and find a ton of different stacktraces | |
| 16:32:23 | mriedem | do we reschedule on all of those? | |
| 16:32:30 | bauzas | yup | |
| 16:32:34 | mriedem | we don't reschedule on boot from volume failure, for example | |
| 16:32:38 | dansmith | I'm seriously asking.. what kind of those failures aren't global? | |
| 16:32:38 | jaypipes | mriedem: no, we ERROR the instance, IIRC. | |
| 16:32:42 | bauzas | *all* is maybe presumptuous | |
| 16:32:59 | bauzas | we *reschedule* on exceptions that are raised by the driver | |
| 16:33:01 | mriedem | i'm thinking of the vif plug timeout case | |
| 16:33:05 | dansmith | I don't really understand why it's not okay for us to ERROR a build if we're far into the process and fail | |
| 16:33:05 | mriedem | bauzas: not for all | |
| 16:33:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: doc: Separate the releasenotes guide from the code-review section https://review.openstack.org/451870 | |
| 16:33:34 | bauzas | mriedem: well, I'm mostly sure it's a catch-all exception handling | |
| 16:33:37 | jaypipes | bauzas: only in the instance_claim() context manager. once the claim is completed (and the "allocations" written to the compute_nodes table), we don't reschedule. | |
| 16:33:40 | dansmith | because if it is global, we bounce to three compute nodes, generate a bunch of glance download traffic, and then ultimately fail, having wasted a bunch of time and resources | |
| 16:33:55 | dansmith | jaypipes: orly, well, that helps | |
| 16:33:55 | mriedem | bauzas: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1971 | |
| 16:33:56 | bauzas | sec, the code will tell us the truth | |
| 16:34:00 | mriedem | raise exception.BuildAbortException | |
| 16:34:10 | dansmith | because if we claim in the scheduler, then after that we're committed to ACTIVE or ERROR | |
| 16:34:17 | jaypipes | right. | |
| 16:34:27 | bauzas | mriedem: jaypipes: nope, nope, general exception https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003-L2010 | |
| 16:34:58 | mriedem | bauzas: but you go through the specific errors before that, | |
| 16:34:59 | mriedem | like VirtualInterfaceCreateException | |
| 16:35:00 | dansmith | bauzas: that's only for we-haveno-idea | |
| 16:35:03 | mriedem | which is translated to an abort | |
| 16:35:12 | mriedem | VirtualInterfaceCreateException is raised for vif plug timeout | |
| 16:35:14 | mriedem | or error | |
| 16:35:40 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5090-L5106 | |
| 16:35:41 | bauzas | dansmith: correct me if I'm wrong, but for other cases, we raise exceptions and enter that section ? | |
| 16:35:44 | mriedem | anyway, that's one class of error | |
| 16:35:51 | mriedem | i know we don't reschedule on boot from volume failures | |
| 16:36:08 | dansmith | bauzas: I dunno | |
| 16:36:09 | mriedem | bauzas: we get into that general block for shit we aren't expecting | |
| 16:36:30 | mriedem | like i bet ProcessExecutionError gets into the generic retry block | |
| 16:36:42 | dansmith | which might be misconfiguration things, but more likely it's crazy shit failing that will affect the reschedule too I bet | |
| 16:36:55 | jaypipes | well, bauzas was correct that I was wrong about the instance_claim() not wrapping the entire spawn() operation. | |
| 16:37:56 | mriedem | yeah i would think most of these are localized misconfiguration, or global | |
| 16:38:12 | mriedem | like the one about not reserving enough resources on a given compute for ovs to run | |
| 16:38:16 | mriedem | remember that one? | |
| 16:38:27 | dansmith | jaypipes: it wraps all of driver.spawn() or did you mean the compute manager level? | |
| 16:38:52 | mriedem | if you need to reserve space for a compute to run ovs, then i'm not sure why you wouldn't make the reservation on the inventory record for the compute resource provider | |
| 16:38:53 | jaypipes | dansmith: yeah for some reason I didn't think instance_claim() wrapped all of spawn(). | |
| 16:38:53 | mriedem | that's the point | |
| 16:38:58 | jaypipes | dansmith: I was wrong. | |
| 16:39:16 | bauzas | so, the flag is clear, we only reschedule if we raise RescheduledException | |
| 16:39:17 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1779 | |
| 16:39:32 | dansmith | jaypipes: I'm confused, what doesn't get wrapped? | |
| 16:39:44 | dansmith | jaypipes: we've started the instance in the driver before we return | |
| 16:40:07 | jaypipes | dansmith: right, I understand that now. I implied with my comment above that that was not the case, and I'm saying I was wrong. | |
| 16:40:16 | bauzas | dansmith: mriedem: so you could be right, some exceptions couldn't be transformed into that specific exception and we could just error out the instance | |
| 16:40:47 | bauzas | but that leaves the whole world of unexpected exceptions that *are* rescheduled | |
| 16:41:07 | bauzas | unexpected exceptions being the ones raised by the driver | |
| 16:42:10 | mriedem | so if we wanted to reschedule within the cell, we'd need two schedulers right? one top level scheduler across all cells, and one for the cell itself, which is basically redoing cells v1 | |
| 16:42:26 | alaski | It's been a while since I looked at the code, but I remember that generally we try to reschedule if there's a chance that the instance could start on another host. Typically this is for the case where there are not enough resources on the host for the instance, either due to a scheduler race or due to bad resource accounting on the compute | |
| 16:42:32 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | |
| 16:42:46 | dansmith | alaski: right | |
| 16:42:56 | mriedem | alaski: but we also just have the generic 'we have no idea' | |
| 16:42:57 | dansmith | alaski: I was thinking that's really the case we most care about today | |
| 16:43:32 | mriedem | this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003 | |
| 16:43:33 | alaski | mriedem: I don't think you need two schedulers. I think the scheduler shouldn't live at the API level. Think of it more like cinder, outside of cells fully | |
| 16:43:42 | alaski | dansmith: yep, agreed | |
| 16:44:01 | dansmith | alaski: um, scheduler shouldn't live at the api level? what does that mean? | |
| 16:44:05 | mriedem | if you have one scheduler, and you want to retry, then we'd have to upcall to that top level scheduler | |