| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-03 | |||
| 16:32:42 | bauzas | *all* is maybe presumptuous | |
| 16:32:59 | bauzas | we *reschedule* on exceptions that are raised by the driver | |
| 16:33:01 | mriedem | i'm thinking of the vif plug timeout case | |
| 16:33:05 | dansmith | I don't really understand why it's not okay for us to ERROR a build if we're far into the process and fail | |
| 16:33:05 | mriedem | bauzas: not for all | |
| 16:33:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: doc: Separate the releasenotes guide from the code-review section https://review.openstack.org/451870 | |
| 16:33:34 | bauzas | mriedem: well, I'm mostly sure it's a catch-all exception handling | |
| 16:33:37 | jaypipes | bauzas: only in the instance_claim() context manager. once the claim is completed (and the "allocations" written to the compute_nodes table), we don't reschedule. | |
| 16:33:40 | dansmith | because if it is global, we bounce to three compute nodes, generate a bunch of glance download traffic, and then ultimately fail, having wasted a bunch of time and resources | |
| 16:33:55 | dansmith | jaypipes: orly, well, that helps | |
| 16:33:55 | mriedem | bauzas: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1971 | |
| 16:33:56 | bauzas | sec, the code will tell us the truth | |
| 16:34:00 | mriedem | raise exception.BuildAbortException | |
| 16:34:10 | dansmith | because if we claim in the scheduler, then after that we're committed to ACTIVE or ERROR | |
| 16:34:17 | jaypipes | right. | |
| 16:34:27 | bauzas | mriedem: jaypipes: nope, nope, general exception https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003-L2010 | |
| 16:34:58 | mriedem | bauzas: but you go through the specific errors before that, | |
| 16:34:59 | mriedem | like VirtualInterfaceCreateException | |
| 16:35:00 | dansmith | bauzas: that's only for we-haveno-idea | |
| 16:35:03 | mriedem | which is translated to an abort | |
| 16:35:12 | mriedem | VirtualInterfaceCreateException is raised for vif plug timeout | |
| 16:35:14 | mriedem | or error | |
| 16:35:40 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5090-L5106 | |
| 16:35:41 | bauzas | dansmith: correct me if I'm wrong, but for other cases, we raise exceptions and enter that section ? | |
| 16:35:44 | mriedem | anyway, that's one class of error | |
| 16:35:51 | mriedem | i know we don't reschedule on boot from volume failures | |
| 16:36:08 | dansmith | bauzas: I dunno | |
| 16:36:09 | mriedem | bauzas: we get into that general block for shit we aren't expecting | |
| 16:36:30 | mriedem | like i bet ProcessExecutionError gets into the generic retry block | |
| 16:36:42 | dansmith | which might be misconfiguration things, but more likely it's crazy shit failing that will affect the reschedule too I bet | |
| 16:36:55 | jaypipes | well, bauzas was correct that I was wrong about the instance_claim() not wrapping the entire spawn() operation. | |
| 16:37:56 | mriedem | yeah i would think most of these are localized misconfiguration, or global | |
| 16:38:12 | mriedem | like the one about not reserving enough resources on a given compute for ovs to run | |
| 16:38:16 | mriedem | remember that one? | |
| 16:38:27 | dansmith | jaypipes: it wraps all of driver.spawn() or did you mean the compute manager level? | |
| 16:38:52 | mriedem | if you need to reserve space for a compute to run ovs, then i'm not sure why you wouldn't make the reservation on the inventory record for the compute resource provider | |
| 16:38:53 | jaypipes | dansmith: yeah for some reason I didn't think instance_claim() wrapped all of spawn(). | |
| 16:38:53 | mriedem | that's the point | |
| 16:38:58 | jaypipes | dansmith: I was wrong. | |
| 16:39:16 | bauzas | so, the flag is clear, we only reschedule if we raise RescheduledException | |
| 16:39:17 | bauzas | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1779 | |
| 16:39:32 | dansmith | jaypipes: I'm confused, what doesn't get wrapped? | |
| 16:39:44 | dansmith | jaypipes: we've started the instance in the driver before we return | |
| 16:40:07 | jaypipes | dansmith: right, I understand that now. I implied with my comment above that that was not the case, and I'm saying I was wrong. | |
| 16:40:16 | bauzas | dansmith: mriedem: so you could be right, some exceptions couldn't be transformed into that specific exception and we could just error out the instance | |
| 16:40:47 | bauzas | but that leaves the whole world of unexpected exceptions that *are* rescheduled | |
| 16:41:07 | bauzas | unexpected exceptions being the ones raised by the driver | |
| 16:42:10 | mriedem | so if we wanted to reschedule within the cell, we'd need two schedulers right? one top level scheduler across all cells, and one for the cell itself, which is basically redoing cells v1 | |
| 16:42:26 | alaski | It's been a while since I looked at the code, but I remember that generally we try to reschedule if there's a chance that the instance could start on another host. Typically this is for the case where there are not enough resources on the host for the instance, either due to a scheduler race or due to bad resource accounting on the compute | |
| 16:42:32 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | |
| 16:42:46 | dansmith | alaski: right | |
| 16:42:56 | mriedem | alaski: but we also just have the generic 'we have no idea' | |
| 16:42:57 | dansmith | alaski: I was thinking that's really the case we most care about today | |
| 16:43:32 | mriedem | this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003 | |
| 16:43:33 | alaski | mriedem: I don't think you need two schedulers. I think the scheduler shouldn't live at the API level. Think of it more like cinder, outside of cells fully | |
| 16:43:42 | alaski | dansmith: yep, agreed | |
| 16:44:01 | dansmith | alaski: um, scheduler shouldn't live at the api level? what does that mean? | |
| 16:44:05 | mriedem | if you have one scheduler, and you want to retry, then we'd have to upcall to that top level scheduler | |
| 16:44:36 | dansmith | mriedem: I definitely don't want to have multiple schedulers to handle the reschedule case | |
| 16:44:41 | alaski | dansmith: I mean start to think of the scheduler as outside of Nova. Just as cinder doesn't live at the API level I wouldn't lump the scheduler there either | |
| 16:45:17 | mriedem | API level == global to me | |
| 16:45:21 | alaski | otherwise you do have to have an upcall to reschedule, or have two levels | |
| 16:45:22 | dansmith | alaski: well, there's the architecture diagram and then there's the plumbing of how to talk to it :) | |
| 16:45:23 | mriedem | neutron, cinder, placement, keystone, etc | |
| 16:45:58 | dansmith | scheduler isn't a REST service which means we're a little constrained from just talking to it from wherever | |
| 16:46:12 | alaski | dansmith: yep. And this is where my knowledge has gaps, but I thought a scheduler API was being stood up, which would allow talking to it from anywhere | |
| 16:46:12 | dansmith | although, I will re-point out the scheduler-is-just-a-library helps with this :) | |
| 16:46:21 | dansmith | alaski: that's placement | |
| 16:46:30 | dansmith | alaski: and it's only part of the act of scheduling a thing | |
| 16:46:44 | alaski | gotcha | |
| 16:47:00 | bauzas | dansmith: I'm basically +1 on the idea of merging filtering/weighting in the conductor | |
| 16:47:12 | bauzas | so I think it's reasonable | |
| 16:47:28 | bauzas | but here, I'm more concerned by having the claims being implemented *before* that | |
| 16:47:31 | dansmith | it'd be super interesting to use scheduler as a library for reschedules | |
| 16:47:45 | bauzas | we sorta do that already | |
| 16:47:54 | dansmith | bauzas: well, then I think you have to nuke the reschedules if you're going to do it first | |
| 16:47:59 | bauzas | we have a scheduler.utils module that computes use for that purpose | |
| 16:48:06 | dansmith | bauzas: temporarily, if you want, to be reopened later | |
| 16:48:21 | bauzas | I dunno, and my wife is yelling at me because of dinner time :(' | |
| 16:48:22 | dansmith | hey I have a crazy idea | |
| 16:48:28 | dansmith | how about we ask an operator? | |
| 16:48:32 | mriedem | i was going to say, | |
| 16:48:43 | bauzas | can someone could just press the "pause" button for real life things? | |
| 16:48:45 | mriedem | we can agree we need to pass a flag down to the compute to tell it to not claim because we already did it at the top | |
| 16:49:00 | mriedem | for the reschedule thing, we have a session proposed for claims in the scheduler already, | |
| 16:49:02 | bauzas | that, I'm all good with | |
| 16:49:13 | mriedem | i think we could hash out the reschedule thing at that session since operators are supposed to be there | |
| 16:49:20 | bauzas | good lord | |
| 16:49:22 | dansmith | mriedem: sounds good to me | |
| 16:49:45 | bauzas | okay, let me just write some stuff and put that in the spec as a big PITA | |
| 16:49:49 | mriedem | that's assuming (1) that session is picked and (2) anyone shows up | |
| 16:50:14 | bauzas | yeah | |
| 16:50:14 | dansmith | mriedem: well, if it's not picked we can do something else | |
| 16:50:26 | dansmith | mriedem: if it is and they don't show up, then, I dunno, I guess they approve of whatever outcome :P | |
| 16:50:30 | bauzas | open a thread and leave people arguing forever \o/ | |
| 16:50:57 | bauzas | either way, I really need to drop off for family business :( | |
| 16:50:59 | mriedem | dansmith: and yell at us 18 months later :) | |
| 16:51:05 | mriedem | that's all par for the course really | |
| 16:51:11 | dansmith | mriedem: I'll be flippin burgers by then | |
| 16:51:21 | mriedem | beet farming for me | |