Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-04-03
16:36:55 jaypipes well, bauzas was correct that I was wrong about the instance_claim() not wrapping the entire spawn() operation.
16:37:56 mriedem yeah i would think most of these are localized misconfiguration, or global
16:38:12 mriedem like the one about not reserving enough resources on a given compute for ovs to run
16:38:16 mriedem remember that one?
16:38:27 dansmith jaypipes: it wraps all of driver.spawn() or did you mean the compute manager level?
16:38:52 mriedem if you need to reserve space for a compute to run ovs, then i'm not sure why you wouldn't make the reservation on the inventory record for the compute resource provider
16:38:53 mriedem that's the point
16:38:53 jaypipes dansmith: yeah for some reason I didn't think instance_claim() wrapped all of spawn().
16:38:58 jaypipes dansmith: I was wrong.
16:39:16 bauzas so, the flag is clear, we only reschedule if we raise RescheduledException
16:39:17 bauzas https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1779
16:39:32 dansmith jaypipes: I'm confused, what doesn't get wrapped?
16:39:44 dansmith jaypipes: we've started the instance in the driver before we return
16:40:07 jaypipes dansmith: right, I understand that now. I implied with my comment above that that was not the case, and I'm saying I was wrong.
16:40:16 bauzas dansmith: mriedem: so you could be right, some exceptions couldn't be transformed into that specific exception and we could just error out the instance
16:40:47 bauzas but that leaves the whole world of unexpected exceptions that *are* rescheduled
16:41:07 bauzas unexpected exceptions being the ones raised by the driver
16:42:10 mriedem so if we wanted to reschedule within the cell, we'd need two schedulers right? one top level scheduler across all cells, and one for the cell itself, which is basically redoing cells v1
16:42:26 alaski It's been a while since I looked at the code, but I remember that generally we try to reschedule if there's a chance that the instance could start on another host. Typically this is for the case where there are not enough resources on the host for the instance, either due to a scheduler race or due to bad resource accounting on the compute
16:42:32 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872
16:42:46 dansmith alaski: right
16:42:56 mriedem alaski: but we also just have the generic 'we have no idea'
16:42:57 dansmith alaski: I was thinking that's really the case we most care about today
16:43:32 mriedem this https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2003
16:43:33 alaski mriedem: I don't think you need two schedulers. I think the scheduler shouldn't live at the API level. Think of it more like cinder, outside of cells fully
16:43:42 alaski dansmith: yep, agreed
16:44:01 dansmith alaski: um, scheduler shouldn't live at the api level? what does that mean?
16:44:05 mriedem if you have one scheduler, and you want to retry, then we'd have to upcall to that top level scheduler
16:44:36 dansmith mriedem: I definitely don't want to have multiple schedulers to handle the reschedule case
16:44:41 alaski dansmith: I mean start to think of the scheduler as outside of Nova. Just as cinder doesn't live at the API level I wouldn't lump the scheduler there either
16:45:17 mriedem API level == global to me
16:45:21 alaski otherwise you do have to have an upcall to reschedule, or have two levels
16:45:22 dansmith alaski: well, there's the architecture diagram and then there's the plumbing of how to talk to it :)
16:45:23 mriedem neutron, cinder, placement, keystone, etc
16:45:58 dansmith scheduler isn't a REST service which means we're a little constrained from just talking to it from wherever
16:46:12 dansmith although, I will re-point out the scheduler-is-just-a-library helps with this :)
16:46:12 alaski dansmith: yep. And this is where my knowledge has gaps, but I thought a scheduler API was being stood up, which would allow talking to it from anywhere
16:46:21 dansmith alaski: that's placement
16:46:30 dansmith alaski: and it's only part of the act of scheduling a thing
16:46:44 alaski gotcha
16:47:00 bauzas dansmith: I'm basically +1 on the idea of merging filtering/weighting in the conductor
16:47:12 bauzas so I think it's reasonable
16:47:28 bauzas but here, I'm more concerned by having the claims being implemented *before* that
16:47:31 dansmith it'd be super interesting to use scheduler as a library for reschedules
16:47:45 bauzas we sorta do that already
16:47:54 dansmith bauzas: well, then I think you have to nuke the reschedules if you're going to do it first
16:47:59 bauzas we have a scheduler.utils module that computes use for that purpose
16:48:06 dansmith bauzas: temporarily, if you want, to be reopened later
16:48:21 bauzas I dunno, and my wife is yelling at me because of dinner time :('
16:48:22 dansmith hey I have a crazy idea
16:48:28 dansmith how about we ask an operator?
16:48:32 mriedem i was going to say,
16:48:43 bauzas can someone could just press the "pause" button for real life things?
16:48:45 mriedem we can agree we need to pass a flag down to the compute to tell it to not claim because we already did it at the top
16:49:00 mriedem for the reschedule thing, we have a session proposed for claims in the scheduler already,
16:49:02 bauzas that, I'm all good with
16:49:13 mriedem i think we could hash out the reschedule thing at that session since operators are supposed to be there
16:49:20 bauzas good lord
16:49:22 dansmith mriedem: sounds good to me
16:49:45 bauzas okay, let me just write some stuff and put that in the spec as a big PITA
16:49:49 mriedem that's assuming (1) that session is picked and (2) anyone shows up
16:50:14 dansmith mriedem: well, if it's not picked we can do something else
16:50:14 bauzas yeah
16:50:26 dansmith mriedem: if it is and they don't show up, then, I dunno, I guess they approve of whatever outcome :P
16:50:30 bauzas open a thread and leave people arguing forever \o/
16:50:57 bauzas either way, I really need to drop off for family business :(
16:50:59 mriedem dansmith: and yell at us 18 months later :)
16:51:05 mriedem that's all par for the course really
16:51:11 dansmith mriedem: I'll be flippin burgers by then
16:51:21 mriedem beet farming for me
16:51:28 bauzas I got a new project
16:51:36 bauzas Burgers-As-a-Service
16:51:42 dansmith bauzas: you really like us better than your wife?
16:51:53 bauzas I do *love* my wife
16:51:56 bauzas even my kids
16:52:14 bauzas but sometimes, work/life balance is hard :)
16:52:20 mriedem dude, just go
16:56:26 mriedem ok, i think he's gone. now let's undo all of the claims stuff behind his back!
16:56:46 dansmith heh
17:03:24 namnh dansmith: Hi Mr.Smith, may I confirm with you a point?
17:06:26 dansmith namnh: you can try :)
17:06:47 namnh dansmith: thanks for your reply
17:06:52 namnh dansmith: Currenlty, Nova supported rolling-upgrade feature, it means old services in Nova can interact with new services. For example nova-api(Mitaka) can interact with nova-conductor(Nova). Is that right?
17:07:22 dansmith namnh: no. That statement is true for control services vs. computes, but that's all currently
17:07:30 dansmith namnh: i.e. api and conductor must be the same level, but compute may be old
17:10:37 namnh dansmith: during upgrade, I can use the "pinning version" option in nova.conf. I.e, during I upgrade M -> Newton, I can pin version in nova.conf at mitaka release
17:11:55 namnh dansmith: so in my understading, conductor can still interact with api at old release. am i right?
17:15:37 dansmith namnh: no
17:15:49 dansmith namnh: at no point can you have different versions of conductor and api
17:17:28 openstackgerrit Matt Riedemann proposed openstack/python-novaclient master: Remove deprecated network APIs https://review.openstack.org/447728
17:19:33 namnh dansmith: so it means that I have turn off all conductors and apis at the same time and then upgrade and start these services at the same time.
17:19:54 namnh dansmith: so it means that I have to turn off all conductors and apis at the same time and then upgrade and start these services at the same time.
17:20:03 mriedem sdague: i addressed the stuff you and alex_xu wanted to see in the os-services/os-hypervisors spec https://review.openstack.org/#/c/447149/, it's got a +2 from dansmith
17:20:16 mriedem dtp: have you started on the change to add uuid to the services table?
17:20:35 dansmith namnh: yes
17:25:14 namnh dansmith: thanks for your answer, I think I need to research more deeply Nova's document for this case
17:25:49 mriedem namnh: https://docs.openstack.org/developer/nova/upgrade.html in case you haven't seen that yet
17:27:14 sdague mriedem: looking
17:28:34 openstackgerrit Maxim Nestratov proposed openstack/nova master: libvirt: Virtuozzo containers config drive support https://review.openstack.org/449818
17:29:08 namnh mriedem: thanks. Surely, I will research more deeply the link.

Earlier   Later