| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-17 | |||
| 16:19:58 | johnthetubaguy | sorry, I missed all that chat there | |
| 16:20:11 | cburgess | johnthetubaguy Welcome to the party! | |
| 16:20:38 | cburgess | johnthetubaguy Wait its after 4PM on a Friday there... why are you at the pub? | |
| 16:20:47 | cburgess | Sorry why aren't you | |
| 16:20:47 | cfriesen | my concern about just moving to cast is that if one of those gets lost we're stuck with a migration in progress | |
| 16:20:59 | cburgess | cfriesen Yes thats the rub. | |
| 16:21:02 | johnthetubaguy | heh, I am around for a few hours yet | |
| 16:21:17 | johnthetubaguy | got band rehersal before I get close to a pub anyways! | |
| 16:21:20 | cburgess | cfriesen We would have to be able to cancel it. Could the migration records accomplish that? | |
| 16:21:24 | cdent | figleaf I think that's inevitable unless we want solely use the external interface of the compute manager (which might be fine), but even with there would still need to be something to causing the period job to run, and to call the external interface | |
| 16:21:29 | mriedem | cfriesen: cburgess: so i just looked at oslo.messaging, and yeah it looks trivial to pass a new timeout in when constructing the client | |
| 16:21:42 | cfriesen | I wonder if the existing live migration abort API would be sufficient | |
| 16:21:59 | figleaf | cdent: yeah, I guess I'm just too idealistic :) | |
| 16:21:59 | cburgess | mriedem I don't even think you have to do it when constructing the client. I think with an existing client its just an arg you can pass on call. | |
| 16:22:05 | johnthetubaguy | mriedem: I thought that was a per call thing we can tweak? | |
| 16:22:14 | mriedem | cburgess: oh that's even better b/c we cache the clients | |
| 16:22:21 | johnthetubaguy | yeah, what cburgess said | |
| 16:22:22 | cdent | figleaf: no, you just are trying to apply principles that might work elsewhere, here | |
| 16:22:34 | cburgess | cfriesen Right thats what I'm thinking. If we know the state is "we made the cast to run pre live migration" then we should be able to cancel that by making the right call to abort it. | |
| 16:22:36 | johnthetubaguy | cburgess: I am +1 increasing the timeout for the lots of volumes case | |
| 16:22:44 | figleaf | cdent: like I said, too idealistic | |
| 16:23:04 | superdan | mriedem: no caching of the client anymore right? | |
| 16:23:09 | superdan | well, for compute anyway | |
| 16:23:23 | cburgess | johnthetubaguy Some type of equation you thjink? Before call is made look at size of BDM and make some type of auto-adjust? Or just a new config value for pre-live timeout? | |
| 16:23:31 | cfriesen | mriedem: I think there's an example of the per-call timeout in nova/cells/rpcapi.py | |
| 16:23:34 | superdan | well, sometimes in compute actually | |
| 16:23:38 | johnthetubaguy | cburgess: just a large one I think | |
| 16:23:51 | mriedem | yeah i see the timeout kwarg to the client.prepare method | |
| 16:23:53 | johnthetubaguy | cburgess: its only there as a "oh dear, you must have given up" kind of timeout | |
| 16:23:58 | johnthetubaguy | mriedem: yeah, thats the one | |
| 16:24:04 | cburgess | johnthetubaguy Just an arbitrary "This call can suck, wait 10 minutes" type thing? | |
| 16:24:11 | johnthetubaguy | mriedem: I thought we used that already for something, I can't remember now | |
| 16:24:13 | superdan | it'd be nice if we had a per-call heartbeat mechanism, | |
| 16:24:17 | superdan | which o.msg could provide for us | |
| 16:24:27 | superdan | send a "still working" reply to the reply queue which would reset the timer | |
| 16:24:29 | cburgess | johnthetubaguy The timeout would be a stop gap really. I think mriedem and cfriesen idea about moving to cast is the right solution but probably going to take more effort. | |
| 16:24:31 | johnthetubaguy | superdan: thats the "task" framework in my head | |
| 16:24:37 | cburgess | superdan OMG yes | |
| 16:24:40 | superdan | johnthetubaguy: no, I mean something at the o.msg layer | |
| 16:24:47 | superdan | johnthetubaguy: we could solve it in nova with a task too | |
| 16:24:50 | johnthetubaguy | superdan: yeah, check if job is progressing, and you know the owner | |
| 16:24:54 | mriedem | cctxt = self.client.prepare(timeout=timeout) | |
| 16:24:54 | mriedem | arg_p = jsonutils.to_primitive(arg) | |
| 16:24:54 | mriedem | def ping(self, context, arg, timeout=None): | |
| 16:24:57 | mriedem | we do have something using it | |
| 16:25:00 | superdan | but it'd be nice for lots of things if o.msg would heartbeat a particular call | |
| 16:25:11 | johnthetubaguy | superdan: oh I see what you mean now | |
| 16:25:29 | cfriesen | mriedem: you can set it in the client init but also in the cctxt.call() | |
| 16:25:32 | superdan | johnthetubaguy: you could just send something to the reply queue like "this isn't the reply, but I ain't dead yet" | |
| 16:25:50 | mriedem | cfriesen: i don't actually see the timeout kwarg in the rpc client call() method in oslo.messaging | |
| 16:25:50 | cburgess | mriedem cfriesen johnthetubaguy superdan: So 1 bug and 1 spec here? bug = set a "large" timeout for the pre_live call today, spec = move to casting? | |
| 16:25:51 | johnthetubaguy | superdan: yeah, like a "continue" or whatever | |
| 16:25:53 | mriedem | it's in the prepare() method | |
| 16:26:18 | superdan | johnthetubaguy: or lease renewal | |
| 16:26:31 | mriedem | cfriesen: http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/rpc/client.py#n178 | |
| 16:26:45 | mriedem | cfriesen: call() doesn't use a timeout kwarg http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/rpc/client.py#n151 | |
| 16:27:16 | cfriesen | mriedem: we're using it in nova/cells/rpcapi.py, line 332 | |
| 16:27:40 | johnthetubaguy | superdan: yeah, reset the timeout timer | |
| 16:27:45 | mriedem | cfriesen: that looks like a mistake | |
| 16:28:06 | cburgess | mriedem cfriesen Yeah the kwargs just gets pased all the way down and I think that one gets dropped. | |
| 16:28:08 | mriedem | cfriesen: the oslo.messaging rpc client call() method takes any **kwargs | |
| 16:28:16 | mriedem | right, the call() method doesn't use the timeout | |
| 16:28:18 | mriedem | the prepare() method does | |
| 16:28:32 | mriedem | which is nova/cells/rpcapi.py L326 | |
| 16:28:48 | cburgess | mriedem Right. | |
| 16:29:08 | cburgess | mriedem I still suspect its doable... we just have to modify things a bit more to make it work. | |
| 16:29:18 | cfriesen | mriedem: I think maybe the call() routine used to take a timeout | |
| 16:29:23 | mriedem | maybe | |
| 16:29:27 | mriedem | anyway yeah this is easy | |
| 16:29:48 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L720 | |
| 16:29:59 | cburgess | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L720 | |
| 16:29:59 | mriedem | you use your nova config pre_lie_migration_timeout value there | |
| 16:30:01 | cburgess | LOL yeah | |
| 16:30:08 | cburgess | mriedem Just found that same line. | |
| 16:30:09 | mriedem | default to None to get the default from oslo.messaging | |
| 16:30:14 | mriedem | bump to 10 minutes if you need to | |
| 16:30:18 | cburgess | mriedem Yup | |
| 16:30:23 | cburgess | mriedem So fixed value or config option? | |
| 16:30:27 | mriedem | config | |
| 16:30:33 | cburgess | Where is sdague to yell at me about config options? | |
| 16:30:37 | cburgess | mriedem Bug? or Spec? | |
| 16:30:41 | mriedem | this is completely operator side config | |
| 16:30:48 | mriedem | this is a bug i think | |
| 16:31:01 | mriedem | the spec is changing the blocking call to two casts | |
| 16:31:07 | cburgess | mriedem Right | |
| 16:31:27 | cburgess | mriedem OK... I shall file bug with code since this is easy and allow me to get ATC again. :P | |
| 16:31:44 | mriedem | cburgess: ATC isn't a thing anymore | |
| 16:31:45 | mriedem | remember | |
| 16:31:50 | mriedem | your ATC pass is your PTG $100 | |
| 16:32:49 | openstackgerrit | Chris Friesen proposed openstack/nova master: remove unused parameter in rpc call https://review.openstack.org/447055 | |
| 16:33:07 | openstackgerrit | Sarafraj Singh proposed openstack/nova-specs master: Spec to add per instance timeout for LM https://review.openstack.org/442809 | |
| 16:33:17 | cburgess | mriedem ATC = voting rights still | |
| 16:33:39 | cburgess | mriedem Semi-related... any idea who to contact about my ATC pass? I never got that email with details for Boston. | |
| 16:34:14 | cburgess | mriedem I can't vote in elections for TC, or nova unless I'm ATC. Or unless a PTL goes through the special ATC nomination process to make someone an ATC. | |
| 16:34:46 | dtp | johnthetubaguy and jroll, are you guys around? | |
| 16:35:20 | raj_singh | johnthetubaguy: I tried to address your concerns on https://review.openstack.org/#/c/442809/. Can you please have another look, when you get a chance? | |
| 16:35:28 | johnthetubaguy | dtp: I am around | |
| 16:35:39 | cfriesen | cburgess: I had an issue too...ended up emailing summit@openstack.org | |
| 16:35:42 | johnthetubaguy | raj_singh: cool | |
| 16:35:48 | cburgess | cfriesen Cool thanks. | |