Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-17
15:56:19 superdan mriedem: I guess I missed the point at which we started having stable branches for clients
15:56:25 mriedem superdan: years ago
15:56:48 mriedem so we could backport the fix to ocata and release 7.1.1
15:56:52 efried That's interesting - the docs on autospec claim it's lazy out of the box.
15:56:59 superdan mriedem: regardless of when, I guess I missed it
15:57:02 mriedem so your customer can pick that up w/o the backward incompatible changes that will be in 8.0.0 in pike
15:57:45 mriedem we needed stable branches on all of the clients and libraries just to get fixes back, plus we needed it at times to unscrew the gate back before upper-constraints
15:59:16 claudiub ooh, autospec
15:59:20 claudiub i'm actually working on it
15:59:29 claudiub I want it to become a permanent thingy
16:00:05 claudiub mriedem: something like this should do the trick: https://github.com/testing-cabal/mock/pull/394
16:00:33 claudiub there's still an issue with mock.patch autospecing, on which i'm working on.
16:00:53 mriedem cool
16:01:38 efried claudiub Second sentence here claims it's already lazy: https://docs.python.org/3/library/unittest.mock.html#autospeccing
16:02:02 efried claudiub So you've discovered that's a false claim under certain conditions?
16:02:21 claudiub it is a false claim. :)
16:02:42 claudiub efried: https://github.com/testing-cabal/mock/issues/393
16:02:45 efried claudiub Down a page or so, more explicitly: "Because the speccing is done “lazily” (the spec is created as attributes on the mock are accessed) you can use it with very complex or deeply nested objects (like modules that import modules that import modules) without a big performance hit."
16:02:55 efried Imagine my disappointment.
16:03:12 claudiub i know, right? :)
16:04:04 claudiub anyways, the only thing that specing / autospecing does at the moment, is ensure that your mock only have the same attributes / members as the spec itself, raising an AttributeError if non-existent attributes are accesed.
16:04:11 claudiub so, there's at least that.
16:04:19 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693
16:04:19 openstackgerrit Lee Yarwood proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520
16:04:32 efried claudiub That issue doesn't look related to performance/laziness
16:04:33 lyarwood mdbooth: ^ remove_volume_connection behaviour should remain the same now
16:05:05 melwitt claudiub, efried: well, it's expensive when you go and autospec every possible attribute, which is what you're doing right?
16:05:07 claudiub efried: but, you can give my PR i've linked above a try. it fixes this issue. check the added unit tests. There are still some unit tests failing for pypy, because the unittests themselves are faulty.
16:05:13 mdbooth lyarwood: Thanks, will look.
16:05:50 claudiub melwitt: actually, the way mock works, is create a child mock when a mock's attrbute is accessed, and it wasn't accessed before, and then caches it.
16:06:25 claudiub so, what I've done, is ensure that that child mock is properly spec'ed when created.
16:06:45 claudiub so, it won't autospec everything, it will only autospec accessed attributes / methods.
16:07:41 melwitt claudiub: okay, but you don't use every child mock in every test, right? or did you only spec the childs that are going to be asserted in tests?
16:08:40 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009
16:09:01 claudiub right, only spec the childs that are going to be asserted in the tests.
16:10:18 claudiub anyways. will try out some new experiments on nova with mock autospecs. will see then the performance impact.
16:11:17 claudiub and if it's a reasonable impact, autospecs should be used pretty much everywhere in openstack. :)
16:11:31 melwitt okay, cool
16:11:50 cfriesen so going back to the issue I raised yesterday about RPC timeouts during live migration waiting for large images to download from glance to the compute node...is this just a case of "we don't support that"?
16:12:02 mriedem cfriesen: did you see my reply?
16:12:57 cfriesen mriedem: yes, though that was dealing with a service token. in my case it's an RPC timeout that's the issue, not authentication
16:13:20 cfriesen if it takes more than 60 seconds to download the glance image, pre_live_migration() times out
16:13:51 figleaf cdent: regarding https://review.openstack.org/#/c/427309 - it seems like you are also doing a lot of stuff manually, like telling RT to update usages, etc. Do you think it would be possible to more directly simulate the actual processes in order to ensure that the right code paths get executed?
16:13:58 mriedem cfriesen: ok so https://docs.openstack.org/developer/nova/api_microversion_history.html#id31 won't help
16:14:16 openstackgerrit Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009
16:14:21 cfriesen mriedem: nope.
16:14:22 mriedem cfriesen: i'd ask tdurakov,
16:14:23 cburgess cfriesen Worse if *anything* takes more then 60 seconds. lets says you have an instance with 10 volumes all using iscsi + multipath, that can take 2 or 3 minutes to build all the luns.
16:14:52 mriedem i'd think that we could make conductor rpc cast to the compute to do pre-live migration, and have the compute cast back to conductor when it's done
16:15:38 cburgess mriedem That might work. The problem right now is we have to set the RPC timeout to a very large value, like 5-10 minutes in some AZs to work around this. Honestly I would like to see that become async like you just described or have a seperate timeout for just that operation.
16:15:44 cdent figleaf: my intent was to start narrow and then get more broad
16:16:08 figleaf cdent: understood, but it feels sorta unit test-y
16:16:31 mriedem cburgess: i'd have to dig into the oslo.messaging internals to find out how easy it is to adjust the rpc timeout on the fly like that for a single operation
16:16:43 cfriesen mriedem: it's easy
16:16:47 cburgess mriedem Just pass it as an arg
16:17:01 cburgess I think we might have to expand one nova wrapper class to support the arg but otherwise its just an arg to the call to change the timeout for that specific call.
16:17:32 cfriesen mriedem: but the thing is that you don't necessarily know up front what a reasonable value is since it depends on the IO load on your computes and glance-api
16:17:46 cfriesen and other stuff like cburgess pointed out
16:17:56 cburgess mriedem cfriesen But a change to have conductor/compute cast to/from each other and simply take action once the call comes in rather then waiting on it and forcing it to take some fixed amount of time is a way way better solution.
16:18:23 cburgess cfriesen Right... which is why I think moving everything to cast calls to/from conductor is way better.
16:18:31 cfriesen cburgess: mriedem: agreed...would we need to change the handling of the overall migration completion timeout? (checking code)
16:18:56 cburgess cfriesen Not sure if that timer starts during pre flight.
16:19:07 mriedem the timer is while it's moving data right?
16:19:14 cfriesen cburgess: mriedem: looks like the migration completion timeout is just a libvirt thing
16:19:15 mriedem to kill it if it gets stuck
16:19:17 cburgess mriedem I thought that timer was going away?
16:19:23 mriedem cburgess: i believe it is
16:19:27 cfriesen cburgess: the progress timer is going away
16:19:31 mriedem johnthetubaguy and the osic fok are looking at that
16:19:32 cburgess Ahhh
16:19:33 mriedem *folk
16:19:58 johnthetubaguy sorry, I missed all that chat there
16:20:11 cburgess johnthetubaguy Welcome to the party!
16:20:38 cburgess johnthetubaguy Wait its after 4PM on a Friday there... why are you at the pub?
16:20:47 cburgess Sorry why aren't you
16:20:47 cfriesen my concern about just moving to cast is that if one of those gets lost we're stuck with a migration in progress
16:20:59 cburgess cfriesen Yes thats the rub.
16:21:02 johnthetubaguy heh, I am around for a few hours yet
16:21:17 johnthetubaguy got band rehersal before I get close to a pub anyways!
16:21:20 cburgess cfriesen We would have to be able to cancel it. Could the migration records accomplish that?
16:21:24 cdent figleaf I think that's inevitable unless we want solely use the external interface of the compute manager (which might be fine), but even with there would still need to be something to causing the period job to run, and to call the external interface
16:21:29 mriedem cfriesen: cburgess: so i just looked at oslo.messaging, and yeah it looks trivial to pass a new timeout in when constructing the client
16:21:42 cfriesen I wonder if the existing live migration abort API would be sufficient
16:21:59 figleaf cdent: yeah, I guess I'm just too idealistic :)
16:21:59 cburgess mriedem I don't even think you have to do it when constructing the client. I think with an existing client its just an arg you can pass on call.
16:22:05 johnthetubaguy mriedem: I thought that was a per call thing we can tweak?
16:22:14 mriedem cburgess: oh that's even better b/c we cache the clients
16:22:21 johnthetubaguy yeah, what cburgess said
16:22:22 cdent figleaf: no, you just are trying to apply principles that might work elsewhere, here
16:22:34 cburgess cfriesen Right thats what I'm thinking. If we know the state is "we made the cast to run pre live migration" then we should be able to cancel that by making the right call to abort it.
16:22:36 johnthetubaguy cburgess: I am +1 increasing the timeout for the lots of volumes case
16:22:44 figleaf cdent: like I said, too idealistic
16:23:04 superdan mriedem: no caching of the client anymore right?
16:23:09 superdan well, for compute anyway
16:23:23 cburgess johnthetubaguy Some type of equation you thjink? Before call is made look at size of BDM and make some type of auto-adjust? Or just a new config value for pre-live timeout?
16:23:31 cfriesen mriedem: I think there's an example of the per-call timeout in nova/cells/rpcapi.py
16:23:34 superdan well, sometimes in compute actually
16:23:38 johnthetubaguy cburgess: just a large one I think
16:23:51 mriedem yeah i see the timeout kwarg to the client.prepare method
16:23:53 johnthetubaguy cburgess: its only there as a "oh dear, you must have given up" kind of timeout

Earlier   Later