Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-07-12
20:27:31 mriedem i'm totally guessing
20:27:32 openstackgerrit Merged openstack/nova master: api-ref: mark instance action events parameter as optional https://review.openstack.org/480792
20:27:33 jdandrea But messy, I admit.
20:27:46 jdandrea Gonna hack on this a li'l bit ...
20:27:46 jdandrea biab
20:29:34 melwitt guh, more random "OpenStackApiAuthenticationException: Authentication error" functional test fails
20:29:39 melwitt gonna have to figure that out soon
20:32:33 melwitt in the gate
20:32:43 mriedem eventlet
20:33:05 mriedem it's something with the compute api and placement api client helper things in the functional tests getting screwed up over eventlet, i think
20:33:22 mriedem cdent had a patch at one point for a related issue where he put a yield in there
20:33:32 melwitt oh, that sounds familiar
20:33:39 mriedem i'm sure i can find it
20:33:47 mriedem like a hog hunting for truffles
20:33:54 melwitt lol
20:40:13 mriedem https://review.openstack.org/#/c/459735/ is what we went with
20:40:30 mriedem https://review.openstack.org/#/c/459733/ was cdents
20:41:18 melwitt hm
20:41:57 mriedem maybe we should set that globally,
20:42:04 mriedem if it's failing on other tests that aren't using the integrated helpers stuf
20:42:07 mriedem *stuff
20:42:17 melwitt oh yeah, lemme see what this one was
20:42:40 mriedem i had one earlier here http://logs.openstack.org/22/454322/37/check/gate-nova-tox-functional-ubuntu-xenial/76cbda6/console.html#_2017-07-12_17_44_14_514884
20:42:42 melwitt api_sample_tests, thought those were using integrated_helpers http://logs.openstack.org/74/467774/7/gate/gate-nova-tox-functional-ubuntu-xenial/4a00841/testr_results.html.gz
20:42:56 mriedem api sample tests are yeah
20:43:58 mriedem mine was something else
20:44:00 mriedem NoValidHost
20:44:17 mriedem i think because the compute node never checked in
20:44:20 mriedem because of some eventlet timeout
20:48:21 jaypipes mriedem, dansmith, edleafe: did we want to return a list of alternates *for each instance in num_instances*? of a single list of alternate allocation requests for the entire request?
20:48:31 jaypipes s/of a single/or a single/g
20:48:37 jdandrea mriedem jaypipes So what I did was look at the value and if it is a string, I try to json.loads the thing. If that fails, I leave it as is. Mischief managed.
20:49:09 dansmith jaypipes: well, if all of them can be rescheduled max_attempts time, then it needs to be num_instances * max_attempts
20:49:39 dansmith jaypipes: but we could also take some percentage or something, and provide some number of bet-hedging alternates, but not enough for all of them to be rescheduled max times
20:50:25 jaypipes dansmith: no, sorry, I meant should I return a single list of alternate allocation requests for the entire call to select_destinations() or should I return a list of alternate allocation requests for each host returned by select_destinations()?
20:51:00 dansmith jaypipes: independent of the number?
20:51:12 dansmith jaypipes: an alternate has to go with a given primary host, because it has to be the same cell,
20:51:29 dansmith so a single list of num*max alternates won't help us once we lose context of which host is in which cell
20:51:39 jaypipes right
20:51:47 jaypipes ok, that answers that
20:52:25 melwitt cells meeting in 8 minutes
20:52:34 dansmith melwitt: correction
20:52:46 dansmith melwitt: short-ass cells meetinglet in 8 minutes
20:54:26 melwitt meetinglet, I like it
20:56:34 edleafe jaypipes: yeah, I think that a single list per instance is best, where each list is the selected (claimed) host followed by the alternates.
20:56:54 efried In Portuguese, meetazinho. In Spanish, meetito. In French, meetette. In German, meetchen.
20:57:05 dansmith ooh, meetito is great
20:57:29 dansmith like dorito, but less yummy
20:57:42 efried Nothing is less yummy than a cooler ranch dorito
20:57:48 jaypipes edleafe: I'm going for a dict, keyed by instance_uuid, of two-tuples where the tuple is (WeighedHost(claimed_host), list(alternate allocation requests))
20:57:57 jaypipes edleafe, dansmith: ^^ make sense to you?
20:58:08 dansmith sure
20:59:16 efried jaypipes Is this yet another thing we're going to regret not being an object later?
20:59:19 melwitt efried: phew, that was close. I misread at first that you were saying cooler ranch was the best dorito
20:59:46 efried melwitt Rest easy.
21:00:00 edleafe jaypipes: this is what scheduler is returning to the conductor?
21:00:35 jaypipes edleafe: well, not *yet*... I'm currently referring to what filter scheduler _schedule() returns to filter scheduler's select_destinations() call.
21:00:45 jaypipes edleafe: still need to figure out what's best for the RPC changes.
21:01:01 jaypipes edleafe: it *may* be that structure...
21:01:12 jaypipes haven't fully thought the RPC layer changes through yet
21:01:26 edleafe jaypipes: I'm still not clear why the selected host has to be in a separate structure than the alternates
21:03:30 jaypipes edleafe: the alternates are meant to be opaque structures. the selected host is the host identifier. the allocation requests are alternate things that can be passed to placement to claim resources if the original destination host fails to fully launch the instance.
21:04:14 jaypipes edleafe: or are you asking to just have a list where the first element of the list is a WeighedHost and the other elements of the list are allocation request blobs?
21:05:16 edleafe jaypipes: the distinction seems unnecessary. Why not send a list of (host, alloc) for each instance, where the first item in the list is the selected host? This way, if the build fails, the cell conductor can easily unclaim the resources already claimed
21:06:05 jaypipes dansmith: thoughts? ^
21:06:19 dansmith jaypipes: we're in the cells meeting right now
21:06:26 jaypipes oh, sorry.
21:06:44 edleafe jaypipes: passing a heterogenous structure seems like it will lead to a lot of conditional blocks to handle the different cases
21:07:19 jaypipes edleafe: I don't disagree with you on that
21:22:48 dansmith jaypipes: I would expect the compute to un-claim before passing the reschedule attempt to conductor, since it will un-claim always on failed boot, and then punt to conductor if it thinks it should be retried
21:23:02 mriedem dansmith: dtp: melwitt: https://etherpad.openstack.org/p/nova-pike-cells-v2-todos
21:23:07 mriedem i'll start dumping
21:23:23 melwitt cool, thanks
21:23:45 dansmith jaypipes: so I'm not sure that we need anything too complicated, and was kinda expecting the conductor to do the next claim in order, and then take the would-be-versioned _response_ from placement as an indication of which compute node to use
21:24:05 dansmith jaypipes: however, if passing (hostname, allocation) around is more convenient or symmetric then that's fine
21:24:18 dansmith jaypipes: your select-destinations can't return a HostState outside of the scheduler though
21:24:27 jaypipes dansmith: yeah, I think edleafe's suggestion is a good one.
21:25:15 dansmith jaypipes: I'm not sure what he's suggesting, other than just making it a list where the implication is that we try the first one as the "selected host"
21:25:44 dansmith I assumed your example of the weighedhost as the selected one was for efficiency of the next step in the process
21:25:49 edleafe dansmith: yeah, that's pretty much it :)
21:26:07 jaypipes dansmith: he's suggesting returning a list of (host, allocation_request) tuples for each instance_uuid in the request
21:26:40 dansmith obviously that's the most you can return from the rpc call, if that's what you're talking about, but you said WeighedHost() above, so I figured you were talking internal still
21:27:31 dansmith but yeah, I think I said before that just returning a list of destinations per instance, and processing them in order in each phase is the way I think of it
21:27:41 dansmith like, I would expect each round to just pop the first one off the list,
21:27:47 dansmith try it, and move on
21:27:54 dansmith doing claim in conductor made that super symmetric
21:28:09 dansmith doing it in scheduler means it's pre-claimed before we return, which is what I was arguing against in the first place :)
21:30:17 jaypipes dansmith: but we can't claim in the conductor when we have num_instances > 0 unfortunately. see: instance groups, affinity, and similar issues.
21:31:25 edleafe jaypipes: so assuming a retry is necessary, how will the cell conductor deal with those things?
21:32:16 jaypipes edleafe: unfortunately, the only way to deal with those things on retry is the last-minute check on the alternate target host :(
21:32:35 dansmith jaypipes: I don't think I understand why that would be, but you guys steamrolled past that anyway, so it doesn't matter
21:33:19 jaypipes dansmith: because of this line: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L127-L128
21:33:29 jaypipes dansmith: those little lines are the bane of my existence.
21:34:24 dansmith I don't see how that affects claiming in the conductor, but whatever, not important
21:52:49 mriedem melwitt: so i didn't get to the resize stuff yet https://review.openstack.org/#/c/416521/ but with fresh eyes i think i've found 2 issues already discussed about over quota on multiple create not cleaning up, and restore
22:04:00 melwitt mriedem: hm, okay. will look at that
22:21:12 melwitt mriedem: replied about restore. reading through multi create
22:28:03 melwitt mriedem: though now I see the change to state "soft-deleted" doesn't happen until after an rpc call to compute, so I think that's why I didn't bother with recheck_quota there
22:28:41 melwitt maybe I should change it to set the state right away before the cast. not sure if that would be okay
22:37:34 mriedem ok still going through some other parts of this

Earlier   Later