Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-12
21:47:37 sean-k-mooney fungi: isnt that there to stop all gate capastity been eaten by triplo
21:47:48 mriedem sean-k-mooney: and nova and neutron
21:47:57 fungi that was a big part of it, but yes also nova and neutron ;)
21:48:09 mriedem https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential
21:48:28 sean-k-mooney yes but i think triplo still uses like the equivalent of nova and neutron combined
21:48:39 sean-k-mooney anyway it makes it fairer for everyone else
21:49:19 fungi under the old first-come-first-served scheduling, if a project has someone push a 30-change series in one shot then those all got priority and one-off changes for other projects got to wait until that entire series got the requested resources
21:50:28 sean-k-mooney we nver have 30+ long seriese in nova
21:50:32 efried never
21:50:59 sean-k-mooney only ever have 3 of them together
21:51:06 sean-k-mooney its never just 1
21:51:31 sean-k-mooney also our downstream ci is supper dumb and really does not like it when that is down to it
21:52:24 sean-k-mooney our downstream ci does not first apply the previous patch before the current one
21:52:34 fungi i'll be the first to admit that the round-robin job scheduler algorithm is still painful, but in general being backlogged on builds is going to be painful for someone regardless
21:53:11 sean-k-mooney fungi: yes but at least zuul make series actully work
21:53:20 sean-k-mooney jenkins on the other hand...
21:53:49 fungi if people have ideas for less painful scheduling algorithms/rules we can consider those too
21:54:21 sean-k-mooney the only one i have come up with that i thought was beeter we cant do.
21:54:40 fungi still trying to figure out if some of the packet scheduling algorithms used for adaptive rate limiting in the network space could be applied to job scheduling
21:54:41 sean-k-mooney which is split check into fast-check and check
21:55:09 sean-k-mooney and stick the tox jobs and docs in fast check
21:55:34 fungi we already have an analog of qos in place by prioritizing different pipelines
21:55:51 fungi and perform windowing and exponential backoff on failure in dependent pipelines
21:56:09 fungi so it's not out of the realm of possibility that networking ideas have still more we can steal from
21:56:16 sean-k-mooney fungi: just dont follow tcp's window algortioum
21:56:29 sean-k-mooney but ya
21:56:35 fungi well, yeah, it's not the exact same algorithm, just the basic idea
21:57:19 fungi but basically we only allocate resources to a subset of changes in dependent pipelines like the gate, and then increase or decrease that window based on how many changes pass or fail
21:57:45 sean-k-mooney there are some intersting algortiom in the cache/task execution domain too
21:58:06 fungi so that even though the gate pipeline has priority over the check pipeline, a perpetually failing gate load won't monopolize all available resources and leave check starved entirely
21:58:48 sean-k-mooney well to get to gate you have to go through check
21:58:53 sean-k-mooney so that should not happen anyway
21:58:59 sean-k-mooney it would be self regualting
21:59:27 sean-k-mooney this would poably be better conversation to have on infra or zuul channel
21:59:33 fungi except dependent pipelines can eat orders of magnitude more resources due to gate resets from failures near the front of the queue
21:59:42 efried fungi: would be neat to have a fast-fail toggle... somewhere, somehow.
22:00:08 sean-k-mooney efried: its a conflict between fail fast and report as much info as possibel
22:00:15 efried yeah, I know
22:00:36 sean-k-mooney fungi: how hard would it be to have each job report independly
22:00:39 efried but if I know my patch *should* pass, I turn that toggle on, so if e.g. py27 blows up spuriously, the whole thing gets kicked out right away freeing up the rest.
22:01:18 fungi it might be that fast-fail would be more appropriate in a dependent pipeline if its changes are filtered through an independent pipeline first like happens with the openstack tenant
22:01:30 sean-k-mooney efried: ya if a voting job fails it would be cool if it did nto start any other job in the job set but continue runniinng the ones that are running
22:01:36 openstackgerrit Eric Fried proposed openstack/nova master: DB API changes to get non-matching aggregates from metadata https://review.opendev.org/671074
22:01:37 openstackgerrit Eric Fried proposed openstack/nova master: Add a new request filter to isolate aggregates https://review.opendev.org/671075
22:01:37 openstackgerrit Eric Fried proposed openstack/nova master: Docs for isolated aggregates request filter https://review.opendev.org/667952
22:02:00 fungi given the expectation is that if the change has made it through check then it normally shouldn't fail in the gate either, so as soon as one failure is encountered it can just be reported and ejected
22:02:13 efried like that one --^
22:02:22 sean-k-mooney fungi: isnet there a way to make jobs depend on other jobs in the same pipline
22:02:45 fungi there is, but then you delay starting them until the others finish which increases time to report on success
22:03:12 sean-k-mooney yes but if you only dely it by the 5-10 minute fo the tox jobs
22:03:20 sean-k-mooney that might be worth it
22:04:15 sean-k-mooney fungi: its too bad we cant simulate this
22:04:21 fungi sure, i expect that depends on the project. ultimately though i think there's a lot more throughput to be gained by finding and fixing the bugs that lead to nondeterministic test results than in optimizing for failure
22:04:53 sean-k-mooney well i hope we are trying to do both
22:06:00 fungi sure, but the resources consumed by build failures far outweigh the meager gains from shuffling them around in the available resource pool
22:07:43 stephenfin efried: Alrighty, functional, unit and pep8 tests passing so I'm going to push up this rebase of PCPU onto vPMEM. Can you spin through it and check out the conflicts when I do?
22:08:23 mriedem fungi: i apologize for rousing you into this channel and discussion
22:08:32 mriedem s/rousing/conjuring/
22:09:15 fungi mriedem: no apologies needed. i'm just sorry if i derailed otherwise stimulating discussion in here ;)
22:09:42 fungi i do still recommend devstack changes on the feet though... so soothing
22:09:58 efried stephenfin: will do.
22:10:17 efried stephenfin: actually, hold on
22:10:22 sean-k-mooney fungi: the scheduling probalem for gate jobs has a lot of similar challanges to nova schduler problem for vms btu a lot of difference too
22:10:31 stephenfin holding
22:10:46 efried stephenfin: cpu-resources #1 just passed check and entered gate
22:10:48 mriedem fungi: before you showed up and professionaled it up we were talking about artom's beautiful brown eyes
22:10:56 efried stephenfin: but #2 is failing check
22:11:02 stephenfin efried: where are you seeing this?
22:11:04 mriedem sean-k-mooney: i already said we should use zuul as a nova scheduler driver in -infra months ago
22:11:08 mriedem dibs on that ideea
22:11:21 efried stephenfin: so if possible, just rebase #2+ onto vpmem
22:11:31 sean-k-mooney hehe its all yours
22:11:31 efried I... think that should be possible.
22:11:56 sean-k-mooney mriedem: add a ai or ml to it and you have your self a start up
22:12:06 stephenfin efried: I don't think I can straddle another branch like that
22:12:19 stephenfin I'd need to pull in vpmem between #1 and #2
22:12:28 stephenfin which means pulling that out of the queue
22:12:30 efried yeah, boo
22:14:10 sean-k-mooney even when its a pain im still glad we have the ci that zuul/infra provides. can you imaging merging all this stuff by hand and testing it like the kernel does
22:15:16 efried alex_xu: Yes, please do, I don't think there's anything you need to do here. Thanks for all the work.
22:15:44 alex_xu efried: stephenfin sean-k-mooney thank you all :)
22:16:18 sean-k-mooney alex_xu: night o/
22:16:22 efried stephenfin: I guess the choices are 1) wait for #1 to merge -- gate queue looks to be <2h -- then rebase the remainder on vpmem; or 2) rebase all right now and lose the +V on #1
22:16:39 stephenfin I've just going to rebase
22:16:41 stephenfin it's one patch
22:16:55 stephenfin if I don't, we'll need to rebase vpmem
22:17:09 stephenfin to pick up the newly merged patch from master
22:17:28 efried not sure about that. But.. okay.
22:17:30 sean-k-mooney because of a conflict?
22:17:42 efried no, because 2+ relies on 1
22:17:51 stephenfin ^ that
22:17:56 sean-k-mooney oh ok ya
22:18:10 sean-k-mooney thats our green check policy
22:18:29 stephenfin maybe the gate will rebase for me. idk
22:18:47 sean-k-mooney the gate will merge in some case but never rebase
22:18:48 stephenfin doesn't seem worth worrying about though. it's easy to hit recheck all weekend long
22:19:36 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793
22:19:36 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693
22:19:37 openstackgerrit Stephen Finucane proposed openstack/nova master: objects: Add 'InstanceNUMATopology.cpu_pinning' property https://review.opendev.org/680106
22:19:37 openstackgerrit Stephen Finucane proposed openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107

Earlier   Later