Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-12
21:39:44 mriedem but as melwitt pointed out earlier, nova gets punished by zuul
21:39:52 mriedem devstack probably doesn't
21:39:55 efried true story
21:40:03 melwitt yeah, a devstack change might make it in a few hours. or less
21:40:11 sean-k-mooney or you can cheat. notice that all my jobs completed in like 2 yours
21:40:13 efried I thought you meant push a change to nova/devstack
21:40:19 mriedem i heard fungi takes a devstack change manually and runs it on his own hardware :)
21:40:43 sean-k-mooney the FN special lables im using have there own pool and if you put them in the experimental queue run almost right away
21:40:58 melwitt let's cheat
21:41:55 fungi mriedem: i take devstack changes amd manually rub them on my feet. it's better than a pedicure
21:42:19 mriedem i was not expecting that response
21:42:39 melwitt that is some fungi CI indeed
21:42:53 fungi i no longer have a basement full of server racks
21:42:53 mriedem now i just need to look up the magic incantation to configure a flavor with a required trait
21:42:57 sean-k-mooney melwitt: i mean if we want to repoduce quickly we could. also im ment to be wringin a greneade job....
21:42:59 fungi nor, you know, a basement
21:43:33 sean-k-mooney efried: stephenfin so what patches are where currently
21:43:39 sean-k-mooney vpmem is pending
21:43:47 efried mriedem: extra spec trait:YOUR_TRAIT_HERE='required' ?
21:43:49 sean-k-mooney and stephenfin you have a version of pcpu on that
21:43:58 mriedem efried: WRONG
21:44:02 mriedem trait:<trait>=required
21:44:06 mriedem quotes would kill you
21:44:15 melwitt sean-k-mooney: I only vaguely understand what y'all were talking about earlier. was just saying funny [to me] unhelpful things
21:44:20 efried mriedem: I was being syntax-y
21:44:34 stephenfin sean-k-mooney: I do, yeah. Just running tests locally before I push it up
21:44:35 efried if you're doing it on an osc cli, the quotes will go away
21:44:59 efried s/n osc//
21:45:25 mriedem https://review.opendev.org/681938
21:45:44 sean-k-mooney melwitt: :) i mean if one queue is slow and the other is fast sometimes cheating in the gate is for the greater good
21:46:18 efried sean-k-mooney: keeping in mind that shoving stuff in the queue from another project will push all the nova stuff down a slot too.
21:46:54 fungi yep. the change scheduling round-robin allocates nodes to changes by project, so the more changes there are for a given project requesting resources, the longer the later ones will wait for a turn
21:46:54 sean-k-mooney ya i know that is why os-vif is generaly way quicker to test stuff in the nova
21:46:59 mriedem https://bugs.launchpad.net/nova/+bug/1843836 tagged for train-rc-potential so we can start using that tag
21:47:00 openstack Launchpad bug 1843836 in OpenStack Compute (nova) "Failure to schedule if flavor contains non-CPU flag traits" [Undecided,In progress] - Assigned to Stephen Finucane (stephenfinucane)
21:47:37 sean-k-mooney fungi: isnt that there to stop all gate capastity been eaten by triplo
21:47:48 mriedem sean-k-mooney: and nova and neutron
21:47:57 fungi that was a big part of it, but yes also nova and neutron ;)
21:48:09 mriedem https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential
21:48:28 sean-k-mooney yes but i think triplo still uses like the equivalent of nova and neutron combined
21:48:39 sean-k-mooney anyway it makes it fairer for everyone else
21:49:19 fungi under the old first-come-first-served scheduling, if a project has someone push a 30-change series in one shot then those all got priority and one-off changes for other projects got to wait until that entire series got the requested resources
21:50:28 sean-k-mooney we nver have 30+ long seriese in nova
21:50:32 efried never
21:50:59 sean-k-mooney only ever have 3 of them together
21:51:06 sean-k-mooney its never just 1
21:51:31 sean-k-mooney also our downstream ci is supper dumb and really does not like it when that is down to it
21:52:24 sean-k-mooney our downstream ci does not first apply the previous patch before the current one
21:52:34 fungi i'll be the first to admit that the round-robin job scheduler algorithm is still painful, but in general being backlogged on builds is going to be painful for someone regardless
21:53:11 sean-k-mooney fungi: yes but at least zuul make series actully work
21:53:20 sean-k-mooney jenkins on the other hand...
21:53:49 fungi if people have ideas for less painful scheduling algorithms/rules we can consider those too
21:54:21 sean-k-mooney the only one i have come up with that i thought was beeter we cant do.
21:54:40 fungi still trying to figure out if some of the packet scheduling algorithms used for adaptive rate limiting in the network space could be applied to job scheduling
21:54:41 sean-k-mooney which is split check into fast-check and check
21:55:09 sean-k-mooney and stick the tox jobs and docs in fast check
21:55:34 fungi we already have an analog of qos in place by prioritizing different pipelines
21:55:51 fungi and perform windowing and exponential backoff on failure in dependent pipelines
21:56:09 fungi so it's not out of the realm of possibility that networking ideas have still more we can steal from
21:56:16 sean-k-mooney fungi: just dont follow tcp's window algortioum
21:56:29 sean-k-mooney but ya
21:56:35 fungi well, yeah, it's not the exact same algorithm, just the basic idea
21:57:19 fungi but basically we only allocate resources to a subset of changes in dependent pipelines like the gate, and then increase or decrease that window based on how many changes pass or fail
21:57:45 sean-k-mooney there are some intersting algortiom in the cache/task execution domain too
21:58:06 fungi so that even though the gate pipeline has priority over the check pipeline, a perpetually failing gate load won't monopolize all available resources and leave check starved entirely
21:58:48 sean-k-mooney well to get to gate you have to go through check
21:58:53 sean-k-mooney so that should not happen anyway
21:58:59 sean-k-mooney it would be self regualting
21:59:27 sean-k-mooney this would poably be better conversation to have on infra or zuul channel
21:59:33 fungi except dependent pipelines can eat orders of magnitude more resources due to gate resets from failures near the front of the queue
21:59:42 efried fungi: would be neat to have a fast-fail toggle... somewhere, somehow.
22:00:08 sean-k-mooney efried: its a conflict between fail fast and report as much info as possibel
22:00:15 efried yeah, I know
22:00:36 sean-k-mooney fungi: how hard would it be to have each job report independly
22:00:39 efried but if I know my patch *should* pass, I turn that toggle on, so if e.g. py27 blows up spuriously, the whole thing gets kicked out right away freeing up the rest.
22:01:18 fungi it might be that fast-fail would be more appropriate in a dependent pipeline if its changes are filtered through an independent pipeline first like happens with the openstack tenant
22:01:30 sean-k-mooney efried: ya if a voting job fails it would be cool if it did nto start any other job in the job set but continue runniinng the ones that are running
22:01:36 openstackgerrit Eric Fried proposed openstack/nova master: DB API changes to get non-matching aggregates from metadata https://review.opendev.org/671074
22:01:37 openstackgerrit Eric Fried proposed openstack/nova master: Docs for isolated aggregates request filter https://review.opendev.org/667952
22:01:37 openstackgerrit Eric Fried proposed openstack/nova master: Add a new request filter to isolate aggregates https://review.opendev.org/671075
22:02:00 fungi given the expectation is that if the change has made it through check then it normally shouldn't fail in the gate either, so as soon as one failure is encountered it can just be reported and ejected
22:02:13 efried like that one --^
22:02:22 sean-k-mooney fungi: isnet there a way to make jobs depend on other jobs in the same pipline
22:02:45 fungi there is, but then you delay starting them until the others finish which increases time to report on success
22:03:12 sean-k-mooney yes but if you only dely it by the 5-10 minute fo the tox jobs
22:03:20 sean-k-mooney that might be worth it
22:04:15 sean-k-mooney fungi: its too bad we cant simulate this
22:04:21 fungi sure, i expect that depends on the project. ultimately though i think there's a lot more throughput to be gained by finding and fixing the bugs that lead to nondeterministic test results than in optimizing for failure
22:04:53 sean-k-mooney well i hope we are trying to do both
22:06:00 fungi sure, but the resources consumed by build failures far outweigh the meager gains from shuffling them around in the available resource pool
22:07:43 stephenfin efried: Alrighty, functional, unit and pep8 tests passing so I'm going to push up this rebase of PCPU onto vPMEM. Can you spin through it and check out the conflicts when I do?
22:08:23 mriedem fungi: i apologize for rousing you into this channel and discussion
22:08:32 mriedem s/rousing/conjuring/
22:09:15 fungi mriedem: no apologies needed. i'm just sorry if i derailed otherwise stimulating discussion in here ;)
22:09:42 fungi i do still recommend devstack changes on the feet though... so soothing
22:09:58 efried stephenfin: will do.
22:10:17 efried stephenfin: actually, hold on
22:10:22 sean-k-mooney fungi: the scheduling probalem for gate jobs has a lot of similar challanges to nova schduler problem for vms btu a lot of difference too
22:10:31 stephenfin holding
22:10:46 efried stephenfin: cpu-resources #1 just passed check and entered gate

Earlier   Later