diff options
| author | David James <davidjames@google.com> | 2014-02-11 11:29:55 -0800 |
|---|---|---|
| committer | chrome-internal-fetch <chrome-internal-fetch@google.com> | 2014-02-12 02:05:04 +0000 |
| commit | e967e925df9a1ac07e5cf3fdafe3c346549eb4a2 (patch) | |
| tree | d197a2c26cf52a40ceb3a92d578fadb238dbea3b /lib/parallel.py | |
| parent | 06015fdbca7bd93c180dd2abb06d5bb825059af6 (diff) | |
| download | chromite-e967e925df9a1ac07e5cf3fdafe3c346549eb4a2.tar.gz | |
Use at least 16 processes by default for process pools.
Right now we calculate the number of processes for a process pool by
looking at the number of processors. This works for CPU-bound operations
but doesn't work as well for network-bound operations.
This should speed up the CQ by a few minutes.
BUG=chromium:342909
TEST=Run it
Change-Id: Iaf41190e5344abc3bec41f4b94aea87c888ec8bb
Reviewed-on: https://chromium-review.googlesource.com/185923
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Matt Tennant <mtennant@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Diffstat (limited to 'lib/parallel.py')
| -rw-r--r-- | lib/parallel.py | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/lib/parallel.py b/lib/parallel.py index 77db90116..f971ea780 100644 --- a/lib/parallel.py +++ b/lib/parallel.py @@ -660,7 +660,9 @@ def RunTasksInProcessPool(task, inputs, processes=None, onexit=None): """ if not processes: - processes = min(multiprocessing.cpu_count(), len(inputs)) + # - Use >=16 processes by default, in case it's a network-bound operation. + # - Try to use all of the CPUs, in case it's a CPU-bound operation. + processes = min(max(16, multiprocessing.cpu_count()), len(inputs)) with BackgroundTaskRunner(task, processes=processes, onexit=onexit) as queue: for x in inputs: |