godoc: fix runaway goroutine use

Previously we would spin up a new goroutine for every directory
we needed to parse. Since parent directories cannot exit until
children have exited, in large GOPATH directories we could have 8000+
goroutines evaluating directories concurrently. The race detector uses
tsan, and tsan can handle a maximum of 8192 concurrent threads at one
time.

Limit the number of concurrent threads. It is difficult to do this
in a way that does not involve a child being blocked on a parent
completing to spin up goroutines for itself to complete. Solve this by
completing work on the main thread if there are no additional workers
available to help complete work.

I was expecting limiting the concurrency to hurt performance, but it
actually significantly improves it. Benchmarks were performed using CL
94904 at tip, and with goroutines at 2, 4, and 8 times the number of
CPU's.

$ benchstat tip.benchmark gated-2.benchmark gated-4.benchmark gated-8.benchmark
name \ time/op    tip.benchmark  gated-2.benchmark  gated-4.benchmark  gated-8.benchmark
NewDirectory-4       293ms ± 2%         262ms ± 4%         252ms ± 4%         253ms ± 2%

name \ alloc/op   tip.benchmark  gated-2.benchmark  gated-4.benchmark  gated-8.benchmark
NewDirectory-4       218MB ± 0%         218MB ± 0%         218MB ± 0%         218MB ± 0%

name \ allocs/op  tip.benchmark  gated-2.benchmark  gated-4.benchmark  gated-8.benchmark
NewDirectory-4        513k ± 0%          508k ± 0%          509k ± 0%          510k ± 0%

Fixes golang/go#22110.

Change-Id: If01f78f1fc53cd195e4f8f6988c3c39b3c275992
Reviewed-on: https://go-review.googlesource.com/94955
Reviewed-by: Yury Smolsky <yury@smolsky.by>
Reviewed-by: Kevin Burke <kev@inburke.com>

0 files changed, 0 insertions, 0 deletions