[ClusterFuzzLite][docs] Start writing docs (#6165)

I started writing some important docs, such as overview, build_integration, and running_clusterfuzzlite. However, most important docs are still TODO.
This PR at least adds a skeleton for those docs.

9 files changed, 506 insertions, 0 deletions

diff --git a/docs/clusterfuzzlite/advanced/adding_ci_support.md b/docs/clusterfuzzlite/advanced/adding_ci_support.md
new file mode 100644
index 000000000..2e855edd8
--- /dev/null
+++ b/docs/clusterfuzzlite/advanced/adding_ci_support.md
@@ -0,0 +1,12 @@
+---
+layout: default
+parent: Advanced Topics
+grand_parent: ClusterFuzzLite
+title: Adding Support for a New CI
+nav_order: 1
+permalink: /clusterfuzzlite/advanced-topics/adding-ci-support/
+---
+
+# Adding Support for a New CI
+
+TODO
diff --git a/docs/clusterfuzzlite/advanced/architecture.md b/docs/clusterfuzzlite/advanced/architecture.md
new file mode 100644
index 000000000..a564f378d
--- /dev/null
+++ b/docs/clusterfuzzlite/advanced/architecture.md
@@ -0,0 +1,12 @@
+---
+layout: default
+parent: Advanced Topics
+grand_parent: ClusterFuzzLite
+title: Architecture
+nav_order: 1
+permalink: /clusterfuzzlite/advanced-topics/architecture/
+---
+
+# Architecture
+
+TODO
diff --git a/docs/clusterfuzzlite/advanced_topics.md b/docs/clusterfuzzlite/advanced_topics.md
new file mode 100644
index 000000000..c5b49e125
--- /dev/null
+++ b/docs/clusterfuzzlite/advanced_topics.md
@@ -0,0 +1,10 @@
+---
+layout: default
+parent: ClusterFuzzLite
+title: Advanced Topics
+has_children: true
+nav_order: 4
+permalink: /clusterfuzzlite/advanced-topics/
+---
+
+# Advanced topics
diff --git a/docs/clusterfuzzlite/build_integration.md b/docs/clusterfuzzlite/build_integration.md
new file mode 100644
index 000000000..e52bee1c1
--- /dev/null
+++ b/docs/clusterfuzzlite/build_integration.md
@@ -0,0 +1,286 @@
+---
+layout: default
+parent: ClusterFuzzLite
+title: Build Integration
+has_children: true
+nav_order: 2
+permalink: /clusterfuzzlite/build-integration/
+---
+# Build integration
+{: .no_toc}
+
+- TOC
+{:toc}
+---
+
+## Prerequisites
+ClusterFuzzLite supports statically linked [libFuzzer targets]({{ site.baseurl }}/reference/glossary/#fuzz-target) on Linux.
+
+Before you can start setting up your new project for fuzzing, you must do the following:
+- [Integrate]({{ site.baseurl }}/advanced-topics/ideal-integration/) one or more [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target)
+  with the project you want to fuzz. For examples, see TODO.
+- [Install Docker](https://docs.docker.com/engine/installation)
+  (Googlers can visit [go/installdocker](https://goto.google.com/installdocker)).
+  [Why Docker?]({{ site.baseurl }}/faq/#why-do-you-use-docker)
+
+  If you want to run `docker` without `sudo`, you can
+  [create a docker group](https://docs.docker.com/engine/installation/linux/ubuntulinux/#/create-a-docker-group).
+
+  **Note:** Docker images can consume significant disk space. Run
+  [docker-cleanup](https://gist.github.com/mikea/d23a839cba68778d94e0302e8a2c200f)
+  periodically to garbage-collect unused images.
+
+- Clone the OSS-Fuzz repo: `git clone https://github.com/google/oss-fuzz.git`
+
+## Generating an empty build integration
+Build integrations consist of two files, a `Dockerfile` that can install
+dependencies for your project, and a `build.sh` file that actually builds your
+project. These must live in the `.clusterfuzzlite` directory in the root of your
+project's source code checkout.
+You can generate empty versions of these files with the following command:
+
+```bash
+$ cd /path/to/oss-fuzz
+$ export PATH_TO_PROJECT=<path_to_your_project>
+$ python infra/helper.py generate $PATH_TO_PROJECT --external
+```
+
+Once the configuration files are generated, you should modify them to fit your
+project.
+
+## Dockerfile {#dockerfile}
+
+This integration file defines the Docker image for your project.
+Your [build.sh](#buildsh) script will be executed in inside the container you
+define.
+For most projects, the image is simple:
+```docker
+FROM gcr.io/oss-fuzz-base/base-builder          # base image with clang toolchain
+RUN apt-get update && apt-get install -y ...    # install required packages to build your project
+COPY . $SRC/<project_name>                      # checkout all sources needed to build your project
+WORKDIR $SRC/<project_name>                     # current directory for the build script
+COPY ./clusterfuzzlite/build.sh fuzzer.cc $SRC/ # copy build script into src dir
+```
+TODO: Provide examples.
+
+## build.sh {#buildsh}
+
+This file defines how to build binaries for [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target) in your project.
+The script is executed within the image built from your [Dockerfile](#Dockerfile).
+If you are familiar with `build.sh` files from OSS-Fuzz, they work exactly the
+same in ClusterFuzzLite.
+
+In general, this script should do the following:
+
+- Build the project using your build system with the ClusterFuzzLite's compiler.
+- Provide ClusterFuzzLite's compiler flags (defined as [environment variables](#Requirements)) to the build system.
+- Build your [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target)
+  and link your project's build with `$LIB_FUZZING_ENGINE` (libFuzzer).
+
+Resulting binaries should be placed in `$OUT`.
+
+Here's an example from Expat
+([source](https://github.com/google/oss-fuzz/blob/master/projects/expat/build.sh)):
+
+```bash
+#!/bin/bash -eu
+
+./buildconf.sh
+# configure scripts usually use correct environment variables.
+./configure
+
+make clean
+make -j$(nproc) all
+
+$CXX $CXXFLAGS -std=c++11 -Ilib/ \
+    $SRC/parse_fuzzer.cc -o $OUT/parse_fuzzer \
+    $LIB_FUZZING_ENGINE .libs/libexpat.a
+
+cp $SRC/*.dict $SRC/*.options $OUT/
+```
+
+If your project is written in Go, check out the [Integrating a Go project]({{ site.baseurl }}//getting-started/new-project-guide/go-lang/) page.
+
+**Note:**
+1. Make sure that the binary names for your [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target) contain only
+alphanumeric characters, underscore(_) or dash(-). Otherwise, they won't run.
+1. Don't remove source code files. They are needed for code coverage.
+
+
+### Temporarily disabling code instrumentation during builds
+
+In some cases, it's not necessary to instrument every 3rd party library or tool that supports the build target. Use the following snippet to build tools or libraries without instrumentation:
+
+```
+CFLAGS_SAVE="$CFLAGS"
+CXXFLAGS_SAVE="$CXXFLAGS"
+unset CFLAGS
+unset CXXFLAGS
+export AFL_NOOPT=1
+
+#
+# build commands here that should not result in instrumented code.
+#
+
+export CFLAGS="${CFLAGS_SAVE}"
+export CXXFLAGS="${CXXFLAGS_SAVE}"
+unset AFL_NOOPT
+```
+TODO: Figure out if we should include this AFL code.
+
+### build.sh script environment
+
+When your build.sh script is executed, the following locations are available within the image:
+
+| Location| Env Variable | Description |
+|---------| ------------ | ----------  |
+| `/out/` | `$OUT`         | Directory to store build artifacts (fuzz targets, dictionaries, options files, seed corpus archives). |
+| `/src/` | `$SRC`         | Directory to checkout source files. |
+| `/work/`| `$WORK`        | Directory to store intermediate files. |
+
+Although the files layout is fixed within a container, environment variables are
+provided so you can write retargetable scripts.
+
+In case your fuzz target uses the [FuzzedDataProvider] class, make sure it is
+included via `#include <fuzzer/FuzzedDataProvider.h>` directive.
+
+[FuzzedDataProvider]: https://github.com/google/fuzzing/blob/master/docs/split-inputs.md#fuzzed-data-provider
+
+### build.sh requirements {#Requirements}
+
+Only binaries without an extension are accepted as targets. Extensions are reserved for other artifacts, like .dict.
+
+You *must* use the special compiler flags needed to build your project and fuzz targets.
+These flags are provided in the following environment variables:
+
+| Env Variable           | Description
+| -------------          | --------
+| `$CC`, `$CXX`, `$CCC`  | The C and C++ compiler binaries.
+| `$CFLAGS`, `$CXXFLAGS` | C and C++ compiler flags.
+| `$LIB_FUZZING_ENGINE`  | C++ compiler argument to link fuzz target against the prebuilt engine library (e.g. libFuzzer).
+
+You *must* use `$CXX` as a linker, even if your project is written in pure C.
+
+Most well-crafted build scripts will automatically use these variables. If not,
+pass them manually to the build tool.
+
+See the [Provided Environment Variables](https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/README.md#provided-environment-variables) section in
+`base-builder` image documentation for more details.
+
+## Fuzzer execution environment
+
+For more on the environment that
+your [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target) run in, and the assumptions you can make, see the [fuzzer environment]({{ site.baseurl }}/further-reading/fuzzer-environment/) page.
+
+## Testing locally
+
+You can build your docker image and fuzz targets locally, so you can test them
+before running ClusterFuzzLite.
+1. Run the same helper script you used to create your directory structure, this time using it to build your docker image and [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target):
+
+    ```bash
+    $ cd /path/to/oss-fuzz
+    $ python infra/helper.py build_image $PATH_TO_PROJECT --external
+    $ python infra/helper.py build_fuzzers $PATH_TO_PROJECT --sanitizer <address/undefined/coverage> --external
+    ```
+
+    The built binaries appear in the `/path/to/oss-fuzz/build/out/$PROJECT_NAME`
+    directory on your machine (and `$OUT` in the container). Note that
+    `$PROJECT_NAME` is the name of the directory of your project (e.g. if
+    `$PATH_TO_PROJECT` is `/path/to/systemd`, `$PROJECT_NAME` is systemd.
+
+    **Note:** You *must* run your fuzz target binaries inside the base-runner docker
+    container to make sure that they work properly.
+
+2. Find failures to fix by running the `check_build` command:
+
+    ```bash
+    $ python infra/helper.py check_build $PATH_TO_PROJECT --external
+    ```
+
+3. If you want to test changes against a particular fuzz target, run the following command:
+
+    ```bash
+    $ python infra/helper.py run_fuzzer --external --corpus-dir=<path-to-temp-corpus-dir> $PATH_TO_PROJECT <fuzz_target>
+    ```
+
+4. We recommend taking a look at your code coverage as a test to ensure that
+your fuzz targets get to the code you expect. This would use the corpus
+generated from the previous `run_fuzzer` step in your local corpus directory.
+
+    ```bash
+    $ python infra/helper.py build_fuzzers --sanitizer coverage $PATH_TO_PROJECT
+    $ python infra/helper.py coverage $PATH_TO_PROJECT --fuzz-target=<fuzz_target> --corpus-dir=<path-to-temp-corpus-dir> --external
+    ```
+
+You may need to run `python infra/helper.py pull_images` to use the latest
+coverage tools. Please refer to
+[code coverage]({{ site.baseurl }}/advanced-topics/code-coverage/) for detailed
+information on code coverage generation.
+
+**Note:** Currently, ClusterFuzzLite only supports AddressSanitizer (address)
+and UndefinedBehaviorSanitizer (undefined) configurations.
+<b>Make sure to test each
+of the supported build configurations with the above commands (build_fuzzers -> run_fuzzer -> coverage).</b>
+
+If everything works locally, it should also work on ClusterFuzzLite. If you
+check in your files and experience failures, review your [dependencies]({{site.baseurl }}/further-reading/fuzzer-environment/#dependencies).
+
+## Debugging Problems
+
+If you run into problems, the [Debugging page]({{ site.baseurl }}/advanced-topics/debugging/) lists ways to debug your build scripts and
+[fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target).
+
+## Efficient fuzzing
+
+To improve your fuzz target ability to find bugs faster, you should consider the
+following ways:
+
+### Seed Corpus
+
+OSS-Fuzz uses evolutionary fuzzing algorithms. Supplying seed corpus consisting
+of good sample inputs is one of the best ways to improve [fuzz target]({{ site.baseurl }}/reference/glossary/#fuzz-target)'s coverage.
+
+To provide a corpus for `my_fuzzer`, put `my_fuzzer_seed_corpus.zip` file next
+to the [fuzz target]({{ site.baseurl }}/reference/glossary/#fuzz-target)'s binary in `$OUT` during the build. Individual files in this
+archive will be used as starting inputs for mutations. The name of each file in the corpus is the sha1 checksum (which you can get using the `sha1sum` or `shasum` command) of its contents. You can store the corpus
+next to source files, generate during build or fetch it using curl or any other
+tool of your choice.
+(example: [boringssl](https://github.com/google/oss-fuzz/blob/master/projects/boringssl/build.sh#L41)).
+
+Seed corpus files will be used for cross-mutations and portions of them might appear
+in bug reports or be used for further security research.
+
+### Dictionaries
+
+Dictionaries hugely improve fuzzing efficiency for inputs with lots of similar
+sequences of bytes. [libFuzzer documentation](http://libfuzzer.info#dictionaries)
+
+Put your dict file in `$OUT`. If the dict filename is the same as your target
+binary name (i.e. `%fuzz_target%.dict`), it will be automatically used. If the
+name is different (e.g. because it is shared by several targets), specify this
+in .options file:
+
+```
+[libfuzzer]
+dict = dictionary_name.dict
+```
+
+It is common for several [fuzz targets]({{ site.baseurl }}/reference/glossary/#fuzz-target)
+to reuse the same dictionary if they are fuzzing very similar inputs.
+(example: [expat](https://github.com/google/oss-fuzz/blob/master/projects/expat/parse_fuzzer.options)).
+
+### Input Size
+
+By default, the fuzzing engine will generate input of any arbitrary length.
+This might be useful to try corner cases that could lead to a
+security vulnerability. However, if large inputs are not necessary to
+increase the coverage of your target API, it is important to add a limit
+here to significantly improve performance.
+
+```cpp
+if (size < kMinInputLength || size > kMaxInputLength)
+  return 0;
+```
+
+TODO(metzman): We probably want a TOC for lang-specific guides (which we still need to add).
diff --git a/docs/clusterfuzzlite/clusterfuzzlite.md b/docs/clusterfuzzlite/clusterfuzzlite.md
new file mode 100644
index 000000000..9dfd30a2b
--- /dev/null
+++ b/docs/clusterfuzzlite/clusterfuzzlite.md
@@ -0,0 +1,27 @@
+---
+layout: default
+title: ClusterFuzzLite
+has_children: true
+nav_order: 8
+permalink: /clusterfuzzlite/
+<!-- Hide for now by setting "published: false" -->
+published: false
+---
+
+# ClusterFuzzLite
+ClusterFuzzLite is a lightweight, easy-to-use, fuzzing infrastructure that is
+based off [ClusterFuzz]. ClusterFuzzLite is designed to run on [continuous integration] (CI)
+systems, which means it is easy to set up and provides a familiar interface for
+users.
+Currently CIFuzz fully supports [GitHub Actions]. However ClusterFuzzLite is
+designed so that supporting new CI systems is trivial and core features can be
+used on any CI system without any additional effort.
+
+See [Overview] for a more detailed description of how ClusterFuzzLite works and
+how you can use it.
+
+[continous integration]: https://en.wikipedia.org/wiki/Continuous_integration
+[fuzzing]: https://en.wikipedia.org/wiki/Fuzzing
+[ClusterFuzz]: https://google.github.io/clusterfuzz/
+[GitHub Actions]: https://docs.github.com/en/actions
+[Overview]: {{ site.baseurl }}/clusterfuzzlite/overview/
diff --git a/docs/clusterfuzzlite/overview.md b/docs/clusterfuzzlite/overview.md
new file mode 100644
index 000000000..5e1e2497d
--- /dev/null
+++ b/docs/clusterfuzzlite/overview.md
@@ -0,0 +1,29 @@
+---
+layout: default
+parent: ClusterFuzzLite
+title: Overview
+nav_order: 1
+permalink: /clusterfuzzlite/overview/
+---
+
+# Overview
+
+ClusterFuzzLite makes fuzzing more valuable by:
+* Fuzzing continuously.
+* Catching bugs before they land in your codebase by fuzzing pull
+  requests/commits.
+* Providing coverage reports on which code is fuzzed.
+* Managing your corpus, pruning it daily or a specified-interval.
+
+ClusterFuzzLite supports [libFuzzer], [AddressSanitizer], and
+[UndefinedBehavior].
+ClusterFuzzLite is modular, so you can decide which features you want to use.
+Using ClusterFuzzLite entails two major steps:
+1. [Integrating with ClusterFuzzLite's build system] so ClusterFuzzLite can
+   build your project's fuzzers.
+2. [Running ClusterFuzzLite].
+[libFuzzer]: https://libfuzzer.info
+[AddressSanitizer]: https://clang.llvm.org/docs/AddressSanitizer.html
+[UndefinedBehaviorSanitizer]: https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html
+[Integrating with ClusterFuzzLite's build system]: {{ site.baseurl }}/clusterfuzzlite/build-integration/
+[Running ClusterFuzzLite]: {{ site.baseurl }}/clusterfuzzlite/running-clusterfuzzlite/
diff --git a/docs/clusterfuzzlite/running-clusterfuzzlite/generic.md b/docs/clusterfuzzlite/running-clusterfuzzlite/generic.md
new file mode 100644
index 000000000..045cd2438
--- /dev/null
+++ b/docs/clusterfuzzlite/running-clusterfuzzlite/generic.md
@@ -0,0 +1,17 @@
+---
+layout: default
+parent: Running ClusterFuzzLite
+grand_parent: ClusterFuzzLite
+title: Running ClusterFuzzLite in a Generic Environment
+nav_order: 2
+permalink: /clusterfuzzlite/running-clusterfuzzlite/generic/
+---
+
+# Running ClusterFuzzLite in a Generic Environment
+TODO: Rename run_cifuzz.py to run_clusterfuzzlite.py
+
+This page describes how to run clusterfuzzlite using the run_clusterfuzzlite.py
+script, a low level script that is used by CI systems to run ClusterFuzzLite.
+This guide is useful if you want to run ClusterFuzzLite outside of CI or if you
+want to run it on a CI system that isn't supported yet.
+TODO
diff --git a/docs/clusterfuzzlite/running-clusterfuzzlite/github_actions.md b/docs/clusterfuzzlite/running-clusterfuzzlite/github_actions.md
new file mode 100644
index 000000000..430f67526
--- /dev/null
+++ b/docs/clusterfuzzlite/running-clusterfuzzlite/github_actions.md
@@ -0,0 +1,12 @@
+---
+layout: default
+parent: Running ClusterFuzzLite
+grand_parent: ClusterFuzzLite
+title: GitHub Actions
+nav_order: 1
+permalink: /clusterfuzzlite/running-clusterfuzzlite/github-actions/
+---
+
+# GitHub Actions
+
+TODO
diff --git a/docs/clusterfuzzlite/running_clusterfuzzlite.md b/docs/clusterfuzzlite/running_clusterfuzzlite.md
new file mode 100644
index 000000000..068fc03eb
--- /dev/null
+++ b/docs/clusterfuzzlite/running_clusterfuzzlite.md
@@ -0,0 +1,101 @@
+---
+layout: default
+parent: ClusterFuzzLite
+title: Running ClusterFuzzLite
+has_children: true
+nav_order: 3
+permalink: /clusterfuzzlite/running-clusterfuzzlite/
+---
+# Running ClusterFuzzLite
+{: .no_toc}
+
+- TOC
+{:toc}
+---
+
+## Overview
+TODO: add a diagram.
+
+Once your project's fuzzers can be built and run by the helper script, it is
+ready to be fuzzed by ClusterFuzzLite.
+The exact method for doing this will depend on the how you are running
+ClusterFuzzLite. For guides on how to run ClusterFuzzLite in your particular
+environment (e.g. GitHub Actions) see the subguides.
+The rest of this page will explain concepts configuration options and that are
+agnostic to how ClusterFuzzLite is being run.
+
+## ClusterFuzzLite Tasks
+
+ClusterFuzzLite has the concept of tasks which instruct ClusterFuzzLite what to
+do when running.
+
+### Code Review Fuzzing
+
+TODO(metzman): Work on a generic name for CIFuzz/PR fuzzing.
+
+One of the core ways for ClusterFuzzLite to be used is for fuzzing code that is
+in review that was just commited.
+This use-case is important because it allows ClusterFuzzLite to find bugs before
+they are commited into your code and while they are easiest to fix.
+To use Code Review Fuzzing, set the configuration option `clusterfuzzlite-task`
+to `code-review`.
+If you are familiar with OSS-Fuzz's CIFuzz, this task is similar to CIFuzz.
+Running other ClusterFuzzLite tasks enhances ClusterFuzzLite's ability to do
+Code Review Fuzzing.
+
+If [Batch Fuzzing] is enabled, Code Review Fuzzing will report only newly
+introduced bugs and use the corpus developed during batch fuzzing.
+If [Code Coverage Reporting] is enabled, Code Review Fuzzing will try to only
+run the fuzzers affected by the code change.
+
+### Batch Fuzzing
+
+ClusterFuzzLite can also run in a batch fuzzing mode where all fuzzers are run
+for a long amount of time. Unlike Code Review Fuzzing, this task is not meant to
+be interactive, it is meant to be long-lasting and generally is more similar to
+fuzzing in ClusterFuzz than Code Review Fuzzing. Batch Fuzzing allows
+ClusterFuzzLite to build up a corpus for each of your fuzz targets. This corpus
+will be used in Code Coverage Reporting as well as Code Review Fuzzing.
+
+### Corpus Prune
+
+If multiple Batch Fuzzing tasks are run concurrently then we strongly recommend
+running a pruning task as well. This task is run according to some set schedule
+(once a day is probably sufficient) to prune the corpus of redundant testcases,
+which can happen if multiple Batch Fuzzing jobs are done concurrently.
+
+### Code Coverage Report
+
+The last task ClusterFuzzLite offers is Code Coverage Reports. This task will
+run your fuzzers on the corpus developed during Batch Fuzzing and will generate
+an HTML report that shows you which part of your code is covered by batch
+fuzzing.
+
+## Configuration Options
+
+Below are some configuration options that you can set when running
+ClusterFuzzLite.
+We will explain how to set these in each of the subguides.
+
+`language`: (optional) The language your target program is written in. Defaults
+to `c++`. This should be the same as the value you set in `project.yaml`. See
+[this explanation]({{ site.baseurl }}//getting-started/new-project-guide/#language)
+for more details.
+
+`fuzz-time`: Determines how long ClusterFuzzLite spends fuzzing your project in
+seconds. The default is 600 seconds.
+
+`sanitizer`: Determines a sanitizer to build and run fuzz targets with. The
+choices are `'address'`, and `'undefined'`. The default is `'address'`.
+
+`task`: The task for ClusterFuzzLite to execute. `code-review`
+by default. See [ClusterFuzzLite Tasks] for more details on how to run different
+tasks.
+TODO(metzman): change run_fuzzers_mode to this.
+
+`dry-run`: Determines if ClusterFuzzLite surfaces bugs/crashes. The default
+value is `false`. When set to `true`, ClusterFuzzLite will never report a
+failure even if it finds a crash in your project. This requires the user to
+manually check the logs for detected bugs.
+
+TODO(metzman): We probably want a TOC on this page for subguides.