From b9db6b1254c9bf3a47c171bb96468628e9bd00f2 Mon Sep 17 00:00:00 2001 From: Chris Ball Date: Mon, 2 Oct 2023 03:23:09 -0700 Subject: benchmark: add a README, lower default runs from 5 to 3 --- benchmark/README.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 benchmark/README.md (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md new file mode 100644 index 00000000..66f7f59e --- /dev/null +++ b/benchmark/README.md @@ -0,0 +1,46 @@ +# American Fuzzy Lop plus plus (AFL++) + +## benchmarking + +This directory contains benchmarking tools that allow you to compare one machine +with another in terms of raw ability to execute a fuzzing target repeatedly. + +To achieve this, we use a sample program ("test-instr.c") where each path is +equally likely, supply it a single seed, and tell AFL to exit after one run of +deterministic mutations against that seed. + +Usage: + +``` +cd aflplusplus/benchmark +python3 benchmark.py + [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override) + [*] Ready, starting benchmark... + [*] Compiling the test-instr-persist-shmem fuzzing harness for the benchmark to use. + [*] multicore test-instr-persist-shmem run 1 of 3, execs/s: 846065.81 + [*] multicore test-instr-persist-shmem run 2 of 3, execs/s: 849694.03 + [*] multicore test-instr-persist-shmem run 3 of 3, execs/s: 850757.52 + [*] Average AFL execs/sec for this test across all runs was: 848839.12 + [*] Average total execs/sec for this test across all runs was: 833138.28 + [*] Results have been written to benchmark-results.jsonl +``` + +By default, the script will use a number of parallel fuzzers equal to your +available CPUs/threads (change with `--fuzzers`), and will perform each test +three times and average the result (change with `--runs`). + +The script will use multicore fuzzing instead of singlecore by default (change +with `--mode singlecore`) and use a persistent-mode shared memory harness for +optimal speed (change with `--target test-instr`). + +Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) +in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other +tools such as [jq -cs](https://jqlang.github.io/jq/) or +[pandas](https://pandas.pydata.org/) for analysis. + +## Data analysis + +There is sample data in [benchmark-results.jsonl](benchmark-results.jsonl), and +a Jupyter notebook for exploring the results and suggesting their meaning at +[benchmark.ipynb](benchmark.ipynb). + -- cgit v1.2.3 From f2cbcdf3ff7349ab505e1fcebc3242c9252f2176 Mon Sep 17 00:00:00 2001 From: Chris Ball Date: Sun, 19 Nov 2023 15:10:23 -0800 Subject: benchmark: update README --- benchmark/README.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md index 66f7f59e..e37abad2 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -9,20 +9,22 @@ To achieve this, we use a sample program ("test-instr.c") where each path is equally likely, supply it a single seed, and tell AFL to exit after one run of deterministic mutations against that seed. -Usage: +Usage example: ``` cd aflplusplus/benchmark python3 benchmark.py - [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override) [*] Ready, starting benchmark... [*] Compiling the test-instr-persist-shmem fuzzing harness for the benchmark to use. - [*] multicore test-instr-persist-shmem run 1 of 3, execs/s: 846065.81 - [*] multicore test-instr-persist-shmem run 2 of 3, execs/s: 849694.03 - [*] multicore test-instr-persist-shmem run 3 of 3, execs/s: 850757.52 - [*] Average AFL execs/sec for this test across all runs was: 848839.12 - [*] Average total execs/sec for this test across all runs was: 833138.28 - [*] Results have been written to benchmark-results.jsonl + [*] singlecore test-instr-persist-shmem run 1 of 2, execs/s: 124883.62 + [*] singlecore test-instr-persist-shmem run 2 of 2, execs/s: 126704.93 + [*] Average execs/sec for this test across all runs was: 125794.28 + [*] Using 16 fuzzers for multicore fuzzing (use --fuzzers to override). + [*] multicore test-instr-persist-shmem run 1 of 2, execs/s: 1179822.66 + [*] multicore test-instr-persist-shmem run 2 of 2, execs/s: 1175584.09 + [*] Average execs/sec for this test across all runs was: 1177703.38 + [*] Results have been written to the benchmark-results.jsonl file. + [*] Results have been written to the COMPARISON file. ``` By default, the script will use a number of parallel fuzzers equal to your @@ -33,6 +35,9 @@ The script will use multicore fuzzing instead of singlecore by default (change with `--mode singlecore`) and use a persistent-mode shared memory harness for optimal speed (change with `--target test-instr`). +Feel free to submit the resulting line for your CPU added to the COMPARISON +file back to aflplusplus in a pull request. + Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other tools such as [jq -cs](https://jqlang.github.io/jq/) or -- cgit v1.2.3 From aabbdac86d6215833391a54fa7d3a474ad41e3fd Mon Sep 17 00:00:00 2001 From: vanhauser-thc Date: Mon, 20 Nov 2023 09:56:09 +0100 Subject: add benchmarks --- benchmark/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md index e37abad2..c7d75e42 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -9,6 +9,14 @@ To achieve this, we use a sample program ("test-instr.c") where each path is equally likely, supply it a single seed, and tell AFL to exit after one run of deterministic mutations against that seed. +**Note that this is not a real-world scenario!** +Because the target does basically nothing this is rather a stress test on +Kernel I/O / context switching. +For this reason you will not see a difference if you run the multicore test +with 20 or 40 threads - or even see the performance decline the more threads +(`-f` parameter) you use. In a real-world scenario you can expect to gain +exec/s until 40-60 threads (if you have that many available on your CPU). + Usage example: ``` -- cgit v1.2.3 From 01e0d4aa1c9e856124491d1f23deea0ae443d8ea Mon Sep 17 00:00:00 2001 From: vanhauser-thc Date: Sun, 3 Dec 2023 13:12:22 +0100 Subject: comparison -> comparison.md --- benchmark/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) (limited to 'benchmark/README.md') diff --git a/benchmark/README.md b/benchmark/README.md index c7d75e42..12f4763e 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -32,7 +32,7 @@ python3 benchmark.py [*] multicore test-instr-persist-shmem run 2 of 2, execs/s: 1175584.09 [*] Average execs/sec for this test across all runs was: 1177703.38 [*] Results have been written to the benchmark-results.jsonl file. - [*] Results have been written to the COMPARISON file. + [*] Results have been written to the COMPARISON.md file. ``` By default, the script will use a number of parallel fuzzers equal to your @@ -43,8 +43,8 @@ The script will use multicore fuzzing instead of singlecore by default (change with `--mode singlecore`) and use a persistent-mode shared memory harness for optimal speed (change with `--target test-instr`). -Feel free to submit the resulting line for your CPU added to the COMPARISON -file back to aflplusplus in a pull request. +Feel free to submit the resulting line for your CPU added to the COMPARISON.md +and benchmark-results.jsonl files back to AFL++ in a pull request. Each run writes results to [benchmark-results.jsonl](benchmark-results.jsonl) in [JSON Lines](https://jsonlines.org/) format, ready to be pulled in to other -- cgit v1.2.3