linux: Smarter unique crashes

When fuzzing with unique crashes enabled in Linux arch, if the same bug (same PC value)
is triggered under different execution path, fuzzer will miss it since the constructed output
filename will be the same and thus skipped.

Following the same philosophy with MAC arch, a callstack signature hash is created from
the last 3 nibbles of each frame's PC. This will effectively cover both the ASLR disabled
targets and attached PID targets which have (probably) ASLR enabled. Callstack hash is
added as part of both the output filename and report file.

However, since most modern targets use a pool of worker processes (e.g. media parsers),
whitelisting the entire backtrace might result into significant noise for cases where the bug
is triggered from the main thread or a worker. Plus other cases (e.g. matroska) where same
bug can be triggered with different exec paths based on how some locks were acquired at
runtime.

For previous reasons only the 7 most significant frames are participating into the callstack
signature. This value is controlled with NMAJORFRAMES macro and can be easily altered.
Another idea would be to export as a calling argument, although since args are quite a few already
I've left it hardcoded.

As a PoC to verify the signature behavior badcode2 target has been added at the examples.

Signed-off-by: Anestis Bechtsoudis <anestis@census-labs.com>

1 files changed, 1 insertions, 0 deletions

diff --git a/.gitignore b/.gitignore
index b34d6ad7..489a8f28 100644
--- a/.gitignore
+++ b/.gitignore
@@ -9,3 +9,4 @@ mach_excServer.c
 libs
 obj
 examples/targets/badcode1
+examples/targets/badcode2