| Age | Commit message (Collapse) | Author |
|---|
|
This CL moves Zucchini-gen invocation code from zucchini_commands.cc
(in target zucchini) to zucchini_integration.cc (in target zucchini_io)
to clean up layering in Zucchini API, i.e.:
- zucchini_lib: Operates on buffers only.
- zucchini_io: Adds files interface, uses memory-mapped I/O.
- zucchini: Stand-alone executable that parses command-line arguments.
Other changes:
- Rename zucchini_lib functions (zuchcini.h), to dedup names and
emphasize that these functions operate on buffers:
- GenerateEnsemble() -> GenerateBuffer(),
- GenerateEnsembleWithImposedMatches() -> GenerateBufferImposed(),
- GenerateRaw() -> GenerateBufferRaw(),
- Apply() -> ApplyBuffer().
These renames only affect Zucchini and various tests.
- Variable renames and parameter reordering in zucchini_integration.cc.
- Remove '-dd' param in help text of Zucchini-detect (was never ported
from Trunk, and has been recently deleted there as well).
- Replace all base::File&& with base::File.
- Miscellaneous cleanup for header include.
- Update README.md.
Change-Id: I835b80d4d3d7b291fa822a7a89dab225bf9171e9
Reviewed-on: https://chromium-review.googlesource.com/1105625
Reviewed-by: Samuel Huang <huangs@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Commit-Queue: Samuel Huang <huangs@chromium.org>
Cr-Commit-Position: refs/heads/master@{#569274}
NOKEYCHECK=True
GitOrigin-RevId: 9f0f325d6e2595deb8d50c77e0464946f3bf1ec1
|
|
The fuzzer is really smart; it discovered the worst case patch scenario
of alternating 16 byte regions of ZTF and Raw regions. This resulted in
a 310 B testcase (17 B source file) generating a 2.5 ~kB patch
(uncompressed) (470 B compressed). This is the absolute worst case
behavior which requires an intentionally badly designed archive/input.
In reality this would never occur with valid binaries. It is good to
know that this case exists but there isn't much that can be done to
prevent it in Zucchini so we can just disable this check.
Two solutions to this could be:
1. Make Zucchini smart enough to try multiple patches, compare the
compressed size and choose the best option.
2. Ignore it and in infra compare:
- Compressed ensemble patch
- Compressed raw patch
- Compressed image
Then just ship the smallest.
Option 1 adds a lot of complexity. Ideally, Zucchini should remain
naive with regards to generating compressed patches so that the infra
can choose the preferred compression and keep Zucchini fast.
The case for making the infra smarter is compelling and probably the
solution to pursue. However, because we have control over the input
binaries and this case will realistically not occur it isn't a
priority.
Bug: 848503
Change-Id: Ic505db49fd89f12dbd1eb5b100a59832f6054b2e
Reviewed-on: https://chromium-review.googlesource.com/1082008
Reviewed-by: Samuel Huang <huangs@chromium.org>
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563662}
NOKEYCHECK=True
GitOrigin-RevId: 54bf0d1da91c3db9a3682b840f7c76faa93c0072
|
|
As discussed in the tracking bug this covers buffer_sink and
patch_writer for fuzzing by serializing the generated patch data into a
buffer. Locally this increased fuzzing coverage by ~2%. exec/s for ZTF
files is still > 1500 which is sufficient for ClusterFuzz performance.
Eventually fuzzing should be added for:
- imposed_ensemble_matcher
- disassembler_dex
But is out of scope for Windows Launch.
It may also be worth adding an apply seed for a tiny Windows binary.
Chromium doesn't contain one small enough so we may need to make a
custom test binary to patch (likely Hello World or similar).
Bug: 835341
Change-Id: Id7208f30b09cd7443287cfe10f8ef1fcda6327d1
Reviewed-on: https://chromium-review.googlesource.com/1076949
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563266}
NOKEYCHECK=True
GitOrigin-RevId: 49613c27bcd84170395588a6d93fc009eb988658
|
|
This is part of a series of Fuzzers to be added to Zucchini for
security review. This tests the full patch application logic
exercising the patch reader and apply process. It covers ~33% of code
in 1000000 runs. The bulk of remaining code ~40% is covered by ZTF Gen
Fuzzer. With the remainder (~30%) being for DEX Disassembly (not in
launch scope), patch serialization (trusted input), and other
testing/debugging/error handling code which isn't triggered.
This already found a couple bugs fixed in
https://chromium-review.googlesource.com/c/chromium/src/+/1072272
With the supplied seed corpus the fuzzer reaches approximately 12000
execs/s.
The file format for the seed is a FilePair proto of a ZTF base file
and a patch file as used in Raw Apply. This reuses the same generator
and fuzzer as Raw Apply as the type of application is encoded in the
patch itself.
Bug: 835341
Change-Id: I00f28c768a6e1c7b8c5e95979b279d64785ef515
Reviewed-on: https://chromium-review.googlesource.com/1072231
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562260}
NOKEYCHECK=True
GitOrigin-RevId: 7206487ebd05fd4f30226ec59b730bb41c5013f2
|
|
This is part of a series of Fuzzers to be added to Zucchini for
security review. This tests the full patch generation logic
exercising the patch writer and gen process. It covers ~44% of code in
100000 runs. The remaining code is split between ZTF Apply Fuzzer
(~30%) and the aggregate of DEX Disassembly (not in launch scope),
patch serialization (trusted input), and other testing/debugging/error
handling code which isn't triggered.
With the supplied seed corpus the fuzzer reaches approximately 4000
execs/s.
The file format for the seed is a FilePair proto of a ZTF base file
and a ZTF updated file as used in Raw Gen.
Also fix bug where wrong fuzzer was running for apply.
Bug: 835341
Change-Id: Ib99dd70ba01820b874d72fecb2b543ea7082f649
Reviewed-on: https://chromium-review.googlesource.com/1072229
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561978}
NOKEYCHECK=True
GitOrigin-RevId: 8b5e3a4b59cfc86fc888726e29dea5d9cb1c1a09
|
|
zucchini_raw_apply_seed is failing on ToTWinCFI this disables building
the target on Windows. A first attempt at a fix failed so until a better
method of diagnosing the problem is determined we will disable it as it
wasn't used by ClusterFuzz anyway (Linux only).
Bug: 844826
Change-Id: I47820432ddb94334006ee727120a242684152b97
Reviewed-on: https://chromium-review.googlesource.com/1072506
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561641}
NOKEYCHECK=True
GitOrigin-RevId: 77d87185250262afe6e43367fd2d95aef5eb186e
|
|
This is an attempted fix for ToTWinCFI. Looks like sys.executable logic
isn't working well on some Python configurations so directly using
sys.executable.
If this doesn't fix ToTCFI I'll revert the previous change until I can
determine what is happening:
https://chromium.googlesource.com/chromium/src/+/8e7c08d3d11c61d08ad05d3ebc283aa2d6bf7c91
Thanks to thakis@ for solution suggestion.
Bug: 844826
Change-Id: Ie1cb36fe19401e954cb84bc905cd95c9eefa1622
Reviewed-on: https://chromium-review.googlesource.com/1072159
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Cr-Commit-Position: refs/heads/master@{#561567}
NOKEYCHECK=True
GitOrigin-RevId: ee3c0b0a933b9d9466620186bfeff8f62ea0af5f
|
|
This is part of a series of Fuzzers to be added to Zucchini for security
review. This tests the raw data patch generation logic exercising the
patch writer and gen process. It only covers ~20% of code in 100000
executions as the bulk of the remaining code is associated with the much
more complex and expensive to fuzz reference related code.
With the supplied seed corpus the fuzzer reaches approximately 5000
execs/s. There were no bug/stability issues found in raw gen.
The file format for the seed is a FilePair proto as used in raw Apply;
however, it is static so doesn't need regeneration. The files within
the FilePair seed are ZTF (Zucchini Text Format) files based on the code
that will be landed in:
https://chromium-review.googlesource.com/c/chromium/src/+/1056147
This way the source files can be reused in the ZTF gen and apply
fuzzers which will be used to more efficiently fuzz the reference
handling code in Zucchini.
Remaining Security Review Fuzzers (For Windows Launch)
- ZTF Gen
- ZTF Apply
(ZTF = Zucchini text format)
Note that suffix array while originally discussed is already implictly
fuzzed by this fuzzer and as such shouldn't require a standalone
fuzzer.
Other remaining fuzzers (Not shipped so non-blocking of Windows Launch)
- Disassembler DEX
- Disassembler ELF (when merged from Trunk)
To create the seed file pair run the following from
components/zucchini/fuzzers/
./create_seed_file_pair.py ../../../out/Release/protoc \
testdata/old.ztxt testdata/new.ztxt testdata/seed_proto.bin
Note: you need to first build protoc in out/Release/
Bug: 835341
Change-Id: I1bf5c2a4251093bbf5bfc92904afc376a2832dbd
Reviewed-on: https://chromium-review.googlesource.com/1062412
Commit-Queue: Calder Kitagawa <ckitagawa@chromium.org>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#559676}
NOKEYCHECK=True
GitOrigin-RevId: 133fb1221e6c293dc43ff438567b2834b2e5c798
|
|
This is a reland of f4a598ff5adfe27f8153bd36984ee9cb549f99e9
Windows cannot resolve #!/usr/bin/env python depending on how it is
configured. To fix this explicitly use python in the subprocess call.
Interestingly, the Tryjobs didn't catch this and only the official
build waterfall does...
Original change's description:
> [Zucchini] (raw) Apply fuzzer
>
> This is part of a series of Fuzzers to be added to Zucchini for
> security review. This tests the raw data patch application logic
> exercising the patch reader and apply process. It only covers ~20%
> of code in 100000 executions as the bulk of the remaining code is
> associated with the much more complex and expensive to fuzz reference
> related code.
>
> With the supplied seed corpus the fuzzer reaches approximately 11000
> execs/s.
>
> This found a couple bugs which are fixed in:
> https://chromium-review.googlesource.com/c/chromium/src/+/1028575
>
>
> Bug: 835341
> Change-Id: Idc1d862bfaa6eb6313f39e10536f4750c05ab863
> Reviewed-on: https://chromium-review.googlesource.com/1028570
> Commit-Queue: Calder Kitagawa <ckitagawa@google.com>
> Reviewed-by: Samuel Huang <huangs@chromium.org>
> Reviewed-by: Greg Thompson <grt@chromium.org>
> Reviewed-by: Max Moroz <mmoroz@chromium.org>
> Reviewed-by: Jonathan Metzman <metzman@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#557185}
Bug: 835341
Change-Id: I24e94dd0c2035d84c84636f0a0a30756ae7f0c36
Reviewed-on: https://chromium-review.googlesource.com/1052567
Commit-Queue: Calder Kitagawa <ckitagawa@google.com>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557286}
NOKEYCHECK=True
GitOrigin-RevId: 8e7c08d3d11c61d08ad05d3ebc283aa2d6bf7c91
|
|
This reverts commit f4a598ff5adfe27f8153bd36984ee9cb549f99e9.
Reason for revert: Does not compile on Win 64, see
https://ci.chromium.org/buildbot/chromium/Win%20x64/22249
Original change's description:
> [Zucchini] (raw) Apply fuzzer
>
> This is part of a series of Fuzzers to be added to Zucchini for
> security review. This tests the raw data patch application logic
> exercising the patch reader and apply process. It only covers ~20%
> of code in 100000 executions as the bulk of the remaining code is
> associated with the much more complex and expensive to fuzz reference
> related code.
>
> With the supplied seed corpus the fuzzer reaches approximately 11000
> execs/s.
>
> This found a couple bugs which are fixed in:
> https://chromium-review.googlesource.com/c/chromium/src/+/1028575
>
>
> Bug: 835341
> Change-Id: Idc1d862bfaa6eb6313f39e10536f4750c05ab863
> Reviewed-on: https://chromium-review.googlesource.com/1028570
> Commit-Queue: Calder Kitagawa <ckitagawa@google.com>
> Reviewed-by: Samuel Huang <huangs@chromium.org>
> Reviewed-by: Greg Thompson <grt@chromium.org>
> Reviewed-by: Max Moroz <mmoroz@chromium.org>
> Reviewed-by: Jonathan Metzman <metzman@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#557185}
TBR=huangs@chromium.org,mmoroz@chromium.org,grt@chromium.org,metzman@chromium.org,ckitagawa@google.com
Change-Id: Ia1790a01d3f31f25b243ce6e4ec5b52e423e3f6e
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 835341
Reviewed-on: https://chromium-review.googlesource.com/1052287
Reviewed-by: vitaliii <vitaliii@chromium.org>
Commit-Queue: vitaliii <vitaliii@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557196}
NOKEYCHECK=True
GitOrigin-RevId: 0101ff77e665851a5fab50b19427cbb5cdb54954
|
|
This is part of a series of Fuzzers to be added to Zucchini for
security review. This tests the raw data patch application logic
exercising the patch reader and apply process. It only covers ~20%
of code in 100000 executions as the bulk of the remaining code is
associated with the much more complex and expensive to fuzz reference
related code.
With the supplied seed corpus the fuzzer reaches approximately 11000
execs/s.
This found a couple bugs which are fixed in:
https://chromium-review.googlesource.com/c/chromium/src/+/1028575
Bug: 835341
Change-Id: Idc1d862bfaa6eb6313f39e10536f4750c05ab863
Reviewed-on: https://chromium-review.googlesource.com/1028570
Commit-Queue: Calder Kitagawa <ckitagawa@google.com>
Reviewed-by: Samuel Huang <huangs@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Jonathan Metzman <metzman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#557185}
NOKEYCHECK=True
GitOrigin-RevId: f4a598ff5adfe27f8153bd36984ee9cb549f99e9
|
|
More fuzzers need to be added to Zucchini for launch including
adding support for protobuf based fuzzers. To facilitate this a new
fuzzers/ subdirectory will help to separate Zucchini from its fuzz
related infrastructure.
Bug: 835341
Change-Id: Ib18bfe9bb0b0050e94fa7bdca22fb99c735d9141
Reviewed-on: https://chromium-review.googlesource.com/1026475
Reviewed-by: Samuel Huang <huangs@chromium.org>
Commit-Queue: Calder Kitagawa <ckitagawa@google.com>
Cr-Commit-Position: refs/heads/master@{#553254}
NOKEYCHECK=True
GitOrigin-RevId: 4725b4fbb75b0f4b2dda8f56e644ca6ef546cd0e
|
