bsdiff: support for lazy reading from extent files in bspatch

Previously, bspatch used to read the whole content of a file to be
patched into a memory buffer, only to copy this data while generating
the new file content, which is written to a second buffer.  The said
read buffer entails an unnecessary memory allocation requirement,
especially since it's being indexed almost linearly.  This behavior was
further extended to support extents (i.e. a list of <offset, length>
pairs), which are used extensively during Chrome OS updates.

This change introduces extent files, which let users open files through
a layer of extents as ordinary glibc file handles. This in turn allows
us to easily convert reads from a memory buffer into direct reads from
an extent file. Extent files are buffered on the outer level (done for
us by glibc), but otherwise use a system file descriptor for the
underlying I/O; this proved to be the most efficient combination when
applying actual update payloads. Since we are reading a single byte at
a time using fread(2), and since the program is decidedly
single-threaded, we shift to using _unlocked variants, which appear to
reduce the total update time significantly without otherwise affecting
memory/CPU footprint.

We expect this to cut bspatch's memory usage by nearly one half.  Note
that in general it is possible to use the same abstraction for
implementing direct writing to the target file; however, due to the way
we implement delta updates, there is risk that such writes might clobber
the read data, and so further support is needed to mark safe operations
(i.e. no read/write dependencies) as such.

This CL obsoletes the previous ebuild patch for handling extent
arguments, which is therefore removed. It also (i) gets rid of logic for
special handling of /dev/fd filenames, which is deemed redundant; (ii)
fixes the Makefile (via a separate patch) and changes the ebuild to use
it, for uniformity; (iii) updates the ebuild to EAPI version 4; (iv)
sets -Wall -Werror and fixes eliminates a warning due to bsdiff.c; (v)
enhances man pages for both bsdiff/bspatch.

BUG=chromium:229705
TEST=Passes update engine unittests with new bspatch
TEST=delta payload with BSDIFF operations updates correctly on x86-alex

Change-Id: I4bb4afa42e43279048093e7a7f0ef96406b0c9e0
Reviewed-on: https://gerrit.chromium.org/gerrit/49595
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Commit-Queue: Gilad Arnold <garnold@chromium.org>

1 files changed, 22 insertions, 0 deletions

diff --git a/extents.h b/extents.h
new file mode 100644
index 0000000..7af1458
--- /dev/null
+++ b/extents.h
@@ -0,0 +1,22 @@
+#ifndef __EXTENTS_H
+#define __EXTENTS_H
+
+#include "exfile.h"
+
+
+/* Parses a string representation |ex_str| and populates an array |ex_arr|
+ * consisting of |*ex_count_p| extents. The string is expected to be a
+ * comma-separated list of pairs of the form "offset:length". An offset may be
+ * -1 or a non-negative integer; the former indicates a sparse extent
+ * (consisting of zeros). A length is a positive integer.  If |ex_arr| is NULL,
+ * |*ex_count_p| is ignored and a new array is allocated based on the actual
+ * number of extents parsed.  Upon success, returns a pointer to the populated
+ * array of extents and stores the actual number of extents at the location
+ * pointed to be |ex_count_p| (if provided).  If the string parses correctly but
+ * the operation otherwise fails (allocation error, array too small), returns
+ * NULL but still store the number of parsed extents.  Otherwise, returns NULL
+ * and does not store anything. If a new array was allocated, then it should be
+ * deallocated with free(3). */
+ex_t *extents_parse(const char *ex_str, ex_t *ex_arr, size_t *ex_count_p);
+
+#endif /* __EXTENTS_H */