This CL upgrades OpenSSL to 1.0.1e with a twist :-)

It introduces a new way to completely automate the process of importing the sources from Android,
applying Chromium-specific patches, generating a 64-bit configuration header, and a gyp-specific build file.

For all details, please read README.chromium.
Review URL: https://codereview.chromium.org/12573005

git-svn-id: http://src.chromium.org/svn/trunk/deps/third_party/openssl@188714 4ff67af0-8c30-449e-8e8b-ad334ec8d88c

1 files changed, 144 insertions, 70 deletions

diff --git a/README.chromium b/README.chromium
index 6fba58f..6b55237 100644
--- a/README.chromium
+++ b/README.chromium
@@ -1,6 +1,6 @@
 Name: openssl
 URL: http://openssl.org/source/
-Version: 1.0.1c
+Version: 1.0.1e
 License: BSDish
 License File: openssl/LICENSE
 License Android Compatible: yes
@@ -24,94 +24,168 @@ the following cases:
 
      http://dev.chromium.org/spdy/running_flipinmemserver   
 
-This means that the library must be built, at a minimum, for Android, Linux
-and Darwin systems.
+This means that the library must be built for these systems:
 
-  Android/ARM Android/x86 Linux/x86 Linux/x86_64 Darwin/x86 Darwin/x86_64
+  Android/ARM
+  Android/x86
+  Linux/x86
+  Linux/x86_64
+  Darwin/x86
+  Darwin/x86_64
 
-This source shall track the state of the Android platform's openssl version
-of the library, with some important details listed here:
+Whenever you change it, try to rebuild Chromium for all these systems.
 
- - The Android version lives under $ANDROID/external/openssl, while
-   the Chromum version (these files) live under
-   $CHROMIUM/src/third_party/openssl.
+**************************************************************************
+Automatic generation of source tree.
 
- - The Android version corresponds to the upstream original sources with
-   the patches under $ANDROID/external/openssl/patches applied. It also
-   does not include many files and directories that are not necessary for
-   the Android build.
+Most of the sources in this directory are auto-generated and come from
+the Android version of the OpenSSL sources, with a few Chromium-specific
+patches applied.
 
- - The Chromium version corresponds to the upstream original sources with
-   the patches under $CHROMIUM/src/third_party/openssl/patches/ applied.
-   Most of these patches come from the Android tree, with a few additional
-   ones.
+Said Android sources are themselves a patched subset of the official
+OpenSSL release sources, generated by a special import script.
 
-   The file patches/README lists the purpose of each Android patch.
-   Additional Chromium-specific patches also exist and are described
-   at the end of this document.
+To update the sources for Chromium, one has to modify
+openssl-chromium.config or the content of patches.chromium/ then run:
 
- - The Chromium openssl.gyp tries to match the Android.mk when it comes
-   to listing all source files. There is no direct mapping due to many
-   differences in their structure. To make this slightly easier, this file
-   doesn't use the convention of listing *all* source files under openssl,
-   then conditionally removing them for an Android build.
+  ./import_from_android.sh
 
-   Instead, only the files needed for the build are listed in 'sources'.
-   Given that there is a very large number of files that are not used in
-   the build, this simplified the task of keeping both files in sync.
+Before doing that, you should understand how everything works:
 
- - The Android tree also includes auto-generated assembly files for
-   ARM, x86 and MIPS, used to speed up some crypto computations. They
-   are generated by the $ANDROID/external/openssl/import_openssl.sh
-   script (which itself invokes a bunch of Perl generations scripts)
-   and have a file extension of .s
+  1) Android-specific files are taken from a given commit from the
+     AOSP git servers. See how 'openssl-chromium.config' defines the
+     following variables:
 
-   These files are also copied into the Chromium tree, with an .S file
-   extension, because they require being sent to the C preprocessor
-   before the assembler.
+       ANDROID_OPENSSL_GIT_SOURCE  -> point to source git server.
+       ANDROID_OPENSSL_GIT_COMMIT  -> point to git commit
 
-   The original Android.mk uses a feature not supported by gyp, which is:
+  2) All downloaded Android-specific files are placed under the openssl/
+     sub-directory. The most important files are the following:
 
-     LOCAL_AS_FLAGS := -x assembler-with-cpp
+      openssl/openssl.version
+          Configuration file telling which upstream version of
+          OpenSSL sources to use.
 
-   It tells the build system to apply the '-x assembler-with-cpp' flag
-   only to assembler files.
+      openssl/patches/
+          Directory containing several Android-specific patches to
+          apply to the official OpenSSL sources to create the
+          Android ones. See openssl/patches/README for a description
+          of what each of these patches do.
 
-   By using .S instead, the build 'just works' on Chromium.
+      openssl/openssl.config
+          Configuration file describing which build-time options
+          to enable, what patches to apply, which source files to compile
+          (including CPU architecture-specific variants), and which
+          sources to keep in the final source directory.
 
- - The Android build holds all configuration in
-   external/openssl/android-config.mk which is a Makefile fragment used
-   to define compiler flags that define configuration macros, to be used
-   when building *and* using the library.
+      openssl/import_openssl.sh
+          Import script used to regenerate all other Android-specific
+          source files, based on the configuration files above
+          and a tarball of the official OpenSSL source release.
 
-   The Chromium version uses config/<name>/openssl/opensslconf.h instead,
-   where name can be 'android', 'piii' or 'k8', corresponding to the
-   Android, Posix/ia32 or Posix/x64 builds.
+     For example, to rebuild the full Android source tree (without any
+     Chromium patches), one would do something like:
 
-   These headers hold the definition of these configuration variables.
-   Their parent directory must appear *before* openssl/include when
-   building and using the library. This is taken care of by openssl.gyp
-   here.
+        cd openssl/
+        ./import_openssl.sh import /path/to/openssl-<version>.tar.gz
 
-   Due to this, the Chromium tree also does not include obsolete copies
-   of opensslconf.h which appear in the Android source tree (and do not
-   seem to be really included during the build).
+     where <version> matches the definition found in 'openssl.version'.
 
-***************************************************************************
-The following patches are needed to compile this openssl on Chromium and
-pass the related net unit tests. They are applied on top of the
-Android-specific patches described under patches/README:
+  3) Chromium adds a few of its own files:
 
-x509_hash_name_algorithm_change.patch
+    openssl-chromium.config
+        Configuration file which indicates:
+          - The reference Android OpenSSL git repository and commit.
+          - The download location of official OpenSSL source tarballs.
+          - The corresponding SHA-1 sum, for sanity checking.
 
-There are many symbolic links under /etc/ssl/certs created by using hash
-of the PEM certificates in order for OpenSSL to find those certificates.
-Openssl has a tool to help you create hash symbolic links (tools/c_rehash).
-However the new openssl changed the hash algorithm. Unless you
-compile/install the latest openssl library and re-create all related
-symbolic links, the new openssl can not find some certificates because
-the links of those certificates were created by using old hash algorithm,
-which causes some tests failed. This patch gives a way to find a
-certificate according to its hash by using both new algorithm and old
-algorithm. crbug.com/111045 is used to track this issue.
+    patches.chromium/
+        A set of additional patches to apply to the openssl/ tree
+        after it has been downloaded from the Android git repository.
 
+        These patches are applied _before_ import_openssl.sh is run to
+        re-generate the final set of sources. This allows modifying the
+        content of any Android configuration file easily.
+
+    openssl.gyp
+        A gyp build file for the library. Manually maintained, this file
+        includes openssl.gypi below.
+
+    openssl.gypi
+        An *auto-generated* gyp include file that contains the required
+        definitions used to describe the library's sources to the
+        Chromium build system. Its content mirrors openssl/openssl.config
+        in a gyp-compatible way.
+
+    config/x64/openssl/opensslconf.h
+        Another *auto-generated* file used for 64-bit builds of the library
+        only. This is required for correctness because the Android sources
+        only come with a single generic header which is tailored for
+        32-bit builds. Using the latter results either in a broken build,
+        or even worse, in a library that doesn't work correctly.
+
+        The content of this file is a simple copy of
+        openssl/include/openssl/opensslconf.h, with a few lines
+        altered to reflect that the target has 64-bit types.
+
+    import_from_android.sh
+        The top-level script that will automatically perform the full
+        Chromium download + patching + import + auto-generation process.
+
+
+More specifically, calling 'import_from_android.sh' will do the following:
+
+  1) Download a specific Android commit from AOSP git servers to openssl/
+  2) Download the corresponding official OpenSSL release tarball.
+  3) Sainty check its SHA-1 against a hard-coded value.
+  4) Apply chromium-specific patches.
+  5) Re-run the Android 'import_openssl.sh' script.
+  6) Auto-generate config/x64/openssl/opensslconf.h
+  7) Auto-generate openssl.gypi
+
+Once the script is done, all you need to do is launch gyp again, rebuild
+and run unit tests. Use the --verbose option to see what the script does,
+or --help to see a detailed scription and a list of valid options.
+
+**************************************************************************
+Chromium-specific patches:
+
+The list of Chromium-specific patches to apply to the Android tree is
+located in patches.chromium/. Currently this consists of:
+
+  enable-dtls1.patch:
+    Enable DTLSv1, which is disabled by default in the Android platform
+    configuration.
+
+**************************************************************************
+Adding new Chromium patches:
+
+In the event you need to add a new Chromium-specific patch, follow this
+procedure:
+
+  1) Use the --temp-dir option to download everything to a known directory
+     (by default, import_from_android.sh downloads everything into a
+     temporary directory that is erased when the script exits, even in
+     case of error).
+
+       ./import_from_android.sh --temp-dir=/tmp/aaa
+
+  2) Save the "original" Android sources:
+
+       cp -rp /tmp/aaa/build/android-openssl /tmp/aaa/build/android-openssl.orig
+
+  3) Modify the content of /tmp/aaa/build/android-openssl appropriately.
+     You do *not* have to run 'import_openssl.sh'
+
+  4) Create new patch:
+
+     (cd /tmp/aaa/build && diff -burN android-openssl.orig android-openssl) > patches.chromium/my-new-change.patch
+
+  5) Re-run the script:
+
+      ./import_from_android.sh
+
+Generally speaking, consider sending your patch directly to the Android
+open-source review servers too. Once submitted there, you can update
+the git commit in openssl-chromium.org and remove your local patch in
+one new CL.