path: root/3.8/README.md

Android Clang/LLVM
==================

Platform Projects
-----------------

#### external/llvm
**Branch**: *aosp/dev*

* This branch tracks LLVM upstream directly and contains our Android-specific
patches that do not exist in upstream.

**Branch**: *aosp/master*

* This branch receives updates from the *aosp/dev* branch as a squashed single
commit.
Any patches submitted here need to be replayed in *aosp/dev* as well for
consistency with future rebases.

#### external/clang
**Branch**: *aosp/dev*

* This branch tracks Clang upstream directly and contains our Android-specific
patches that do not exist in upstream.

**Branch**: *aosp/master*

* This branch receives updates from the *aosp/dev* branch as a squashed single
commit.
Any patches submitted here need to be replayed in *aosp/dev* as well for
consistency with future rebases.

#### external/compiler-rt
**Branch**: *aosp/dev*

* This branch tracks compiler-rt upstream directly and contains our
Android-specific patches that do not exist in upstream.

**Branch**: *aosp/master*

* This branch receives updates from the *aosp/dev* branch as a squashed single
commit.
Any patches submitted here need to be replayed in *aosp/dev* as well for
consistency with future rebases.


**Note**: Similar branching strategies can be used for **external/libcxx** and
**external/libcxxabi**, but these projects are less divergent and may not need
special LLVM rebase behavior.


Development Flow
----------------

Rebases take place in the **external/** *aosp/dev* branches by merging
upstream CLs directly.
Conflicts are resolved manually and then a patch is produced to adapt
Android.mk files (as well as to add/delete any updated source files).
Prebuilts are then generated using these projects and placed into the proper
**prebuilts/clang/host** subprojects.

For example, these projects for LLVM 3.8 would be:

    prebuilts/clang/host/linux-x86/3.8
    prebuilts/clang/host/darwin-x86/3.8
    prebuilts/clang/host/windows-x86/3.8

In order to prepare for the actual rebase (including updating dependent
projects), we will copy each **external/** *aosp/dev* project to its
corresponding **external/** *aosp/master* project as a squashed single CL.
This makes rollbacks simpler, and it also reduces churn on the Android build
servers.
This also has the side effect of not spamming non-Android @google.com
committers to upstream LLVM projects, since their commits will be batched up
into a single copy commit on each tracked **external/** project.

Prebuilts for llvm-rs-cc and bcc\_compat also need to be generated for
prebuilts/sdk.
This is done by running **frameworks/rs/update\_rs\_prebuilts.sh** on both Mac
and Linux.
After this completes, the **prebuilts/sdk** project will have a prepared
branch/CL that can be uploaded for review/commit.


Fixing Bugs
-----------

If we find a host-side bug that needs to be fixed, it may trigger an update of
the host prebuilts (i.e. rebase).
Device-side fixes can be pushed directly to **external/** *aosp/master* and then
copied to **external/** *aosp/dev* to speed up the process (assuming that it
doesn’t affect the host builds).


Looking at Upstream
-------------------

Here are the remotes to add in your **external/** projects. You can synchronize
them with `git fetch upstream`.


    Clang: git remote add upstream http://llvm.org/git/clang.git
    LLVM:  git remote add upstream http://llvm.org/git/llvm.git
    compiler-rt: git remote add upstream http://llvm.org/git/compiler-rt.git


LLVM 3.8 era rebase process
---------------------------

Loop over llvm, clang, compiler-rt (in this order):

1. We are working from a separate untracked/merged branch called *aosp/dev*,
so we can’t really use repo start.

        git branch -D working_dev
        git checkout -b working_dev aosp/dev

2. **OPTIONAL FIXUPS**.
These aren't really necessary if you remember to always keep *aosp/dev* and
*aosp/master* synchronized otherwise, but very often someone will forget to
merge back a change.

  1. Grab the squashed commit that went into *aosp/master* and mark it
  committed to *aosp/dev* too.

     **Note**: If there were changes to *aosp/master* before the squashed
     commit, grab those changes (using step 2), before applying this step,
     and finally repeat step 2 for changes after the squashed commit.

          git branch -D clean_master
          git checkout -b clean_master <SHA_FOR_SQUASH>
          git checkout working_dev
          git merge -s ours clean_master
          git push aosp refs/heads/working_dev:refs/heads/dev
          git branch -D clean_master

  2. Grab all outstanding changes that went into *aosp/master* and put them
  into *aosp/dev* too.

          git branch -D clean_master
          git checkout -b clean_master aosp/master
          git checkout working_dev
          git merge clean_master
          git push aosp refs/heads/working_dev:refs/heads/dev
          git branch -D clean_master

3. Merge the upstream branch.
Use `git log upstream/master` to browse upstream commits and find a SHA.

        git merge <upstream_sha>

4. Fix conflicts.

5. Update build rules and commit that patch on top.

6. Test everything before pushing.

7. Submit your work to *aosp/dev*.

        git push aosp refs/heads/working_dev:refs/heads/dev

8. Squash your work for *aosp/master*.

        repo start update_38 .
        git merge --squash working_dev
        git commit -a
        repo upload .

9. Test everything before submitting the patch from the previous step.

10. Grab the squashed commit and replay it in *aosp/dev*.

        repo sync .
        git remote update
        git branch -D clean_master
        git checkout -b clean_master aosp/master
        git checkout working_dev

     Use -s ours to ensure that we skip the squashed set of changes
     If/when we forget this, we have to do it later

        git merge -s ours clean_master
        git push aosp refs/heads/working_dev:refs/heads/dev
        git branch -D clean_master

11. Clean up after our working branch.

        git checkout --detach
        git branch -D working_dev

This works better because we can keep full history in *aosp/dev*, while
maintaining easy reverts/commits through *aosp/master*.

Generating new prebuilts
------------------------

1. Build all of llvm/clang/compiler-rt.

        cd $ANDROID_BUILD_TOP/
        mmma external/llvm external/clang external/compiler-rt -j48

        cd $ANDROID_BUILD_TOP/prebuilts/clang/host/linux-x86/3.8
        ./build_toolchain.sh
        ./build_profile_rt.sh

2. Copy over our new prebuilts.
Note that these were built with the previous clang.

        ./update.sh

3. Now we need to actually rebuild everything with the new clang.
This includes clang itself!

        rm -rf out/
        ./build_toolchain.sh
        ./build_profile_rt.sh

4. Now we need to copy over our actual self-hosted prebuilts.

        ./update.sh

5. Rebuild/test everything one more time to ensure correctness.
There may be necessary fixups here, to handle .ll reading or other projects
where new warnings/errors are firing.

        m -j48 checkbuild

6. Upload the changes produced in **prebuilts/clang/host**.
This may entail more than a simple `git commit -a`, so look at `git status`
before finally uploading/committing.

        repo start updated_toolchain .
        git commit -a
        repo upload --cbr .

7. Update RenderScript prebuilts.

        cd $ANDROID_BUILD_TOP/frameworks/rs
        ./update_rs_prebuilts.sh

8. The prebuilts get copied to **prebuilts/sdk**, so we must upload the
relevant bits from there.

        cd $ANDROID_BUILD_TOP/prebuilts/sdk
        git commit -a
        repo upload .

9. Submit CLs.


Steps/Checklist for testing:
----------------------------

1. Do a Clang-only build and ensure it is not broken:

        USE_CLANG_PLATFORM_BUILD=true m -j48

2. Go to **external/llvm** and run `./android_test.sh` (no known failures
as of 2015-10-08).
3. Ensure successful build for all architectures: 32- and 64- bit ARM, x86 and
Mips.
4. Run 32- and 64- bit RenderScript CTS at least for ARM and AArch64.
5. Test RenderScript apps: RsTest, ImageProcessing, and finally
RSTest\_Compatlib in compatibility mode.
6. Test old APKs with rebased tools: grab the above apps from a different tree
(i.e. without the rebase), push them to a device with the rebased tools, and
test.
This ensures that the rebased BitcodeReader can read the output of old
BitcodeWriters.
7. Test new APKs on an old device: test freshly built APKs for
RSTest\_V{11,14,16}, and ImageProcessing\_2 on an old device (say Manta) and
ensure they pass.
This ensures that the rebase did not break the 2.9 and 3.2 BitcodeWriters.
8. Run ART host tests.
This was broken by a rebase once, and worth testing after every rebase.

        croot && cd art && mma -j40 test-art-host

9. Run ART device tests.

        croot && cd art && mma -j4 test-art-device


Checklist for CLs
-----------------

The following projects will almost always have CLs as a part of the rebase.
Depending on the changes in LLVM, there might be updates to other projects as
well.

* External projects
  * **external/clang**
  * **external/compiler-rt**
  * **external/llvm**
  * **frameworks/compile/mclinker**
* RenderScript projects
  * **frameworks/compile/libbcc**
  * **frameworks/compile/slang**
  * **frameworks/rs**
* Prebuilts
  * **prebuilts/clang/host/darwin-x86/**
  * **prebuilts/clang/host/linux-x86/**
  * **prebuilts/clang/host/windows-x86/**
  * **prebuilts/sdk**
* CTS tests
  * **cts/tests/tests/renderscript**
  * **cts/tests/tests/renderscriptlegacy**
  * **cts/tests/tests/rscpp**