Swift ideal integration (#6312)

* Helper flags for swift compilation

* Documentation for swift project integration

* Adds swift to the languages with coverage

* Only thread sanitizer is supported

* Fixes swift coverage target compilation

* fixup flags facotring

* swift: run on new ubuntu

* fixup

* swift: right copy for symbolizer

16 files changed, 159 insertions, 48 deletions

diff --git a/docs/getting-started/new-project-guide/swift.md b/docs/getting-started/new-project-guide/swift.md
new file mode 100644
index 000000000..dbb127b03
--- /dev/null
+++ b/docs/getting-started/new-project-guide/swift.md
@@ -0,0 +1,78 @@
+---
+layout: default
+title: Integrating a Swift project
+parent: Setting up a new project
+grand_parent: Getting started
+nav_order: 1
+permalink: /getting-started/new-project-guide/swift/
+---
+
+# Integrating a Swift project
+{: .no_toc}
+
+- TOC
+{:toc}
+---
+
+The process of integrating a project written in Swift with OSS-Fuzz is very similar
+to the general
+[Setting up a new project]({{ site.baseurl }}/getting-started/new-project-guide/)
+process. The key specifics of integrating a Swift project are outlined below.
+
+## Project files
+
+First, you need to write a Swift fuzz target that accepts a stream of bytes and
+calls the program API with that. This fuzz target should reside in your project
+repository.
+
+The structure of the project directory in OSS-Fuzz repository doesn't differ for
+projects written in Swift. The project files have the following Swift specific
+aspects.
+
+### project.yaml
+
+The `language` attribute must be specified.
+
+```yaml
+language: swift
+```
+
+The only supported fuzzing engine is `libfuzzer`
+
+The supported sanitizers are and `address`, `thread`
+
+[Example](https://github.com/google/oss-fuzz/blob/2a15c3c88b21f4f1be2a7ff115f72bd7a08e34ac/projects/swift-nio/project.yaml#L9):
+
+```yaml
+fuzzing_engines:
+  - libfuzzer
+sanitizers:
+  - address
+  - thread
+```
+
+### Dockerfile
+
+The Dockerfile should start by `FROM gcr.io/oss-fuzz-base/base-builder-swift`
+instead of using the simple base-builder
+
+### build.sh
+
+A `precompile_swift` generates an environment variable `SWIFTFLAGS`
+This can then be used in the building command such as `swift build -c release $SWIFTFLAGS`
+
+
+A usage example from swift-protobuf project is
+
+```sh
+. precompile_swift
+# build project
+cd FuzzTesting
+swift build -c debug $SWIFTFLAGS
+
+(
+cd .build/debug/
+find . -maxdepth 1 -type f -name "*Fuzzer" -executable | while read i; do cp $i $OUT/"$i"-debug; done
+)
+
+```
diff --git a/infra/base-images/base-builder-swift/Dockerfile b/infra/base-images/base-builder-swift/Dockerfile
index 1554fd7f9..2b063bb2f 100644
--- a/infra/base-images/base-builder-swift/Dockerfile
+++ b/infra/base-images/base-builder-swift/Dockerfile
@@ -14,6 +14,8 @@
 #
 ################################################################################
 
-FROM gcr.io/oss-fuzz-base/base-builder-new:xenial
+FROM gcr.io/oss-fuzz-base/base-builder
 
 RUN install_swift.sh
+
+COPY precompile_swift /usr/local/bin/
diff --git a/infra/base-images/base-builder-swift/precompile_swift b/infra/base-images/base-builder-swift/precompile_swift
new file mode 100755
index 000000000..366b0dfb5
--- /dev/null
+++ b/infra/base-images/base-builder-swift/precompile_swift
@@ -0,0 +1,34 @@
+#!/bin/bash -eu
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+################################################################################
+
+cp /usr/local/bin/llvm-symbolizer-swift $OUT/llvm-symbolizer
+
+export SWIFTFLAGS="-Xswiftc -parse-as-library -Xswiftc -static-stdlib -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib"
+if [ "$SANITIZER" = "coverage" ]
+then
+    export SWIFTFLAGS="$SWIFTFLAGS -Xswiftc -profile-generate -Xswiftc -profile-coverage-mapping -Xswiftc -sanitize=fuzzer"
+else
+    export SWIFTFLAGS="$SWIFTFLAGS -Xswiftc -sanitize=fuzzer,$SANITIZER --sanitize=$SANITIZER"
+fi
+
+for f in $CFLAGS; do
+    export SWIFTFLAGS="$SWIFTFLAGS -Xcc=$f"
+done
+
+for f in $CXXFLAGS; do
+    export SWIFTFLAGS="$SWIFTFLAGS -Xcxx=$f"
+done
diff --git a/infra/base-images/base-builder/install_swift.sh b/infra/base-images/base-builder/install_swift.sh
index c88a46caf..d88a7b5cd 100755
--- a/infra/base-images/base-builder/install_swift.sh
+++ b/infra/base-images/base-builder/install_swift.sh
@@ -16,17 +16,31 @@
 ################################################################################
 
 
-SWIFT_PACKAGES="wget binutils libc6-dev libcurl3 libedit2 libgcc-5-dev libpython2.7 libsqlite3-0 libstdc++-5-dev libxml2 pkg-config tzdata zlib1g-dev"
+SWIFT_PACKAGES="wget \
+          binutils \
+          git \
+          gnupg2 \
+          libc6-dev \
+          libcurl4 \
+          libedit2 \
+          libgcc-9-dev \
+          libpython2.7 \
+          libsqlite3-0 \
+          libstdc++-9-dev \
+          libxml2 \
+          libz3-dev \
+          pkg-config \
+          tzdata \
+          zlib1g-dev"
 SWIFT_SYMBOLIZER_PACKAGES="build-essential make cmake ninja-build git python3 g++-multilib binutils-dev zlib1g-dev"
 apt-get update && apt install -y $SWIFT_PACKAGES && \
   apt install -y $SWIFT_SYMBOLIZER_PACKAGES --no-install-recommends  
 
 
-wget https://swift.org/builds/swift-5.3.3-release/ubuntu1604/swift-5.3.3-RELEASE/swift-5.3.3-RELEASE-ubuntu16.04.tar.gz
-tar xzf swift-5.3.3-RELEASE-ubuntu16.04.tar.gz
-cp -r swift-5.3.3-RELEASE-ubuntu16.04/usr/* /usr/
-rm -rf swift-5.3.3-RELEASE-ubuntu16.04.tar.gz
-
+wget https://swift.org/builds/swift-5.4.2-release/ubuntu2004/swift-5.4.2-RELEASE/swift-5.4.2-RELEASE-ubuntu20.04.tar.gz
+tar xzf swift-5.4.2-RELEASE-ubuntu20.04.tar.gz
+cp -r swift-5.4.2-RELEASE-ubuntu20.04/usr/* /usr/
+rm -rf swift-5.4.2-RELEASE-ubuntu20.04.tar.gz
 # TODO: Move to a seperate work dir
 git clone --depth 1 https://github.com/llvm/llvm-project.git
 cd llvm-project
@@ -42,7 +56,7 @@ cmake -G "Ninja" \
     -DLLVM_BUILD_TESTS=OFF \
     -DLLVM_INCLUDE_TESTS=OFF llvm
 ninja -j$(nproc) llvm-symbolizer
-cp bin/llvm-symbolizer $OUT/
+cp bin/llvm-symbolizer /usr/local/bin/llvm-symbolizer-swift
 
 cd $SRC
 rm -rf llvm-project llvmsymbol.diff
diff --git a/infra/base-images/base-builder/llvmsymbol.diff b/infra/base-images/base-builder/llvmsymbol.diff
index c53c978df..70181bf39 100644
--- a/infra/base-images/base-builder/llvmsymbol.diff
+++ b/infra/base-images/base-builder/llvmsymbol.diff
@@ -1,8 +1,8 @@
 diff --git a/llvm/lib/DebugInfo/Symbolize/CMakeLists.txt b/llvm/lib/DebugInfo/Symbolize/CMakeLists.txt
-index acfb3bd0e..5c4cf9763 100644
+index acfb3bd0e..a499ee2e0 100644
 --- a/llvm/lib/DebugInfo/Symbolize/CMakeLists.txt
 +++ b/llvm/lib/DebugInfo/Symbolize/CMakeLists.txt
-@@ -12,4 +12,12 @@ add_llvm_component_library(LLVMSymbolize
+@@ -12,4 +12,11 @@ add_llvm_component_library(LLVMSymbolize
    Object
    Support
    Demangle
@@ -10,14 +10,13 @@ index acfb3bd0e..5c4cf9763 100644
 +
 +  LINK_LIBS
 +  /usr/lib/swift_static/linux/libswiftCore.a
-+  /usr/lib/swift_static/linux/libswiftImageInspectionShared.a
 +  /usr/lib/swift_static/linux/libicui18nswift.a
 +  /usr/lib/swift_static/linux/libicuucswift.a
 +  /usr/lib/swift_static/linux/libicudataswift.a
 +  /usr/lib/x86_64-linux-gnu/libstdc++.so.6
 +)
 diff --git a/llvm/lib/DebugInfo/Symbolize/Symbolize.cpp b/llvm/lib/DebugInfo/Symbolize/Symbolize.cpp
-index 4c3f3a3767e1..aa7b9f0f5abb 100644
+index fb4875f79..0030769ee 100644
 --- a/llvm/lib/DebugInfo/Symbolize/Symbolize.cpp
 +++ b/llvm/lib/DebugInfo/Symbolize/Symbolize.cpp
 @@ -36,6 +36,13 @@
@@ -34,7 +33,7 @@ index 4c3f3a3767e1..aa7b9f0f5abb 100644
  namespace llvm {
  namespace symbolize {
  
-@@ -632,6 +639,14 @@ LLVMSymbolizer::DemangleName(const std::string &Name,
+@@ -678,6 +685,14 @@ LLVMSymbolizer::DemangleName(const std::string &Name,
      free(DemangledName);
      return Result;
    }
diff --git a/infra/build/functions/build_and_run_coverage.py b/infra/build/functions/build_and_run_coverage.py
index 275a66d12..1195776d9 100755
--- a/infra/build/functions/build_and_run_coverage.py
+++ b/infra/build/functions/build_and_run_coverage.py
@@ -42,7 +42,7 @@ COVERAGE_BUCKET_NAME = 'oss-fuzz-coverage'
 LATEST_REPORT_INFO_CONTENT_TYPE = 'application/json'
 
 # Languages from project.yaml that have code coverage support.
-LANGUAGES_WITH_COVERAGE_SUPPORT = ['c', 'c++', 'go', 'jvm', 'rust']
+LANGUAGES_WITH_COVERAGE_SUPPORT = ['c', 'c++', 'go', 'jvm', 'rust', 'swift']
 
 
 class Bucket:  # pylint: disable=too-few-public-methods
diff --git a/infra/constants.py b/infra/constants.py
index d4b9f7d76..a323a4368 100644
--- a/infra/constants.py
+++ b/infra/constants.py
@@ -30,7 +30,7 @@ LANGUAGES = [
     'rust',
     'swift',
 ]
-LANGUAGES_WITH_COVERAGE_SUPPORT = ['c', 'c++', 'go', 'jvm', 'rust']
+LANGUAGES_WITH_COVERAGE_SUPPORT = ['c', 'c++', 'go', 'jvm', 'rust', 'swift']
 SANITIZERS = [
     'address', 'none', 'memory', 'undefined', 'dataflow', 'thread', 'coverage'
 ]
diff --git a/projects/grpc-swift/Dockerfile b/projects/grpc-swift/Dockerfile
index a34d30faf..8c56c0b2d 100644
--- a/projects/grpc-swift/Dockerfile
+++ b/projects/grpc-swift/Dockerfile
@@ -14,11 +14,7 @@
 #
 ################################################################################
 
-# Using Ubuntu 16.04 because of breakage on Ubuntu 20.04.
-# See https://github.com/google/oss-fuzz/issues/6291 for more details.
-FROM gcr.io/oss-fuzz-base/base-builder-swift:xenial
-# Delete line above and uncomment line below to upgrade to 20.04.
-# FROM gcr.io/oss-fuzz-base/base-builder-swift
+FROM gcr.io/oss-fuzz-base/base-builder-swift
 
 # specific to project
 RUN git clone --depth 1 https://github.com/grpc/grpc-swift
diff --git a/projects/grpc-swift/build.sh b/projects/grpc-swift/build.sh
index f9f415062..a27f3719c 100755
--- a/projects/grpc-swift/build.sh
+++ b/projects/grpc-swift/build.sh
@@ -15,25 +15,16 @@
 #
 ################################################################################
 
+. precompile_swift
 # build project
 cd FuzzTesting
-# Maybe we should have a helper script to set $SWIFT_FLAGS
-# for instance about -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION in -Xcc
-swift build -c debug -Xswiftc -sanitize=fuzzer,address \
-    -Xswiftc -parse-as-library -Xswiftc -static-stdlib \
-    -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib \
-    --sanitize=address -Xcc="-fsanitize=fuzzer-no-link,address" \
-    -Xcxx="-fsanitize=fuzzer-no-link,address"
+swift build -c debug $SWIFTFLAGS
 
 (
 cd .build/debug/
 find . -maxdepth 1 -type f -name "*Fuzzer" -executable | while read i; do cp $i $OUT/"$i"-debug; done
 )
-swift build -c release -Xswiftc -sanitize=fuzzer,address \
-    -Xswiftc -parse-as-library -Xswiftc -static-stdlib \
-    -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib \
-    --sanitize=address -Xcc="-fsanitize=fuzzer-no-link,address" \
-    -Xcxx="-fsanitize=fuzzer-no-link,address"
+swift build -c release $SWIFTFLAGS
 (
 cd .build/release/
 find . -maxdepth 1 -type f -name "*Fuzzer" -executable | while read i; do cp $i $OUT/"$i"-release; done
diff --git a/projects/grpc-swift/project.yaml b/projects/grpc-swift/project.yaml
index 2825267a3..2b5c23049 100644
--- a/projects/grpc-swift/project.yaml
+++ b/projects/grpc-swift/project.yaml
@@ -9,4 +9,5 @@ fuzzing_engines:
 - libfuzzer
 sanitizers:
 - address
+- thread
 main_repo: 'https://github.com/grpc/grpc-swift'
diff --git a/projects/swift-nio/Dockerfile b/projects/swift-nio/Dockerfile
index 19a34fc17..1e47174c0 100644
--- a/projects/swift-nio/Dockerfile
+++ b/projects/swift-nio/Dockerfile
@@ -14,11 +14,7 @@
 #
 ################################################################################
 
-# Using Ubuntu 16.04 because of breakage on Ubuntu 20.04.
-# See https://github.com/google/oss-fuzz/issues/6291 for more details.
-FROM gcr.io/oss-fuzz-base/base-builder-swift:xenial
-# Delete line above and uncomment line below to upgrade to 20.04.
-# FROM gcr.io/oss-fuzz-base/base-builder-swift
+FROM gcr.io/oss-fuzz-base/base-builder-swift
 
 # specific swift-nio
 RUN git clone --depth 1 https://github.com/google/fuzzing
diff --git a/projects/swift-nio/build.sh b/projects/swift-nio/build.sh
index 833efc816..1544e2b66 100755
--- a/projects/swift-nio/build.sh
+++ b/projects/swift-nio/build.sh
@@ -23,14 +23,14 @@ rm -Rf Sources/swift-nio-fuzz
 mkdir Sources/swift-nio-http1-fuzz
 cp $SRC/fuzz_http1.swift Sources/swift-nio-http1-fuzz/main.swift
 cp $SRC/Package.swift Package.swift
-# Maybe we should have a helper script to set $SWIFT_FLAGS
-# for instance about -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION in -Xcc
-swift build -c debug -Xswiftc -sanitize=fuzzer,address -Xswiftc -parse-as-library -Xswiftc -static-stdlib -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib  --sanitize=address -Xcc="-fsanitize=fuzzer-no-link,address"
+
+. precompile_swift
+swift build -c debug $SWIFTFLAGS
 (
 cd .build/debug/
 find . -maxdepth 1 -type f -name "*fuzz" -executable | while read i; do cp $i $OUT/"$i"-debug; done
 )
-swift build -c release -Xswiftc -sanitize=fuzzer,address -Xswiftc -parse-as-library -Xswiftc -static-stdlib -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib  --sanitize=address -Xcc="-fsanitize=fuzzer-no-link,address"
+swift build -c release $SWIFTFLAGS
 (
 cd .build/release/
 find . -maxdepth 1 -type f -name "*fuzz" -executable | while read i; do cp $i $OUT/"$i"-release; done
diff --git a/projects/swift-nio/project.yaml b/projects/swift-nio/project.yaml
index ea9e22737..6c9a01ddc 100644
--- a/projects/swift-nio/project.yaml
+++ b/projects/swift-nio/project.yaml
@@ -10,4 +10,5 @@ fuzzing_engines:
 - libfuzzer
 sanitizers:
 - address
+- thread
 main_repo: 'https://github.com/apple/swift-nio.git'
diff --git a/projects/swift-protobuf/Dockerfile b/projects/swift-protobuf/Dockerfile
index ade552a86..d8be91344 100644
--- a/projects/swift-protobuf/Dockerfile
+++ b/projects/swift-protobuf/Dockerfile
@@ -14,11 +14,7 @@
 #
 ################################################################################
 
-# Using Ubuntu 16.04 because of breakage on Ubuntu 20.04.
-# See https://github.com/google/oss-fuzz/issues/6291 for more details.
-FROM gcr.io/oss-fuzz-base/base-builder-swift:xenial
-# Delete line above and uncomment line below to upgrade to 20.04.
-# FROM gcr.io/oss-fuzz-base/base-builder-swift
+FROM gcr.io/oss-fuzz-base/base-builder-swift
 
 RUN git clone --depth 1 https://github.com/apple/swift-protobuf.git
 COPY build.sh $SRC
diff --git a/projects/swift-protobuf/build.sh b/projects/swift-protobuf/build.sh
index 7b2c6e00d..ee291dac4 100755
--- a/projects/swift-protobuf/build.sh
+++ b/projects/swift-protobuf/build.sh
@@ -15,14 +15,16 @@
 #
 ################################################################################
 
+
+. precompile_swift
 # build project
 cd FuzzTesting
-swift build -c debug -Xswiftc -sanitize=address,fuzzer -Xswiftc -parse-as-library -Xswiftc -static-stdlib -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib --sanitize=address
+swift build -c debug $SWIFTFLAGS
 (
 cd .build/debug/
 find . -maxdepth 1 -type f -name "Fuzz*" -executable | while read i; do cp $i $OUT/"$i"_debug; done
 )
-swift build -c release -Xswiftc -sanitize=address,fuzzer -Xswiftc -parse-as-library -Xswiftc -static-stdlib -Xswiftc -use-ld=/usr/bin/ld --static-swift-stdlib --sanitize=address
+swift build -c release $SWIFTFLAGS
 (
 cd .build/release/
 find . -maxdepth 1 -type f -name "Fuzz*" -executable | while read i; do cp $i $OUT/"$i"_release; done
diff --git a/projects/swift-protobuf/project.yaml b/projects/swift-protobuf/project.yaml
index 71794bcc2..002ee8726 100644
--- a/projects/swift-protobuf/project.yaml
+++ b/projects/swift-protobuf/project.yaml
@@ -9,4 +9,5 @@ fuzzing_engines:
 - libfuzzer
 sanitizers:
 - address
+- thread
 main_repo: 'https://github.com/apple/swift-protobuf.git'