Replace chroot based debootstrap with QEMU

To better support foreign debootstraps, and to debootstrap hermetically,
move away from using chroots. The chroot feature for foreign archs needs
qemu-user, and some versions of these tools crash randomly, which has
caused problems generating new filesystems.

This new method is quite a bit more complicated, but it also provides a
lot more validation. The user must specify a prebuilt kernel and
initramfs, and the initramfs is combined with one produced by the stage2
script, which allows the suite script to run under the kernel it will be
used with, which guarantees the combination will work.

Since we require the user to specify a kernel, we can also support
embedding the kernel, ramdisk (and for arm64 dtb) into the rootfs at
build time, because `uname -r` expands to the true kernel that will be
used.

The script supports building filesystems for non-virtual devices, but
the kernel provided must have at least virtio pci, blk, net and rng to
complete successfully.

These changes also finally enable a long requested capability to boot
the net tests root filesystem with a modular kernel, so that the GKI
kernel prebuilts can be used directly against it, without having to
build a custom "built-in" kernel equivalent. This in turn will enable
changes to run_net_test.sh to skip the kernel build and use cuttlefish
to host the test, which lets us integrate the testing with ATP.

Finally, these changes have been used to build Other OS root filesystems
for cuttlefish, and the root filesystem for the RockPi devices we are
using for ARM64 lab testing.

The bootstrapping process is conducted in three stages:

1) The debootstrap tool is run in --foreign mode offline to download
   the needed packages for installation. However, no installation is
   done and the binaries are simply unpacked, which works without
   needing qemu-user to be installed.

2) QEMU is used to boot the user-provided kernel and a basic root
   filesystem from debootstrap with a stage1.sh control script. The root
   filesystem (512M max) is passed as a ramdisk, so no special drivers
   are required. The stage1.sh script loads some virtual machine modules
   and sets up the psuedo-filesystems, like the old debootstrap second
   stage chroot did. Then, chains to stage2.sh to complete the
   debootstrap second stage on *another* copy of the root filesystem
   from virtio_blk. This process generates an initrd fragment which can
   be combined with the user's initramfs to boot the final system.

3) Finally, the "installed" root filesystem is booted with the Debian
   initial ramdisk and user-provided kernel modules. Once the root
   filesystem is mounted, the suite script runs ("stage3"). This script
   makes the usual customizations from our original chroot method.

If the -e option is specified, the kernel and "baked" initramfs are
stored in /boot. This process works for cuttlefish and rockpi but for
the normal net tests setup, embedding is not preferred (as the kernel
will always be changing). The grub2-common package is also installed to
create a grub.cfg for bootloader config chaining on cuttlefish.

Change-Id: I53f68031103e95b4b3c26da286195260ccd12f88

5 files changed, 305 insertions, 56 deletions

diff --git a/net/test/build_rootfs.sh b/net/test/build_rootfs.sh
index d6a2d91..4034f4e 100755
--- a/net/test/build_rootfs.sh
+++ b/net/test/build_rootfs.sh
@@ -20,17 +20,11 @@ set -u
 
 SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)
 
-# Make sure we're in C locale so build inside chroot does not complain
-# about missing files
-unset LANG LANGUAGE \
-  LC_ADDRESS LC_ALL LC_COLLATE LC_CTYPE LC_IDENTIFICATION LC_MEASUREMENT \
-  LC_MESSAGES LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME
-export LC_ALL=C
-
 usage() {
   echo -n "usage: $0 [-h] [-s bullseye|bullseye-cuttlefish] "
-  echo -n "[-a i386|amd64|armhf|arm64] "
-  echo -n "[-m http://mirror/debian] [-n rootfs] [-r initrd]"
+  echo -n "[-a i386|amd64|armhf|arm64] -k /path/to/kernel "
+  echo -n "-i /path/to/initramfs.gz [-d /path/to/dtb:subdir] "
+  echo "[-m http://mirror/debian] [-n rootfs] [-r initrd] [-e]"
   exit 1
 }
 
@@ -38,10 +32,13 @@ mirror=http://ftp.debian.org/debian
 suite=bullseye
 arch=amd64
 
+embed_kernel_initrd_dtb=
+dtb_subdir=
 ramdisk=
 rootfs=
+dtb=
 
-while getopts ":hs:a:m:n:r:" opt; do
+while getopts ":hs:a:m:n:r:k:i:d:e" opt; do
   case "${opt}" in
     h)
       usage
@@ -54,15 +51,7 @@ while getopts ":hs:a:m:n:r:" opt; do
       suite="${OPTARG}"
       ;;
     a)
-      case "${OPTARG}" in
-        i386|amd64|armhf|arm64)
-          arch="${OPTARG}"
-          ;;
-        *)
-          echo "Invalid arch: ${OPTARG}" >&2
-          usage
-          ;;
-      esac
+      arch="${OPTARG}"
       ;;
     m)
       mirror="${OPTARG}"
@@ -73,6 +62,21 @@ while getopts ":hs:a:m:n:r:" opt; do
     r)
       ramdisk="${OPTARG}"
       ;;
+    k)
+      kernel="${OPTARG}"
+      ;;
+    i)
+      initramfs="${OPTARG}"
+      ;;
+    d)
+      dtb="${OPTARG%:*}"
+      if [ "${OPTARG#*:}" != "${dtb}" ]; then
+        dtb_subdir="${OPTARG#*:}/"
+      fi
+      ;;
+    e)
+      embed_kernel_initrd_dtb=1
+      ;;
     \?)
       echo "Invalid option: ${OPTARG}" >&2
       usage
@@ -84,6 +88,45 @@ while getopts ":hs:a:m:n:r:" opt; do
   esac
 done
 
+# Disable Debian's "persistent" network device renaming
+cmdline="net.ifnames=0 rw PATH=/usr/sbin:/usr/bin"
+
+# Pass down embedding option, if specified
+if [ -n "${embed_kernel_initrd_dtb}" ]; then
+  cmdline="${cmdline} embed_kernel_initrd_dtb=${embed_kernel_initrd_dtb}"
+fi
+
+case "${arch}" in
+  i386)
+    cmdline="${cmdline} exitcode=/dev/ttyS1"
+    machine="pc-i440fx-2.8,accel=kvm"
+    qemu="qemu-system-i386"
+    cpu="max"
+    ;;
+  amd64)
+    cmdline="${cmdline} exitcode=/dev/ttyS1"
+    machine="pc-i440fx-2.8,accel=kvm"
+    qemu="qemu-system-x86_64"
+    cpu="max"
+    ;;
+  armhf)
+    cmdline="${cmdline} exitcode=/dev/ttyS0"
+    machine="virt,gic-version=2"
+    qemu="qemu-system-arm"
+    cpu="cortex-a15"
+    ;;
+  arm64)
+    cmdline="${cmdline} exitcode=/dev/ttyS0"
+    machine="virt,gic-version=2"
+    qemu="qemu-system-aarch64"
+    cpu="cortex-a53" # "max" is too slow
+    ;;
+  *)
+    echo "Invalid arch: ${OPTARG}" >&2
+    usage
+    ;;
+esac
+
 if [[ -z "${rootfs}" ]]; then
   rootfs="rootfs.${arch}.${suite}.$(date +%Y%m%d)"
 fi
@@ -94,6 +137,22 @@ if [[ -z "${ramdisk}" ]]; then
 fi
 ramdisk=$(realpath "${ramdisk}")
 
+if [[ -z "${kernel}" ]]; then
+  echo "$0: Path to kernel image must be specified (with '-k')"
+  usage
+elif [[ ! -e "${kernel}" ]]; then
+  echo "$0: Kernel image not found at '${kernel}'"
+  exit 2
+fi
+
+if [[ -z "${initramfs}" ]]; then
+  echo "Path to initial ramdisk image must be specified (with '-i')"
+  usage
+elif [[ ! -e "${initramfs}" ]]; then
+  echo "Initial ramdisk image not found at '${initramfs}'"
+  exit 3
+fi
+
 # Sometimes it isn't obvious when the script fails
 failure() {
   echo "Filesystem generation process failed." >&2
@@ -125,23 +184,90 @@ sudo debootstrap --arch="${arch}" --variant=minbase --include="${packages}" \
 # Copy some bootstrapping scripts into the rootfs
 sudo cp -a "${SCRIPT_DIR}"/rootfs/*.sh root/
 sudo cp -a "${SCRIPT_DIR}"/rootfs/net_test.sh sbin/net_test.sh
-sudo chown -R root:root root/ sbin/net_test.sh
+sudo chown root:root sbin/net_test.sh
+
+# Extract the ramdisk to bootstrap with to /
+lz4 -lcd "${initramfs}" | sudo cpio -idum lib/modules/*
 
 # Create /host, for the pivot_root and 9p mount use cases
 sudo mkdir host
 
-sudo chroot . root/stage2.sh
-sudo chroot . root/${suite}.sh
-raw_initrd="${PWD}"/boot/initrd.img
+# Leave the workdir, to build the filesystem
+cd -
 
-# Workarounds for bugs in the debootstrap suite scripts
-for mount in $(cat /proc/mounts | cut -d' ' -f2 | grep -e "^${workdir}"); do
-  echo "Unmounting mountpoint ${mount}.." >&2
+# For the initial ramdisk, and later for the final rootfs
+mount=$(mktemp -d)
+mount_remove() {
+  rmdir "${mount}"
+  tmpdir_remove
+}
+trap mount_remove EXIT
+
+# The initial ramdisk filesystem must be <=512M, or QEMU's -initrd
+# option won't touch it
+initrd=$(mktemp)
+initrd_remove() {
+  rm -f "${initrd}"
+  mount_remove
+}
+trap initrd_remove EXIT
+truncate -s 512M "${initrd}"
+mke2fs -F -t ext3 -L ROOT "${initrd}"
+
+# Mount the new filesystem locally
+sudo mount -o loop -t ext3 "${initrd}" "${mount}"
+image_unmount() {
   sudo umount "${mount}"
-done
+  initrd_remove
+}
+trap image_unmount EXIT
 
-# Leave the workdir, to process the initrd
-cd -
+# Copy the patched debootstrap results into the new filesystem
+sudo cp -a "${workdir}"/* "${mount}"
+sudo rm -rf "${workdir}"
+
+# Unmount the initial ramdisk
+sudo umount "${mount}"
+trap initrd_remove EXIT
+
+# Copy the initial ramdisk to the final rootfs name and extend it
+sudo cp -a "${initrd}" "${rootfs}"
+truncate -s 2G "${rootfs}"
+e2fsck -p -f "${rootfs}" || true
+resize2fs "${rootfs}"
+
+# Create another fake block device for initrd.img writeout
+raw_initrd=$(mktemp)
+raw_initrd_remove() {
+  rm -f "${raw_initrd}"
+  initrd_remove
+}
+trap raw_initrd_remove EXIT
+truncate -s 64M "${raw_initrd}"
+
+# Complete the bootstrap process using QEMU and the specified kernel
+${qemu} -machine "${machine}" -cpu "${cpu}" -m 2048 >&2 \
+  -kernel "${kernel}" -initrd "${initrd}" -no-user-config -nodefaults \
+  -no-reboot -display none -nographic -serial stdio -parallel none \
+  -smp 8,sockets=8,cores=1,threads=1 \
+  -object rng-random,id=objrng0,filename=/dev/urandom \
+  -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 \
+  -drive file="${rootfs}",format=raw,if=none,aio=threads,id=drive-virtio-disk0 \
+  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0 \
+  -drive file="${raw_initrd}",format=raw,if=none,aio=threads,id=drive-virtio-disk1 \
+  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk1 \
+  -chardev file,id=exitcode,path=exitcode \
+  -device pci-serial,chardev=exitcode \
+  -append "root=/dev/ram0 ramdisk_size=524288 init=/root/stage1.sh ${cmdline}"
+[[ -s exitcode ]] && exitcode=$(cat exitcode | tr -d '\r') || exitcode=2
+rm -f exitcode
+if [ "${exitcode}" != "0" ]; then
+  echo "Second stage debootstrap failed (err=${exitcode})"
+  exit "${exitcode}"
+fi
+
+# Fix up any issues from the unclean shutdown
+e2fsck -p -f "${rootfs}" || true
 
 # New workdir for the initrd extraction
 workdir="${tmpdir}/initrd"
@@ -153,8 +279,7 @@ sudo chown root:root "${workdir}"
 cd "${workdir}"
 
 # Process the initrd to remove kernel-specific metadata
-lz4 -lcd "${raw_initrd}" | sudo cpio -idum
-sudo rm -f "${raw_initrd}"
+kernel_version=$(basename $(lz4 -lcd "${raw_initrd}" | sudo cpio -idumv 2>&1 | grep usr/lib/modules/ - | head -n1))
 sudo rm -rf usr/lib/modules
 sudo mkdir -p usr/lib/modules
 
@@ -168,32 +293,66 @@ sudo ln -s /lib usr/lib
 # Repack the ramdisk to the final output
 find * | sudo cpio -H newc -o --quiet | lz4 -lc9 >"${ramdisk}"
 
-# Leave the workdir, to build the filesystem
-workdir="${tmpdir}/_"
+# Pack another ramdisk with the combined artifacts, for boot testing
+cat "${ramdisk}" "${initramfs}" >"${initrd}"
+
+# Leave workdir to boot-test combined initrd
 cd -
 
-# For the final image mount
-mount=$(mktemp -d)
-mount_remove() {
-  rmdir "${mount}"
-  tmpdir_remove
+# Mount the new filesystem locally
+sudo mount -o loop -t ext3 "${rootfs}" "${mount}"
+image_unmount2() {
+  sudo umount "${mount}"
+  raw_initrd_remove
 }
-trap mount_remove EXIT
+trap image_unmount2 EXIT
 
-# Create a 1G empty ext3 filesystem
-truncate -s 1G "${rootfs}"
-mke2fs -F -t ext3 -L ROOT "${rootfs}"
+# Embed the kernel and dtb images now, if requested
+if [ -n "${embed_kernel_initrd_dtb}" ]; then
+  if [ -n "${dtb}" ]; then
+    sudo mkdir -p "${mount}/boot/dtb/${dtb_subdir}"
+    sudo cp -a "${dtb}" "${mount}/boot/dtb/${dtb_subdir}"
+    sudo chown -R root:root "${mount}/boot/dtb/${dtb_subdir}"
+  fi
+  sudo cp -a "${kernel}" "${mount}/boot/vmlinuz-${kernel_version}"
+  sudo chown root:root "${mount}/boot/vmlinuz-${kernel_version}"
+fi
 
-# Mount the new filesystem locally
+# Unmount the initial ramdisk
+sudo umount "${mount}"
+trap raw_initrd_remove EXIT
+
+# Boot test the new system and run stage 3
+${qemu} -machine "${machine}" -cpu "${cpu}" -m 2048 >&2 \
+  -kernel "${kernel}" -initrd "${initrd}" -no-user-config -nodefaults \
+  -no-reboot -display none -nographic -serial stdio -parallel none \
+  -smp 8,sockets=8,cores=1,threads=1 \
+  -object rng-random,id=objrng0,filename=/dev/urandom \
+  -device virtio-rng-pci-non-transitional,rng=objrng0,id=rng0,max-bytes=1024,period=2000 \
+  -drive file="${rootfs}",format=raw,if=none,aio=threads,id=drive-virtio-disk0 \
+  -device virtio-blk-pci-non-transitional,scsi=off,drive=drive-virtio-disk0 \
+  -chardev file,id=exitcode,path=exitcode \
+  -device pci-serial,chardev=exitcode \
+  -netdev user,id=usernet0,ipv6=off \
+  -device virtio-net-pci-non-transitional,netdev=usernet0,id=net0 \
+  -append "root=LABEL=ROOT init=/root/${suite}.sh ${cmdline}"
+[[ -s exitcode ]] && exitcode=$(cat exitcode | tr -d '\r') || exitcode=2
+rm -f exitcode
+if [ "${exitcode}" != "0" ]; then
+  echo "Root filesystem finalization failed (err=${exitcode})"
+  exit "${exitcode}"
+fi
+
+# Fix up any issues from the unclean shutdown
+e2fsck -p -f "${rootfs}" || true
+
+# Mount the final rootfs locally
 sudo mount -o loop -t ext3 "${rootfs}" "${mount}"
-image_unmount() {
+image_unmount3() {
   sudo umount "${mount}"
-  mount_remove
+  raw_initrd_remove
 }
-trap image_unmount EXIT
-
-# Copy the patched debootstrap results into the new filesystem
-sudo cp -a "${workdir}"/* "${mount}"
+trap image_unmount3 EXIT
 
 # Fill the rest of the space with zeroes, to optimize compression
 sudo dd if=/dev/zero of="${mount}/sparse" bs=1M 2>/dev/null || true
diff --git a/net/test/rootfs/bullseye-cuttlefish.sh b/net/test/rootfs/bullseye-cuttlefish.sh
index a4a909b..b61f8e5 100755
--- a/net/test/rootfs/bullseye-cuttlefish.sh
+++ b/net/test/rootfs/bullseye-cuttlefish.sh
@@ -44,5 +44,7 @@ install_and_cleanup_iptables
 
 create_systemd_getty_symlinks ttyS0 hvc1
 
+setup_grub "net.ifnames=0"
+
 apt-get purge -y vim-tiny
 bullseye_cleanup
diff --git a/net/test/rootfs/common.sh b/net/test/rootfs/common.sh
index 3c9d8d7..c86c39c 100644
--- a/net/test/rootfs/common.sh
+++ b/net/test/rootfs/common.sh
@@ -15,6 +15,8 @@
 # limitations under the License.
 #
 
+trap "echo 3 >${exitcode}" ERR
+
 # $1 - Suite name for apt sources
 update_apt_sources() {
   # Add the needed debian sources
@@ -49,6 +51,11 @@ remove_installed_packages() {
 }
 
 setup_static_networking() {
+  # Temporarily bring up static QEMU SLIRP networking (no DHCP)
+  ip link set dev eth0 up
+  ip addr add 10.0.2.15/24 broadcast 10.0.2.255 dev eth0
+  ip route add default via 10.0.2.2 dev eth0
+
   # Permanently update the resolv.conf with the Google DNS servers
   echo "nameserver 8.8.8.8"  >/etc/resolv.conf
   echo "nameserver 8.8.4.4" >>/etc/resolv.conf
@@ -106,6 +113,23 @@ create_systemd_getty_symlinks() {
   done
 }
 
+# $1 - Additional default command line
+setup_grub() {
+  if [ -n "${embed_kernel_initrd_dtb}" ]; then
+    # For testing the image with a virtual device
+    apt-get install -y grub2-common
+    cat >/etc/default/grub <<EOF
+GRUB_DEFAULT=0
+GRUB_TIMEOUT=5
+GRUB_DISTRIBUTOR=Debian
+GRUB_CMDLINE_LINUX_DEFAULT="quiet"
+GRUB_CMDLINE_LINUX="\\\$cmdline $1"
+EOF
+    mkdir /boot/grub
+    update-grub
+  fi
+}
+
 cleanup() {
   # Prevents systemd boot issues with read-only rootfs
   mkdir -p /var/lib/systemd/{coredump,linger,rfkill,timesync}
@@ -114,10 +138,17 @@ cleanup() {
   # If embedding isn't enabled, remove the embedded modules and initrd and
   # uninstall the tools to regenerate the initrd, as they're unlikely to
   # ever be used
-  apt-get purge -y initramfs-tools initramfs-tools-core klibc-utils kmod
+  if [ -z "${embed_kernel_initrd_dtb}" ]; then
+    apt-get purge -y initramfs-tools initramfs-tools-core klibc-utils kmod
+    rm -f "/boot/initrd.img-$(uname -r)"
+    rm -rf "/lib/modules/$(uname -r)"
+  fi
 
   # Miscellaneous cleanup
   rm -rf /var/lib/apt/lists/* || true
   rm -f /root/* || true
   apt-get clean
+
+  echo 0 >"${exitcode}"
+  sync && poweroff -f
 }
diff --git a/net/test/rootfs/stage1.sh b/net/test/rootfs/stage1.sh
new file mode 100755
index 0000000..ccf54f1
--- /dev/null
+++ b/net/test/rootfs/stage1.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+#
+# Copyright (C) 2021 The Android Open Source Project
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+set -u
+
+trap "echo 1 >${exitcode}" ERR
+
+# So we have a rw location to extract kmod
+mount -t tmpfs tmpfs /tmp
+
+# Extract kmod utility to /tmp
+dpkg-deb -x /var/cache/apt/archives/kmod*.deb /tmp
+ln -s /tmp/bin/kmod /tmp/insmod
+
+# Load just enough to get the rootfs from virtio_blk
+module_dir=/lib/modules/$(uname -r)/kernel
+# virtio_pci_modern_dev was split out in 5.12
+/tmp/insmod ${module_dir}/drivers/virtio/virtio_pci_modern_dev.ko || true
+/tmp/insmod ${module_dir}/drivers/virtio/virtio_pci.ko
+/tmp/insmod ${module_dir}/drivers/block/virtio_blk.ko
+/tmp/insmod ${module_dir}/drivers/char/hw_random/virtio-rng.ko
+
+# Mount devtmpfs so we can see /dev/vda
+mount -t devtmpfs devtmpfs /dev
+
+# Mount /dev/vda over the top of /root
+mount /dev/vda /root
+
+# Switch to the new root and start stage 2
+mount -n --move /dev /root/dev
+mount -n --move /tmp /root/tmp
+mount -n -t proc none /root/proc
+mount -n -t sysfs none /root/sys
+mount -n -t tmpfs tmpfs /root/run
+pivot_root /root /root/host
+exec chroot / /root/stage2.sh ${exitcode} </dev/console >/dev/console 2>&1
diff --git a/net/test/rootfs/stage2.sh b/net/test/rootfs/stage2.sh
index 26be79e..7f2e76f 100755
--- a/net/test/rootfs/stage2.sh
+++ b/net/test/rootfs/stage2.sh
@@ -18,6 +18,11 @@
 set -e
 set -u
 
+trap "echo 2 >${exitcode}" ERR
+
+# Remove the old ramdisk root; we don't need it any more
+umount -l /host
+
 # Complete the debootstrap process
 /debootstrap/debootstrap --second-stage
 
@@ -50,14 +55,15 @@ EOF
 mkdir -p /var/empty
 
 # Clean up any other junk created by the imaging process
-rm -rf /root/stage2.sh /tmp/*
+rm -rf /root/stage1.sh /root/stage2.sh /root/lib /tmp/*
 find /var/log -type f -exec rm -f '{}' ';'
 find /var/tmp -type f -exec rm -f '{}' ';'
 
 # Create an empty initramfs to be combined with modules later
 sed -i 's,^COMPRESS=gzip,COMPRESS=lz4,' /etc/initramfs-tools/initramfs.conf
-mkdir -p /lib/modules/0.0
-depmod -a 0.0
-update-initramfs -c -k 0.0
-mv /boot/initrd.img-0.0 /boot/initrd.img
-rm -rf /lib/modules/0.0
+depmod -a $(uname -r)
+update-initramfs -c -k $(uname -r)
+dd if=/boot/initrd.img-$(uname -r) of=/dev/vdb conv=fsync
+
+echo 0 >"${exitcode}"
+sync && poweroff -f