diff options
author | android-build-team Robot <android-build-team-robot@google.com> | 2019-01-06 04:10:29 +0000 |
---|---|---|
committer | android-build-team Robot <android-build-team-robot@google.com> | 2019-01-06 04:10:29 +0000 |
commit | 49a0124bdb040c5f336f0ddd328f9a7a2675608b (patch) | |
tree | d83ba8e94dfb8a97ba5ed887acda6e738008acf1 | |
parent | 692d7ecef5d5b5f77f77dbd78cda52d86b0df15f (diff) | |
parent | 5fe2a3912cb822a5f890d2bc1245b09de3b9ae18 (diff) | |
download | adeb-49a0124bdb040c5f336f0ddd328f9a7a2675608b.tar.gz |
Snap for 5212161 from 5fe2a3912cb822a5f890d2bc1245b09de3b9ae18 to qt-release
Change-Id: I9c15183618086c6c394990958995617f0a96c216
37 files changed, 2305 insertions, 0 deletions
@@ -0,0 +1,186 @@ +BCC (BPF compiler collection) for Android +========================================= + +Introduction +------------ +BCC is a compiler and a toolkit, containing powerful kernel tracing tools that +trace at the lowest levels, including adding hooks to functions in kernel space +and user space to deeply understand system behavior while being low in +overhead. [Here's a presentation with an +overview](http://www.joelfernandes.org/resources/bcc-ospm.pdf) and visit [BCC's +project page](https://github.com/iovisor/bcc) for the official BCC +documentation. + +Quick Start +----------- +adeb is the primary vehicle for running BCC on Android. It supports +preparing the target Android device with necessary kernel headers, cloning and +building BCC on device, and other setup. Take a look a quick look at [adeb +README](https://github.com/joelagnel/adeb/blob/master/README.md) so that +you're familiar with what it is. + +To download a prebuilt filesystem with BCC already built/installed for an ARM64 +device, you can just run: +``` +adeb prepare --full +``` + +This downloads the FS and also downloads prebuilt kernel headers after +detecting your device's kernel version. Running BCC this way may cause a warning +at startup since the headers may not be an *exact* match for your kernel's +sublevel (only version and patchlevel are matched), however it works well in +our testing and could be used, as long as you can tolerate the warning. + +If you would like to setup your own kernel headers and prevent the warning, +you can point adeb to the kernel sources which will extract headers from there: +``` +adeb prepare --full --kernelsrc /path/to/kernel-source/ +``` +For targets other than ARM64, see the [Other Architectures +section](https://github.com/joelagnel/adeb/blob/master/BCC.md#other-architectures-other-than-arm64) + +Now to run BCC, just start an adeb shell: `adeb shell`. This uses adb +as the backend to start a shell into your adeb environment. Try running +`opensnoop` or any of the other BCC tracers to confirm that the setup worked +correctly. + +If building your own kernel, following are the kernel requirements: + +You need kernel 4.9 or newer. Anything less needs backports. Your kernel needs +to be built with the following config options at the minimum: +``` +CONFIG_KPROBES=y +CONFIG_KPROBE_EVENT=y +CONFIG_BPF_SYSCALL=y +``` +Optionally, +``` +CONFIG_UPROBES=y +CONFIG_UPROBE_EVENT=y +``` +Additionally, for the criticalsection BCC tracer to work, you need: +``` +CONFIG_DEBUG_PREEMPT=y +CONFIG_PREEMPTIRQ_EVENTS=y +``` + +Build BCC during adeb install (Optional) +-------------------------------------------- +If you would like the latest upstream BCC built and installed on your Android +device, you can run: +``` +adeb prepare --build --bcc --kernelsrc /path/to/kernel-source/ +``` +NOTE: This is a slow process and can take a long time. Since it not only builds +BCC but also installs all non-BCC debian packages onto the filesystem and configures them. + +Other Architectures (other than ARM64) +----------------------- +By default adeb assumes the target Android device is based on ARM64 +processor architecture. For other architectures, use the --arch option. For +example for x86_64 architecture, run: +``` +adeb prepare --arch amd64 --build --bcc --kernelsrc /path/to/kernel-source/ +``` +Note: The --download option ignores the --arch flag. This is because we only +provide pre-built filesystems for ARM64 at the moment. +Note: If you pass --arch, you have to pass --build, because prebuilt +filesystems are not available for non-arm64 devices. + +Common Issues +------------- +Here are some common issues you may face when running different BCC tools. + +* Issue 1: Headers are missing on the target device. + +Symptom: This will usually result in an error like the following: +``` +root@localhost:/# criticalstat + +In file included from <built-in>:2 +In file included from /virtual/include/bcc/bpf.h:12: +In file included from include/linux/types.h:5: +include/uapi/linux/types.h:4:10: fatal error: 'asm/types.h' file not found + +#include <asm/types.h> + + ^~~~~~~~~~~~~ +1 error generated. +Traceback (most recent call last): + + File "./criticalstat.py", line 138, in <module> + b = BPF(text=bpf_text) + File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 297, in __init__ + raise Exception("Failed to compile BPF text:\n%s" % text) +Exception: Failed to compile BPF text: + +#include <uapi/linux/ptrace.h> +#include <uapi/linux/limits.h> +#include <linux/sched.h> + +extern char _stext[]; +``` + +* Issue 2: `CONFIG_KPROBES` isn't enabled. + +Symptom: This will result in an error like the following: +``` +Traceback (most recent call last): + File "/usr/share/bcc/tools/cachetop", line 263, in <module> + curses.wrapper(handle_loop, args) + File "/usr/lib/python2.7/curses/wrapper.py", line 43, in wrapper + return func(stdscr, *args, **kwds) + File "/usr/share/bcc/tools/cachetop", line 172, in handle_loop + b.attach_kprobe(event="add_to_page_cache_lru", fn_name="do_count") + File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 543, in +attach_kprobe + fn = self.load_func(fn_name, BPF.KPROBE) + File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 355, in +load_func + (func_name, errstr)) +Exception: Failed to load BPF program do_count: Invalid argument +``` + +* Issue 3: `CONFIG_BPF_SYSCALL` isn't enabled. + +Symptom: This may result in a compilation error like the following: +``` +root@localhost:/# cachetop +Traceback (most recent call last): + File "/usr/share/bcc/tools/cachetop", line 263, in <module> + curses.wrapper(handle_loop, args) + File "/usr/lib/python2.7/curses/wrapper.py", line 43, in wrapper + return func(stdscr, *args, **kwds) + File "/usr/share/bcc/tools/cachetop", line 171, in handle_loop + b = BPF(text=bpf_text) + File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 297, in __init__ + raise Exception("Failed to compile BPF text:\n%s" % text) +Exception: Failed to compile BPF text: + + + #include <uapi/linux/ptrace.h> + struct key_t { + u64 ip; + u32 pid; + u32 uid; + char comm[16]; + }; + + BPF_HASH(counts, struct key_t); + + int do_count(struct pt_regs *ctx) { + struct key_t key = {}; + u64 zero = 0 , *val; + u64 pid = bpf_get_current_pid_tgid(); + u32 uid = bpf_get_current_uid_gid(); + + key.ip = PT_REGS_IP(ctx); + key.pid = pid & 0xFFFFFFFF; + key.uid = uid & 0xFFFFFFFF; + bpf_get_current_comm(&(key.comm), 16); + + val = counts.lookup_or_init(&key, &zero); // update counter + (*val)++; + return 0; + } +``` @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/METADATA b/METADATA new file mode 100644 index 0000000..90881ff --- /dev/null +++ b/METADATA @@ -0,0 +1,19 @@ +name: "adeb" +description: "adeb (also known as androdeb) provides a powerful Linux shell +environment where one can run popular and mainstream Linux tracing, compiling, +editing and other development tools on an existing Android device. All the +commands typically available on a modern Linux system are supported in adeb." + +third_party { + url { + type: HOMEPAGE + value: "https://github.com/joelagnel/adeb/README.md" + } + url { + type: GIT + value: "https://github.com/joelagnel/adeb.git" + } + version: "v0.99f" + last_upgrade_date { year: 2018 month: 7 day: 25 } + license_type: NOTICE +} diff --git a/MODULE_LICENSE_APACHE2 b/MODULE_LICENSE_APACHE2 new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/MODULE_LICENSE_APACHE2 @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..23ae44d --- /dev/null +++ b/README.md @@ -0,0 +1,173 @@ +adeb +-------- + +**adeb** (also known as **androdeb**) provides a powerful Linux shell +environment where one can run popular and mainstream Linux tracing, compiling, +editing and other development tools on an existing Android device. All the +commands typically available on a modern Linux system are supported in +adeb. + +Usecases +-------- +1. Powerful development environment with all tools ready to go (editors, +compilers, tracers, perl/python etc) for your on-device development. + +2. No more cross-compiler needed: Because it comes with gcc and clang, one can +build target packages natively without needing to do any cross compilation. We even +ship git, and have support to run apt-get to get any missing development packages +from the web. + +3. Using these one can run popular tools such as BCC that are difficult to run +in an Android environment due to lack of packages, dependencies and +cross-compilation needed for their operation. [Check BCC on Android using +adeb](https://github.com/joelagnel/adeb/blob/master/BCC.md) for more +information on that. + +4. No more crippled tools: Its often a theme to build a static binary with +features disabled, because you couldn't cross-compile the feature's dependencies. One +classic example is perf. However, thanks to adeb, we can build perf natively +on device without having to cripple it. + +Requirements for running +------------------------ +Target: +An ARM64 android N or later device which has "adb root" supported. Typically +this is a build in a userdebug configuration. Device should have atleast 2 GB +free space in the data partition. If you would like to use other architectures, +see the [Other Architectures](https://github.com/joelagnel/adeb/blob/master/README.md#how-to-use-adeb-for-other-architectures-other-than-arm64) section. + +Host: +A machine running recent Ubuntu or Debian, with 4GB of memory and 4GB free space. +Host needs debootstrap and qemu-debootstrap packages. +To install it, run `sudo apt-get install qemu-user-static debootstrap`. +Other distributions may work but they are not tested. + +Quick Start Instructions +------------------------ +* First clone this repository into adb and cd into it. +``` +cd adeb + +# Add some short cuts: +sudo ln -s $(pwd)/adeb /usr/bin/adeb + +# Cached image downloads result in a huge speed-up. These are automatic if you +# cloned the repository using git. However, if you downloaded the repository +# as a zip file (or you want to host images elsewere), you could set the +# ADEB_REPO_URL environment variable in your bashrc file. +# Disclaimer: Google is not liable for the below URL and this +# is just an example. +export ADEB_REPO_URL="github.com/joelagnel/adeb/" +``` + +* Installing adeb onto your device: +First make sure device is connected to system +Then run, for the base image: +``` +adeb prepare +``` +The previous command only downloads and installs the base image. +Instead if you want to download and install the full image, do: +``` +adeb prepare --full +``` + +* Now run adeb shell to enter your new environment!: +``` +adeb shell +``` + +* Once done, hit `CTRL + D` and you will exit out of the shell. +To remove adeb from the device, run: +``` +adeb remove +``` +If you have multiple devices connected, please add `-s <serialnumber>`. +Serial numbers of all devices connected can be obtained by `adb devices`. + +* To update an existing adeb clone on your host, run: +``` +adeb git-pull +``` + +More advanced usage instructions +-------------------------------- +### Install kernel headers in addition to preparing adeb device: +``` +adeb prepare --kernelsrc /path/to/kernel-source +``` + +### Update kernel headers onto an already prepared device: + +If you need to put kernel sources for an existing install, run: +``` +adeb prepare --kernelsrc /path/to/kernel-source --skip-install +``` +Note: The kernel sources should have been built (atleast build should have started). + +### Build and prepare device with a custom rootfs locally: + +The adeb fs will be prepared locally by downloading packages as needed: +``` +adeb prepare --build +``` +This is unlike the default behavior, where the adeb rootfs is itself pulled from the web. + +If you wish to do a full build (that is locally prepare a rootfs with all packages, including bcc, then do): +``` +adeb prepare --full --build +``` + +### Add kernel headers to device in addition to building locally: +``` +adeb prepare --build --kernelsrc /path/to/kernel-source/ +``` + +### Build/install a base image with BCC: +``` +adeb prepare --build --bcc --kernelsrc /path/to/kernel-source/ +``` +Note: BCC is built from source. Also `--kernelsrc` is recommended for tools to +function unless device has them already. + +### Extract the FS from the device, after its prepared: +``` +adeb prepare --buildtar /path/ +``` +After device is prepared, it will extract the root fs from it +and store it as a tar archive at `/path/adeb-fs.tgz`. This +can be used later. + +### Use a previously prepared adeb rootfs tar from local: +``` +adeb prepare --archive /path/adeb-fs.tgz +``` + +### Build a standalone raw EXT4 image out of the FS: +``` +adeb prepare --build-image /path/to/image.img +``` +This can then be passed to Qemu as -hda. Note: This option doesn't need a +device connected. + +### How to use adeb for other Architectures (other than ARM64) +By default adeb assumes the target Android device is based on ARM64 +processor architecture. For other architectures, use the --arch option. For +example for x86_64 architecture, run: +``` +adeb prepare --build --arch amd64 --bcc --kernelsrc /path/to/kernel-source/ +``` +Note: The --download option ignores the --arch flag. This is because we only +provide pre-built filesystems for ARM64 at the moment. + +Common Trouble shooting +----------------- +1. Installing g++ with `apt-get install g++` fails. + +Solution: Run `adeb shell apt-get update` after the `adeb prepare` stage. + +2. It's too slow to use debootstrap to create debian fs + +Solution: Use a local mirror, for example in China you could use +https://mirror.tuna.tsinghua.edu.cn/debian/ instead of debian official website +http://deb.debian.org/debian/ diff --git a/README.version b/README.version new file mode 100644 index 0000000..3b30f4a --- /dev/null +++ b/README.version @@ -0,0 +1 @@ +URL: https://github.com/joelagnel/adeb/ @@ -0,0 +1,5 @@ +TODO: + - update androdeb with latest bcc (which will include criticalstat) + - symlink whichever perf is installed to /usr/bin/perf + - patch perf with the new futex contention script + (probably it can just be dropped into a path) diff --git a/addons/bashrc b/addons/bashrc new file mode 100644 index 0000000..5ca6461 --- /dev/null +++ b/addons/bashrc @@ -0,0 +1,21 @@ +# The bannered bashrc +source .bashrc.common + +if [ ! -f .banner.shown ]; then + +echo "" +echo "##########################################################" +echo "# Welcome to androdeb environment running on Android! #" +echo "# Questions to: Joel Fernandes <joel@joelfernandes.org> #" +echo " #" +echo " Try running vim, gcc, clang, git, make, perf, filetop #" +echo " ..etc or apt-get install something. #" +echo "##########################################################" +echo "" + +touch .banner.shown + +fi + +homedir=$( getent passwd "$USER" | cut -d: -f6 ) +export HOME=$homedir diff --git a/addons/bashrc.common b/addons/bashrc.common new file mode 100644 index 0000000..14e09d3 --- /dev/null +++ b/addons/bashrc.common @@ -0,0 +1,37 @@ +export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/share/bcc/tools/ +export TMPDIR=/tmp/ + +psk=/proc/sys/kernel/kptr_restrict +if [ -f $psk ]; then echo 0 > $psk; fi + +override_vers=1 +this_vers=$(/get_kvers.sh) +if [ -d /kernel-headers/ ]; then + export BCC_KERNEL_SOURCE=/kernel-headers/; + + h_vers=0 + vers_line=$(grep LINUX_VERSION_CODE /kernel-headers/include/generated/uapi/linux/version.h) + if [ "x$vers_line" != "x" ]; then + h_vers=$(echo $vers_line | cut -d ' ' -f3) + fi + + if [ $this_vers -eq $h_vers ]; then + override_vers=0 + else + this_major=$(($this_vers / (256 * 256))) + this_minor=$(($this_vers / 256 % 256)) + h_major=$(($h_vers / (256 * 256))) + h_minor=$(($h_vers / 256 % 256)) + if [ $this_major -ne $h_major ] || [ $this_minor -ne $h_minor ]; then + echo "Error: kernel version does not match installed headers ($this_major.$this_minor <> $h_major.$h_minor)" + echo "BCC will not work" + override_vers=0 + fi + fi +fi + +# Override kernel version if we haven't found headers with them. +# Needed for BCC to work on slightly mismatched kernels. +if [ $override_vers -eq 1 ]; then + export BCC_LINUX_VERSION_CODE=$this_vers +fi diff --git a/addons/bashrc.silent b/addons/bashrc.silent new file mode 100644 index 0000000..1e52500 --- /dev/null +++ b/addons/bashrc.silent @@ -0,0 +1,4 @@ +source .bashrc.common + +homedir=$( getent passwd "$USER" | cut -d: -f6 ) +export HOME=$homedir diff --git a/addons/build-debian-tar b/addons/build-debian-tar new file mode 100755 index 0000000..2c58c37 --- /dev/null +++ b/addons/build-debian-tar @@ -0,0 +1,19 @@ +#!/system/bin/sh +set -e +# Build a tarball out of a android environment which I can then +# upload to places for people who want to expedite the install + +# This script runs on the device + +spath=$( cd "$(dirname "$0")" ; pwd -P ) +cd $spath + +./device-umount-all + +if [ ! -d debian ]; then echo "Error: environment to tar doesn't exist"; exit 1; fi + +rm -rf debian-tar; cp -r debian debian-tar; +rm -rf debian/debian; mv debian-tar debian/debian +./run-command "tar -zcf androdeb-fs.tgz --exclude='debian/kernel-headers' debian" +mv debian/androdeb-fs.tgz . +rm -rf debian/debian-tar diff --git a/addons/device-umount-all b/addons/device-umount-all new file mode 100755 index 0000000..fe3b092 --- /dev/null +++ b/addons/device-umount-all @@ -0,0 +1,16 @@ +#!/system/bin/sh + +if [ "$1x" == "--debugx" ]; then + set -x +fi + +umount_all() { + mpoints=$(mount|cut -d ' ' -f3|grep debian) + for m in $mpoints; + do umount $m 2>&1 > /dev/null + done +} + +for i in $(seq 0 6); do + umount_all 2>&1 > /dev/null +done diff --git a/addons/device-unpack b/addons/device-unpack new file mode 100755 index 0000000..14ba258 --- /dev/null +++ b/addons/device-unpack @@ -0,0 +1,28 @@ +#!/system/bin/sh + +set -e + +# Script to do unpack of rootfs, ensures proper tear down +# of existing environment. Expects debian rootfs in +# /data/deb.tar.gz which it will delete after successful +# unpack of rootfs. + +spath=$( cd "$(dirname "$0")" ; pwd -P ) + +if [ ! -f /data/androdeb/deb.tar.gz ]; then + echo "Debian rootfs tar doesn't existing at /data/deb.tar.gz" + echo "Run androdeb with device connected first" + exit 1 +fi + +if [ -d /data/androdeb/debian ]; then + echo "androdeb environment already exists, doing a tear down" + /data/androdeb/device-umount-all + rm -rf /data/androdeb/debian +fi + + +tar -zxf /data/androdeb/deb.tar.gz -C /data/androdeb/ || die 2 "Couldn't unpack due to tar -x errors" +rm /data/androdeb/deb.tar.gz + +echo "Unpack of rootfs successful!" diff --git a/addons/get_kvers.sh b/addons/get_kvers.sh new file mode 100755 index 0000000..3d18e32 --- /dev/null +++ b/addons/get_kvers.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +kvers=$(uname -r) + +MAJOR=$(echo $kvers | awk -F. '{ print $1 }') +MINOR=$(echo $kvers | awk -F. '{ print $2 }') +SUBVR=$(echo $kvers | awk -F. '{ print $3 }' | awk -F- '{ print $1 }' | sed 's/[^0-9]*//g') + +maj_num=$(($MAJOR * 65536)) +min_num=$(($MINOR * 256)) + +echo $(($maj_num + $min_num + $SUBVR)) diff --git a/addons/run b/addons/run new file mode 100755 index 0000000..9738504 --- /dev/null +++ b/addons/run @@ -0,0 +1,7 @@ +#!/system/bin/sh +spath=$( cd "$(dirname "$0")" ; pwd -P ) +cd $spath + +source run.common + +chroot debian/ /bin/bash diff --git a/addons/run-command b/addons/run-command new file mode 100755 index 0000000..6d5a8ca --- /dev/null +++ b/addons/run-command @@ -0,0 +1,10 @@ +#!/system/bin/sh +spath=$( cd "$(dirname "$0")" ; pwd -P ) +cd $spath + +source run.common + +# Directly execute a command within the chroot of an Android device +CMD="$*" + +chroot debian /bin/bash --rcfile '.bashrc.silent' -i -c "$CMD" diff --git a/addons/run.common b/addons/run.common new file mode 100644 index 0000000..5c45308 --- /dev/null +++ b/addons/run.common @@ -0,0 +1,42 @@ +do_mounts() +{ + mount --bind /proc debian/proc/ > /dev/null + + mount --bind /dev debian/dev/ > /dev/null + mount --bind /dev/pts debian/dev/pts > /dev/null + + mount --bind /sys debian/sys/ > /dev/null + mount --bind /sys/fs/bpf/ debian/sys/fs/bpf/ > /dev/null + mount --bind /sys/kernel/debug/ debian/sys/kernel/debug/ > /dev/null + mount --bind /sys/kernel/debug/tracing/ debian/sys/kernel/debug/tracing/ + + # Fix up weirdness with debugfs permission changing because of + # above mounts. + chmod 0777 /sys/kernel/debug > /dev/null + chmod 0777 debian/sys/kernel/debug > /dev/null + chmod 0777 /sys/kernel/debug/tracing > /dev/null + chmod 0777 debian/sys/kernel/debug/tracing > /dev/null + + # Mount Android partitions + if [ -d /d/ ]; then + if [ ! -d debian/d ]; then ln -s /sys/kernel/debug debian/d; fi + fi + + if [ -d /data/ ]; then + mkdir -p debian/data/ + mount --bind /data debian/data/ + fi + + if [ -d /system/ ]; then + mkdir -p debian/system/ + mount --bind /system debian/system/ + fi + + if [ -d /vendor/ ]; then + mkdir -p debian/vendor/ + mount --bind /vendor debian/vendor/ + fi +} + +mount | grep debian > /dev/null +if [ $? -ne 0 ]; then do_mounts; fi @@ -0,0 +1,6 @@ +#!/bin/bash -e +# +# (c) Joel Fernandes <joel@joelfernandes.org> + +spath="$(dirname "$(readlink -f "$0")")" +source $spath/androdeb diff --git a/androdeb b/androdeb new file mode 100755 index 0000000..3600a5b --- /dev/null +++ b/androdeb @@ -0,0 +1,386 @@ +#!/bin/bash -e +# +# (c) Joel Fernandes <joel@joelfernandes.org> + +VERSION=v0.99g + +spath="$(dirname "$(readlink -f "$0")")" +# spath=$( cd "$(dirname "$0")" ; pwd -P ) +curdir=$( pwd -P ) +source $spath/utils/support +source $spath/utils/banners + +# Set default vars +DISTRO=buster; ARCH=arm64 +ADB="adb" +FULL=0 # Default to a minimal install +DOWNLOAD=1 # Default to downloading from web +SKIP_DEVICE=0 # Skip device preparation +INSTALL_BCC=0 # Decide if BCC is to be installed + +# Default packages +PACKAGES="" +DEFAULT_PACKAGES="bash ca-certificates apt net-tools iputils-ping procps vim" + +EXTRA_FILES="none" + +config_full_build() { + for f in $(ls $spath/packages); do source $spath/packages/$f; done; +} + +# Parse command line parameters +if [ $# -lt 1 ]; then usage; fi; POSITIONAL=() +while [[ $# -gt 0 ]]; do key="$1"; + +# If its shell mode, any future args become shell's args +if [ "x$ASHELL" == "x1" ]; then + if [ -z "$SHELL_ARGS" ]; then + SHELL_ARGS=$key + else + SHELL_ARGS="$SHELL_ARGS $key" + fi + shift || true; continue +fi + +case $key in + shell) ASHELL=1; shift || true; ;; + remove) REMOVE=1; shift || true; ;; + git-pull) GIT_PULL=1; shift || true; ;; + pull) PULL=1; shift || true; break ;; + push) PUSH=1; shift || true; break ;; + prepare) PREPARE=1; shift || true; ;; + --full) FULL=1; config_full_build; shift || true; ;; + --arch) ARCH=$2; shift || true; shift || true; ;; + --archive) DOWNLOAD=0; TARF=$2; shift || true; shift || true; ;; + --bcc) FULL=1; source $spath/packages/bcc; shift || true; ;; + --kernelsrc) KERNELSRC="$2"; shift || true; shift || true; ;; + --skip-install) SKIP_INSTALL=1; shift || true; ;; + --kernel-headers-targz) KERNELHDRS=$2; shift || true; shift || true; ;; + --tempdir) TDIR="$2"; shift || true; shift || true; ;; + --build) DOWNLOAD=0; shift || true; ;; + --buildtar) BTAR=1; DOWNLOAD=0; TARDIR="$2"; shift || true; shift || true; ;; + --device|-s) ADB="$ADB -s $2"; shift || true; shift || true; ;; + --build-image) BI=1; BUILD_IMAGEF=$2; SKIP_DEVICE=1; DOWNLOAD=0; shift || true; shift || true; ;; + --debug) set -x; shift || true; ;; + *) c_error "Unknown option ($1)"; usage; ;; +esac +done + +[ -z $ASHELL ] && box_out "adeb: $VERSION" + +if [ $FULL -eq 1 ]; then + FNAME=androdeb-fs.tgz.zip + FNAME_UZ=androdeb-fs.tgz +else + FNAME=androdeb-fs-minimal.tgz.zip + FNAME_UZ=androdeb-fs-minimal.tgz +fi + +if [ ! -z $BTAR ] && [ -z $TARDIR ]; then + TARDIR=$spath +fi + +if [ ! -z "$GIT_PULL" ]; then + c_info "Updating androdeb by git pull" + cd $spath + git pull + c_info "Done." + exit 0 +fi + +if [ ! -z "$PULL" ]; then + if [ $1 == "-a" ]; then + PRESERVE="-a" + c_info "Preserving filestamps and mode" + shift || true + fi + file_count=`count_sources $@` + i=0 + while [ $i -lt $file_count ]; do + files["$i"]=/data/androdeb/debian/$1 + shift || true + i=$((i + 1)) + done + $ADB pull $PRESERVE "${files[@]}" "$@" + exit 0 +fi + +if [ ! -z "$PUSH" ]; then + file_count=`count_sources $@` + i=0 + while [ $i -lt $file_count ]; do + files["$i"]=$1 + shift || true + i=$((i + 1)) + done + dest=/data/androdeb/debian/$1 + $ADB push $sync "${files[@]}" $dest + exit 0 +fi + +if [[ ! -z ${TARDIR+x} ]] && [[ ! -d $TARDIR ]]; then die 7 "Tar dir specified doesn't exist"; fi + +if [ -z $BI ]; then + [ -z $ASHELL ] && c_info "Looking for device.." + set +e + do_adb_root "$ADB" + + if [ $? -ne 0 ]; then + c_error "adb root failed, make sure:" + c_error " * If multiple devices connected, provide --device <serialno> (or -s <serialno>)" + c_error " * Try to run \"adb root\" manually and see if it works. Typically this needs a userdebug build." + c_error "" + c_error "Note: adb can be typically obtained using the android-tools-adb or the adb" + c_error "packages on your distro, or by installing the Android SDK." + die 3 "Exiting." + fi + set -e +else + [ ! -z $BUILD_IMAGEF ] || die 8 "--build-image passed but no image file provided" +fi + +if [ ! -z "$REMOVE" ]; then + die_if_no_androdeb "Nothing to remove." + $ADB shell /data/androdeb/device-umount-all || true; + $ADB shell rm -rf /data/androdeb; exit 0; fi + +########################################################## +# SHELL +########################################################## +if [ ! -z ${ASHELL+x} ]; then + set +e; $ADB shell ls /data/androdeb/debian/.bashrc > /dev/null 2>&1 + if [ $? -ne 0 ]; then + die 2 "Device doesn't have an androdeb environment, run \"./androdeb prepare\" first"; + fi; set -e + + if [ ! -z ${SHELL_ARGS+x} ]; then + # Explanation of quotes: + # Outer quote is so that androdeb's bash passes the SHELL_ARGS as a single + # argument to $ADB shell. Inner quotes is so that run-command can receive all + # the args even though they may be separated by spaces. \m/ + $ADB shell -t /data/androdeb/run-command "\"$SHELL_ARGS\"" + else + $ADB shell -t /data/androdeb/run + fi + + exit 0 +fi + +########################################################## +# PREPARE +########################################################## + +function do_cleanup() { + rm -rf $TDIR/*; if [ $MKTEMP -eq 1 ]; then rm -rf $TDIR; fi +} + +function push_unpack_headers() { + die_if_no_androdeb "Couldn't update headers." + + c_info "Storing kernel headers into androdeb /kernel-headers/" + $ADB shell rm -rf /data/androdeb/debian/kernel-headers/ + $ADB shell mkdir /data/androdeb/debian/kernel-headers/ + run_quiet $ADB push $TDIR_ABS/kh.tgz /data/androdeb/ + $ADB shell tar -xvf /data/androdeb/kh.tgz -C /data/androdeb/debian/kernel-headers/ > /dev/null + $ADB shell rm /data/androdeb/kh.tgz +} + +function push_unpack_tarred_headers() { + die_if_no_androdeb "Couldn't update headers." + + $ADB shell rm -rf /data/androdeb/debian/kernel-headers/ + $ADB shell mkdir /data/androdeb/debian/kernel-headers/ + + c_info "Pushing headers tar onto device" + run_quiet $ADB push $1 /data/androdeb/ + + c_info "Storing kernel headers into androdeb root directory" + $ADB shell tar -xvf /data/androdeb/$(basename $1) -C /data/androdeb/debian/ > /dev/null + + $ADB shell rm /data/androdeb/$(basename $1) +} + +function all_done_banner() { + c_info "All done! Run \"adeb shell\" to enter environment" +} + +function detect_repo_url() { + ADEB_REPO_URL=`cd $spath && git config -l | grep -m1 remote | grep url | sed -e "s/.*url=//" \ + -e "s/.*@//" \ + -e "s/https:\/\///" \ + -e "s/:/\//" \ + -e "s/\.git$//"`"/" + c_info "Detected URL: $ADEB_REPO_URL" +} + +function check_repo_url () { + if [ -z $ADEB_REPO_URL ]; then + c_info "No repository URL provided in enviromnent. Attempting to auto-detect it" + detect_repo_url + fi + + if [ -z $ADEB_REPO_URL ]; then + c_warning "Automatic download is disabled. To enable it, please set the \$ADEB_REPO_URL" + c_warning "environment variable as recommended in the setup instructions in the README.md" + do_cleanup + exit 0 + fi +} + +function download_headers() { + KERNEL_MAJOR=`$ADB shell uname -r | cut -d - -f 1 | cut -d . -f 1` + KERNEL_MINOR=`$ADB shell uname -r | cut -d - -f 1 | cut -d . -f 2` + KERNEL_VERSION="$KERNEL_MAJOR.$KERNEL_MINOR" + PREBUILT_HEADERS_FILE=headers-$ARCH-$KERNEL_VERSION.tar.gz.zip + + check_repo_url + + curl -L https://$ADEB_REPO_URL/releases/download/$VERSION/$PREBUILT_HEADERS_FILE --output $TDIR_ABS/$PREBUILT_HEADERS_FILE || + die 9 "Failed to download kernel headers. Please check your internet connection and repository URL" + + unzip -e $TDIR_ABS/$PREBUILT_HEADERS_FILE -d $TDIR_ABS/ || + die 10 "Failed to download kernel headers. Kernel $KERNEL_VERSION for $ARCH may not be supported or ADEB_REPO_URL is incorrect." + KERNELHDRS=$TDIR_ABS/`echo "$PREBUILT_HEADERS_FILE" | sed "s/.zip//"` +} + +# Prepare is the last command checked +if [ -z "$PREPARE" ]; then usage; fi + +if [ ! -z "$TARF" ] && [ ! -f $TARF ] && [ -z "$DOWNLOAD" ]; then die 5 "archive provided doesn't exist"; fi + +if [ ! -z "$KERNELSRC" ] && [ ! -d $KERNELSRC ]; then die 6 "Kernel source directory provided doesn't exist"; fi + +if [ ! -z "$KERNELHDRS" ] && [ ! -f $KERNELHDRS ]; then die 7 "Kernel headers tar.gz doesn't exist"; fi + +print_prepare_banner + +# Where do we want to store temporary files +MKTEMP=0; if [[ -z ${TDIR+x} ]] || [[ ! -d "${TDIR}" ]]; then + TDIR=`mktemp -d`; MKTEMP=1; fi +rm -rf $TDIR/* +TDIR_ABS=$( cd "$TDIR" ; pwd -P ) + +if [ $DOWNLOAD -eq 1 ]; then + c_info "Downloading Androdeb from the web..."; c_info "" + + # Github dropped tar gz support! ##?#??#! Now we've to zip everything. + check_repo_url + + curl -L https://$ADEB_REPO_URL/releases/download/$VERSION/$FNAME --output $TDIR_ABS/$FNAME || + die 9 "Failed to download adeb release." + + unzip -e $TDIR_ABS/$FNAME -d $TDIR_ABS/ || + die 10 "Failed to download adeb release. Double check the ADEB_REPO_URL value." + TARF=$TDIR_ABS/$FNAME_UZ +fi + +if [ ! -z "$FULL" ] && [ -z "$KERNELSRC" ] && [ -z "$KERNELHDRS" ] && [ -z "$BI" ]; then + c_info "Kernel headers are needed but none were provided. Downloading pre-built headers" + download_headers +fi + +OUT_TMP=$TDIR/debian; rm -rf $OUT_TMP; mkdir -p $OUT_TMP + +# Unpack the supplied kernel headers tar.gz directly into androdeb root +if [ ! -z "$KERNELHDRS" ]; then + c_info "Building updating kernel headers from supplied tar.gz ($KERNELHDRS)" + + # Is header tar gz update the only thing left to do? + if [[ ! -z "$SKIP_INSTALL" ]]; then + c_info "Skipping install" + push_unpack_tarred_headers $KERNELHDRS; do_cleanup; all_done_banner; exit 0; fi + + tar -xvf $KERNELHDRS -C $OUT_TMP/ > /dev/null +fi + +# Package kernel headers +if [ ! -z "$KERNELSRC" ]; then + c_info "Building and updating kernel headers from kernel source dir ($KERNELSRC)" + $spath/bcc/build-kheaders-targz.sh ${KERNELSRC} $TDIR_ABS/kh.tgz > /dev/null + + # Is header update the only thing left to do? + if [[ ! -z "$SKIP_INSTALL" ]]; then + c_info "Skipping install" + push_unpack_headers; do_cleanup; all_done_banner; exit 0; fi + + mkdir $OUT_TMP/kernel-headers + tar -xvf $TDIR_ABS/kh.tgz -C $OUT_TMP/kernel-headers/ > /dev/null +fi + +# Build FS from existing tar, very simple. +if [ ! -z "$TARF" ]; then + c_info "Using archive at $TARF for filesystem preparation" + $ADB shell mkdir -p /data/androdeb/ + + c_info "Pushing filesystem to device.." + run_quiet $ADB push $TARF /data/androdeb/deb.tar.gz + + c_info "Pushing addons to device.." + run_quiet $ADB push $spath/addons/* /data/androdeb/ + + c_info "Unpacking filesystem in device.." + run_quiet $ADB shell /data/androdeb/device-unpack + + if [ ! -z "$KERNELHDRS" ]; then push_unpack_tarred_headers $KERNELHDRS; fi + if [ ! -z "$KERNELSRC" ]; then push_unpack_headers; fi + + do_cleanup; all_done_banner; exit 0 +fi + +PACKAGES+="$DEFAULT_PACKAGES" +c_info "Using temporary directory: $TDIR" + +if [[ $EUID -ne 0 ]]; then c_info "The next stage runs as sudo, please enter password if asked."; fi + +ex_files=$(mktemp); echo $EXTRA_FILES > $ex_files + +sudo $spath/buildstrap $ARCH $DISTRO $TDIR $OUT_TMP \ + "$(make_csv "$PACKAGES")"\ + $ex_files $INSTALL_BCC $SKIP_DEVICE +rm $ex_files + +# If we only wanted to prepare a rootfs and don't have +# a device connected, then just echo that and skip cleanup +if [ $SKIP_DEVICE -eq 1 ]; then + c_info "Device preparation is being skipped for the selected options" + c_info "any builds that need to happen on device may be cloned but not built." + + if [ ! -z $BI ]; then + sudo $spath/buildimage $OUT_TMP $(dirname $BUILD_IMAGEF)/$(basename $BUILD_IMAGEF) + sudo chmod a+rw $(dirname $BUILD_IMAGEF)/$(basename $BUILD_IMAGEF) + c_info "Your .img has been built! Enjoy!" + fi + + do_cleanup + exit 0 +fi + +# Push tar to device and start unpack +$ADB shell mkdir -p /data/androdeb/ +$ADB push $TDIR/deb.tar.gz /data/androdeb/ +$ADB push $spath/addons/* /data/androdeb/ +$ADB shell /data/androdeb/device-unpack + +# Build BCC and install bcc on device if needed +if [ $INSTALL_BCC -eq 1 ]; then + $ADB shell /data/androdeb/run-command /bcc-master/build-bcc.sh; +fi + +# Extract a tar of the built, compiled and installed androdeb env +if [[ ! -z ${TARDIR+x} ]]; then + c_info "Creating tarball" + pushd $TARDIR + if [ $INSTALL_BCC -eq 0 ]; then + mv $TDIR/deb.tar.gz $FNAME_UZ + else + $ADB shell /data/androdeb/build-debian-tar + $ADB pull /data/androdeb/androdeb-fs.tgz $FNAME_UZ + $ADB shell rm /data/androdeb/androdeb-fs.tgz; + fi + zip -r $FNAME $FNAME_UZ + popd +fi + +do_cleanup + +all_done_banner diff --git a/bcc/build-bcc.sh b/bcc/build-bcc.sh new file mode 100755 index 0000000..a3ddc83 --- /dev/null +++ b/bcc/build-bcc.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# This script should run within a bcc checkout + +spath=$( cd "$(dirname "$0")" ; pwd -P ) +cd $spath + +rm -rf build && mkdir -p build && cd build +cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_C_COMPILER=clang-6.0 -DCMAKE_CXX_COMPILER=clang++-6.0 +make -j4 +make install +cd .. +rm -rf build diff --git a/bcc/build-kheaders-targz.sh b/bcc/build-kheaders-targz.sh new file mode 100755 index 0000000..06eed93 --- /dev/null +++ b/bcc/build-kheaders-targz.sh @@ -0,0 +1,39 @@ +#!/bin/bash +script_full_path=$( cd "$(dirname "$0")" ; pwd -P ) + +if [ $# -ne 2 ]; then + echo "illegal number of parameters, usage: ./build KERNEL_PATH out.tar.gz" + exit 1 +fi + +# Please provide absolute paths +KERNEL_PATH=$1 +OUT_TAR=$2 + +KERNEL_PATH="$(dirname $(readlink -e $KERNEL_PATH))/$(basename $KERNEL_PATH)" +if [ ! -d "$KERNEL_PATH" ]; then + echo "Kernel directory couldn't be found" + exit 3 +fi + +# kdir=$(basename $KERNEL_PATH) + +cd $KERNEL_PATH +find arch -name include -type d -print | xargs -n1 -i: find : -type f > /tmp/kernel-headers.h +find include >> /tmp/kernel-headers.h + +grep "include/generated/autoconf.h" /tmp/kernel-headers.h > /dev/null 2>&1 +retgrep=$? +if [ $retgrep -ne 0 ]; then + >&2 echo "" + >&2 echo "The kernel sources at ${KERNEL_PATH} you pointed to aren't configured and built." + >&2 echo "Please atleast run in your kernel sources:" + >&2 echo $'make defconfig\nmake' + >&2 echo $'\nNote: You dont need to do the full build since headers are generated early on.\n' + >&2 echo "Note: Please build your kernel in tree (build and source should be in same directory)" + >&2 echo "" + exit $retgrep +fi + +cat /tmp/kernel-headers.h | tar -zcf $OUT_TAR -T - +rm /tmp/kernel-headers.h diff --git a/bcc/misc/android-futex-contention-record b/bcc/misc/android-futex-contention-record new file mode 100644 index 0000000..a12ff79 --- /dev/null +++ b/bcc/misc/android-futex-contention-record @@ -0,0 +1,2 @@ +#!/bin/bash +perf record -a -g -e syscalls:sys_enter_futex -e syscalls:sys_exit_futex -e sched:sched_waking $@ diff --git a/bcc/misc/android-futex-contention-report b/bcc/misc/android-futex-contention-report new file mode 100644 index 0000000..7a8c122 --- /dev/null +++ b/bcc/misc/android-futex-contention-report @@ -0,0 +1,4 @@ +#!/bin/bash +# description: futext contention measurement + +perf script $@ -s "$PERF_EXEC_PATH"/scripts/python/android-futex-contention.py diff --git a/bcc/misc/android-futex-contention.py b/bcc/misc/android-futex-contention.py new file mode 100644 index 0000000..5ef3841 --- /dev/null +++ b/bcc/misc/android-futex-contention.py @@ -0,0 +1,115 @@ +# futex contention +# (c) 2010, Arnaldo Carvalho de Melo <acme@redhat.com> +# Licensed under the terms of the GNU GPL License version 2 +# +# Translation of: +# +# http://sourceware.org/systemtap/wiki/WSFutexContention +# +# to perf python scripting. +# +# Measures futex contention + +import os, sys +sys.path.append(os.environ['PERF_EXEC_PATH'] + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') +from Util import * + +process_names = {} +thread_thislock = {} +thread_blocktime = {} + +lock_waits = {} # long-lived stats on (tid,lock) blockage elapsed time +waker_wakee = {} # maps the futex waker to wakee +max_waits = {} # Details about a maximum contention like owner, owner chain +process_names = {} # long-lived pid-to-execname mapping + +def android_lock(callchain): + for c in callchain: + if 'sym' in c and 'name' in c['sym']: + name = c['sym']['name'] + else: + continue + + if 'art::Monitor::Lock' in name: + return True + return False + +def print_callchain(callchain): + for c in callchain: + if 'sym' in c and 'name' in c['sym']: + name = c['sym']['name'] + else: + continue + + print(" %s" % (name)) + +def sched__sched_waking(event_name, context, common_cpu, + common_secs, common_nsecs, common_pid, common_comm, + common_callchain, comm, pid, prio, success, + target_cpu): + waker_wakee[pid] = [common_pid, common_callchain] + +def syscalls__sys_enter_futex(event, ctxt, cpu, s, ns, tid, comm, callchain, + nr, uaddr, op, val, utime, uaddr2, val3): + + cmd = op & FUTEX_CMD_MASK + if cmd != FUTEX_WAIT or android_lock(callchain) == False: + return # we don't care about originators of WAKE events + # or futex uses that aren't android locks. + + process_names[tid] = comm + thread_thislock[tid] = uaddr + thread_blocktime[tid] = nsecs(s, ns) + +def syscalls__sys_exit_futex(event, ctxt, cpu, s, ns, tid, comm, callchain, + nr, ret): + + waker_pid = -1 + waker_chain = "[no call chain]" + + if thread_blocktime.has_key(tid): + # Gather stats about the contention (sum, min, max) + elapsed = nsecs(s, ns) - thread_blocktime[tid] + add_stats(lock_waits, (tid, thread_thislock[tid]), elapsed) + + # Track details about the maximum contention seen + # including owner and its callchain + if (tid, thread_thislock[tid]) in max_waits: + prev_wait = max_waits[(tid, thread_thislock[tid])][0] + else: + prev_wait = 0 + + if elapsed > prev_wait: + if tid in waker_wakee: + waker_pid = waker_wakee[tid][0] + waker_chain = waker_wakee[tid][1] + + max_waits[(tid, thread_thislock[tid])] = [elapsed, waker_pid, waker_chain, callchain] + + del thread_blocktime[tid] + del thread_thislock[tid] + +def trace_begin(): + print "Press control+C to stop and show the summary" + +def trace_end(): + for (tid, lock) in lock_waits: + print("\n==============================================================\n") + min, max, avg, count = lock_waits[tid, lock] + print "%s[%d] lock %x contended %d times, %d avg ns, %d max ns" % \ + (process_names[tid], tid, lock, count, avg, max) + print "" + + if not (tid, lock) in max_waits: + print"Max contention info not available" + continue + + print "Callstack of suffering task:" + print_callchain(max_waits[tid, lock][3]) + print "" + + waker_pid = max_waits[tid, lock][1] + waker_name = process_names[waker_pid] if waker_pid in process_names else "nameless-owner" + print "Owner %s caused this contention of %d ns. Owner's Call stack below:" % (waker_name, max_waits[tid, lock][0]) + print_callchain(max_waits[tid, lock][2]) + diff --git a/bcc/misc/futex-contention.py b/bcc/misc/futex-contention.py new file mode 100644 index 0000000..5c96780 --- /dev/null +++ b/bcc/misc/futex-contention.py @@ -0,0 +1,63 @@ +# futex contention +# (c) 2010, Arnaldo Carvalho de Melo <acme@redhat.com> +# Licensed under the terms of the GNU GPL License version 2 +# +# Translation of: +# +# http://sourceware.org/systemtap/wiki/WSFutexContention +# +# to perf python scripting. +# +# Measures futex contention + +import os, sys +sys.path.append(os.environ['PERF_EXEC_PATH'] + '/scripts/python/Perf-Trace-Util/lib/Perf/Trace') +from Util import * + +process_names = {} +thread_thislock = {} +thread_blocktime = {} + +lock_waits = {} # long-lived stats on (tid,lock) blockage elapsed time +process_names = {} # long-lived pid-to-execname mapping + +def android_lock(callchain): + for c in callchain: + if 'sym' in c and 'name' in c['sym']: + name = c['sym']['name'] + else: + continue + + if 'art::Monitor::Lock' in name: + return True + return False + +def syscalls__sys_enter_futex(event, ctxt, cpu, s, ns, tid, comm, callchain, + nr, uaddr, op, val, utime, uaddr2, val3): + + cmd = op & FUTEX_CMD_MASK + if cmd != FUTEX_WAIT or android_lock(callchain) == False: + return # we don't care about originators of WAKE events + # or futex uses that aren't android locks. + + process_names[tid] = comm + thread_thislock[tid] = uaddr + thread_blocktime[tid] = nsecs(s, ns) + +def syscalls__sys_exit_futex(event, ctxt, cpu, s, ns, tid, comm, callchain, + nr, ret): + if thread_blocktime.has_key(tid): + elapsed = nsecs(s, ns) - thread_blocktime[tid] + add_stats(lock_waits, (tid, thread_thislock[tid]), elapsed) + del thread_blocktime[tid] + del thread_thislock[tid] + +def trace_begin(): + print "Press control+C to stop and show the summary" + +def trace_end(): + for (tid, lock) in lock_waits: + min, max, avg, count = lock_waits[tid, lock] + print "%s[%d] lock %x contended %d times, %d avg ns" % \ + (process_names[tid], tid, lock, count, avg) + diff --git a/bcc/misc/lockstat.py b/bcc/misc/lockstat.py new file mode 100644 index 0000000..fa3edda --- /dev/null +++ b/bcc/misc/lockstat.py @@ -0,0 +1,300 @@ +#!/usr/bin/python +# +# lockstat Trace and display lock contention stats +# +# USAGE: lockstat + +# Licensed under the Apache License, Version 2.0 (the "License") +# 28-Jul-2017 Gisle Dankel Created this. + +from bcc import BPF +from ctypes import c_int +from time import sleep +from datetime import datetime +import argparse +import subprocess +import os + +# One Lock object per TGID and uaddr. +class Lock(object): + def __init__(self): + self.contention_count = 0 + self.elapsed_blocked = 0 + self.thread_count = 0 + self.last_stack_syms = [] + + def update(self, count, block_time, last_stack_syms): + self.contention_count += count + self.elapsed_blocked += block_time + self.thread_count += 1 + self.last_stack_syms = last_stack_syms + +def run_command_get_pid(command): + p = subprocess.Popen(command.split()) + return p.pid + +examples = """ +EXAMPLES: + +./lockstat + Trace calls to sys_futex and display contented locks every 5 seconds + for all processes running on the system +./lockstat -p <pid> + Trace only for the specified pid and display contended locks + every 5 seconds +./lockstat -p <pid> -t + Trace for a specified pid and print a message on each entry and exit to + sys_futex until interrupted or killed +./lockstat -p <pid> 10 + Trace the specified pid and show a message every 10 seconds +./lockstat -c <command> 1 30 + Run the specified command and display contended locks every 1 second + for a total of 30 times +""" + +description = """ +Trace kernel futex events. +These often occur because of lock contention, e.g. involving a pthread_mutex. +This script resemblers the following SystemTap example: +https://sourceware.org/systemtap/SystemTap_Beginners_Guide/futexcontentionsect.html +""" + +parser = argparse.ArgumentParser(description=description, + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=examples) +parser.add_argument("-p", "--pid", type=int, default=-1, + help="the PID to trace; if not specified, trace all") +parser.add_argument("-t", "--trace", action="store_true", + help="print trace messages for each futex enter/exit") +parser.add_argument("interval", nargs="?", default=5, type=int, + help="interval in seconds to print summary") +parser.add_argument("count", nargs="?", type=int, + help="number of times to print the report before exiting") +parser.add_argument("-c", "--command", + help="execute and trace the specified command") + +args = parser.parse_args() + +pid = args.pid +command = args.command +interval = args.interval +num_prints = args.count +trace_all = args.trace + +if command is not None: + print("Executing '%s' and tracing the resulting process." % command) + pid = run_command_get_pid(command) + +bpf_source = """ +#include <uapi/linux/futex.h> +#include <uapi/linux/ptrace.h> +#include <linux/sched.h> +#include <linux/time.h> + +struct comm_t { + char name[TASK_COMM_LEN]; +}; + +struct lock_key_t { + u64 uaddr; + u32 pid; + u32 tgid; +}; + +struct lock_info_t { + u64 elapsed_blocked; + u64 contention_count; + u64 sid; +}; + +BPF_HASH(pid_lock, u32, u64); +BPF_HASH(pid_blocktime, u32, u64); +BPF_HASH(tgid_comm, u32, struct comm_t); +BPF_HASH(lock_stats, struct lock_key_t, struct lock_info_t, 1000000); +BPF_STACK_TRACE(stack_traces, 16384); + +static inline int update_stats(u32 pid, u32 tgid, u64 uaddr, u64 block_time, u64 sid) { + struct lock_key_t key = {}; + struct lock_info_t zero = {}; + struct lock_info_t *info; + + key.pid = pid; + key.tgid = tgid; + key.uaddr = uaddr; + info = lock_stats.lookup_or_init(&key, &zero); + info->elapsed_blocked += block_time; + info->contention_count++; + info->sid = sid; + + if (0 == tgid_comm.lookup(&tgid)) { + struct comm_t comm; + bpf_get_current_comm(&comm.name, sizeof(comm.name)); + tgid_comm.update(&tgid, &comm); + } + return 0; +} + +// FIXME: Should attach to sys_enter_futex and sys_exit_futex tracepoints here, +// but that does not currently work +int sys_futex_enter(struct pt_regs *ctx, u32 *uaddr, int op, u32 val, + struct timespec *utime, u32 *uaddr2, u32 val3) { + int cmd = op & FUTEX_CMD_MASK; + if (cmd != FUTEX_WAIT) + return 0; + + u64 pid_tgid = bpf_get_current_pid_tgid(); + u32 pid = pid_tgid; + u32 tgid = pid_tgid >> 32; + + if (!(THREAD_FILTER)) + return 0; + + u64 timestamp = bpf_ktime_get_ns(); + u64 uaddr64 = (u64) uaddr; + pid_lock.update(&pid, &uaddr64); + pid_blocktime.update(&pid, ×tamp); + + if (SHOULD_PRINT) + bpf_trace_printk("enter sys_futex, pid = %u, uaddr = %x, " + "cmd = %u\\n", pid, uaddr64, cmd); + return 0; +} + +int sys_futex_exit(struct pt_regs *ctx) { + u64 pid_tgid = bpf_get_current_pid_tgid(); + u32 pid = pid_tgid; + u32 tgid = pid_tgid >> 32; + if (!(THREAD_FILTER)) + return 0; + + u64 *blocktime = pid_blocktime.lookup(&pid); + u64 *uaddr = pid_lock.lookup(&pid); + u64 timestamp = bpf_ktime_get_ns(); + u64 elapsed; + u64 sid; + + if (blocktime == 0 || uaddr == 0) + return 0; // not FUTEX_WAIT, or (less likely) missed futex_enter + + elapsed = timestamp - *blocktime; + + sid = stack_traces.get_stackid(ctx, BPF_F_USER_STACK); + update_stats(pid, tgid, *uaddr, elapsed, sid); + pid_lock.delete(&pid); + pid_blocktime.delete(&pid); + + if (SHOULD_PRINT) { + bpf_trace_printk("exit sys_futex, uaddr = %x, elapsed = %uns\\n", + uaddr == 0 ? 0 : *uaddr, elapsed); + } + return 0; +} + +""" + +bpf_source = bpf_source.replace("SHOULD_PRINT", "1" if trace_all else "0") + +thread_filter = '1' +if pid != -1: + print("Tracing pid %d, Ctrl+C to quit." % pid) + # 'tgid' in kernel space is what people thin of as 'pid' in userspace + thread_filter = "tgid == %d" % pid +else: + print("Tracing all processes, Ctrl+C to quit.") + +bpf_source = bpf_source.replace("THREAD_FILTER", thread_filter) + +bpf_program = BPF(text=bpf_source) +bpf_program.attach_kprobe(event="SyS_futex", fn_name="sys_futex_enter") +bpf_program.attach_kretprobe(event="SyS_futex", fn_name="sys_futex_exit") + +def get_syms(stack, pid): + global bpf_program + syms = [] + for addr in stack: + s = bpf_program.sym(addr, pid, show_offset=True) + syms.append(s) + return syms + +def print_syms(syms): + print("=========") + for f in syms: + print(f) + print("=========") + +def is_android_monitor_lock(syms): + for s in syms: + if 'art::Monitor::Lock' in s: + return True + return False + +def disp_stack(stack, pid): + for addr in stack: + s = bpf_program.sym(addr, pid, show_offset=True) + +def create_tgid_stats(): + global bpf_program + stats = bpf_program["lock_stats"] + res = {} + stack_traces = bpf_program['stack_traces'] + for key, val in stats.items(): + # Only display Android monitor locks + if val.sid >= 0: + ust = stack_traces.walk(val.sid) + syms = get_syms(ust, key.pid) + if not is_android_monitor_lock(syms): + continue + else: + continue + + if not key.tgid in res: + res[key.tgid] = {} + if not key.uaddr in res[key.tgid]: + res[key.tgid][key.uaddr] = Lock() + + lock = res[key.tgid][key.uaddr] + lock.update(val.contention_count, val.elapsed_blocked, syms) + return res + +def print_comm_stats(stats): + if stats == {}: + return + + comms = bpf_program["tgid_comm"] + print("\n%s:" % (datetime.now().strftime("%H:%M:%S"))) + for tgid, locks in stats.items(): + comm = comms[c_int(tgid)].name + print("\n %s (%d):" % (comm, tgid)) + sorted_locks = sorted(locks.items(), + key=lambda x: x[1].elapsed_blocked, + reverse=True) + for addr, stats in sorted_locks: + print(" %x: %dms (%d contentions involving %d threads, avg %dus)" % + (addr, stats.elapsed_blocked / 1000000, + stats.contention_count, stats.thread_count, + stats.elapsed_blocked / stats.contention_count / 1000)) + + # No use of displaying lock stacks since we're only + # filtering for Android monitor locks. + # + # print("Last stack for this lock:") + # print_syms(stats.last_stack_syms) + +count_so_far = 0 +while True: + if trace_all: + print(bpf_program.trace_fields()) + else: + try: + sleep(interval) + except KeyboardInterrupt: + exit() + print_comm_stats(create_tgid_stats()) + count_so_far += 1 + bpf_program['tgid_comm'].clear() + bpf_program['lock_stats'].clear() + bpf_program['pid_lock'].clear() + bpf_program['pid_blocktime'].clear() + + if num_prints is not None and count_so_far >= num_prints: + exit() diff --git a/buildimage b/buildimage new file mode 100755 index 0000000..ccdc65d --- /dev/null +++ b/buildimage @@ -0,0 +1,11 @@ +#!/bin/bash +set -e + +# Given a path, make an ext4 image out of it, store it in $2 +dd if=/dev/zero of=$2 bs=4k count=$((256 * 1024 * 2)) +mkfs.ext4 $2 + +OUT=`mktemp -d` +mount -o loop $2 $OUT/ +rsync -ra $1/ $OUT/ +umount $OUT/ diff --git a/buildstrap b/buildstrap new file mode 100755 index 0000000..b33b71e --- /dev/null +++ b/buildstrap @@ -0,0 +1,76 @@ +#!/bin/bash + +spath="$(dirname "$(readlink -f "$0")")" +source $spath/utils/support +source $spath/utils/banners + +ARCH=$1 +DISTRO=$2 +TDIR=$3 +OUT_TMP=$4 +PACKAGES=$5 +EXTRA_FILES="$(cat $6)" +INSTALL_BCC=$7 +SKIP_DEVICE=$8 # Skip any device-specific stages +VARIANT="--variant=minbase" + +time qemu-debootstrap --arch $ARCH --include=$PACKAGES $VARIANT \ + $DISTRO $OUT_TMP http://deb.debian.org/debian/ + +# Some reason debootstrap leaves these mounted +umount $OUT_TMP/proc/sys/fs/binfmt_misc || true +umount $OUT_TMP/proc || true + +# Make bash the default shell +chroot $OUT_TMP rm /bin/sh || true +chroot $OUT_TMP ln -s /bin/bash /bin/sh || true +cp $spath/addons/bashrc $OUT_TMP/.bashrc +cp $spath/addons/bashrc.common $OUT_TMP/.bashrc.common +cp $spath/addons/bashrc.silent $OUT_TMP/.bashrc.silent +cp $spath/addons/get_kvers.sh $OUT_TMP/ + +for f in $EXTRA_FILES; do + if [ $f == "none" ]; then continue; fi + cp $f $OUT_TMP/ +done + +# Cleanup +rm -rf $OUT_TMP/lib/udev/* +rm -rf $OUT_TMP/var/lib/apt/lists/* +rm -rf $OUT_TMP/var/cache/apt/archives/*deb +rm -rf $OUT_TMP/usr/share/locale/* +rm -rf $OUT_TMP/usr/lib/share/locale/* +rm -rf $OUT_TMP/usr/share/doc/* +rm -rf $OUT_TMP/usr/lib/share/doc/* +rm -rf $OUT_TMP/usr/share/ieee-data/* +rm -rf $OUT_TMP/usr/lib/share/ieee-data/* +rm -rf $OUT_TMP/usr/share/man/* +rm -rf $OUT_TMP/usr/lib/share/man/* + +# Fix apt-get issue: Android requires _apt user to be in the +# AID_INET group which is also android specific. +grep -ri _apt:x:100:65534 $OUT_TMP/etc/passwd > /dev/null 2>&1 +if [ $? -ne 0 ]; then + c_warning "_apt user cannot be added to AID_INET group" +else + sed -i -e 's/_apt:x:100:65534/_apt:x:100:3003/' $OUT_TMP/etc/passwd +fi + +# Add a default DNS server +echo "nameserver 4.2.2.2" > $OUT_TMP/etc/resolv.conf + +# Clone BCC if needed +if [ $INSTALL_BCC -eq 1 ]; then + git clone https://github.com/iovisor/bcc.git $TDIR/debian/bcc-master + cp $spath/bcc/build-bcc.sh $TDIR/debian/bcc-master/; +fi + +# Should be really do this? +chmod -R 0777 $TDIR/ + +[ $SKIP_DEVICE -eq 0 ] || exit 0 + +c_info "Compressing new filesystem to prepare to push to Android /data/androdeb/" +tar -zcf $TDIR/deb.tar.gz -C $TDIR debian + +chmod 0777 $TDIR/deb.tar.gz diff --git a/packages/bcc b/packages/bcc new file mode 100644 index 0000000..1d91354 --- /dev/null +++ b/packages/bcc @@ -0,0 +1,25 @@ +PACKAGES+=" +llvm-6.0-dev +libclang-6.0-dev +libelf-dev +libfl-dev +libunwind-dev +libdw-dev +git +gcc +libtool +autoconf +make +cmake +iperf +arping +ethtool +flex +bison +python +clang-6.0 +python-netaddr +python-pyroute2 +" + +INSTALL_BCC=1 diff --git a/packages/compilers b/packages/compilers new file mode 100644 index 0000000..f380dd3 --- /dev/null +++ b/packages/compilers @@ -0,0 +1,9 @@ +PACKAGES+="\ +git +clang-6.0 +gcc +libtool +autoconf +make +cmake +" diff --git a/packages/editors b/packages/editors new file mode 100644 index 0000000..99865fa --- /dev/null +++ b/packages/editors @@ -0,0 +1,5 @@ +PACKAGES+="\ +vim +nano +git +" diff --git a/packages/scheduler b/packages/scheduler new file mode 100644 index 0000000..a5df38c --- /dev/null +++ b/packages/scheduler @@ -0,0 +1,4 @@ +PACKAGES+="\ +git +rt-app +" diff --git a/packages/tracers b/packages/tracers new file mode 100644 index 0000000..38c1d43 --- /dev/null +++ b/packages/tracers @@ -0,0 +1,5 @@ +PACKAGES+="\ +linux-perf +trace-cmd +strace +" diff --git a/utils/banners b/utils/banners new file mode 100755 index 0000000..5f975bc --- /dev/null +++ b/utils/banners @@ -0,0 +1,59 @@ +#!/bin/bash -x + +print_prepare_banner() { + c_info "Preparing device..." + if [ $FULL -eq 1 ]; then + c_info "Doing a full install." + else + c_info "Doing a base install." + fi + c_info "" +} + +usage() { + c_info "USAGE:" + c_info "adeb" + c_info " shell Enter the androdeb shell environment and get to work!" + c_info " remove Remove androdeb from the device" + c_info " git-pull Git pull androdeb to update it on your host" + c_info " pull Copy files from the androdeb filesystem in the device" + c_info " push Copy files to the androdeb filesystem in the device" + c_info "" + c_info " prepare Prepare the device (when running for the first time)" + c_info " By default, this will download and install a base image." + c_info " ** Folowing are the prepare options **" + c_info " --full Pass this to prepare to download and install the full image which" + c_info " contains compilers, editors, tracers etc." + c_info "" + c_info " --build Instead of download, build and install the image onto the device" + c_info "" + c_info " --archive Use archive for root fs (overrides all other prepare options)" + c_info "" + c_info " --buildtar While preparing, also build a tar.gz.zip file of the filesystem," + c_info " this is how images that are downloaded by prepare are built" + c_info "" + c_info " --build-image Build an ext4 .img with the base image and BCC (useful for Qemu)" + c_info "" + c_info " ** Folowing are misc build options **" + c_info " --tempdir Use a specific temporary directory for build operation" + c_info " --arch Specify an ARCH to build for (default arm64)" + c_info " --distro Debian distro to base on (default is buster)" + c_info "" + c_info " ** Folowing are the options for BCC **" + c_info " --bcc Build and install BCC onto the device, from source" + c_info " BCC is already included in 'prepare --full'" + c_info "" + c_info " --kernelsrc Extract kernel headers for BCC from this directory" + c_info "" + c_info " --skip-install Pass this to --kernelsrc if you wish to only extra/install kernel headers" + c_info " and would like to exit after that (skips build/install of everything else)" + c_info "" + c_info " ** Folowing are device specific options ** " + c_info " --device Serial number of adb device." + c_info " -s Serial number of adb device." + c_info "" + c_info " --debug Debug all execution." + exit 1 +} + + diff --git a/utils/packheaders.sh b/utils/packheaders.sh new file mode 100755 index 0000000..d10b81b --- /dev/null +++ b/utils/packheaders.sh @@ -0,0 +1,25 @@ +#!/bin/bash + +# Utility to build kernel headers tar/zip file +# must be run from the top level of a kernel source directory +# and supplied an output file name + +MKTEMP=0; if [[ -z ${TDIR+x} ]] || [[ ! -d "${TDIR}" ]]; then + TDIR=`mktemp -d`; MKTEMP=1; fi +rm -rf $TDIR/* +TDIR_ABS=$( cd "$TDIR" ; pwd -P ) + + +if [ $# -ne 1 ]; then + echo "usage: makeheaders.sh <output file name>" + exit 0 +fi + +mkdir -p $TDIR_ABS/kernel-headers + +find arch -name include -type d -print | xargs -n1 -i: find : -type f -exec cp --parents {} $TDIR_ABS/kernel-headers/ \; +find include -exec cp --parents {} $TDIR_ABS/kernel-headers/ 2> /dev/null \; +tar -zcf $1 --directory=$TDIR_ABS kernel-headers + +zip -r $1.zip $1 +rm -rf $TDIR/*; if [ $MKTEMP -eq 1 ]; then rm -rf $TDIR; fi diff --git a/utils/support b/utils/support new file mode 100755 index 0000000..ad72e05 --- /dev/null +++ b/utils/support @@ -0,0 +1,175 @@ +#!/bin/bash -x +# Utilities to interact with android more easily + +function run_quiet() { eval "$* >/dev/null 2>&1"; } + +make_csv() { + out="" + in=$1 + for p in $in; do + if [ "x$out" == "x" ]; then + out=$p + else + out="$out,$p" + fi + done + echo $out +} + +make_spaces() { + out="" + in=$1 + for p in $in; do + if [ "x$out" == "x" ]; then + out=$p + else + out="$out $p" + fi + done + echo $out +} + +cmd_exists() { + which $1 > /dev/null + return $? +} + +do_adb_root() { + ADB="$1" + $ADB root > /dev/null 2>&1 + return $? +} + +die() { + exit_code=$1 + msg=$2 + c_error "$msg" + exit $exit_code +} + +die_if_no_androdeb() { + set +e + $ADB shell ls /data/androdeb/debian > /dev/null 2>&1 + if [ $? -ne 0 ]; then die 8 "Existing androdeb env not found on device. $1"; fi + set -e +} + +# Helper function to count number of source arguments in a list +# when more than one argument is supplied, it is assumed the last argument +# is a destination +count_sources() { + local source_count=$# + if [ $source_count -gt 1 ]; then + source_count=$((source_count - 1)) + fi + echo "$source_count" +} + +# Borrowed from project LISA. +################################################################################ +# Logging functions +################################################################################ +c_error() { + NOW=$(date +"%H:%m:%S") + # If there is only one parameter, let's assume it's just the message + if [ $# -gt 1 ]; then + local parent_lineno="$1" + local message="$2" + echo -e "${red}$NOW - ERROR: on or near line ${parent_lineno}: ${message}${nocol}" + return + fi + + local message="$1" + echo -e "${red}$NOW - ERROR : ${message}${nocol}" +} + +c_warning() { + NOW=$(date +"%H:%m:%S") + # If there is only one parameter, let's assume it's just the message + if [ $# -gt 1 ]; then + local parent_lineno="$1" + local message="$2" + echo -e "${yellow}$NOW - WARNING: on or near line ${parent_lineno}: ${message}${nocol}" + return + fi + local message="$1" + echo -e "${yellow}$NOW - WARNING : ${message}${nocol}" +} + +c_info() { + NOW=$(date +"%H:%m:%S") + # If there is only one parameter, let's assume it's just the message + if [ $# -gt 1 ]; then + local parent_lineno="$1" + local message="$2" + echo -e "${blue}$NOW - INFO: on or near line ${parent_lineno}: ${message}${nocol}" + return + fi + local message="$1" + echo -e "${blue}$NOW - INFO : ${message}${nocol}" +} + +d_notify() { + MESSAGE=$1 + ICON=$2 + # Let's try to send a desktop notification, + # silently fails if there is not support. + notify-send \ + --icon=$ICON \ + --urgency=critical \ + --expire-time=1500 \ + "Test Series" \ + "$MESSAGE" \ + 2>/dev/null +} + +my_tput() { + if [ "${TERM-dumb}" == dumb ]; then + return + fi + tput $* +} + +box_out() +{ + local s=("$@") b w + for l in "${s[@]}"; do + ((w<${#l})) && { b="$l"; w="${#l}"; } + done + my_tput setaf 3 + echo -e "|-${b//?/-}-|" + for l in "${s[@]}"; do + printf '| %s%*s%s |\n' "$(my_tput setaf 4)" "-$w" "$l" "$(my_tput setaf 3)" + # echo "|-${b//?/-}-|" + done + echo "|-${b//?/-}-|" + my_tput sgr 0 +} + + +################################################################################ +# Colors +################################################################################ + +if [ -t 1 ]; then + ncolors=$(my_tput colors) + if [ -n "${ncolors}" ] && [ ${ncolors} -ge 8 ]; then + nocol='\e[0m' # No Color + white='\e[1;37m' + black='\e[0;30m' + blue='\e[0;34m' + lblue='\e[1;34m' + green='\e[0;32m' + lgreen='\e[1;32m' + cyan='\e[0;36m' + lcyan='\e[1;36m' + red='\e[0;31m' + lred='\e[1;31m' + purple='\e[0;35m' + lpurple='\e[1;35m' + brown='\e[0;33m' + yellow='\e[1;33m' + grey='\e[0;30m' + lgrey='\e[0;37m' + fi +fi |