1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
|
<html devsite>
<head>
<title>Debugging Native Android Platform Code</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>This section summarizes useful tools and related commands for debugging,
tracing, and profiling native Android platform code when developing
platform-level features.</p>
<p class="note"><strong>Note:</strong> The pages in this section and elsewhere
within this site recommend use of <code>adb</code> in conjunction with the
<code>setprop</code> argument to debug certain aspects of Android. Please note,
on pre-O versions of the Android OS, property names had a length limit of 32
characters. This meant that to create a wrap property with the name of the app,
it was necessary to truncate the name to fit. In Android O and later, this
limit is much greater and should require no truncation.</p>
<p>This page covers the basics surrounding crash dumps found in logcat output.
Other pages have far more detail about
<a href="/devices/tech/debug/native-crash.html">diagnosing native crashes</a>,
exploring system services with
<a
href="https://developer.android.com/studio/command-line/dumpsys.html">
<code>dumpsys</code></a>, viewing
<a href="/devices/tech/debug/native-memory.html">native memory</a>,
<a href="https://developer.android.com/studio/command-line/dumpsys.html#network">network</a>, and
<a href="https://developer.android.com/studio/command-line/dumpsys.html#procstats">RAM</a> usage, using
<a href="/devices/tech/debug/asan.html">AddressSanitizer</a> to detect memory
bugs in native code, evaluating
<a href="/devices/tech/debug/eval_perf.html"> performance issues</a> (includes
<a href="/devices/tech/debug/systrace">systrace</a>), and using the
<a href="/devices/tech/debug/gdb.html">GNU debugger (GDB)</a> and
other debugging tools.</p>
<h2 id=debuggerd>Crash dumps</h2>
<p>When a dynamically linked executable starts, several signal handlers are
registered that, in the event of a crash, cause a basic crash dump to be written to logcat
and a more detailed "tombstone" file to be written to <code>/data/tombstones/</code>.
The tombstone is a file with extra data about the crashed process. In particular, it contains
stack traces for all the threads in the crashing process (not just the thread that caught the
signal), a full memory map, and a list of all open file descriptors.</p>
Before Android 8.0, crashes were handled by the
<code>debuggerd</code> and <code>debuggerd64</code> daemons. In Android O and later,
<code>crash_dump32</code> and <code>crash_dump64</code> are spawned as needed.</p>
<p>It's possible for the crash dumper to attach only if nothing else is
already attached, which means using tools such as <code>strace</code> or
<code>gdb</code> will prevent crash dumps from occurring.</p>
<p>Example output (with timestamps and extraneous information removed):</p>
<pre class="devsite-click-to-copy">
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
Revision: '0'
ABI: 'arm'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
backtrace:
#00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
#01 pc 0001aa1b /system/lib/libc.so (readdir+10)
#02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
#03 pc 0000184b /system/xbin/crasher (do_action+978)
#04 pc 00001459 /system/xbin/crasher (thread_callback+24)
#05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
#06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
Tombstone written to: /data/tombstones/tombstone_06
</pre>
<p>The last line of output gives the location of the full <em>tombstone</em> on disk.</p>
<p>Assuming you have the unstripped binaries available, you can get a more detailed
unwind with line number information by pasting the stack into
<code>development/scripts/stack</code>:</p>
<p class="key-point"><strong>Tip:</strong> For convenience, if you've run <code>lunch</code>,
then <code>stack</code> will be on your $PATH already so you don't need to give the
full path.</p>
<pre class="devsite-terminal devsite-click-to-copy">
development/scripts/stack
</pre>
<p>Example output (based on the logcat output above):</p>
<pre class="devsite-click-to-copy">
Reading native crash info from stdin
03-02 23:53:49.477 17951 17951 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
03-02 23:53:49.477 17951 17951 F DEBUG : Build fingerprint: 'Android/aosp_angler/angler:7.1.1/NYC/enh12211018:eng/test-keys'
03-02 23:53:49.477 17951 17951 F DEBUG : Revision: '0'
03-02 23:53:49.477 17951 17951 F DEBUG : ABI: 'arm'
03-02 23:53:49.478 17951 17951 F DEBUG : pid: 17946, tid: 17949, name: crasher >>> crasher <<<
03-02 23:53:49.478 17951 17951 F DEBUG : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
03-02 23:53:49.478 17951 17951 F DEBUG : r0 0000000c r1 00000000 r2 00000000 r3 00000000
03-02 23:53:49.478 17951 17951 F DEBUG : r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
03-02 23:53:49.478 17951 17951 F DEBUG : r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
03-02 23:53:49.478 17951 17951 F DEBUG : ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
03-02 23:53:49.491 17951 17951 F DEBUG :
03-02 23:53:49.491 17951 17951 F DEBUG : backtrace:
03-02 23:53:49.492 17951 17951 F DEBUG : #00 pc 0004793e /system/lib/libc.so (pthread_mutex_lock+1)
03-02 23:53:49.492 17951 17951 F DEBUG : #01 pc 0001aa1b /system/lib/libc.so (readdir+10)
03-02 23:53:49.492 17951 17951 F DEBUG : #02 pc 00001b91 /system/xbin/crasher (readdir_null+20)
03-02 23:53:49.492 17951 17951 F DEBUG : #03 pc 0000184b /system/xbin/crasher (do_action+978)
03-02 23:53:49.492 17951 17951 F DEBUG : #04 pc 00001459 /system/xbin/crasher (thread_callback+24)
03-02 23:53:49.492 17951 17951 F DEBUG : #05 pc 00047317 /system/lib/libc.so (_ZL15__pthread_startPv+22)
03-02 23:53:49.492 17951 17951 F DEBUG : #06 pc 0001a7e5 /system/lib/libc.so (__start_thread+34)
03-02 23:53:49.492 17951 17951 F DEBUG : Tombstone written to: /data/tombstones/tombstone_06
Reading symbols from /huge-ssd/aosp-arm64/out/target/product/angler/symbols
Revision: '0'
pid: 17946, tid: 17949, name: crasher >>> crasher <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xc
r0 0000000c r1 00000000 r2 00000000 r3 00000000
r4 00000000 r5 0000000c r6 eccdd920 r7 00000078
r8 0000461a r9 ffc78c19 sl ab209441 fp fffff924
ip ed01b834 sp eccdd800 lr ecfa9a1f pc ecfd693e cpsr 600e0030
Using arm toolchain from: /huge-ssd/aosp-arm64/prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin/
Stack Trace:
RELADDR FUNCTION FILE:LINE
0004793e pthread_mutex_lock+2 bionic/libc/bionic/pthread_mutex.cpp:515
v------> ScopedPthreadMutexLocker bionic/libc/private/ScopedPthreadMutexLocker.h:27
0001aa1b readdir+10 bionic/libc/bionic/dirent.cpp:120
00001b91 readdir_null+20 system/core/debuggerd/crasher.cpp:131
0000184b do_action+978 system/core/debuggerd/crasher.cpp:228
00001459 thread_callback+24 system/core/debuggerd/crasher.cpp:90
00047317 __pthread_start(void*)+22 bionic/libc/bionic/pthread_create.cpp:202 (discriminator 1)
0001a7e5 __start_thread+34 bionic/libc/bionic/clone.cpp:46 (discriminator 1)
</pre>
<p class="note"><strong>Note:</strong> Some system libraries are built with
<code>LOCAL_STRIP_MODULE := keep_symbols</code> to provide usable backtraces
directly without taking up anywhere near as much space as an unstripped version.</p>
<p>You can also <code>stack</code> an entire tombstone. Example:</p>
<pre class="devsite-terminal devsite-click-to-copy">
stack < FS/data/tombstones/tombstone_05
</pre>
<p>This is useful if you've just unzipped a bugreport in the current directory.
For more information about diagnosing native crashes and tombstones, see
<a href="/devices/tech/debug/native-crash.html">Diagnosing Native Crashes</a>.
</p>
<h2 id="tombstone">Getting a stack trace/tombstone from a running process</h2>
<p>You can also use the <code>debuggerd</code> tool to get a stack dump from a running process.
From the command line, invoke <code>debuggerd</code> using a process ID (PID) to dump a
full tombstone to <code>stdout</code>. To get just the stack for every thread in
the process, include the <code>-b</code> or <code>--backtrace</code> flag.</p>
<h2 id="complex">Understanding a complex unwind</h2>
<p>When an app crashes, the stack tends to be pretty complex.
The following detailed example highlights many of the complexities:</p>
<pre class="devsite-click-to-copy">
#00 pc 00000000007e6918 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#01 pc 00000000001845cc /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#02 pc 00000000001847e4 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000)
#03 pc 00000000001805c0 /system/priv-app/Velvet/Velvet.apk (offset 0x346b000) (Java_com_google_speech_recognizer_AbstractRecognizer_nativeRun+176)
</pre>
<p>Frames #00-#03 are from native JNI code that was stored uncompressed in the APK to save disk
space rather than being extracted into a separate <code>.so</code> file. The stack unwinder in
Android 9 doesn’t need the extracted <code>.so</code> file to be able to cope with this common
Android-specific case.</p>
<p>Frames #00-#02 don’t have symbol names because they were stripped by the developer.</p>
<p>Frame #03 shows that where symbols are available, the unwinder uses them.</p>
<pre class="devsite-click-to-copy">
#04 pc 0000000000117550 /data/dalvik-cache/arm64/system@priv-app@Velvet@Velvet.apk@classes.dex (offset 0x108000) (com.google.speech.recognizer.AbstractRecognizer.nativeRun+160)
</pre>
<p>Frame #04 is ahead-of-time compiled Java code. The old unwinder would have stopped here, unable
to unwind through Java.</p>
<pre class="devsite-click-to-copy">
#05 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#06 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#07 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#08 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#09 pc 000000000052abc0 /system/lib64/libart.so (MterpInvokeDirect+296)
#10 pc 000000000054c614 /system/lib64/libart.so (ExecuteMterpImpl+14484)
</pre>
<p>Frames #05-#10 are from the ART interpreter implementation.
The stack unwinder in releases prior to Android 9 would have shown these frames without the context
of frame #11 explaining what code the interpreter was interpreting. These frames are useful if
you're debugging ART itself. If you're debugging an app, you can ignore them. Some tools, such as
<code>simpleperf</code>, will automatically omit these frames.</p>
<pre class="devsite-click-to-copy">
#11 pc 00000000001992d6 /system/priv-app/Velvet/Velvet.apk (offset 0x26cf000) (com.google.speech.recognizer.AbstractRecognizer.run+18)
</pre>
<p>Frame #11 is the Java code being interpreted.</p>
<pre class="devsite-click-to-copy">
#12 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#13 pc 000000000025a328 /system/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+216)
#14 pc 000000000027ac90 /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+920)
#15 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584)
#16 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228)
</pre>
<p>Frames #12-#16 are more of the interpreter implementation itself.</p>
<pre class="devsite-click-to-copy">
#17 pc 00000000002454a0 /system/priv-app/Velvet/Velvet.apk (offset 0x1322000) (com.google.android.apps.gsa.speech.e.c.c.call+28)
</pre>
<p>Frame #17 is more Java code being interpreted. This Java method corresponds to interpreter frames #12-#16.</p>
<pre class="devsite-click-to-copy">
#18 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#19 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#20 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
</pre>
<p>Frames #18-#20 are more of the VM itself, code to transition from compiled Java code to interpreted Java code.</p>
<pre class="devsite-click-to-copy">
#21 pc 00000000002ce44c /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.FutureTask.run+204)
</pre>
<p>Frame #21 is the compiled Java method that calls the Java method in #17.</p>
<pre class="devsite-click-to-copy">
#22 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#23 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#24 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#25 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#26 pc 0000000000529880 /system/lib64/libart.so (MterpInvokeVirtual+584)
#27 pc 000000000054c514 /system/lib64/libart.so (ExecuteMterpImpl+14228)
</pre>
<p>Frames #22-#27 are the interpreter implementation, making a method invocation from interpreted
code to a compiled method.</p>
<pre class="devsite-click-to-copy">
#28 pc 00000000003ed69e /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.e.run+22)
</pre>
<p>Frame #28 is the Java code being interpreted.</p>
<pre class="devsite-click-to-copy">
#29 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#30 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#31 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
</pre>
<p>Frames #29-#31 are another transition between compiled code and interpreted code.</p>
<pre class="devsite-click-to-copy">
#32 pc 0000000000329284 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor.runWorker+996)
#33 pc 00000000003262a0 /system/framework/arm64/boot.oat (offset 0xdc000) (java.util.concurrent.ThreadPoolExecutor$Worker.run+64)
#34 pc 00000000002037e8 /system/framework/arm64/boot.oat (offset 0xdc000) (java.lang.Thread.run+72)
</pre>
<p>Frames #32-#34 are compiled Java frames calling each other directly. In this case the native call
stack is the same as the Java call stack.</p>
<pre class="devsite-click-to-copy">
#35 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#36 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#37 pc 0000000000280cf0 /system/lib64/libart.so (art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*)+344)
#38 pc 000000000027acac /system/lib64/libart.so (bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)+948)
#39 pc 0000000000529f10 /system/lib64/libart.so (MterpInvokeSuper+1408)
#40 pc 000000000054c594 /system/lib64/libart.so (ExecuteMterpImpl+14356)
</pre>
<p>Frames #35-#40 are the interpreter itself.</p>
<pre class="devsite-click-to-copy">
#41 pc 00000000003ed8e0 /system/priv-app/Velvet/Velvet.apk (com.google.android.apps.gsa.shared.util.concurrent.b.i.run+20)
</pre>
<p>Frame #41 is the Java code being interpreted.</p>
<pre class="devsite-click-to-copy">
#42 pc 00000000002547a8 /system/lib64/libart.so (_ZN3art11interpreterL7ExecuteEPNS_6ThreadERKNS_20CodeItemDataAccessorERNS_11ShadowFrameENS_6JValueEb.llvm.780698333+496)
#43 pc 0000000000519fd8 /system/lib64/libart.so (artQuickToInterpreterBridge+1032)
#44 pc 00000000005630fc /system/lib64/libart.so (art_quick_to_interpreter_bridge+92)
#45 pc 0000000000559f88 /system/lib64/libart.so (art_quick_invoke_stub+584)
#46 pc 00000000000ced40 /system/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+200)
#47 pc 0000000000460d18 /system/lib64/libart.so (art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*)+104)
#48 pc 0000000000461de0 /system/lib64/libart.so (art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*)+424)
#49 pc 000000000048ccb0 /system/lib64/libart.so (art::Thread::CreateCallback(void*)+1120)
</pre>
<p>Frames #42-#49 are the VM itself. This time it's the code that starts running Java on a new thread.</p>
<pre class="devsite-click-to-copy">
#50 pc 0000000000082e24 /system/lib64/libc.so (__pthread_start(void*)+36)
#51 pc 00000000000233bc /system/lib64/libc.so (__start_thread+68)
</pre>
<p>Frames #50-#51 are how all threads should actually start. This is the <code>libc</code>
new thread start code.
A handy tip is that you know you have a successful and complete unwind if you see these frames.
Any time you don't see these frames at the bottom of your stack, you should be suspicious that
you're either looking at a truncated stack or at a corrupted stack: you shouldn't blindly
assume that the stack is correct if it doesn't end this way.</p>
</body>
</html>
|
