1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
|
<!-- Copyright 2015 The Chromium Authors. All rights reserved.
Use of this source code is governed by a BSD-style license that can be
found in the LICENSE file.
-->
# Telemetry
Telemetry is the performance testing framework used by Chrome. It allows you
to perform arbitrary actions on a set of web pages (or any android application!)
and report metrics about it. The framework abstracts:
* Launching a browser with arbitrary flags on any platform.
* Opening a tab and navigating to the page under test.
* Launching an Android application with intents through ADB.
* Fetching data via the Inspector timeline and traces.
* Using [Web Page Replay](https://github.com/chromium/web-page-replay) to
cache real-world websites so they don’t change when used in benchmarks.
## Design Principles
* Write one performance test that runs on major platforms - Windows, Mac,
Linux, Chrome OS, and Android for both Chrome and ContentShell.
* Run on browser binaries, without a full Chromium checkout, and without
having to build the browser yourself.
* Use Web Page Replay to get repeatable test results.
* Clean architecture for writing benchmarks that keeps measurements and use
cases separate.
**Telemetry is designed for measuring performance rather than checking
correctness. If you want to check for correctness,
[browser tests](http://www.chromium.org/developers/testing/browser-tests) are
your friend.**
**If you are a Chromium developer looking to add a new Telemetry benchmark to
[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/),
please make sure to read our
[Benchmark Policy](https://docs.google.com/document/d/1bBKyYCW3VlUUPDpQE4xvrMFdA6tovQMZoqO9KCcmqqQ/preview)
first.**
## Code Concepts
Telemetry provides two major functionality groups: those that provide test
automation, and those that provide the capability to collect data.
### Test Automation
The test automation facilities of Telemetry provide Python wrappers for a number
of different system concepts.
* _Platforms_ use a variety of libraries & tools to abstract away the OS
specific logic.
* _Browser_ wraps Chrome's
[DevTools Remote Debugging Protocol](https://developer.chrome.com/devtools/docs/remote-debugging)
to perform actions and extract information from the browser.
* _Android App_ is a Python wrapper around
[`adb shell`](http://developer.android.com/tools/help/adb.html).
The Telemetry framework lives in
[`src/third_party/catapult/telemetry/`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/)
and performance benchmarks that use Telemetry live in
[`src/tools/perf/`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/).
### Data Collection
Telemetry offers a framework for collecting metrics that quantify the
performance of automated actions in terms of benchmarks, measurements, and story
sets.
* A
[_benchmark_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark.py)
combines a _measurement_ together with a _story set_, and optionally a set
of browser options.
* We strongly discourage benchmark authors from using command-line flags
to specify the behavior of benchmarks, since benchmarks should be
cross-platform.
* Benchmarks are discovered and run by the
[benchmark runner](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/benchmark_runner.py),
which is wrapped by scripts like
[`run_benchmark`](https://code.google.com/p/chromium/codesearch#chromium/src/tools/perf/run_benchmark)
in `tools/perf`.
* A _measurement_ (called
[`StoryTest`](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/story_test.py)
in the code) is responsible for setting up and tearing down the testing
platform, and for collecting _metrics_ that quantify the application
scenario under test.
* Measurements need to work with all story sets, to provide consistency
and prevent benchmark rot.
* You probably don't need to override `StoryTest` (see "Timeline Based
Measurement" below). If you think you do, please talk to us.
* A
[_story set_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story_set.py)
is a set of _stories_ together with a
[_shared state_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/shared_state.py)
that describes application-level configuration options.
* A
[_story_](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/story/story.py)
is an application scenario and a set of actions to run in that scenario. In
the typical Chromium use case, this will be a web page together with actions
like scrolling, clicking, or executing JavaScript.
* A _metric_ describes how to collect data about the story run and compute
results.
* New metrics should generally be
[timeline-based](https://cs.chromium.org/chromium/src/third_party/catapult/telemetry/telemetry/web_perf/metrics/timeline_based_metric.py).
* Metrics can specify many different types of results, including numbers,
histograms, traces, and failures.
* _Timeline Based Measurement_ is a built-in `StoryTest` that runs all
available timeline-based metrics, and benchmarks that use it can filter
relevant results.
## Next Steps
* [Run Telemetry benchmarks locally](/telemetry/docs/run_benchmarks_locally.md)
* [Record a story set](https://sites.google.com/a/chromium.org/dev/developers/telemetry/record_a_page_set)
with Web Page Replay
* [Add a measurement](https://sites.google.com/a/chromium.org/dev/developers/telemetry/add_a_measurement)
* [Feature guidelines](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-feature-guidelines)
* [Profiling with Telemetry](https://sites.google.com/a/chromium.org/dev/developers/telemetry/profiling)
* [Profile generation](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-profile-generation)
* [Telemetry unittests](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-unittests)
## Contact Us or Follow Along
If you have questions, please email telemetry@chromium.org.
You can keep up with Telemetry related discussions by joining the
[telemetry group](https://groups.google.com/a/chromium.org/forum/#!forum/telemetry).
[For Googlers](http://go/telemetry)
## Frequently Asked Questions
### I get an error when I try to use recorded story sets.
The recordings are not included in the Chromium source tree. If you are a Google
partner, run `gsutil config` to authenticate, then try running the test again.
If you don't have `gsutil` installed on your machine, you can find it in
`build/third_party/gsutil/gsutil`.
If you are not a Google partner, you can run on live sites with
--use-live-sites` or
[record your own](http://dev.chromium.org/developers/telemetry/record_a_page_set)
story set archive.
### I get mysterious errors about device\_forwarder failing.
Your forwarder binary may be outdated. If you have built the forwarder in
src/out that one will be used. if there isn't anything there Telemetry will
default to downloading a pre-built binary. Try re-building the forwarder, or
alternatively wiping the contents of `src/out/` and running `run_benchmark`,
which should download the latest binary.
### I'm having problems with keychain prompts on Mac.
Make sure that your keychain is
[correctly configured](https://sites.google.com/a/chromium.org/dev/developers/telemetry/telemetry-mac-keychain-setup).
|
