pw_tokenizer: Replace string literals with tokens

pw_tokenizer provides macros that replace printf-style string literals
with 32-bit hashes at compile time. The string literals are removed
from the resulting binary, which dramatically reduces the binary size.
Like any printf-style string, binary versions of the strings can be
formatted with arguments and then transmitted or stored.

The pw_tokenizer module is general purpose, but its most common use case
is binary logging. In binary logging, human-readable text logs are
replaced with binary tokens. These are decoded off-device.

This commit includes the C and C++ code for tokenizing strings. It also
includes a C++ library for decoding tokenized strings.

Change-Id: I6d5737ab2d6dfdd76dcf70c852b547fdcd68d683

1 files changed, 1 insertions, 0 deletions

diff --git a/docs/BUILD.gn b/docs/BUILD.gn
index d4c376717..050b3266e 100644
--- a/docs/BUILD.gn
+++ b/docs/BUILD.gn
@@ -57,5 +57,6 @@ pw_doc_gen("docs") {
     "$dir_pw_status:docs",
     "$dir_pw_string:docs",
     "$dir_pw_target_runner:docs",
+    "$dir_pw_tokenizer:docs",
   ]
 }