path: root/READMEs/README.cbor-lecp.md

# RFC8949 CBOR Stream Parsing and Writing

|||
|---|---|---|
|cmake| `LWS_WITH_CBOR`, `LWS_WITH_CBOR_FLOAT`|
|Header| ./include/libwebsockets/lws-lecp.h|
|api-test| ./minimal-examples/api-tests/api-test-lecp/|
|test app| ./test-apps/test-lecp.c -> libwebsockets-test-lecp|

LECP is the RFC8949 CBOR stream parsing counterpart to LEJP for JSON.

## Features

 - Completely immune to input fragmentation, give it any size blocks of CBOR as
   they become available; 1 byte, or 100K at a time give identical parsing
   results
 - Input chunks discarded as they are parsed, whole CBOR never needed in memory
 - Nonrecursive, fixed stack usage of a few dozen bytes
 - No heap allocations at all, just requires ~500 byte context usually on
   caller stack
 - Creates callbacks to a user-provided handler as members are parsed out
 - No payload size limit, supports huge / endless strings or blobs bigger than
   system memory
 - Collates utf-8 text and blob payloads into a 250-byte chunk buffer for ease
   of access
 - Write apis don't use any heap allocations or recursion either
 - Write apis use an explicit context with its own lifecycle, and printf style
   vaargs including sized blobs, C strings, double, int, unsigned long etc
 - Completely immune to output fragmentation, supports huge strings and blobs
   into small buffers, api returns to indicates unfinished if it needs to be
   called again to continue; 1 byte or 100K output buffer give same results
 - Write apis completely fill available buffer and if unfinished, continues
   into same or different buffer when called again with same args; no
   requirement for subsequent calls to be done sequentially or even from same
   function

## Type limits

CBOR allows negative integers of up to 64 bits, these do not fit into a `uint64_t`.
LECP has a union for numbers that includes the types `uint64_t` and `int64_t`,
but it does not separately handle negative integers.  Only -2^63.. 2^64 -1 can
be handled by the C types, the oversize negative numbers wrap and should be
avoided.

## Floating point support

Floats are handled using the IEEE memory format, it means they can be parsed
from the CBOR without needing any floating point support in the build.  If
floating point is available, you can also enable `LWS_WITH_CBOR_FLOAT` and
a `float` and `double` types are available in the number item union.  Otherwise
these are handled as `ctx->item.u.u32` and `ctx->item.u.u64` union members.

Half-float (16-bit) is defined in CBOR and always handled as a `uint16_t`
number union member `ctx->item.u.hf`.

## Callback reasons

The user callback does not have to handle any callbacks, it only needs to
process the data for the ones it is interested in.

|Callback reason|CBOR structure|Associated data|
|---|---|---|
|`LECPCB_CONSTRUCTED`|Created the parse context||
|`LECPCB_DESTRUCTED`|Destroyed the parse context||
|`LECPCB_COMPLETE`|The parsing completed OK||
|`LECPCB_FAILED`|The parsing failed||
|`LECPCB_VAL_TRUE`|boolean true||
|`LECPCB_VAL_FALSE`|boolean false||
|`LECPCB_VAL_NULL`|explicit NULL||
|`LECPCB_VAL_NUM_INT`|signed integer|`ctx->item.u.i64`|
|`LECPCB_VAL_STR_START`|A UTF-8 string is starting||
|`LECPCB_VAL_STR_CHUNK`|The next string chunk|`ctx->npos` bytes in `ctx->buf`|
|`LECPCB_VAL_STR_END`|The last string chunk|`ctx->npos` bytes in `ctx->buf`|
|`LECPCB_ARRAY_START`|An array is starting||
|`LECPCB_ARRAY_END`|An array has ended||
|`LECPCB_OBJECT_START`|A CBOR map is starting||
|`LECPCB_OBJECT_END`|A CBOR map has ended||
|`LECPCB_TAG_START`|The following data has a tag index|`ctx->item.u.u64`|
|`LECPCB_TAG_END`|The end of the data referenced by the last tag||
|`LECPCB_VAL_NUM_UINT`|Unsigned integer|`ctx->item.u.u64`|
|`LECPCB_VAL_UNDEFINED`|CBOR undefined||
|`LECPCB_VAL_FLOAT16`|half-float available as host-endian `uint16_t`|`ctx->item.u.hf`|
|`LECPCB_VAL_FLOAT32`|`float` (`uint32_t` if no float support) available|`ctx->item.u.f`|
|`LECPCB_VAL_FLOAT64`|`double` (`uint64_t` if no float support) available|`ctx->item.u.d`|
|`LECPCB_VAL_SIMPLE`|CBOR simple|`ctx->item.u.u64`|
|`LECPCB_VAL_BLOB_START`|A binary blob is starting||
|`LECPCB_VAL_BLOB_CHUNK`|The next blob chunk|`ctx->npos` bytes in `ctx->buf`|
|`LECPCB_VAL_BLOB_END`|The last blob chunk|`ctx->npos` bytes in `ctx->buf`|
|`LECPCB_ARRAY_ITEM_START`|A logical item in an array is starting|
|`LCEPDB_ARRAY_ITEM_END`|A logical item in an array has completed|

## CBOR indeterminite lengths

Indeterminite lengths are supported, but are concealed in the parser as far as
possible, the CBOR lengths or its indeterminacy are not exposed in the callback
interface at all, just chunks of data that may be the start, the middle, or the
end.

## Handling CBOR UTF-8 strings and blobs

When a string or blob is parsed, an advisory callback of `LECPCB_VAL_STR_START` or
`LECPCB_VAL_BLOB_START` occurs first.  The `_STR_` callbacks indicate the
content is a CBOR UTF-8 string, `_BLOB_` indicates it is binary data.

Strings or blobs may have indeterminite length, but if so, they are composed
of logical chunks which must have known lengths.  When the `_START` callback
occurs, the logical length either of the whole string, or of the sub-chunk if
indeterminite length, can be found in `ctx->item.u.u64`.

Payload is collated into `ctx->buf[]`, the valid length is in `ctx->npos`.

For short strings or blobs where the length is known, the whole payload is
delivered in a single `LECPCB_VAL_STR_END` or `LECPCB_VAL_BLOB_END` callback.

For payloads larger than the size of `ctx->buf[]`, `LECPCB_VAL_STR_CHUNK` or
`LECPCB_VAL_BLOB_CHUNK` callbacks occur delivering each sequential bufferload.
If the CBOR indicates the total length, the last chunk is delievered in a
`LECPCB_VAL_STR_END` or `LECPCB_VAL_BLOB_END`.

If the CBOR indicates the string end after the chunk, a zero-length `..._END`
callback is provided.

## Handling CBOR tags

CBOR tags are exposed as `LECPCB_TAG_START` and `LECPCB_TAG_END` pairs, at
the `_START` callback the tag index is available in `ctx->item.u.u64`.

## CBOR maps

You can check if you are on the "key" part of a map "key:value" pair using the
helper api `lecp_parse_map_is_key(ctx)`.

## Parsing paths

LECP maintains a "parsing path" in `ctx->path` that represents the context of
the callback events.  As a convenience, at LECP context creation time, you can
pass in an array of path strings you want to match on, and have any match
checkable in the callback using `ctx->path_match`, it's 0 if no active match,
or the match index from your path array starting from 1 for the first entry.

|CBOR element|Representation in path|
|---|---|
|CBOR Array|`[]`|
|CBOR Map|`.`|
|CBOR Map entry key string|`keystring`|

## Accessing raw CBOR subtrees

Some CBOR usages like COSE require access to selected raw CBOR from the input
stream.  `lecp_parse_report_raw(ctx, on)` lets you turn on and off buffering of
raw CBOR and reporting it in the parse callback with `LECPCB_LITERAL_CBOR`
callbacks.  The callbacks mean the temp buffer `ctx->cbor[]` has `ctx->cbor_pos`
bytes of raw CBOR available in it.  Callbacks are triggered when the buffer
fills, or reporting is turned off and the buffer has something in it.

By turning the reporting on and off according to the outer CBOR parsing state,
it's possible to get exactly the raw CBOR subtree that's needed.

Capturing and reporting the raw CBOR does not change that the same CBOR is being
passed to the parser as usual as well.

## Comparison with LEJP (JSON parser)

LECP is based on the same principles as LEJP and shares most of the callbacks.
The major differences:

 - LEJP value callbacks all appear in `ctx->buf[]`, ie, floating-point is
   provided to the callback in ascii form like `"1.0"`.  CBOR provides a more
   strict typing system, and the different type values are provided either in
   `ctx->buf[]` for blobs or utf-8 text strtings, or the `item.u` union for
   converted types, with additional callback reasons specific to each type.

 - CBOR "maps" use `_OBJECT_START` and `_END` parsing callbacks around the
   key / value pairs.  LEJP has a special callback type `PAIR_NAME` for the
   key string / integer, but in LECP these are provided as generic callbacks
   dependent on type, ie, generic string callbacks or integer ones, and the
   value part is represented according to whatever comes.


# Writing CBOR

CBOR is written into a `lws_lec_pctx_t` object that has been initialized to
point to an output buffer of a specified size, using printf type formatting.

Output is paused if the buffer fills, and the write api may be called again
later with the same context object, to resume emitting to the same or different
buffer.

This allows bufferloads of encoded CBOR to be produced on demand, it's designed
to fit usage in WRITEABLE callbacks and Secure Streams tx() callbacks where the
buffer size for one packet is already fixed.

CBOR array and map lengths are deduced from the format string, as is whether to
use indeterminite length formatting or not.  For indeterminite text or binary
strings, a container of < > 

|Format|Arg(s)|Meaning|
|---|---|---|
|`123`||unsigned literal number|
|`-123`||signed literal number|
|`%u`|`unsigned int`|number|
|`%lu`|`unsigned long int`|number|
|`%llu`|`unsigned long long int`|number|
|`%d`|`signed int`|number|
|`%ld`|`signed long int`|number|
|`%lld`|`signed long long int`|number|
|`%f`|`double`|floating point number|
|`123(...)`||literal tag and scope|
|`%t(...)`|`unsigned int`|tag and scope|
|`%lt(...)`|`unsigned long int`|tag and scope|
|`%llt(...)`|`unsigned long long int`|tag and scope|
|`[...]`||Array (fixed len if `]` in same format string)|
|`{...}`||Map (fixed len if `}` in same format string)|
|`<t...>`||Container for indeterminite text string frags|
|`<b...>`||Container for indeterminite binary string frags|
|`'string'`||Literal text of known length|
|`%s`|`const char *`|NUL-terminated string|
|`%.*s`|`int`, `const char *`|length-specified string|
|`%.*b`|`int`, `const uint8_t *`|length-specified binary|
|`:`||separator between Map items (a:b)|
|`,`||separator between Map pairs or array items|

Backslash is used as an escape in `'...'` literal strings, so `'\\'` represents
a string consisting of a single backslash, and `'\''` a string consisting of a
single single-quote.

For integers, various natural C types are available, but in all cases, the
number is represented in CBOR using the smallest valid way based on its value,
the long or long-long modifiers just apply to the expected C type in the args.

For floats, the C argument is always expected to be a `double` type following
C type promotion, but again it is represented in CBOR using the smallest valid
way based on value, half-floats are used for NaN / Infinity and where possible
for values like 0.0 and -1.0.

## Examples

### Literal ints

```
	uint8_t buf[128];
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "-1");
```
|||
|---|---|
|Return| `LWS_LECPCTX_RET_FINISHED`|
|`ctx->used`|1|
|`buf[]`|20|

### Dynamic ints

```
	uint8_t buf[128];
	lws_lec_pctx_t cbw;
	int n = -1; /* could be long */

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%d", n); /* use %ld for long */
```
|||
|---|---|
|Return| `LWS_LECPCTX_RET_FINISHED`|
|`ctx->used`|1|
|`buf[]`|20|

### Maps, arrays and dynamic ints

```
	...
	int args[3] = { 1, 2, 3 };

	lws_lec_printf(ctx, "{'a':%d,'b':[%d,%d]}", args[0], args[1], args[2]);
```

|||
|---|---|
|Return| `LWS_LECPCTX_RET_FINISHED`|
|`ctx->used`|9|
|`buf[]`|A2 61 61 01 61 62 82 02 03|

### String longer than the buffer

Using `%s` and the same string as an arg gives same results

```
	uint8_t buf[16];
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "'A literal string > one buf'");
	/* not required to be in same function context or same buf,
	 * but the string must remain the same */
	lws_lec_setbuf(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "'A literal string > one buf'");
```

First call

|||
|---|---|
|Return| `LWS_LECPCTX_RET_AGAIN`|
|`ctx->used`|16|
|`buf[]`|78 1A 41 20 6C 69 74 65 72 61 6C 20 73 74 72 69|

Second call

|||
|---|---|
|Return| `LWS_LECPCTX_RET_FINISHED`|
|`ctx->used`|12|
|`buf[]`|6E 67 20 3E 20 6F 6E 65 20 62 75 66|

### Binary blob longer than the buffer

```
	uint8_t buf[16], blob[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 };
	lws_lec_pctx_t cbw;

	lws_lec_init(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);
	/* not required to be in same function context or same buf,
	 * but the length and blob must remain the same */
	lws_lec_setbuf(&cbw, buf, sizeof(buf));
	lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);
```

First call

|||
|---|---|
|Return| `LWS_LECPCTX_RET_AGAIN`|
|`ctx->used`|16|
|`buf[]`|52 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F|

Second call

|||
|---|---|
|Return| `LWS_LECPCTX_RET_FINISHED`|
|`ctx->used`|3|
|`buf[]`|10 11 12|