path: root/READMEs/README.ctest.md

## Using CTest with lws

### Updating ancient cmake

You need a recent cmake to have the CTest tests work properly, if you're on an
older distro you need to update your cmake.  Luckily Kitware provide a repo for
common distros.  These instructions work for bionic and xenial.

First remove the old distro cmake and install the pieces needed to get the new repo keys

```
# apt purge --auto-remove cmake
# apt install gnupg wget apt-transport-https ca-certificates
# wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2>/dev/null | sudo apt-key add -
# apt edit-sources
```

Add the line `deb https://apt.kitware.com/ubuntu/ bionic main` at the end
replacing `bionic` with `xenial` as needed, and save (:wq).  Then

```
# apt update
# apt install cmake
```

## Tests live in CMakeLists.txt

The rules for tests are described in ctest / cmake language inside the minimal
examples and api tests that are enabled by current build options, so you need
to build with `-DLWS_WITH_MINIMAL_EXAMPLES=1` to build the examples along with
the library.

The tests are typically running the examples or api tests and regarding the
process exiting with exit code 0 as success, anything else as failure.

## Generating the tests

The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`.  You can optionally set
`-DLWS_CTEST_INTERNET_AVAILABLE=0` to indicate you can't run the tests that need
internet connectivity.

## Preparing to run the tests

The tests have to be able to run without root and without disturbing any other
install of lws in the build machine.

For that reason you have to do an unprivileged side-install into `../destdir`,
using `make install DESTDIR=../destdir` from the build directory and perform the
tests on the pieces in there.

## Running the tests

We must take care to run the pieces (.so etc) we just built, without having
root access, and not any of the same pieces from some other lws version that may
have been installed on the build machine.  That includes, eg, plugins that
we just built, to ensure precedence of those in the search path we can set our
DESTDIR unprivileged install path in `LD_LIBRARY_PATH`.

Then we can run ctest on the unprivileged install.  The whole step looks
something like this:

```
build $ make -j12 && \
  rm -rf ../destdir && \
  make -j12 DESTDIR=../destdir install && \\
  LD_LIBRARY_PATH=../destdir/usr/local/share/libwebsockets-test-server/plugins ctest -j2 --output-on-failure
```  

On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build
type.

Good results look something like this (which tests can run depend on your
build options)

```
Test project /projects/libwebsockets/build
      Start 71: st_wcs_srv
      Start 43: st_hcp_srv
 1/73 Test #71: st_wcs_srv ..................................   Passed    5.01 sec
      Start 19: st_hcmp_srv
 2/73 Test #43: st_hcp_srv ..................................   Passed    5.01 sec
      Start 17: st_hcm_srv
 3/73 Test #19: st_hcmp_srv .................................   Passed    5.01 sec
      Start 55: st_ssproxyctx
 4/73 Test #17: st_hcm_srv ..................................   Passed    5.01 sec
      Start 52: st_ssproxy
 5/73 Test #55: st_ssproxyctx ...............................   Passed    1.02 sec
      Start 67: st_sstfproxy
 6/73 Test #52: st_ssproxy ..................................   Passed    1.02 sec
      Start 60: st_ssprxsmd_sspc
 7/73 Test #67: st_sstfproxy ................................   Passed    1.01 sec
      Start 63: st_mulssprxsmd_sspc
 8/73 Test #60: st_ssprxsmd_sspc ............................   Passed    1.01 sec
      Start 69: sspc-minimaltf
 9/73 Test #63: st_mulssprxsmd_sspc .........................   Passed    1.02 sec
      Start 73: ws-client-spam
10/73 Test #73: ws-client-spam ..............................   Passed   12.21 sec
      Start 57: sspc-minimaltx
11/73 Test #57: sspc-minimaltx ..............................   Passed    5.90 sec
      Start 65: mulsspcsmd_sspc
12/73 Test #65: mulsspcsmd_sspc .............................   Passed    3.58 sec
      Start 62: sspcsmd_sspc
13/73 Test #62: sspcsmd_sspc ................................   Passed    1.73 sec
      Start 22: http-client-multi-h1
14/73 Test #22: http-client-multi-h1 ........................   Passed    5.04 sec
      Start 25: http-client-multi-stag
15/73 Test #25: http-client-multi-stag ......................   Passed    4.53 sec
      Start 26: http-client-multi-stag-h1
16/73 Test #26: http-client-multi-stag-h1 ...................   Passed    4.40 sec
      Start 21: http-client-multi
17/73 Test #21: http-client-multi ...........................   Passed    4.37 sec
      Start 36: http-client-multi-post-h1
18/73 Test #36: http-client-multi-post-h1 ...................   Passed    2.73 sec
      Start 54: sspc-minimal
19/73 Test #54: sspc-minimal ................................   Passed    0.93 sec
      Start 39: http-client-multi-post-stag
20/73 Test #39: http-client-multi-post-stag .................   Passed    2.29 sec
      Start 40: http-client-multi-post-stag-h1
21/73 Test #69: sspc-minimaltf ..............................   Passed   49.83 sec
      Start 35: http-client-multi-post
22/73 Test #40: http-client-multi-post-stag-h1 ..............   Passed    4.30 sec
      Start 33: http-client-multi-restrict-nopipe-fail
23/73 Test #35: http-client-multi-post ......................   Passed    3.23 sec
      Start 28: http-client-multi-stag-h1-pipe
24/73 Test #33: http-client-multi-restrict-nopipe-fail ......   Passed    2.86 sec
      Start 32: http-client-multi-restrict-stag-h1-pipe
25/73 Test #28: http-client-multi-stag-h1-pipe ..............   Passed    2.86 sec
      Start 27: http-client-multi-stag-pipe
26/73 Test #32: http-client-multi-restrict-stag-h1-pipe .....   Passed    1.51 sec
      Start 31: http-client-multi-restrict-stag-pipe
27/73 Test #27: http-client-multi-stag-pipe .................   Passed    1.52 sec
      Start 34: http-client-multi-restrict-h1-nopipe-fail
28/73 Test #34: http-client-multi-restrict-h1-nopipe-fail ...   Passed    2.78 sec
      Start 46: http-client-post-m
29/73 Test #31: http-client-multi-restrict-stag-pipe ........   Passed    2.80 sec
      Start 42: http-client-multi-post-stag-h1-pipe
30/73 Test #42: http-client-multi-post-stag-h1-pipe .........   Passed    1.51 sec
      Start 41: http-client-multi-post-stag-pipe
31/73 Test #46: http-client-post-m ..........................   Passed    1.59 sec
      Start 48: http-client-post-m-h1
32/73 Test #48: http-client-post-m-h1 .......................   Passed    1.10 sec
      Start 23: http-client-multi-pipe
33/73 Test #41: http-client-multi-post-stag-pipe ............   Passed    1.51 sec
      Start 29: http-client-multi-restrict-pipe
34/73 Test #23: http-client-multi-pipe ......................   Passed    1.09 sec
      Start 24: http-client-multi-h1-pipe
35/73 Test #29: http-client-multi-restrict-pipe .............   Passed    0.74 sec
      Start 30: http-client-multi-restrict-h1-pipe
36/73 Test #24: http-client-multi-h1-pipe ...................   Passed    1.14 sec
      Start 45: http-client-post
37/73 Test #30: http-client-multi-restrict-h1-pipe ..........   Passed    1.14 sec
      Start 38: http-client-multi-post-h1-pipe
38/73 Test #45: http-client-post ............................   Passed    0.30 sec
      Start 37: http-client-multi-post-pipe
39/73 Test #38: http-client-multi-post-h1-pipe ..............   Passed    0.49 sec
      Start 47: http-client-post-h1
40/73 Test #37: http-client-multi-post-pipe .................   Passed    0.31 sec
      Start 50: hs_evlib_foreign_event
41/73 Test #47: http-client-post-h1 .........................   Passed    0.29 sec
      Start 66: ss-tf
42/73 Test #50: hs_evlib_foreign_event ......................   Passed   22.02 sec
      Start 49: hs_evlib_foreign_uv
43/73 Test #49: hs_evlib_foreign_uv .........................   Passed   21.03 sec
      Start 51: ss-warmcat
44/73 Test #51: ss-warmcat ..................................   Passed    2.69 sec
      Start 59: ss-smd
45/73 Test #59: ss-smd ......................................   Passed    1.78 sec
      Start 10: api-test-secure-streams
46/73 Test #10: api-test-secure-streams .....................   Passed    1.34 sec
      Start 11: http-client-warmcat
47/73 Test #11: http-client-warmcat .........................   Passed    0.27 sec
      Start 58: sspost-warmcat
48/73 Test #58: sspost-warmcat ..............................   Passed    0.84 sec
      Start 12: http-client-warmcat-h1
49/73 Test #12: http-client-warmcat-h1 ......................   Passed    0.25 sec
      Start  2: api-test-jose
50/73 Test  #2: api-test-jose ...............................   Passed    0.27 sec
      Start 70: ws-client-rx-warmcat
51/73 Test #70: ws-client-rx-warmcat ........................   Passed    0.27 sec
      Start 56: ki_ssproxyctx
52/73 Test #56: ki_ssproxyctx ...............................   Passed    0.12 sec
      Start 68: ki_ssproxy
53/73 Test #68: ki_ssproxy ..................................   Passed    0.11 sec
      Start 64: ki_mulssprxsmd_sspc
54/73 Test #64: ki_mulssprxsmd_sspc .........................   Passed    0.10 sec
      Start 61: ki_ssprxsmd_sspc
55/73 Test #61: ki_ssprxsmd_sspc ............................   Passed    0.11 sec
      Start 13: http-client-h2-rxflow-warmcat
56/73 Test #13: http-client-h2-rxflow-warmcat ...............   Passed    0.28 sec
      Start 14: http-client-h2-rxflow-warmcat-h1
57/73 Test #14: http-client-h2-rxflow-warmcat-h1 ............   Passed    0.34 sec
      Start 16: http-client-hugeurl-warmcat-h1
58/73 Test #16: http-client-hugeurl-warmcat-h1 ..............   Passed    0.16 sec
      Start 15: http-client-hugeurl-warmcat
59/73 Test #15: http-client-hugeurl-warmcat .................   Passed    0.16 sec
      Start 72: ki_wcs_srv
60/73 Test #72: ki_wcs_srv ..................................   Passed    0.12 sec
      Start 44: ki_hcp_srv
61/73 Test #44: ki_hcp_srv ..................................   Passed    0.11 sec
      Start 20: ki_hcmp_srv
62/73 Test #20: ki_hcmp_srv .................................   Passed    0.11 sec
      Start 18: ki_hcm_srv
63/73 Test #18: ki_hcm_srv ..................................   Passed    0.11 sec
      Start  7: api-test-lws_struct_sqlite
64/73 Test  #7: api-test-lws_struct_sqlite ..................   Passed    0.03 sec
      Start  1: api-test-gencrypto
65/73 Test  #1: api-test-gencrypto ..........................   Passed    0.02 sec
      Start  6: api-test-lws_struct-json
66/73 Test  #6: api-test-lws_struct-json ....................   Passed    0.01 sec
      Start  4: api-test-lws_dsh
67/73 Test  #4: api-test-lws_dsh ............................   Passed    0.01 sec
      Start  8: api-test-lws_tokenize
68/73 Test  #8: api-test-lws_tokenize .......................   Passed    0.01 sec
      Start  9: api-test-lwsac
69/73 Test  #9: api-test-lwsac ..............................   Passed    0.00 sec
      Start  3: api-test-lejp
70/73 Test  #3: api-test-lejp ...............................   Passed    0.00 sec
      Start 53: ki_ssproxy
71/73 Test #53: ki_ssproxy ..................................   Passed    0.11 sec
72/73 Test #66: ss-tf .......................................   Passed   55.51 sec
      Start  5: api-test-lws_smd
73/73 Test  #5: api-test-lws_smd ............................   Passed    4.22 sec

100% tests passed, 0 tests failed out of 73

Total Test time (real) = 137.76 sec
```

## Considerations for creating tests

### Timeout

The default test timeout is 1500s, for that reason it's good practice to set
a more suitable `TIMEOUT` property on every test.

### Working Directory

Server-side test apps usually need to be run from their `./minimal-examples/...`
directory so they can access their assets like index.html etc.

However when building with `-DLWS_WITH_MBEDTLS=1` then even client-side apps
need to be run from their directory, since they need to get the trusted CA for
warmcat.com or libwebsockets.org additionally.

For that reason it's good practice to set the `WORKING_DIRECTORY` property to
the home dir of the example app in all cases.

### Spawning Buddies

Many networking tests need to either spawn a client or a server in order to
have a "buddy" to talk to during the test for the opposing side.  This is a
bit awkward in cmake since it does not directly support spawning daemons as
test dependencies.

Lws provides helper scripts for unix type targets in `./scripts/ctest-background.sh`
and `./scripts/ctest-background-kill.sh`, which spawn background processes,
save the pid in a decorated /tmp file and can later take the process down.  This
also has arrangements to dump the log of any background process that exited
early.

To arrange the buddy to run aligned with the test, you first explain to cmake
how to start and stop the buddy using phony tests to make a "fixture" in cmake
terms.

In this example, taken from minimal-http-client-multi, we arrange for
minimal-http-server-tls to be available for our actual test.  The starting and
stopping definition, for "st_hcm_srv" and "ki_hcm_srv":

```
	add_test(NAME st_hcm_srv COMMAND
		${CMAKE_SOURCE_DIR}/scripts/ctest-background.sh
			hcm_srv $<TARGET_FILE:lws-minimal-http-server-tls>
			--port ${PORT_HCM_SRV} )
	add_test(NAME ki_hcm_srv COMMAND
		${CMAKE_SOURCE_DIR}/scripts/ctest-background-kill.sh
			hcm_srv $<TARGET_FILE_NAME:lws-minimal-http-server-tls>
				--port ${PORT_HCM_SRV})
```

... and binding those together so cmake knows they start and stop a specific
named fixture "hcm_srv", itself with an 800s timeout

```
	set_tests_properties(st_hcm_srv PROPERTIES
       		WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-server/minimal-http-server-tls
		FIXTURES_SETUP hcm_srv
		TIMEOUT 800)
	set_tests_properties(ki_hcm_srv PROPERTIES
		FIXTURES_CLEANUP hcm_srv)
```

... and finally, adding the "hcm_srv" fixture as a requirement on the actual
test (http-client-multi) we are testing

```
	set_tests_properties(http-client-multi
			     PROPERTIES
			     FIXTURES_REQUIRED "hcm_srv"
			     WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-client/minimal-http-client-multi
			     TIMEOUT 50)
```

Once all that explaining is done, ctest itself will take care about starting
and killing hcm_srv before and after http-client-multi test.

### Buddy sockets and test concurrency

For tests with local buddies using tcp sockets inside the same VM or systemd-
nspawn networking context, you cannot just use a well-known port like 7681.

ctest itself is usually executed concurrently, and Sai is typically building
multiple different instances concurrently as well (typically 3), so it may be
running different ctests inside the same VM simultaneously.

Different tests can have their own convention for port ranges, to solve the
problem about Sai running different tests concurrently inside one ctest.

For the case there are multiple ctests running, we can use the env var
`$ENV{SAI_INSTANCE_IDX}`, which is an ordinal like 0 or 1, to further ensure
that port selections won't conflict.  If not using Sai, you can just set this
in the evironment yourself to reflect your build instance index.

```
       #
       # instantiate the server per sai builder instance, they are running in the same
       # machine context in parallel so they can tread on each other otherwise
       #
       set(PORT_HCM_SRV "7670")
       if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "0")
               set(PORT_HCM_SRV 7671)
       endif()
       if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "1")
               set(PORT_HCM_SRV 7672)
       endif()
       if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "2")
               set(PORT_HCM_SRV 7673)
       endif()
       if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "3")
               set(PORT_HCM_SRV 7674)
       endif()
```

This is complicated enough that the best approach is copy an existing simple
case like the CMakeLists.txt for minimal-http-client and change the names and
ports to be unique.