diff options
Diffstat (limited to 'doc/Test-Writing-Guidelines.asciidoc')
| -rw-r--r-- | doc/Test-Writing-Guidelines.asciidoc | 412 |
1 files changed, 412 insertions, 0 deletions
diff --git a/doc/Test-Writing-Guidelines.asciidoc b/doc/Test-Writing-Guidelines.asciidoc new file mode 100644 index 000000000..0db852ae6 --- /dev/null +++ b/doc/Test-Writing-Guidelines.asciidoc @@ -0,0 +1,412 @@ +LTP Test Writing Guidelines +=========================== + +This document describes LTP guidelines and is intended for anybody who want to +write or modify a LTP testcase. It's not a definitive guide and it's not, by +any means, a substitute for common sense. + +NOTE: See also + https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API], + https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API], + https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines[LTP Library API Writing Guidelines]. + +Rules and recommendations which are "machine checkable" should be +tagged with an ID like +LTP-XXX+. There will be a corresponding entry +in +https://github.com/linux-test-project/ltp/tree/master/doc/rules.tsv[doc/rules.tsv]. When +you run 'make check' or 'make check-test' it will display these IDs as +a reference. + +1. Guide to clean and understandable code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For testcases it's required that the source code is as easy to follow as +possible. When a test starts to fail the failure has to be analyzed, clean +test codebase makes this task much easier and quicker. + +Here are some hints on how to write clean and understandable code, a few of +these points are further discussed below: + +* First of all *Keep things simple* + +* Keep function and variable names short but descriptive + +* Keep functions reasonably short and focused on a single task + +* Do not overcomment + +* Be consistent + +* Avoid deep nesting + +* DRY + +1.1 Keep things simple +~~~~~~~~~~~~~~~~~~~~~~ + +For all it's worth keep the testcases simple or better as simple as possible. + +The kernel and libc are tricky beasts and the complexity imposed by their +interfaces is quite high. Concentrate on the interface you want to test and +follow the UNIX philosophy. + +It's a good idea to make the test as self-contained as possible too, ideally +tests should not depend on tools or libraries that are not widely available. + +Do not reinvent the wheel! + +* Use LTP standard interface + +* Do not add custom PASS/FAIL reporting functions + +* Do not write Makefiles from scratch, use LTP build system instead + +* Etc. + +1.2 Keep functions and variable names short +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Choosing a good name for an API functions or even variables is a difficult +task do not underestimate it. + +There are a couple of customary names for different things that help people to +understand code, for example: + +* For loop variables are usually named with a single letter 'i', 'j', ... + +* File descriptors 'fd' or 'fd_foo'. + +* Number of bytes stored in file are usually named as 'size' or 'len' + +* Etc. + +1.3 Do not overcomment +~~~~~~~~~~~~~~~~~~~~~~ + +Comments can sometimes save you day but they can easily do more harm than +good. There has been several cases where comments and actual implementation +drifted slowly apart which yielded into API misuses and hard to find bugs. +Remember there is only one thing worse than no documentation, wrong +documentation. + +Ideally everybody should write code that is obvious, which unfortunately isn't +always possible. If there is a code that requires to be commented keep it +short and to the point. These comments should explain *why* and not *how* +things are done. + +Never ever comment the obvious. + +In case of LTP testcases it's customary to add an asciidoc formatted comment +paragraph with highlevel test description at the beginning of the file right +under the GPL SPDX header. This helps other people to understand the overall +goal of the test before they dive into the technical details. It's also +exported into generated documentation hence it should mostly explain what is +tested. + +1.4 DRY (Code duplication) +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Copy & paste is a good servant but very poor master. If you are about to copy a +large part of the code from one testcase to another, think what would happen if +you find bug in the code that has been copied all around the tree. What about +moving it to a library instead? + +The same goes for short but complicated parts, whenever you are about to copy & +paste a syscall wrapper that packs arguments accordingly to machine +architecture or similarly complicated code, put it into a header instead. + +2 Coding style +~~~~~~~~~~~~~~ + +2.1 C coding style +^^^^^^^^^^^^^^^^^^ + +LTP adopted Linux kernel coding style: +https://www.kernel.org/doc/html/latest/process/coding-style.html + +If you aren't familiar with its rules please read it, it's a well written +introduction. + +Run `make check` in the test's directory and/or use `make check-$TCID`, +it uses (among other checks) our vendored version of +https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl] +script from kernel git tree. + +NOTE: If `make check` does not report any problems, the code still may be wrong + as all tools used for checking only look for common mistakes. + +2.1.1 LTP-004: Test executable symbols are marked static +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Test executables should not export symbols unnecessarily. This means +that all top-level variables and functions should be marked with the +static keyword. The only visible symbols should be those included from +shared object files. + +2.2 Shell coding style +^^^^^^^^^^^^^^^^^^^^^^ + +When writing testcases in shell write in *portable shell* only, it's a good +idea to try to run the test using alternative shell (alternative to bash, for +example dash) too. + +*Portable shell* means Shell Command Language as defined by POSIX with an +exception of few widely used extensions, namely 'local' keyword used inside of +functions and '-o' and '-a' test parameters (that are marked as obsolete in +POSIX). + +You can either try to run the testcases on Debian which has '/bin/sh' pointing +to 'dash' by default or install 'dash' on your favorite distribution and use +it to run the tests. If your distribution lacks 'dash' package you can always +compile it from http://gondor.apana.org.au/~herbert/dash/files/[source]. + +Run `make check` in the test's directory and/or use `make check-$TCID.sh`, +it uses (among other checks) our vendored version of +https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl] +from Debian, that is used to check for non-portable shell code. + +NOTE: If `make check` does not report any problems, the code still may be wrong + as `checkbashisms.pl` used for checking only looks for common mistakes. + +Here are some common sense style rules for shell + +* Keep lines under 80 chars + +* Use tabs for indentation + +* Keep things simple, avoid unnecessary subshells + +* Don't do confusing things (i.e. don't name your functions like common shell + commands, etc.) + +* Quote variables + +* Be consistent + +3 Backwards compatibility +~~~~~~~~~~~~~~~~~~~~~~~~~ + +LTP test should be as backward compatible as possible. Think of an enterprise +distributions with long term support (more than five years since the initial +release) or of an embedded platform that needs to use several years old +toolchain supplied by the manufacturer. + +Therefore LTP test for more current features should be able to cope with older +systems. It should at least compile fine and if it's not appropriate for the +configuration it should return 'TCONF'. + +There are several types of checks we use: + +The *configure script* is usually used to detect availability of a function +declarations in system headers. It's used to disable tests at compile time or +to enable fallback definitions. + +Checking the *errno* value is another type of runtime check. Most of the +syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented +or was disabled upon kernel compilation. + +LTP has kernel version detection that can be used to disable tests at runtime, +unfortunately kernel version does not always corresponds to a well defined +feature set as distributions tend to backport hundreds of patches while the +kernel version stays the same. Use with caution. + +Lately we added kernel '.config' parser, a test can define a boolean +expression of kernel config variables that has to be satisfied in order for a +test to run. This is mostly used for kernel namespaces at the moment. + +Sometimes it also makes sense to define a few macros instead of creating +configure test. One example is Linux specific POSIX clock ids in +'include/lapi/posix_clocks.h'. + +3.1 Dealing with messed up legacy code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +LTP still contains a lot of old and messy code and we are cleaning it up as +fast as we can but despite the decade of efforts there is still a lot. If you +start modifying old or a messy testcase and your changes are more complicated +than simple typo fixes you should convert the test into a new library first. + +It's also much easier to review the changes if you split them into a smaller +logical groups. The same goes for moving files. If you need a rename or move +file do it in a separate patch. + +4 License +~~~~~~~~~ + +Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or +any later version). + +Use `SPDX-License-Identifier: GPL-2.0-or-later` + +5 LTP Structure +~~~~~~~~~~~~~~~ + +The structure of LTP is quite simple. Each test is a binary written either in +portable shell or C. The test gets a configuration via environment variables +and/or command line parameters, it prints additional information into the +stdout and reports overall success/failure via the exit value. + +Tests are generally placed under the 'testcases/' directory. Everything that +is a syscall or (slightly confusingly) libc syscall wrapper goes under +'testcases/kernel/syscalls/'. + +Then there is 'testcases/open_posix_testsuite/' which is a well maintained fork +of the upstream project that has been dead since 2005 and also a number of +directories with tests for more specific features. + +5.1 Runtest Files +^^^^^^^^^^^^^^^^^ + +The list of tests to be executed is stored in runtest files under the +'runtest/' directory. The default set of runtest files to be executed is +stored in 'scenario_groups/default'. When you add a test you should add +corresponding entries into some runtest file(s) as well. + +For syscall tests (these placed under 'testcases/kernel/syscalls/') use +'runtest/syscalls' file, for kernel related tests for memory management we +have 'runtest/mm', etc. + +IMPORTANT: The runtest files should have one entry per a test. Creating a + wrapper that runs all your tests and adding it as a single test + into runtest file is strongly discouraged. + +5.2 Datafiles +^^^^^^^^^^^^^ + +If your test needs datafiles to work, these should be put into a subdirectory +named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to +do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the +'datafiles/Makefile'). + +You can obtain path to datafiles via $TST_DATAROOT provided by test.sh +'$TST_DATAROOT/...' +or via C function 'tst_dataroot()' provided by libltp: + +[source,c] +------------------------------------------------------------------------------- +const char *dataroot = tst_dataroot(); +------------------------------------------------------------------------------- + +Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...', +but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used +when running testcases directly in git tree as well as from install +location. + +The path is constructed according to these rules: + +1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID' +2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles' + (where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()') +3. else return '$CWD/datafiles' + +See 'testcases/commands/file/' for example. + +5.3 Subexecutables +^^^^^^^^^^^^^^^^^^ + +If your test needs to execute a binary, place it in the same directory as the +testcase and name the file starting with '${test_binary_name}_'. Once the +test is executed by the framework, the path to the directory with all LTP +binaries is added to the '$PATH' and you can execute it just by its name. + +TIP: If you need to execute such test from the LTP tree, you can add path to + current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'. + +6 Test Contribution Checklist +------------------------------ + +NOTE: See also + https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist]. + +1. Test compiles and runs fine (check with `-i 10` too) +2. `make check` does not emit any warnings for the test you are working on + (hint: run it in the test's directory and/or use `make check-$TCID`) +3. The runtest entries are in place +4. Test binaries are added into corresponding '.gitignore' files +5. Patches apply over the latest git + +6.1 About .gitignore files +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are numerous '.gitignore' files in the LTP tree. Usually there is a +'.gitignore' file per a group of tests. The reason for this setup is simple. +It's easier to maintain a '.gitignore' file per directory with tests, rather +than having single file in the project root directory. This way, we don't have +to update all the gitignore files when moving directories, and they get deleted +automatically when a directory with tests is removed. + +7 Testing pre-release kernel features +------------------------------------- + +Tests for features not yet in a mainline kernel release are accepted. However +they must only be added to the +staging+ runtest file. Once a feature is part +of the stable kernel ABI the associated test must be moved out of staging. + +This is primarily to help test kernel RCs by avoiding the need to download +separate LTP patchsets. + +8 LTP C And Shell Test API Comparison +------------------------------------- + +Comparison of +https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API] and +https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API]. + +[options="header"] +|================================================================================ +| C API ('struct tst_test' members) | shell API ('$TST_*' environment variables) +| '.all_filesystems' | 'TST_ALL_FILESYSTEMS' +| '.bufs' | – +| '.caps' | – +| '.child_needs_reinit' | not applicable +| '.cleanup' | 'TST_CLEANUP' +| '.dev_extra_opts' | 'TST_DEV_EXTRA_OPTS' +| '.dev_fs_opts' | 'TST_DEV_FS_OPTS' +| '.dev_fs_type' | 'TST_FS_TYPE' +| '.dev_min_size' | not applicable +| '.format_device' | 'TST_FORMAT_DEVICE' +| '.max_runtime' | – +| '.min_cpus' | not applicable +| '.min_kver' | 'TST_MIN_KVER' +| '.min_mem_avail' | not applicable +| '.mnt_flags' | 'TST_MNT_PARAMS' +| '.min_swap_avail' | not applicable +| '.mntpoint', '.mnt_data' | 'TST_MNTPOINT' +| '.mount_device' | 'TST_MOUNT_DEVICE' +| '.needs_cgroup_ctrls' | – +| '.needs_checkpoints' | 'TST_NEEDS_CHECKPOINTS' +| '.needs_cmds' | 'TST_NEEDS_CMDS' +| '.needs_devfs' | – +| '.needs_device' | 'TST_NEEDS_DEVICE' +| '.needs_drivers' | 'TST_NEEDS_DRIVERS' +| '.needs_kconfigs' | 'TST_NEEDS_KCONFIGS' +| '.needs_overlay' | +| '.needs_rofs' | – +| '.needs_root' | 'TST_NEEDS_ROOT' +| '.needs_tmpdir' | 'TST_NEEDS_TMPDIR' +| '.options' | 'TST_PARSE_ARGS', 'TST_OPTS' +| '.resource_files' | – +| '.restore_wallclock' | not applicable +| '.sample' | – +| '.save_restore' | – +| '.scall' | not applicable +| '.setup' | 'TST_SETUP' +| '.skip_filesystems' | 'TST_SKIP_FILESYSTEMS' +| '.skip_in_compat' | – +| '.skip_in_lockdown' | 'TST_SKIP_IN_LOCKDOWN' +| '.skip_in_secureboot' | 'TST_SKIP_IN_SECUREBOOT' +| '.supported_archs' | not applicable +| '.tags' | – +| '.taint_check' | – +| '.tcnt' | 'TST_CNT' +| '.tconf_msg' | not applicable +| '.test', '.test_all' | 'TST_TESTFUNC' +| '.test_variants' | – +| '.timeout' | 'TST_TIMEOUT' +| '.tst_hugepage' | not applicable +| .format_device | 'TST_DEVICE' +| not applicable | 'TST_NEEDS_KCONFIGS_IFS' +| not applicable | 'TST_NEEDS_MODULE' +| not applicable | 'TST_POS_ARGS' +| not applicable | 'TST_USAGE' +|================================================================================ |