Dittosuite
Dittosuite is a work in progress collection of tools that aims at providing a high-level language called Dittolang that defines operations.
The defined Dittolang operations can be interpreted by Dittosim for simulation to provide a simulated performance measurement and quickly identify the goodness of a solution.
Specularly, Dittobench interprets the Dittolang operations and executes them on a real device, tracking the behavior and measuring the performance.
How to run
$ ./dittobench [options] [.ditto file]
To run a benchmark, a well formed .ditto file must be provided, see section How to write .ditto files In addition, these options can be set:
--results-output=<int | string>
(default: report). Select the results output format. Options: report, csv with 0, 1 respectively.--log-stream=<int | string>
(default: stdout). Select the output stream for the log messages. Options: stdout, logcat with 0, 1 respectively.--log-level=<int | string>
(default: INFO). Select to output messages which are at or below the set level. Options: VERBOSE, DEBUG, INFO, WARNING, ERROR, FATAL with 0, 1, 2, 3, 4 and 5 respectively.--parameters=string
. If the benchmark is parametric, all the parameters (separated by commas) can be given through this option.
How to write .ditto files
Every .ditto file should begin with this skeleton:
main: {
...
},
global {
...
}
Optionally, it can contain init
and clean_up
sections:
init: {
...
},
main: {
...
},
clean_up: {
...
},
global {
...
}
global
Global section should contain general benchmark configuration. Currently available options:
- (optional)
string absolute_path
(default = ""
). Specifies the absolute path for the files.
init
init
is optional and can be used to initialize the benchmarking environment. It executes
instructions similar to main
, but the results are not collected in the end.
main
main
is the entry point for the benchmark. It can contain a single instruction
or
instruction_set
(also with nested instruction_set
).
clean_up
clean_up
is optional and can be used to reset the benchmarking environment to the initial state,
e.g, delete benchmark files. Similar to init
, it executes instructions like main
, but results
are not collected in the end.
instruction
{
<name of the instruction>: {
<first argument>,
<second argument>,
...
},
<general instruction options>
}
Currently available options:
- (optional) int repeat
(default = 1
). Specifies how many times the instruction should be
repeated.
instruction_set
{
instruction_set: {
instructions: {
{
<name of the first instruction>: {
<first argument>,
<second argument>,
...
},
<general instruction options>
},
{
<name of the second instruction>: {
<first argument>,
<second argument>,
...
},
<general instruction options>
},
...
},
iterate_options: {...}
},
<general instruction options>
}
Instruction set is an Instruction container that executes the contained instructions sequentially.
Instruction set can optionally iterate over a list and execute the provided set of instructions on
each item from the list. To use it, iterate_options
should be set with these options:
- string list_name
- Shared variable name pointing to a list of values.
- string item_name
- Shared variable name to which a selected value should be stored.
- (optional) Order order
(default = SEQUENTIAL
) - Specifies if
the elements of the list should be accessed sequentially or randomly. Options:
SEQUENTIAL
, RANDOM
.
- (optional) Reseeding reseeding
(default = ONCE
) - Specifies how often the random number
generator should be reseeded with the same provided (or generated) seed. Options: ONCE
,
EACH_ROUND_OF_CYCLES
, EACH_CYCLE
.
- (optional) uint32 seed
- Seed for the random number generator. If the seed is not provided,
current system time is used as the seed.
multithreading
and threads
multithreading: {
threads: [
{
instruction: {...},
spawn: <number of threads to spawn with the provided instruction>
},
...
]
}
Multithreading is another instruction container that executes the specified instructions
(or instruction sets) in different threads. If the optional spawn
option for a specific
instruction (or instruction set) is provided, then the provided number of threads will be created
for it.
Example
main: {
instruction_set: {
instructions: [
{
open_file: {
path_name: "newfile2.txt",
output_fd: "test_file"
}
},
{
close_file: {
input_fd: "test_file"
}
}
]
},
repeat: 10
},
global {
absolute_path: "/data/local/tmp/";
}
See more examples in example/
.
Predefined list of instructions
open_file
Opens the file with a file path or a shared variable name pointing to a file path. If neither of those are provided, a random name consisting of 9 random digits is generated. Optionally saves the file descriptor which can then be used by subsequent instructions. Also, can optionally create the file if it does not already exist.
Arguments:
- (optional)
string path_name
- Specifies the file path.
OR
string input
- Shared variable name pointing to a file path. - (optional)
string output_fd
- Shared variable name to which output file descriptor should be saved. - (optional)
bool create
(default = true
) - Specifies if the file should be created if it does not already exist. If the file exists, nothing happens.
delete_file
Deletes the file with a file path or a shared variable name pointing to a file path.
Uses unlink(2)
.
Arguments:
string path_name
- Specifies the file path.
OR
string input
- Shared variable name pointing to a file path.
close_file
Closes the file with the provided file descriptor.
Uses close(2)
.
Arguments:
string input_fd
- Shared variable name pointing to a file descriptor.
resize_file
Resizes the file with the provided file descriptor and new size. If the provided size is greater
than the current file size, fallocate(2)
is used, while ftruncate(2)
is used if the provided
size is not greater than the current file size.
Arguments:
string input_fd
- Shared variable name pointing to a file descriptor.int64 size
- New file size (in bytes).
resize_file_random
Resizes the file with the provided file descriptor and a range for new size. New file size is
randomly generated in the provided range and if the generated size is greater
than the current file size, fallocate(2)
is used, while ftruncate(2)
is used if the generated
size is not greater than the current file size.
Arguments:
string input_fd
- Shared variable name pointing to a file descriptor.int64 min
- Minimum value (in bytes)int64 max
- Maximum value (in bytes)- (optional)
uint32 seed
- Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional)
Reseeding reseeding
(default = ONCE
). How often the random number generator should be reseeded with the provided (or generated) seed. Options:ONCE
,EACH_ROUND_OF_CYCLES
,EACH_CYCLE
.
write_file
Writes to file with the provided file descriptor. For SEQUENTIAL
access, the blocks of data will be written sequentially and if the end of
the file is reached, new blocks will start from the beginning of the file. For
RANDOM
access, the block offset, to which data should be written, will
be randomly chosen with uniform distribution. 10101010
byte is used for the
write operation to fill the memory with alternating ones and zeroes. Uses
pwrite64(2)
.
Arguments:
string input_fd
- Shared variable name pointing to a file descriptor.- (optional)
int64 size
(default = -1
) - How much data (in bytes) should be written in total. If it is set to-1
, then file size is used. - (optional)
int64 block_size
(default = 4096
) - How much data (in bytes) should be written at once. If it is set to-1
, then file size is used. - (optional)
int64 starting_offset
(default = 0
) - Ifaccess_order
is set toSEQUENTIAL
, then the blocks, to which the data should be written, will start from this starting offset (in bytes). - (optional)
Order access_order
(default = SEQUENTIAL
) - Order of the write. Options:SEQUENTIAL
andRANDOM
. - (optional)
uint32 seed
- Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional)
bool fsync
(default = false
) - If set,fsync(2)
will be called after the execution of all write operations. - (optional)
Reseeding reseeding
(default = ONCE
) - How often the random number generator should be reseeded with the provided (or generated) seed. Options:ONCE
,EACH_ROUND_OF_CYCLES
,EACH_CYCLE
.
read_file
Reads from file with the provided file descriptor. For SEQUENTIAL
access, the blocks of data will be read sequentially and if the end of
the file is reached, new blocks will start from the beginning of the file. For
RANDOM
access, the block offset, from which data should be read, will
be randomly chosen with uniform distribution. Calls posix_fadvise(2)
before
the read operations. Uses pread64(2)
.
Arguments:
string input_fd
- Shared variable name pointing to a file descriptor.- (optional)
int64 size
(default = -1
) - How much data (in bytes) should be read in total. If it is set to-1
, then file size is used. - (optional)
int64 block_size
(default = 4096
) - How much data (in bytes) should be read at once. If it is set to-1
, then file size is used. - (optional)
int64 starting_offset
(default = 0
) - Ifaccess_order
is set toSEQUENTIAL
, then the blocks, from which the data should be read, will start from this starting offset (in bytes). - (optional)
Order access_order
(default = SEQUENTIAL
) - Order of the read. Options:SEQUENTIAL
andRANDOM
. - (optional)
uint32 seed
- Seed for the random number generator. If the seed is not provided, current system time is used as the seed. - (optional)
ReadFAdvise fadvise
(default = AUTOMATIC
) - Sets the argument for theposix_fadvise(2)
operation. Options:AUTOMATIC
,NORMAL
,SEQUENTIAL
andRANDOM
. IfAUTOMATIC
is set, thenPOSIX_FADV_SEQUENTIAL
orPOSIX_FADV_RANDOM
will be used forSEQUENTIAL
andRANDOM
access order respectively. - (optional)
Reseeding reseeding
(default = ONCE
) - How often the random number generator should be reseeded with the provided (or generated) seed. Options:ONCE
,EACH_ROUND_OF_CYCLES
,EACH_CYCLE
.
read_directory
Reads file names from a directory and stores them as a list in a shared variable. Uses readdir(3)
.
Arguments:
string directory_name
- Name of the directorystring output
- Shared variable name to which files names should be saved.
invalidate_cache
Drops kernel clean caches, including, dentry, inode and page caches by calling sync() first and
then writing 3
to /proc/sys/vm/drop_caches
. No arguments.
Dependencies
Android
The project is currently being developed as part of the Android Open Source Project (AOSP) and is supposed to run out-of-the-box.
Linux
The following utilities are required to build the project on Linux:
sudo apt install cmake protobuf-compiler
Testing
Linux
A suite of unit tests is provided in the test/ directory. In Linux these tests can be run with the following commands:
mkdir build
cd build
make
cd test
ctest
Use cases
File operations performance (high priority)
Bandwidth and measurement when dealing with few huge files or many small files. Operations are combinations of sequential/random-offset read/write.
Latency in creating/deleting files/folders.
These operations should be able to be triggered in a multiprogrammed fashion.
Display pipeline
A graph of processes that are communicating with each others in a pipeline of operations that are parallely contributing to the generation of display frames.
Scheduling (low priority)
Spawning tasks (period, duration, deadline) and verifying their scheduling latency and deadline misses count.
Workflow example and implementation nits
In the following scenario, two threads are running.
T1 runs the following operations: Read, Write, Read, sends a request to T2 and waits for the reply, then Write, Read. T2 waits for a request, then Read, Write, then sends the reply to the requester.
Operations are encoded as primitives expressed with ProtoBuffers. The definition of dependencies among threads can be represented as graphs.
Initialization phase
The first graph traversal is performed at initialization time, when all the ProtoBuffer configuration files are distributed among all the binaries so that they can perform all the heavy initialization duties.
Execution phase
After the initialization phase completes the graph can be traversed again to put all the workloads in execution.
Results gathering phase
A final graph traversal can be performed to fetch all the measurements that each entity internally stored.
Post processing
All the measurements must be ordered and later processed to provide useful information to the user.
T1: INIT : [ RD WR RD ] SND RCV [ WR RD ] : END T2: INIT : RCV [ RD WR ] SND : END
Scratch notes
critical path [ READ WRITE READ ] [ READ WRITE ] [ WRITE READ ] --------------------> > < Thread1 III-XXXXXX|X-SSSSSS-XX-TTTT Thread2 III-XXXX|XXX-TTTT ^
> XXXXXXX XX<
XXXX
READ WRITE READ
--->
vector
RECEIVE READ WRITE READ SEND
--->
vector
lock on receive()