test.py is a simple regression testing harness shipped along with
scylla.git, which runs C++ unit and CQL tests.
This is a manual for
test.py, Python 3.7 or higher is required.
./install-dependencies.sh should install all the required Python
install-dependencies.sh does not support your distribution,
please manually install all Python modules it lists with
In order to invoke
test.py, you need to build first.
run all existing tests in all configured build modes:
If you want to specify a specific build mode:
$ ./test.py --mode=dev
If you want to run only a specific test:
$ ./test.py suitename/testname
Build artefacts, such as test output and harness output is stored
./testlog. Scylla data files are stored in
ninja to find out configured build modes. Then
it searches all subdirectories of
suite.yaml files: each
suite.yaml is a test suite, in which
test.py then looks
for tests. All files ending with
_test.cql are considered
A suite must contain tests of the same type, as configured in
The list of found tests is matched with the optional command line test name
filter. A match is registered if filter substring exists anywhere in test
full name. For example:
$ ./test.py cql
cql/lwt_batch_test, as well as
./testlog directory is created if it doesn’t exist, otherwise it is
cleared from the previous run artefacts.
Matched tests are run concurrently, with concurrency factor set to the
number of available CPU cores.
test.py continues until all tests are run
even if any one of them fails.
To run CQL tests,
test.py uses an auxiliary program,
This program reads CQL input from stdin, evaluates it using Scylla CQL
database front-end, and prints output in JSON format to stdout. A default
keyspace is created automatically.
cql_repl providing the test file for its standard
input and redirecting its output to a temporary file in
test.py compares the output stored in the
temporary file with a pre-recorded output stored in
The test is considered failed if executing any CQL statement produced an
error (e.g. because the server crashed during execution) or server output
does not match one recorded in
testname.result, or there is no
testname.result. The latter is possible when it’s the first invocation of
the test ever.
In the event of output mismatch file
is created, and first lines of the diff between the two files are output.
.result file with new output, simply overwrite it with the
Note Since the result file is effectively part of the test, developers
must thorougly examine the diff and understand the reasons for every
change before overwriting
The same unit test can be run in different seastar configurations, i.e. with
different command line arguments. The custom arguments can be set in
custom_args key of the
If a test fails, its log can be found in
By default, all unit tests are built stripped. To build non-stripped tests,
test.py adds some command line arguments to unit tests. The exact way in
which the test is invoked is recorded in
To debug CQL tests, one can invoke
gdb cql_repl, which will start
cql_repl in interactive mode, and then supply faulty CQL statements
on the terminal.
If any of the tests fails,
test.py returns a non-zero exit status.
JUNIT and XUNIT execution status XML files can be found in
For command line help and available options, please see also:
$ ./test.py --help