Testing¶

Introduction¶

test.py is a simple regression testing harness shipped along with scylla.git, which runs C++ unit and CQL tests.

This is a manual for test.py.

Installation¶

To run test.py, Python 3.7 or higher is required. ./install-dependencies.sh should install all the required Python modules. If install-dependencies.sh does not support your distribution, please manually install all Python modules it lists with pip.

Usage¶

In order to invoke test.py, you need to build first. ./test.py will run all existing tests in all configured build modes:

$ ./test.py

If you want to specify a specific build mode:

$ ./test.py --mode=dev

If you want to run only a specific test:

$ ./test.py suitename/testname

Build artefacts, such as test output and harness output is stored in ./testlog. Scylla data files are stored in /tmp.

How it works¶

On start, test.py invokes ninja to find out configured build modes. Then it searches all subdirectories of ./test/ for suite.yaml files: each directory containing suite.yaml is a test suite, in which test.py then looks for tests. All files ending with _test.cc or _test.cql are considered tests.

A suite must contain tests of the same type, as configured in suite.yaml. The list of found tests is matched with the optional command line test name filter. A match is registered if filter substring exists anywhere in test full name. For example:

$ ./test.py cql

runs cql/lwt_test, cql/lwt_batch_test, as well as boost/cql_query_test.

The ./testlog directory is created if it doesn’t exist, otherwise it is cleared from the previous run artefacts.

Matched tests are run concurrently, with concurrency factor set to the number of available CPU cores. test.py continues until all tests are run even if any one of them fails.

CQL tests¶

To run CQL tests, test.py uses an auxiliary program, test/tools/cql_repl.cc. This program reads CQL input from stdin, evaluates it using Scylla CQL database front-end, and prints output in JSON format to stdout. A default keyspace is created automatically.

test.py invokes cql_repl providing the test file for its standard input and redirecting its output to a temporary file in testlog directory.

After cql_repl finishes, test.py compares the output stored in the temporary file with a pre-recorded output stored in test/suitename/testname_test.result.

The test is considered failed if executing any CQL statement produced an error (e.g. because the server crashed during execution) or server output does not match one recorded in testname.result, or there is no testname.result. The latter is possible when it’s the first invocation of the test ever.

In the event of output mismatch file test/suitename/testname_test.reject is created, and first lines of the diff between the two files are output. To update .result file with new output, simply overwrite it with the reject file:

mv test/suitename/testname.re*

Note Since the result file is effectively part of the test, developers must thorougly examine the diff and understand the reasons for every change before overwriting .result files.

Unit tests¶

The same unit test can be run in different seastar configurations, i.e. with different command line arguments. The custom arguments can be set in custom_args key of the suite.yaml file.

Debugging¶

If a test fails, its log can be found in testlog/${mode}/testname.log. By default, all unit tests are built stripped. To build non-stripped tests, ./configure with --tests-debuginfo list-of-tests. test.py adds some command line arguments to unit tests. The exact way in which the test is invoked is recorded in testlog/test.py.log.

To debug CQL tests, one can invoke gdb cql_repl, which will start cql_repl in interactive mode, and then supply faulty CQL statements on the terminal.

Automation¶

If any of the tests fails, test.py returns a non-zero exit status. JUNIT and XUNIT execution status XML files can be found in testlog/${mode}/xml/ directory.