Scylla Documentation Logo Documentation
  • Server
    • Scylla Open Source
    • Scylla Enterprise
    • Scylla Alternator
  • Cloud
    • Scylla Cloud
    • Scylla Cloud Docs
  • Tools
    • Scylla Manager
    • Scylla Monitoring Stack
    • Scylla Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu

Caution

You're viewing documentation for an unstable version of Scylla Dev. Switch to the latest stable version.

Scylla Dev Guides Testing

Testing¶

Introduction¶

test.py is a simple regression testing harness shipped along with scylla.git, which runs C++ unit and CQL tests.

This is a manual for test.py.

Installation¶

To run test.py, Python 3.7 or higher is required. ./install-dependencies.sh should install all the required Python modules. If install-dependencies.sh does not support your distribution, please manually install all Python modules it lists with pip.

Usage¶

In order to invoke test.py, you need to build first. ./test.py will run all existing tests in all configured build modes:

$ ./test.py

If you want to specify a specific build mode:

$ ./test.py --mode=dev

If you want to run only a specific test:

$ ./test.py suitename/testname

Build artefacts, such as test output and harness output is stored in ./testlog. Scylla data files are stored in /tmp.

How it works¶

On start, test.py invokes ninja to find out configured build modes. Then it searches all subdirectories of ./test/ for suite.yaml files: each directory containing suite.yaml is a test suite, in which test.py then looks for tests. All files ending with _test.cc or _test.cql are considered tests.

A suite must contain tests of the same type, as configured in suite.yaml. The list of found tests is matched with the optional command line test name filter. A match is registered if filter substring exists anywhere in test full name. For example:

$ ./test.py cql

runs cql/lwt_test, cql/lwt_batch_test, as well as boost/cql_query_test.

The ./testlog directory is created if it doesn’t exist, otherwise it is cleared from the previous run artefacts.

Matched tests are run concurrently, with concurrency factor set to the number of available CPU cores. test.py continues until all tests are run even if any one of them fails.

CQL tests¶

To run CQL tests, test.py uses an auxiliary program, test/tools/cql_repl.cc. This program reads CQL input from stdin, evaluates it using Scylla CQL database front-end, and prints output in JSON format to stdout. A default keyspace is created automatically.

test.py invokes cql_repl providing the test file for its standard input and redirecting its output to a temporary file in testlog directory.

After cql_repl finishes, test.py compares the output stored in the temporary file with a pre-recorded output stored in test/suitename/testname_test.result.

The test is considered failed if executing any CQL statement produced an error (e.g. because the server crashed during execution) or server output does not match one recorded in testname.result, or there is no testname.result. The latter is possible when it’s the first invocation of the test ever.

In the event of output mismatch file test/suitename/testname_test.reject is created, and first lines of the diff between the two files are output. To update .result file with new output, simply overwrite it with the reject file:

mv test/suitename/testname.re*

Note Since the result file is effectively part of the test, developers must thorougly examine the diff and understand the reasons for every change before overwriting .result files.

Unit tests¶

The same unit test can be run in different seastar configurations, i.e. with different command line arguments. The custom arguments can be set in custom_args key of the suite.yaml file.

Debugging¶

If a test fails, its log can be found in testlog/${mode}/testname.log. By default, all unit tests are built stripped. To build non-stripped tests, ./configure with --tests-debuginfo list-of-tests. test.py adds some command line arguments to unit tests. The exact way in which the test is invoked is recorded in testlog/test.py.log.

To debug CQL tests, one can invoke gdb cql_repl, which will start cql_repl in interactive mode, and then supply faulty CQL statements on the terminal.

Automation¶

If any of the tests fails, test.py returns a non-zero exit status. JUNIT and XUNIT execution status XML files can be found in testlog/${mode}/xml/ directory.

See also¶

For command line help and available options, please see also:

    $ ./test.py --help
PREVIOUS
Logging in Scylla
NEXT
Tracing
  • master
    • 4.6
    • 4.5
  • Scylla Developer Documentation
  • Alternator: DynamoDB API in Scylla
    • Getting Started With ScyllaDB Alternator
    • Scylla Alternator for DynamoDB users
  • Design Notes
    • IDL compiler
    • CDC
    • The Compaction Controller
    • Scylla CQL extensions
    • Scylla CQL extensions
    • CQL3 Type Mapping
    • Hinted Handoff Design
    • Performance Isolation in Scylla
    • CQL to Lua type mapping
    • Scylla Metrics
    • Migrating from users to roles
    • Paged queries
    • Parallel aggregations
    • Per-partition rate limiting
    • Protocol extensions to the Cassandra Native Protocol
    • Ports and protocols in Scylla
    • Intro
    • Raft application in Scylla
    • The group 0
    • Using Group 0 to perform schema changes
    • Redis API in Scylla
    • Repair based node operations
    • Reverse reads
    • Row Cache
    • Row level repair
    • Rust and C++
    • Secondary indexes in Scylla
    • Service Level Distributed Data
    • File format of the Scylla.db sstable component
    • sstables directory structure
    • System keyspace layout
    • Virtual tables in the system keyspace
    • System schema keyspace layout
    • WASM support for user-defined functions
  • Guides
    • Scylla RESTful API V2
    • Building Scylla
    • Debugging with GDB
    • Docker Hub Image
    • Logging in Scylla
    • Testing
    • Tracing
    • Virtual Tables
  • Contribute
    • Contributing to Scylla
    • Backport
    • Maintainer’s handbook
    • Review Checklist
  • Create an issue
  • Edit this page

On this page

  • Testing
    • Introduction
    • Installation
    • Usage
    • How it works
    • CQL tests
    • Unit tests
    • Debugging
    • Automation
    • See also
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon
© 2022, ScyllaDB. All rights reserved.
Last updated on 28 June 2022.
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.2.2