Command Line Usage¶
This package provides a script, zope-testrunner
, that’s available when
installed via pip. It can also be used as a Python module with the
-m
option.
$ python -m zope.testrunner --help
usage: __main__.py [-h] [--package PACKAGE] [--module MODULE] [--test TEST]
[--unit] [--non-unit] [--layer LAYER] [-a AT_LEVEL] [--all]
[--only-level ONLY_LEVEL] [--list-tests] [--require-unique]
[--verbose] [--quiet] [--progress] [--no-progress]
[--auto-progress] [--color] [--no-color] [--auto-color]
[--subunit] [--subunit-v2] [--slow-test N] [-1]
[--show-secondary-failures] [--ndiff] [--udiff] [--cdiff]
[--ignore-new-thread REGEXP] [--buffer] [--xml XMLOUTPUT]
[--stop-on-error] [--post-mortem] [--gc GC]
[--gc-option {DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_SAVEALL,DEBUG_INSTANCES,DEBUG_COLLECTABLE,DEBUG_OBJECTS}]
[--gc-after-test] [--repeat REPEAT] [--report-refcounts]
[--coverage COVERAGE] [--profile {cProfile}]
[--profile-directory PROF_DIR] [--path PATH]
[--test-path TEST_PATH] [--package-path ARG ARG]
[--tests-pattern TESTS_PATTERN] [--suite-name SUITE_NAME]
[--test-file-pattern TEST_FILE_PATTERN]
[--ignore_dir IGNORE_DIR] [--shuffle]
[--shuffle-seed SHUFFLE_SEED] [--version] [-j PROCESSES]
[--keepbytecode] [--usecompiled] [--exit-with-status]
[legacy_module_filter] [legacy_test_filter]
Discover and run unittest tests
positional arguments:
legacy_module_filter DEPRECATED: Prefer to use --module.
legacy_test_filter DEPRECATED: Prefer to use --test.
options:
-h, --help show this help message and exit
Searching and filtering:
Options in this group are used to define which tests to run.
--package PACKAGE, --dir PACKAGE, -s PACKAGE
Search the given package's directories for tests. This
can be specified more than once to run tests in
multiple parts of the source tree. For example, if
refactoring interfaces, you don't want to see the way
you have broken setups for tests in other packages.
You *just* want to run the interface tests. Packages
are supplied as dotted names. For compatibility with
the old test runner, forward and backward slashed in
package names are converted to dots. (In the special
case of packages spread over multiple directories,
only directories within the test search path are
searched. See the --path option.)
--module MODULE, -m MODULE
Specify a test-module filter as a regular expression.
This is a case-sensitive regular expression, used in
search (not match) mode, to limit which test modules
are searched for tests. The regular expressions are
checked against dotted module names. In an extension
of Python regexp notation, a leading "!" is stripped
and causes the sense of the remaining regexp to be
negated (so "!bc" matches any string that does not
match "bc", and vice versa). The option can be
specified multiple test-module filters. Test modules
matching any of the test filters are searched. If no
test-module filter is specified, then all test modules
are used.
--test TEST, -t TEST Specify a test filter as a regular expression. This is
a case-sensitive regular expression, used in search
(not match) mode, to limit which tests are run. In an
extension of Python regexp notation, a leading "!" is
stripped and causes the sense of the remaining regexp
to be negated (so "!bc" matches any string that does
not match "bc", and vice versa). The option can be
specified multiple test filters. Tests matching any of
the test filters are included. If no test filter is
specified, then all tests are run.
--unit, -u Run only unit tests, ignoring any layer options.
--non-unit, -f Run tests other than unit tests.
--layer LAYER Specify a test layer to run. The option can be given
multiple times to specify more than one layer. If not
specified, all layers are run. It is common for the
running script to provide default values for this
option. Layers are specified regular expressions, used
in search mode, for dotted names of objects that
define a layer. In an extension of Python regexp
notation, a leading "!" is stripped and causes the
sense of the remaining regexp to be negated (so "!bc"
matches any string that does not match "bc", and vice
versa). The layer named
'zope.testrunner.layer.UnitTests' is reserved for unit
tests, however, take note of the --unit and non-unit
options.
-a AT_LEVEL, --at-level AT_LEVEL
Run the tests at the given level. Any test at a level
at or below this is run, any test at a level above
this is not run. Level <= 0 runs all tests.
--all Run tests at all levels.
--only-level ONLY_LEVEL
Run only the tests at the given level. Tests on other
levels are not run. Default is None, which means the
level is not restricted by this option. If this option
is used it overrides `--at-level` and `--all` options.
--list-tests List all tests that matched your filters. Do not run
any tests.
--require-unique Require that all test IDs be unique and raise an error
if duplicates are encountered.
Reporting:
Reporting options control basic aspects of test-runner output
--verbose, -v Make output more verbose. Increment the verbosity
level.
--quiet, -q Make the output minimal, overriding any verbosity
options.
--progress, -p Output progress status
--no-progress Do not output progress status. This is the default,
but can be used to counter a previous use of
--progress or -p.
--auto-progress Output progress status, but only when stdout is a
terminal.
--color, -c Colorize the output.
--no-color, -C Do not colorize the output. This is the default, but
can be used to counter a previous use of --color or
-c.
--auto-color Colorize the output, but only when stdout is a
terminal.
--subunit Use subunit v1 output. Will not be colorized.
--subunit-v2 Use subunit v2 output. Will not be colorized.
--slow-test N With -c and -vvv, highlight tests that take longer
than N seconds (default: 10).
-1, --hide-secondary-failures
Report only the first failure in a doctest. (Examples
after the failure are still executed, in case they do
any cleanup.)
--show-secondary-failures
Report all failures in a doctest. This is the default,
but can be used to counter a default use of -1 or
--hide-secondary-failures.
--ndiff When there is a doctest failure, show it as a diff
using the ndiff.py utility.
--udiff When there is a doctest failure, show it as a unified
diff.
--cdiff When there is a doctest failure, show it as a context
diff.
--ignore-new-thread REGEXP
If a thread with this name is left behind, don't
report this at the end. This is a case-sensitive
regular expression, used in match mode. This option
can be used multiple times. If a thread name matches
any of them, it will be ignored.
--buffer Buffer the standard output and standard error streams
during each test. Output during a passing test is
discarded. Output during failing or erroring tests is
echoed. This option is enabled by default if --subunit
or --subunit-v2 is in use, to avoid corrupting the
subunit stream.
--xml XMLOUTPUT If given, XML reports will be written to the specified
directory.
Analysis:
Analysis options provide tools for analysing test output.
--stop-on-error, --stop, -x
Stop running tests after first test failure or error.
--post-mortem, --pdb, -D
Enable post-mortem debugging of test failures
--gc GC, -g GC Set the garbage collector generation threshold. This
can be used to stress memory and gc correctness. Some
crashes are only reproducible when the threshold is
set to 1 (aggressive garbage collection). Do "--gc 0"
to disable garbage collection altogether. The --gc
option can be used up to 3 times to specify up to 3 of
the 3 Python gc_threshold settings.
--gc-option {DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_SAVEALL,DEBUG_INSTANCES,DEBUG_COLLECTABLE,DEBUG_OBJECTS}, -G {DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_SAVEALL,DEBUG_INSTANCES,DEBUG_COLLECTABLE,DEBUG_OBJECTS}
Set a Python gc-module debug flag. This option can be
used more than once to set multiple flags.
--gc-after-test After each test, call 'gc.collect' and record the
return value *rv*; when *rv* is non-zero, output '!'
on verbosity level 1 and '[*rv*]' on higher verbosity
levels. On verbosity level 4 or higher output detailed
cycle information.
--repeat REPEAT, -N REPEAT
Repeat the tests the given number of times. This
option is used to make sure that tests leave their
environment in the state they found it and, with the
--report-refcounts option to look for memory leaks.
--report-refcounts, -r
After each run of the tests, output a report
summarizing changes in refcounts by object type. This
option that requires that Python was built with the
--with-pydebug option to configure.
--coverage COVERAGE Perform code-coverage analysis, saving trace data to
the directory with the given name. A code coverage
summary is printed to standard out.
--profile {cProfile} Run the tests under cProfiler and display the top 50
stats, sorted by cumulative time and number of calls.
--profile-directory PROF_DIR
Directory for temporary profiler files. All files
named tests_profile.*.prof in this directory will be
removed. If you intend to run multiple instances of
the test runner in parallel, be sure to tell them to
use different directories, so they won't step on each
other's toes.
Setup:
Setup options are normally supplied by the testrunner script, although
they can be overridden by users.
--path PATH Specify a path to be added to Python's search path.
This option can be used multiple times to specify
multiple search paths. The path is usually specified
by the test-runner script itself, rather than by users
of the script, although it can be overridden by users.
Only tests found in the path will be run. This option
also specifies directories to be searched for tests.
See the search_directory.
--test-path TEST_PATH
Specify a path to be searched for tests, but not added
to the Python search path. This option can be used
multiple times to specify multiple search paths. The
path is usually specified by the test-runner script
itself, rather than by users of the script, although
it can be overridden by users. Only tests found in the
path will be run.
--package-path ARG ARG
Specify a path to be searched for tests, but not added
to the Python search path. Also specify a package for
files found in this path. This is used to deal with
directories that are stitched into packages that are
not otherwise searched for tests. This option takes 2
arguments specifying the path and the package. This
option can be used multiple times to specify multiple
search paths. The path is usually specified by the
test-runner script itself, rather than by users of the
script, although it can be overridden by users. Only
tests found in the path will be run.
--tests-pattern TESTS_PATTERN
The test runner looks for modules containing tests. It
uses this pattern to identify these modules. The
modules may be either packages or python files. If a
test module is a package, it uses the value given by
the test-file-pattern to identify python files within
the package containing tests.
--suite-name SUITE_NAME
Specify the name of the object in each test_module
that contains the module's test suite.
--test-file-pattern TEST_FILE_PATTERN
Specify a pattern for identifying python files within
a tests package. See the documentation for the
--tests-pattern option.
--ignore_dir IGNORE_DIR
Specifies the name of a directory to ignore when
looking for tests.
--shuffle Shuffles the order in which tests are ran.
--shuffle-seed SHUFFLE_SEED
Value used to initialize the tests shuffler. Specify a
value to create repeatable random ordered tests.
Other:
Other options
--version Print the version of the testrunner, and exit.
-j PROCESSES Use up to given number of parallel processes to
execute tests. May decrease test run time
substantially. Defaults to 1.
--keepbytecode, -k Normally, the test runner scans the test paths and the
test directories looking for and deleting pyc or pyo
files without corresponding py files. This is to
prevent spurious test failures due to finding compiled
modules where source modules have been deleted. This
scan can be time consuming. Using this option disables
this scan. If you know you haven't removed any modules
since last running the tests, can make the test run go
much faster.
--usecompiled Normally, a package must contain an __init__.py file,
and only .py files can contain test code. When this
option is specified, compiled Python files (.pyc and
.pyo) can be used instead: a directory containing
__init__.pyc or __init__.pyo is also considered to be
a package, and if file XYZ.py contains tests but is
absent while XYZ.pyc or XYZ.pyo exists then the
compiled files will be used. This is necessary when
running tests against a tree where the .py files have
been removed after compilation to .pyc/.pyo. Use of
this option implies --keepbytecode.
--exit-with-status DEPRECATED: The test runner will always exit with a
status.