Command Line Usage

This package provides a script, zope-testrunner, that’s available when installed via pip. It can also be used as a Python module with the -m option.

$ python -m zope.testrunner --help
usage: __main__.py [-h] [--package PACKAGE] [--module MODULE] [--test TEST]
                   [--unit] [--non-unit] [--layer LAYER] [-a AT_LEVEL] [--all]
                   [--list-tests] [--require-unique] [--verbose] [--quiet]
                   [--progress] [--no-progress] [--auto-progress] [--color]
                   [--no-color] [--auto-color] [--subunit] [--subunit-v2]
                   [--slow-test N] [-1] [--show-secondary-failures] [--ndiff]
                   [--udiff] [--cdiff] [--ignore-new-thread REGEXP] [--buffer]
                   [--stop-on-error] [--post-mortem] [--gc GC]
                   [--gc-option {DEBUG_OBJECTS,DEBUG_UNCOLLECTABLE,DEBUG_COLLECTABLE,DEBUG_INSTANCES,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_LEAK}]
                   [--gc-after-test] [--repeat REPEAT] [--report-refcounts]
                   [--coverage COVERAGE] [--profile {cProfile}]
                   [--profile-directory PROF_DIR] [--path PATH]
                   [--test-path TEST_PATH] [--package-path ARG ARG]
                   [--tests-pattern TESTS_PATTERN] [--suite-name SUITE_NAME]
                   [--test-file-pattern TEST_FILE_PATTERN]
                   [--ignore_dir IGNORE_DIR] [--shuffle]
                   [--shuffle-seed SHUFFLE_SEED] [--version] [-j PROCESSES]
                   [--keepbytecode] [--usecompiled] [--exit-with-status]
                   [legacy_module_filter] [legacy_test_filter]

Discover and run unittest tests

positional arguments:
  legacy_module_filter  DEPRECATED: Prefer to use --module.
  legacy_test_filter    DEPRECATED: Prefer to use --test.

optional arguments:
  -h, --help            show this help message and exit

Searching and filtering:
  Options in this group are used to define which tests to run.

  --package PACKAGE, --dir PACKAGE, -s PACKAGE
                        Search the given package's directories for tests. This
                        can be specified more than once to run tests in
                        multiple parts of the source tree. For example, if
                        refactoring interfaces, you don't want to see the way
                        you have broken setups for tests in other packages.
                        You *just* want to run the interface tests. Packages
                        are supplied as dotted names. For compatibility with
                        the old test runner, forward and backward slashed in
                        package names are converted to dots. (In the special
                        case of packages spread over multiple directories,
                        only directories within the test search path are
                        searched. See the --path option.)
  --module MODULE, -m MODULE
                        Specify a test-module filter as a regular expression.
                        This is a case-sensitive regular expression, used in
                        search (not match) mode, to limit which test modules
                        are searched for tests. The regular expressions are
                        checked against dotted module names. In an extension
                        of Python regexp notation, a leading "!" is stripped
                        and causes the sense of the remaining regexp to be
                        negated (so "!bc" matches any string that does not
                        match "bc", and vice versa). The option can be
                        specified multiple test-module filters. Test modules
                        matching any of the test filters are searched. If no
                        test-module filter is specified, then all test modules
                        are used.
  --test TEST, -t TEST  Specify a test filter as a regular expression. This is
                        a case-sensitive regular expression, used in search
                        (not match) mode, to limit which tests are run. In an
                        extension of Python regexp notation, a leading "!" is
                        stripped and causes the sense of the remaining regexp
                        to be negated (so "!bc" matches any string that does
                        not match "bc", and vice versa). The option can be
                        specified multiple test filters. Tests matching any of
                        the test filters are included. If no test filter is
                        specified, then all tests are run.
  --unit, -u            Run only unit tests, ignoring any layer options.
  --non-unit, -f        Run tests other than unit tests.
  --layer LAYER         Specify a test layer to run. The option can be given
                        multiple times to specify more than one layer. If not
                        specified, all layers are run. It is common for the
                        running script to provide default values for this
                        option. Layers are specified regular expressions, used
                        in search mode, for dotted names of objects that
                        define a layer. In an extension of Python regexp
                        notation, a leading "!" is stripped and causes the
                        sense of the remaining regexp to be negated (so "!bc"
                        matches any string that does not match "bc", and vice
                        versa). The layer named
                        'zope.testrunner.layer.UnitTests' is reserved for unit
                        tests, however, take note of the --unit and non-unit
                        options.
  -a AT_LEVEL, --at-level AT_LEVEL
                        Run the tests at the given level. Any test at a level
                        at or below this is run, any test at a level above
                        this is not run. Level 0 runs all tests.
  --all                 Run tests at all levels.
  --list-tests          List all tests that matched your filters. Do not run
                        any tests.
  --require-unique      Require that all test IDs be unique and raise an error
                        if duplicates are encountered.

Reporting:
  Reporting options control basic aspects of test-runner output

  --verbose, -v         Make output more verbose. Increment the verbosity
                        level.
  --quiet, -q           Make the output minimal, overriding any verbosity
                        options.
  --progress, -p        Output progress status
  --no-progress         Do not output progress status. This is the default,
                        but can be used to counter a previous use of
                        --progress or -p.
  --auto-progress       Output progress status, but only when stdout is a
                        terminal.
  --color, -c           Colorize the output.
  --no-color, -C        Do not colorize the output. This is the default, but
                        can be used to counter a previous use of --color or
                        -c.
  --auto-color          Colorize the output, but only when stdout is a
                        terminal.
  --subunit             Use subunit v1 output. Will not be colorized.
  --subunit-v2          Use subunit v2 output. Will not be colorized.
  --slow-test N         With -c and -vvv, highlight tests that take longer
                        than N seconds (default: 10).
  -1, --hide-secondary-failures
                        Report only the first failure in a doctest. (Examples
                        after the failure are still executed, in case they do
                        any cleanup.)
  --show-secondary-failures
                        Report all failures in a doctest. This is the default,
                        but can be used to counter a default use of -1 or
                        --hide-secondary-failures.
  --ndiff               When there is a doctest failure, show it as a diff
                        using the ndiff.py utility.
  --udiff               When there is a doctest failure, show it as a unified
                        diff.
  --cdiff               When there is a doctest failure, show it as a context
                        diff.
  --ignore-new-thread REGEXP
                        If a thread with this name is left behind, don't
                        report this at the end. This is a case-sensitive
                        regular expression, used in match mode. This option
                        can be used multiple times. If a thread name matches
                        any of them, it will be ignored.
  --buffer              Buffer the standard output and standard error streams
                        during each test. Output during a passing test is
                        discarded. Output during failing or erroring tests is
                        echoed. This option is enabled by default if --subunit
                        or --subunit-v2 is in use, to avoid corrupting the
                        subunit stream.

Analysis:
  Analysis options provide tools for analysing test output.

  --stop-on-error, --stop, -x
                        Stop running tests after first test failure or error.
  --post-mortem, --pdb, -D
                        Enable post-mortem debugging of test failures
  --gc GC, -g GC        Set the garbage collector generation threshold. This
                        can be used to stress memory and gc correctness. Some
                        crashes are only reproducible when the threshold is
                        set to 1 (aggressive garbage collection). Do "--gc 0"
                        to disable garbage collection altogether. The --gc
                        option can be used up to 3 times to specify up to 3 of
                        the 3 Python gc_threshold settings.
  --gc-option {DEBUG_OBJECTS,DEBUG_UNCOLLECTABLE,DEBUG_COLLECTABLE,DEBUG_INSTANCES,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_LEAK}, -G {DEBUG_OBJECTS,DEBUG_UNCOLLECTABLE,DEBUG_COLLECTABLE,DEBUG_INSTANCES,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_LEAK}
                        Set a Python gc-module debug flag. This option can be
                        used more than once to set multiple flags.
  --gc-after-test       After each test, call 'gc.collect' and record the
                        return value *rv*; when *rv* is non-zero, output '!'
                        on verbosity level 1 and '[*rv*]' on higher verbosity
                        levels. On verbosity level 4 or higher output detailed
                        cycle information.
  --repeat REPEAT, -N REPEAT
                        Repeat the tests the given number of times. This
                        option is used to make sure that tests leave their
                        environment in the state they found it and, with the
                        --report-refcounts option to look for memory leaks.
  --report-refcounts, -r
                        After each run of the tests, output a report
                        summarizing changes in refcounts by object type. This
                        option that requires that Python was built with the
                        --with-pydebug option to configure.
  --coverage COVERAGE   Perform code-coverage analysis, saving trace data to
                        the directory with the given name. A code coverage
                        summary is printed to standard out.
  --profile {cProfile}  Run the tests under cProfiler and display the top 50
                        stats, sorted by cumulative time and number of calls.
  --profile-directory PROF_DIR
                        Directory for temporary profiler files. All files
                        named tests_profile.*.prof in this directory will be
                        removed. If you intend to run multiple instances of
                        the test runner in parallel, be sure to tell them to
                        use different directories, so they won't step on each
                        other's toes.

Setup:
  Setup options are normally supplied by the testrunner script, although
  they can be overridden by users.

  --path PATH           Specify a path to be added to Python's search path.
                        This option can be used multiple times to specify
                        multiple search paths. The path is usually specified
                        by the test-runner script itself, rather than by users
                        of the script, although it can be overridden by users.
                        Only tests found in the path will be run. This option
                        also specifies directories to be searched for tests.
                        See the search_directory.
  --test-path TEST_PATH
                        Specify a path to be searched for tests, but not added
                        to the Python search path. This option can be used
                        multiple times to specify multiple search paths. The
                        path is usually specified by the test-runner script
                        itself, rather than by users of the script, although
                        it can be overridden by users. Only tests found in the
                        path will be run.
  --package-path ARG ARG
                        Specify a path to be searched for tests, but not added
                        to the Python search path. Also specify a package for
                        files found in this path. This is used to deal with
                        directories that are stitched into packages that are
                        not otherwise searched for tests. This option takes 2
                        arguments specifying the path and the package. This
                        option can be used multiple times to specify multiple
                        search paths. The path is usually specified by the
                        test-runner script itself, rather than by users of the
                        script, although it can be overridden by users. Only
                        tests found in the path will be run.
  --tests-pattern TESTS_PATTERN
                        The test runner looks for modules containing tests. It
                        uses this pattern to identify these modules. The
                        modules may be either packages or python files. If a
                        test module is a package, it uses the value given by
                        the test-file-pattern to identify python files within
                        the package containing tests.
  --suite-name SUITE_NAME
                        Specify the name of the object in each test_module
                        that contains the module's test suite.
  --test-file-pattern TEST_FILE_PATTERN
                        Specify a pattern for identifying python files within
                        a tests package. See the documentation for the
                        --tests-pattern option.
  --ignore_dir IGNORE_DIR
                        Specifies the name of a directory to ignore when
                        looking for tests.
  --shuffle             Shuffles the order in which tests are ran.
  --shuffle-seed SHUFFLE_SEED
                        Value used to initialize the tests shuffler. Specify a
                        value to create repeatable random ordered tests.

Other:
  Other options

  --version             Print the version of the testrunner, and exit.
  -j PROCESSES          Use up to given number of parallel processes to
                        execute tests. May decrease test run time
                        substantially. Defaults to 1.
  --keepbytecode, -k    Normally, the test runner scans the test paths and the
                        test directories looking for and deleting pyc or pyo
                        files without corresponding py files. This is to
                        prevent spurious test failures due to finding compiled
                        modules where source modules have been deleted. This
                        scan can be time consuming. Using this option disables
                        this scan. If you know you haven't removed any modules
                        since last running the tests, can make the test run go
                        much faster.
  --usecompiled         Normally, a package must contain an __init__.py file,
                        and only .py files can contain test code. When this
                        option is specified, compiled Python files (.pyc and
                        .pyo) can be used instead: a directory containing
                        __init__.pyc or __init__.pyo is also considered to be
                        a package, and if file XYZ.py contains tests but is
                        absent while XYZ.pyc or XYZ.pyo exists then the
                        compiled files will be used. This is necessary when
                        running tests against a tree where the .py files have
                        been removed after compilation to .pyc/.pyo. Use of
                        this option implies --keepbytecode.
  --exit-with-status    DEPRECATED: The test runner will always exit with a
                        status.