Code Coverage

Note

The coverage reports produced here are those generated by the standard trace module. They are not compatible with those produced by coverage.

Note

You can run zope.testrunner with the common coverage tool to take advantage of its speed and integration with other tools:

coverage run -m zope.testrunner –test-path src …other options…

If the --coverage option is used, test coverage reports will be generated. The directory name given as the parameter will be used to hold the reports.

>>> import os.path, sys, tempfile
>>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
>>> defaults = [
...     '--path', directory_with_tests,
...     '--tests-pattern', '^sampletestsf?$',
...     ]
>>> tempdir = tempfile.mkdtemp(prefix='zope.testrunner-')
>>> coverage_dir = os.path.join(tempdir, 'coverage_dir')
>>> sys.argv = ['test', '--coverage', coverage_dir]
>>> from zope import testrunner
>>> testrunner.run_internal(defaults)
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
  Ran 156 tests with 0 failures, 0 errors and 0 skipped in 0.687 seconds.
Running samplelayers.Layer1 tests:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
  Set up samplelayers.Layer1 in 0.000 seconds.
  Ran 9 tests with 0 failures, 0 errors and 0 skipped in 0.000 seconds.
Running samplelayers.Layer11 tests:
  Set up samplelayers.Layer11 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds.
Running samplelayers.Layer111 tests:
  Set up samplelayers.Layerx in 0.000 seconds.
  Set up samplelayers.Layer111 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds.
Running samplelayers.Layer112 tests:
  Tear down samplelayers.Layer111 in 0.000 seconds.
  Set up samplelayers.Layer112 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds.
Running samplelayers.Layer12 tests:
  Tear down samplelayers.Layer112 in 0.000 seconds.
  Tear down samplelayers.Layerx in 0.000 seconds.
  Tear down samplelayers.Layer11 in 0.000 seconds.
  Set up samplelayers.Layer12 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.140 seconds.
Running samplelayers.Layer121 tests:
  Set up samplelayers.Layer121 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds.
Running samplelayers.Layer122 tests:
  Tear down samplelayers.Layer121 in 0.000 seconds.
  Set up samplelayers.Layer122 in 0.000 seconds.
  Ran 26 tests with 0 failures, 0 errors and 0 skipped in 0.125 seconds.
Tearing down left over layers:
  Tear down samplelayers.Layer122 in 0.000 seconds.
  Tear down samplelayers.Layer12 in 0.000 seconds.
  Tear down samplelayers.Layer1 in 0.000 seconds.
lines   cov%   module   (path)
...
   48   100%   sampletests.test1   (testrunner-ex/sampletests/test1.py)
   74   100%   sampletests.test11   (testrunner-ex/sampletests/test11.py)
   74   100%   sampletests.test111   (testrunner-ex/sampletests/test111.py)
   76   100%   sampletests.test112   (testrunner-ex/sampletests/test112.py)
   74   100%   sampletests.test12   (testrunner-ex/sampletests/test12.py)
   74   100%   sampletests.test121   (testrunner-ex/sampletests/test121.py)
   74   100%   sampletests.test122   (testrunner-ex/sampletests/test122.py)
   48   100%   sampletests.test_one   (testrunner-ex/sampletests/test_one.py)
  112    95%   sampletestsf   (testrunner-ex/sampletestsf.py)
Total: 321 tests, 0 failures, 0 errors and 0 skipped in 0.630 seconds.
False

The directory specified with the --coverage option will have been created and will hold the coverage reports.

>>> os.path.exists(coverage_dir)
True
>>> os.listdir(coverage_dir)
[...]

(We should clean up after ourselves.)

>>> import shutil
>>> shutil.rmtree(tempdir)

Ignoring Tests

The trace module supports ignoring directories and modules based the test selection. Only directories selected for testing should report coverage. The test runner provides a custom implementation of the relevant API.

The TestIgnore class, the class managing the ignoring, is initialized by passing the command line options. It uses the options to determine the directories that should be covered.

>>> class FauxOptions(object):
...   package = None
...   test_path = [('/myproject/src/blah/foo', ''),
...                ('/myproject/src/blah/bar', '')]
>>> from zope.testrunner import coverage
>>> from zope.testrunner.find import test_dirs
>>> ignore = coverage.TestIgnore(test_dirs(FauxOptions(), {}))
>>> ignore._test_dirs
['/myproject/src/blah/foo/', '/myproject/src/blah/bar/']

We can now ask whether a particular module should be ignored:

>>> ignore.names('/myproject/src/blah/foo/baz.py', 'baz')
False
>>> ignore.names('/myproject/src/blah/bar/mine.py', 'mine')
False
>>> ignore.names('/myproject/src/blah/foo/__init__.py', 'foo')
False
>>> ignore.names('/myproject/src/blah/hello.py', 'hello')
True

When running the test runner, modules are sometimes created from text strings. Those should always be ignored:

>>> ignore.names('/myproject/src/blah/hello.rst', '<string>')
True

To make this check fast, the class implements a cache. In an early implementation, the result was cached by the module name, which was a problem, since a lot of modules carry the same name (not the Python dotted name here!). So just because a module has the same name in an ignored and tested directory, does not mean it is always ignored:

>>> ignore.names('/myproject/src/blah/module.py', 'module')
True
>>> ignore.names('/myproject/src/blah/foo/module.py', 'module')
False