Generating Code Coverage¶

Note: This guide details how to generate code coverage when using Scrutinizer as your CI service. Please refer to sending external code coverage if you are using Scrutinizer for automated code reviews only.

Scrutinizer can process code coverage information that you generate during your build. We support various formats for code coverage. As long as your test runner generates one of the compatible formats, Scrutinizer can automatically process and merge coverage data from multiple test runs (even of different runners or languages).

You can also define failure conditions related to coverage like how much coverage a build must have or how well new code should be covered for a build to pass.

When you run a command, simply tell Scrutinizer where to find the coverage file and what format it has:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: ./my-test-run
            coverage:
              file: path/to/coverage-data

              # These formats are supported:
              # clover, cobertura, jacoco, python-cc, ruby-cc, go-cc, lcov
              format: name-of-format

Supported Formats¶

Clover¶

Clover is the most common format, and generated by virtually all modern test runners. When running your tests, simply tell Scrutinizer where to find the clover file and it will pick up the coverage data from there:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: ./run-my-tests
            coverage:
              file: target/coverage.xml   # <- Set this to where we find the coverage data.
                                          #    The path is relative to the current directory.
              format: clover

This format is for example used by phpunit (PHP), istanbul (Javascript), many Java build systems, Go, and many more.

Python Coverage¶

For Python, we support data generated by coverage. When setting up your test commands, simply specify where we find the .coverage file, and potentially your .coveragerc configuration file:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: './run-tests.sh'
            coverage:
              file: '.coverage'
              config_file: '.coveragerc'
              format: 'py-cc'

Ruby Coverage¶

For Ruby, code coverage can be generated using our scrutinizer-ocular gem that uses SimpleCov internally. When you have set-up the scrutinizer-ocular gem, make sure to set the SCRUTINIZER_CC_FILE environment variable:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: 'bundle exec rspec spec'
            environment:
              'SCRUTINIZER_CC_FILE': 'my-coverage'
            coverage:
              file: 'my-coverage'
              format: 'rb-cc'

Cobertura¶

For Java, Scrutinizer also supports the cobertura format:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: mvn clean package
            coverage:
              file: path/to/cobertura
              format: cobertura

JaCoCo¶

As another Java format, Scrutinizer also supports jacoco:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: mvn clean package
            coverage:
              file: '[path-to-coverage-reports]'
              format: 'jacoco'

LCOV¶

The lcov coverage format was originally created for Linux test coverage, but is supported by various other runners, too:

build:
  nodes:
    coverage:
      tests:
        override:
          - command: make test
            coverage:
              file: '/path/to/lcov.coverage'
              format: 'lcov'

Go Coverage¶

For Go projects, Scrutinizer supports the default coverage file format of the go tool Cover <https://golang.org/cmd/cover/> along with the cross language formats like clover or cobertura.

If you use the Cover tool as an example, the configuration will look like:

build:
    nodes:
        coverage:
            tests:
                override:
                    -
                        command: go test -coverprofile=cover.out
                        coverage:
                            file: 'cover.out'
                            format: 'go-cc'