Developer Coverage Overview

Building

The implementation of coverage starts with the "doCoverage" flag on the builder in comp-builder.nix. The doCoverage flag enables and disables the Cabal coverage flag and copies any generated coverage data to "$out/share/hpc".

Mix and tix files

The coverage information for any derivation consists of "mix" and "tix" files.

Mix files record static information about a source file and are generated at build time. They primarily contain a path to the source file and information about expressions and regions of the source file, which are later referenced by tix files.

Tix files contain dynamic information about a test run, recording when a portion of a source file is touched by a test. These are generated when the test is run.

Multiple local packages

In the context of multiple local packages, there are a few types of coverage we might be interested in: - How well does the tests for this package cover the package library? - How well does the tests for this package cover the libraries of other packages in this project? - Both of the above.

To facilitate expressing any of these classifications of coverage, the lib/cover.nix function provides the mixLibraries argument. If you're just interested in how the tests cover the package library, you provide that library as an argument to mixLibraries. If you're interested in how the tests also cover other local packages in the project, you can also provide those libraries as arguments to mixLibraries.

The projectCoverageReport and coverageReport attributes that are provided by default on projects and packages respectively provide coverage information for all local packages in the project. This is to mimic the behaviour of Stack, which seems to be the expectation of most people. Of course, you can use the projectCoverageReport and coverageReport functions to construct your own custom coverage reports (as detailed in the coverage tutorial).

Coverage reports

Package reports

The coverage information generated will look something like this:

/nix/store/...-my-project-0.1.0.0-coverage-report/
└── share
    └── hpc
        └── vanilla
            ├── html
            │   └── my-library-0.1.0.0
            │       ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       │   ├── My.Lib.Config.hs.html
            │       │   ├── My.Lib.Types.hs.html
            │       │   └── My.Lib.Util.hs.html
            │       ├── hpc_index_alt.html
            │       ├── hpc_index_exp.html
            │       ├── hpc_index_fun.html
            │       └── hpc_index.html
            ├── mix
            │   └── my-library-0.1.0.0
            │       └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │           ├── My.Lib.Config.mix
            │           ├── My.Lib.Types.mix
            │           └── My.Lib.Util.mix
            └── tix
                └── my-library-0.1.0.0
                    ├── my-library-0.1.0.0.tix
                    ├── my-test-1
                    │   └── my-test-1.tix
                    └── unit-test
                        └── unit-test.tix
  • The mix files are copied verbatim from the library built with coverage.
  • The tix files for each test are copied from the check run verbatim and are output to ".../tix///.tix".
  • The tix files for each library are generated by summing the tix files for each test, but excluding any test modules. This tix file is output to ".../tix//.tix".
  • Test modules are determined by inspecting the plan for the project (i.e. for the project "my-project" and test-suite "my-test-1", the test modules are read from: my-project.checks.my-test-1.config.modules)
  • The hpc HTML reports for each library are generated from their respective tix files (i.e. the share/hpc/vanilla/html/my-library-0.1.0.0 report is generated from the share/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tix file)

Project-wide reports

The coverage information for an entire project will look something like this:

/nix/store/...-coverage-report
└── share
    └── hpc
        └── vanilla
            ├── html
            │   ├── index.html
            │   ├── all
            │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   │   ├── My.Lib.Config.hs.html
            │   │   │   ├── My.Lib.Types.hs.html
            │   │   │   └── My.Lib.Util.hs.html
            │   │   ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │   │   │   ├── Other.Lib.A.hs.html
            │   │   │   └── Other.Lib.B.hs.html
            │   │   ├── hpc_index_alt.html
            │   │   ├── hpc_index_exp.html
            │   │   ├── hpc_index_fun.html
            │   │   └── hpc_index.html
            │   ├── my-library-0.1.0.0
            │   │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   │   ├── My.Lib.Config.hs.html
            │   │   │   ├── My.Lib.Types.hs.html
            │   │   │   └── My.Lib.Util.hs.html
            │   │   ├── hpc_index_alt.html
            │   │   ├── hpc_index_exp.html
            │   │   ├── hpc_index_fun.html
            │   │   └── hpc_index.html
            │   └── other-libray-0.1.0.0
            │       ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       │   ├── Other.Lib.A.hs.html
            │       │   └── Other.Lib.B.hs.html
            │       ├── hpc_index_alt.html
            │       ├── hpc_index_exp.html
            │       ├── hpc_index_fun.html
            │       └── hpc_index.html
            ├── mix
            │   ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
            │   │   ├── My.Lib.Config.mix
            │   │   ├── My.Lib.Types.mix
            │   │   └── My.Lib.Util.mix
            │   └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
            │       ├── Other.Lib.A.mix
            │       └── Other.Lib.B.mix
            └── tix
                ├── all
                │   └── all.tix
                ├── my-library-0.1.0.0
                │   ├── my-library-0.1.0.0.tix
                │   ├── my-test-1
                │   │   └── my-test-1.tix
                │   └── unit-test
                │       └── unit-test.tix
                └── another-library-0.1.0.0
                    ├── another-library-0.1.0.0.tix
                    ├── my-test-2
                    │   └── my-test-2.tix
                    └── unit-test
                        └── unit-test.tix

All of the coverage information is copied verbatim from the coverage reports for each of the constituent packages. A few additions are made: - tix/all/all.tix is generated from the union of all the library tix files. - We use this file when generating coverage reports for "coveralls.io". - An index page (html/index.html) is generated which links to the HTML coverage reports of the constituent packages. - A synthetic HTML report is generated from the tix/all/all.tix file. This shows the union of all the coverage information generated by each constituent coverage report.