Developer Coverage Overview
Building
The implementation of coverage starts with the "doCoverage" flag on
the builder in comp-builder.nix. The doCoverage flag enables and
disables the Cabal coverage flag and copies any generated coverage
data to "$out/share/hpc".
Mix and tix files
The coverage information for any derivation consists of "mix" and "tix" files.
Mix files record static information about a source file and are generated at build time. They primarily contain a path to the source file and information about expressions and regions of the source file, which are later referenced by tix files.
Tix files contain dynamic information about a test run, recording when a portion of a source file is touched by a test. These are generated when the test is run.
Multiple local packages
In the context of multiple local packages, there are a few types of coverage we might be interested in:
- How well does the tests for this package cover the package library?
- How well does the tests for this package cover the libraries of other packages in this project?
- Both of the above.
To facilitate expressing any of these classifications of coverage, the
lib/cover.nix function provides the mixLibraries argument. If
you're just interested in how the tests cover the package library, you
provide that library as an argument to mixLibraries. If you're
interested in how the tests also cover other local packages in the
project, you can also provide those libraries as arguments to
mixLibraries.
The projectCoverageReport and coverageReport attributes that are
provided by default on projects and packages respectively provide
coverage information for all local packages in the project. This is
to mimic the behaviour of Stack, which seems to be the expectation of
most people. Of course, you can use the projectCoverageReport and
coverageReport functions to construct your own custom coverage
reports (as detailed in the coverage tutorial).
Coverage reports
Package reports
The coverage information generated will look something like this:
/nix/store/...-my-project-0.1.0.0-coverage-report/
└── share
└── hpc
└── vanilla
├── html
│ └── my-library-0.1.0.0
│ ├── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
│ │ ├── My.Lib.Config.hs.html
│ │ ├── My.Lib.Types.hs.html
│ │ └── My.Lib.Util.hs.html
│ ├── hpc_index_alt.html
│ ├── hpc_index_exp.html
│ ├── hpc_index_fun.html
│ └── hpc_index.html
├── mix
│ └── my-library-0.1.0.0
│ └── my-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
│ ├── My.Lib.Config.mix
│ ├── My.Lib.Types.mix
│ └── My.Lib.Util.mix
└── tix
└── my-library-0.1.0.0
├── my-library-0.1.0.0.tix
├── my-test-1
│ └── my-test-1.tix
└── unit-test
└── unit-test.tix
- The mix files are copied verbatim from the library built with coverage.
- The tix files for each test are copied from the check run verbatim
and are output to ".../tix/
/ / .tix". - The tix files for each library are generated by summing the tix
files for each test, but excluding any test modules. This tix file
is output to ".../tix/
/ .tix". - Test modules are determined by inspecting the plan for the project
(i.e. for the project "my-project" and test-suite "my-test-1", the
test modules are read from:
my-project.checks.my-test-1.config.modules)
- Test modules are determined by inspecting the plan for the project
(i.e. for the project "my-project" and test-suite "my-test-1", the
test modules are read from:
- The hpc HTML reports for each library are generated from their
respective tix files (i.e. the
share/hpc/vanilla/html/my-library-0.1.0.0report is generated from theshare/hpc/vanilla/tix/my-library-0.1.0.0/my-library-0.1.0.0.tixfile)
Project-wide reports
The coverage information for an entire project will look something like this:
/nix/store/...-coverage-report
└── share
└── hpc
└── vanilla
├── html
│ ├── index.html
│ ├── all
│ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
│ │ │ ├── My.Lib.Config.hs.html
│ │ │ ├── My.Lib.Types.hs.html
│ │ │ └── My.Lib.Util.hs.html
│ │ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
│ │ │ ├── Other.Lib.A.hs.html
│ │ │ └── Other.Lib.B.hs.html
│ │ ├── hpc_index_alt.html
│ │ ├── hpc_index_exp.html
│ │ ├── hpc_index_fun.html
│ │ └── hpc_index.html
│ ├── my-library-0.1.0.0
│ │ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
│ │ │ ├── My.Lib.Config.hs.html
│ │ │ ├── My.Lib.Types.hs.html
│ │ │ └── My.Lib.Util.hs.html
│ │ ├── hpc_index_alt.html
│ │ ├── hpc_index_exp.html
│ │ ├── hpc_index_fun.html
│ │ └── hpc_index.html
│ └── other-libray-0.1.0.0
│ ├── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
│ │ ├── Other.Lib.A.hs.html
│ │ └── Other.Lib.B.hs.html
│ ├── hpc_index_alt.html
│ ├── hpc_index_exp.html
│ ├── hpc_index_fun.html
│ └── hpc_index.html
├── mix
│ ├── my-library-0.1.0.0-ERSaOroBZhe9awsoBkhmcV
│ │ ├── My.Lib.Config.mix
│ │ ├── My.Lib.Types.mix
│ │ └── My.Lib.Util.mix
│ └── other-library-0.1.0.0-48EVZBwW9Kj29VTaRMhBDf
│ ├── Other.Lib.A.mix
│ └── Other.Lib.B.mix
└── tix
├── all
│ └── all.tix
├── my-library-0.1.0.0
│ ├── my-library-0.1.0.0.tix
│ ├── my-test-1
│ │ └── my-test-1.tix
│ └── unit-test
│ └── unit-test.tix
└── another-library-0.1.0.0
├── another-library-0.1.0.0.tix
├── my-test-2
│ └── my-test-2.tix
└── unit-test
└── unit-test.tix
All of the coverage information is copied verbatim from the coverage reports for each of the constituent packages. A few additions are made:
tix/all/all.tixis generated from the union of all the library tix files.- We use this file when generating coverage reports for "coveralls.io".
- An index page (
html/index.html) is generated which links to the HTML coverage reports of the constituent packages. - A synthetic HTML report is generated from the
tix/all/all.tixfile. This shows the union of all the coverage information generated by each constituent coverage report.