Testing

Testing

Unit Tests

$ cabal test cardano-wallet-core:unit
$ cabal test cardano-wallet:unit

Alternatively, one can run tests of a particular module by running:

$ cabal test cardano-wallet:unit --test-options "--match MyModule"

Integration Tests

Pre-requisites

Install cardano-node and cardano-cli; make sure to use one of the compatible versions.

Alternatively, use cabal test all -j8.

Test

$ cabal test cardano-wallet:integration

Many tests require a cardano network with stake pools. To support this, the integration tests run a local cardano-node cluster with one Ouroboros BFT node and three Ouroboros Praos nodes for the three stake pools.

Environment Variables

Several environment variables control debugging features of the integration tests and test cluster.

VariableTypeMeaningDefault
CARDANO_WALLET_PORTnumberSet a specific port for the wallet HTTP serverRandom unused port
NO_CLEANUPboolLeave the temporary directory after tests have finished.Delete directory on exit
CARDANO_WALLET_TRACING_MIN_SEVERITYseverityLog level for the cardano-wallet server under test.Critical
CARDANO_NODE_TRACING_MIN_SEVERITYseverityLog level for the test cluster nodesInfo
TESTS_TRACING_MIN_SEVERITYseverityLog level for test suite and clusterNotice
TESTS_LOGDIRpathWrite log files in the given directoryLog files are written to the tests temp directory
TESTS_RETRY_FAILEDboolEnable retrying once of failed testsNo retrying
NO_POOLSboolDon’t start any stake pool nodes in the local test cluster.Pool nodes are started
TOKEN_METADATA_SERVERURLUse this URL for querying asset metadataAsset metadata fetching disabled
NO_CACHE_LISTPOOLSboolDo not cache pool listing retrieved from cardano-node. Testing only. Use --no-cache-listpools command line for executable.Stake distribution is cached to improve responsiveness
CACHE_LISTPOOLS_TTLnumberCache time to live (TTL) for pool listing. Testing only. Use --no-cache-listpools command line for executable.6 seconds for test builds

Here are the possible values of different types of environment variables:

TypeValues
boolunset or empty ⇒ false, anything else ⇒ true
severitydebug, info, notice, warning, error, critical

Logging and debugging

If your test has failed, viewing the logs often helps. They are written to file in the integration tests temporary directory.

To inspect this directory after the tests have finished, set the NO_CLEANUP variable.

Here is an example tree
/tmp/test-8b0f3d88b6698b51
├── bft
│   ├── cardano-node.log
│   ├── db
│   ├── genesis.json
│   ├── node.config
│   ├── node-kes.skey
│   ├── node.opcert
│   ├── node.socket
│   ├── node.topology
│   └── node-vrf.skey
├── pool-0
│   ├── cardano-node.log
│   ├── db
│   ├── dlg.cert
│   ├── faucet.prv
│   ├── genesis.json
│   ├── kes.prv
│   ├── kes.pub
│   ├── metadata.json
│   ├── node.config
│   ├── node.socket
│   ├── node.topology
│   ├── op.cert
│   ├── op.count
│   ├── op.prv
│   ├── op.pub
│   ├── pool.cert
│   ├── sink.prv
│   ├── sink.pub
│   ├── stake.cert
│   ├── stake.prv
│   ├── stake.pub
│   ├── tx.raw
│   ├── tx.signed
│   ├── vrf.prv
│   └── vrf.pub
├── pool-1
│   ├── cardano-node.log
│   ├── db
│   ├── dlg.cert
│   ├── faucet.prv
│   ├── genesis.json
│   ├── kes.prv
│   ├── kes.pub
│   ├── metadata.json
│   ├── node.config
│   ├── node.socket
│   ├── node.topology
│   ├── op.cert
│   ├── op.count
│   ├── op.prv
│   ├── op.pub
│   ├── pool.cert
│   ├── sink.prv
│   ├── sink.pub
│   ├── stake.cert
│   ├── stake.prv
│   ├── stake.pub
│   ├── tx.raw
│   ├── tx.signed
│   ├── vrf.prv
│   └── vrf.pub
├── pool-2
│   ├── cardano-node.log
│   ├── db
│   ├── dlg.cert
│   ├── faucet.prv
│   ├── genesis.json
│   ├── kes.prv
│   ├── kes.pub
│   ├── metadata.json
│   ├── node.config
│   ├── node.socket
│   ├── node.topology
│   ├── op.cert
│   ├── op.count
│   ├── op.prv
│   ├── op.pub
│   ├── pool.cert
│   ├── sink.prv
│   ├── sink.pub
│   ├── stake.cert
│   ├── stake.prv
│   ├── stake.pub
│   ├── tx.raw
│   ├── tx.signed
│   ├── vrf.prv
│   └── vrf.pub
├── wallets-b33cfce13ce1ac74
│   └── stake-pools.sqlite
├── cluster.log
└── wallet.log

The default log level for log files is Info.

Only Error level logs are shown on stdout during test execution. To change this, set the *_MIN_SEVERITY variables shown above.

Common Failures and Resolution

No More Wallets

If your test fails with something like:

user error (No more faucet wallet available in MVar!)

Generate more wallet mnemonics and populate the appropriate list in lib/core-integration/src/Test/Integration/Faucet.hs.

Generate new mnemonics with:

nix build .#cardano-wallet
# Size may vary depending on which array you need to add to.
./result/bin/cardano-wallet recovery-phrase generate --size 15

Mock servers

Use the cardano-wallet:mock-token-metadata-server executable as a mock server for asset metadata. See the --help output for full instructions.

Benchmarks

Database

$ cabal bench cardano-wallet-core:db

Restoration

Pre-requisites

  • Follow the pre-requisites from integration above

  • (Optional) Install hp2pretty

    $ cabal install hp2pretty

Test

⚠️ Disclaimer ⚠️

Restoration benchmarks will catch up with the chain before running which can take quite a long time in the case of mainnet. For a better experience, make sure your system isn’t too far behind the tip before running.

$ cabal bench cardano-wallet:restore

Alternatively, one can specify a target network (by default, benchmarks run on testnet):

$ cabal bench cardano-wallet:restore --benchmark-options "mainnet"

Also, it’s interesting to look at heap consumption during the running of the benchmark:

$ cabal bench cardano-wallet:restore --benchmark-options "mainnet +RTS -h -RTS"
$ hp2pretty restore.hp
$ eog restore.svg

Code Coverage

Pre-requisites

  • Follow the pre-requisites from integration above

Test

Running combined code coverage on all components is pretty easy. This generates code coverage reports in an HTML format as well as a short summary in the console. Note that, because code has to be compiled in a particular way to be “instrumentable” by the code coverage engine, it is recommended to run this command using another working directory (--builddir option) so that one can easily switch between coverage testing and standard testing (faster to run):

$ cabal test all --enable-coverage --builddir .dist-coverage

Note that, integration tests are excluded from the basic coverage report because the cardano-wallet server runs in a separate process. It it still possible to combine coverage from various sources (see this article for some examples / details).

E2E Tests

See: README.

QA Schedule

PR PushBors MergeNightlyPre-releaseHydraBuildkiteGitHubRequired forStatus
Unit testsLMwLMWLMwLWMerge
Integation testsLMWLMwLWMerge
E2E testsLDMWLDMWMergeE2E Linux
E2E Docker
E2E MacOS
E2E Windows
E2E Linux –light
E2E MacOS –light
E2E Windows –light
Code lintingXXMerge
Code reviewXXMerge
DB Migration Path tests
Release file contents testsLMWLMWLMWRelease
Manual testsLMWRelease
Wiki docs checksXRelease
Database benchmarkLL
API Latency benchmarkLL
Restore benchmarkLL
Docker image pushL–XLMerge
ChangeLog reviewXRelease
KeyMeaning
LRun for Linux platform
DRun under Docker for Linux platform
MRun fox macOS platform
WRun for Windows platform
wRun under Wine on Linux
-N/A
XTest is executed

Handy things to know

  • Normal PR builds do not run integration tests. This is to save CI resources and time.

  • Bors job builds also execute integration tests. This can be the reason why your PR is green, but Bors still fails.

  • Retrying a build in Buildkite or Hydra will have absolutely no effect if the Bors job has already failed.

Links to this page