Skip to main content

19 posts tagged with "consensus"

View All Tags

· One min read
Damian Nadales

High level summary​

This week the consensus team continued working on the refactoring of the UTxO HD prototype, and design and testing of Genesis. We also extracted the fs-sim package, which provides a file-system abstraction layer that can be used for testing and simulation. This makes the Consensus code base smaller, while providing a package that the community can reuse and contribute to. We also fixed a failing property test related to iterators. We are also working on mempool and VRF improvements.

Low-level details​

· 3 min read
Damian Nadales

High level summary​

The Consensus team continued working on refactoring and improving the UTxO-HD prototype, and introducing improvements to the lmdb related packages. In particular we identified an opportunity to gain performance by handling locks in a more optimal way.

On the Genesis front, we sketched a mitigation for an issue that PNSol and Researchers caught. We also came op with a road map for not only testing the Genesis prototypes, but also for enriching the tests we already have.

Regarding technical debt, next to some minor improvements, we created component-level micro-benchmarks for adding transactions to the mempool. The results of these benchmarks will be published in the ouroboros-consensus web page.

We also finished moving the Consensus documentation to the ouroboros-consensus repository, released ouroboros-consensus 0.3.0.0, and reduced the time GitHub actions take in ouroboros-network.

Workstreams​

UTxO HD Prototype​

We continued working on refactoring and improving the UTxO-HD prototype. As a result of the first round of sytem-level benchmarks, we identified an opportunity to optimise the way we handle locks to improve performance (#4393).

Also, we introduced several improvements to the lmdb related packages:

Genesis​

We sketched out a mitigation of the issue that PNSol and Researchers caught in the Genesis design.

We came up with a road map for testing the Genesis prototypes, including early milestones that are applicable to today's master branch, ie tests that are useful before Genesis, and that will be nicely enriched when we do add Genesis.

We developed the aforementioned tests, specifically a QuickCheck generator for the Honest leader schedule and one as-aggressive-as-possible Adversarial leader schedule that together satisfy the Praos properties that the Consensus design takes as invariants.

We investigated why the improved ChainDB queueing implementation behaves differently in the baseline compared to the prototype, and we are close to having a full picture of how the Consensus components interact during bulk sync.

Technical debt​

We created component-level micro-benchmarks for adding transactions to the mempool. We plan on extending this to more mempool actions and different types of blocks. We store the benchmark data to make it available to the GitHub action that publishes the benchmarks results.

Other minor improvements include:

  • Removal of Test.Util.Classify in favour of Test.StateMachine.Labelling.
  • Addition of -Wunused-packages to the default ghc-options for Consensus packages.

Fostering collaboration​

We finished moving the Consensus documentation from ouroboros-network to ouroboros-consensus, in preparation for migrating the code to the latter repository.

Support​

We released ouroboros-consensus 0.3.0.0.

We reduced the load in the ouroboros-network GitHub actions, thereby reducing the time CI jobs take.

· 3 min read
Damian Nadales

High level summary​

During the past two weeks we got the results from the system level benchmarks for UTxO HD. They showed a substantial performance regression, so we spent some time analyzing the results. We found out the frequency at which ledger snapshots were taken was too high, so we requested the benchmarking team a new run with a more realistic snapshotting policy. We continued refactoring and improving the prototype, and we released UTxO-HD related packages to CHaP.

We met with IOG researchers and networking specialists to discuss the Genesis design, which was well received. We continued working on testing and benchmarking different Genesis prototypes.

We are also working on solving a test failure related to iterators. This work derived in several improvements such as better documentation, a framework for writing unit (and regression) tests, and the possibility of debugging QuickCheck counter examples in the REPL.

Finally, we released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP

Workstreams​

UTxO HD Prototype​

We got the results of the first system level benchmarks for UTxO HD. They seemed to indicate a significant regression in performance. After looking into the benchmark logs we found that the benchmark runs took ledger state snapshots too often, due to the default snapshotting policy depending on k, and k being so small in the benchmark runs. Therefore, the next step is to re-run the benchmarks with a snapshotting policy that more closely resembles the one from mainnet.

At the same time, we continued refactoring and cleaning up the prototype.

Also, we prepared the anti-diff packages (fingertree-rm, diff-containers, simple-semigroupoids) and the lmdb related packages (cardano-lmdb and cardano-lmdb-simple) to CHaP.

Genesis​

The Genesis design was presented to the IOG researchers and Peter Thompson from NSol. It was well received. They pointed out one blindspot, but we think it'll be relatively simple to mitigate.

In parallel, we continued developing test and benchmarks for the Genesis prototypes. I particular we tested and implemented a potential fix for increased ChainDB dequeue timings, which partly behaved as we expected, but still needs further investigation. Also we obtained new benchmarking data for the prototype.

Technical debt​

Related to #4183, we developed a DSL for specifying ChainDB unit tests. This will allow us to better understand the counter-examples returned by QuickCheck tests, and to write regression tests for them. Also, we added a module to enable QuickCheck counter-examples to be run on the REPL, allowing for faster debugging feedback. Also, we improved the documentation related to followers (#4372).

We are also working on a design for optimizing the way we handle blocks from the future.

Support​

We released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP. Remember that we decided to split the packages related to Consensus into two bundles, one with the core functionality, Cardano-agnostic code, and another bundle with instantiations specific to Cardano.

· 2 min read
Damian Nadales

High level summary​

We continue refactoring the UTxO HD prototype while we wait for the system level benchmarks. We have created a new repository that contains the anti-diff packages used in this prototype.

On the Genesis front, we are preparing another meeting with the researchers to audit the implementation design, and we continued working on basic tests and simplifications.

During the past two weeks we also introduced two new tools. One for dumping CBOR encoded blocks to JSON, and another to serve a local immutable DB.

Workstreams​

UTxO HD Prototype​

We are in the process of refactoring the UTxO HD prototype, while we wait for the system level benchmarks to confirm if the performance of the prototype is satisfactory.

We also set up a repository for the anti-diff package, which required us to refactor the code, write documentation, and prepare a release to CHaP.

Genesis​

We worked on basic tests for the Limit on Eagerness property of Genesis. We also introduced further robustness and simplifications in the Genesis Density governor. Finally, we developed a presentation to engage again with the researchers on our Genesis implementation design.

Technical debt​

Fostering collaboration​

We are in the process of polishing the ouroboros-consensus documentation site, which we will use a the entry point for Consensus related documentation. The first version will not be complete, but we plan on systematically improving it.

Support​

We added a tool to ouroboros-consensus-cardano-tools which allows to dump the Chain DB blocks or any given CBOR encoded blocks as JSON.

We also added another tool that serves an existing immutable DB via BlockFetch and ChainSync. This tool can help in assisting our local benchmarking efforts (for instance Genesis' ChainSync jumping prototype).

· 3 min read
Damian Nadales

High level summary​

During the past two weeks, the consensus team finished the testing activities around the UTxO-HD prototype. This is a very important milestone which will enable us to run system-level tests and benchmarks, as well as start refactoring and cleaning the prototype. Regarding our Genesis workstream, we elaborated a roadmap that gives an indication of the remaining work. We also continued our work on benchmarking chain-sync-jumping. We also continued working on improving the way we handle blocks from the future, and advancing the integration of the new VRF and KES crypto.

Workstreams​

UTxO HD Prototype​

As the prototype is nearing its completion, it was important to have enough confidence that we will be able to move additional parts of the ledger state onto disk. We worked together with the Ledger team to elaborate a sketch on how the UTxO-HD design would accommodate the migration of additional data from memory to disk. This gave us enough confidence that the current architecture will be extensible in the future.

On the testing front, we added property-based tests for the UTxO-HD type classes.

We also enabled disabled components, and addressed several technical debt issues:

  • Implement splitSized anti-diff split (#4269), and integrate it into consensus (#4273).
  • Renaming of peekVal to peekMDBVal (#7).

We ran ad-hoc benchmarks for syncing a chain from scratch and replaying. We found a race condition in the LMDB backing store, which we fixed. After the fix we were able to successfully run these benchmarks. The results were published by this pull request.

We used our db-analyser tool to benchmark the cost of reading keys and flushing values to disk. The following plot shows the duration of these disk operation in relation to the main ledger operations, where we can see that the cost of the former are comparatively low. The spike at the beginning of the graph is when, at the start of the Shelley era, the entire UTxO set is flushed to disk.

UTxO-HD read and flush benchmarks

After months of hard work adding tests for the prototype, we are ready to run end-to-end tests on the node, and system level benchmarks. This signals a very important milestone for the UTxO-HD workstream 🎉.

Genesis​

We elaborated a high-level decomposition of the remaining work for Genesis. We also continued benchmarking the chain-sync-jumping happy-path.

Technical debt​

We continued working on improving the way we handle blocks from the future.

Support​

We completed the mapping of Crypto to HeaderCrypto and body Crypto. HeaderCrypto is moved to cardano-protocol-tpraos. We created a draft pull request to facilitate compiling consensus.