Skip to main content

3 posts tagged with "consensus"

View All Tags

· 3 min read
Damian Nadales

Consensus Quarterly Update​

2023-01 - 2023-03​

Main achievements​

UTxO HD​

  • We finished the testing activities for the prototype, which involved adding new tests, and fixing and enabling temporarily disabled tests.
  • We spent a substantial amount of effort refactoring and cleaning the prototype.
  • We audited the UTxO HD prototype to make sure it can accommodate the migration of other tables (eg stake-keys registration) from memory to disk. The result of the audit was positive.
  • We ran ad-hoc benchmarks for reading keys and flushing values to disk. No unexpected costs found.
  • We ran the first system level benchmarks. The performance regressions reported were due to an unrealistic snapshotting rate. We need to re-run them again after we design a more fine grained locking mechanism.

Genesis​

  • We elaborated a roadmap of the remaining work for Genesis.
  • We presented the design to the IOG Researchers and PNSol on February 20. The design was well received. We updated the Genesis design with the researcher's feedback.
    • We plugged the new DoS vector identified during the aforementioned presentation.
  • We developed a generator for adversarial leader schedules that satisfy key Ouroboros properties, which will be used to test the Genesis design.
    • The generator enables use of smaller Ouroboros parameters, which makes extrema more likely and counterxamples easier interpret.
  • We wrote up the latest design iteration.
  • We continued benchmarking the Chain Sync Jumping prototype. In particular:
    • We debugged the prototype's performance regression, and unmasked the actual cause by patching our initial theory (bad queuing behavior)
    • We identified and validated the actual cause (a pathological case in BlockFetch tiebreaker).

Support​

  • We created two new tools. One for dumping CBOR encoded blocks to JSON. Ahother to serve a local immutable DB.

Conway era​

  • We integrated the Conway era into consensus.

Technical debt​

  • We fixed a bug with followers, which was discovered by property tests.
  • We developed a DSL for specifying and running ChainDB test cases.
  • We fixed failing tests with iterators.
  • We created micro-benchmarks for adding transactions to the mempool.

Fostering collaboration​

  • We released a new technical documentation site for consensus.
  • We factored out several packages to external repositories. Some of this work originated in the UTxO HD workstream.

Next steps​

UTxO HD​

Genesis​

Support​

  • Design Consensus side of hardfork-enactment in the Voltaire phase (#4180).
  • Estimate the number of file descriptors Consensus needs #20.

Tech debt​

  • Identify Quantitative Timeliness Agreements (QTAs) metrics that we can define for consensus. Pick one and implement benchmarks for it.

Fostering collaboration​

  • Onboard a new team member.

· 4 min read
Damian Nadales

Consensus Quarterly Update​

2022-12 - 2023-01​

Main achievements​

UTxO HD​

The prototype is feature complete and thoroughly tested at the consensus level. In particular, we invested a lot of time in writing property-test for the mempool, and other crucial new parts of the prototype. Now we are ready to run integration tests and system-level benchmarks.

Genesis​

We identified and fixed a slowdown in cross-era forecasting that was inhibiting our efforts to benchmark the ChainSync Jumping prototype. This resulted in a 7% speedup in full sync times in the baseline.

We also started prototyping a self-contained implementation of the Genesis dynamics (in particular of the parts intentionally not part of the ChainSync Jumping prototype) that furthered our understanding of subtleties and edge cases.

Support​

  • We worked on designing integration of new VRF and KES crypto into consensus.
    • Crypto class was split into two parts: Crypto and HeaderCrypto.
    • With the Ledger team's help, we refactored cardano-ledger to use a proxy type for VRF.

Conway era​

  • PR went through its second review round. It is about to be merged, but it got delayed due to people's availability during Christmas break.

Technical debt​

  • We improved the capabilities of our io-sim library, which is key for testing and simulating Cardano components.
  • We removed thunks from epoch translations in the ledger, which is important for reducing memory consumption of the Cardano node.

Fostering collaboration​

  • We added a tutorial on how to instantiate the Consensus layer to run custom ledgers. This should be a valuable resource to people looking to roll their own custom blockchain (either for commercial or research purposes).
  • We added an overview of consensus to the top level documentation of ouroboros-network. This overview describes the consensus components and adds a hyperlinked map to the modules documentation.

Next steps​

UTxO HD​

  • Evaluate the extensibility of the prototype. Moving the UTxO to disk is only the first step towards reducing the memory requirements of Cardano node, and ensuring its long term sustainability. In the future, we plan on moving other large maps, such as delegation maps. The prototype should be able to accommodate these changes without any major modifications.
  • Start the integration with other downstream components, such as the wallet and db-sync. The idea is to identify and address any potential pain points that might arise during this integration.
  • Run integration tests and system-level benchmarks.

Genesis​

  • Finish benchmarking and tuning the fast-path ChainSync Jumping prototype
  • Expand and optimize the self-contained implementation of the Disconnect Rule (including density comparisons and the LoE)
  • Develop documentation and smoke tests for these components.
  • Start modifying the ChainSync Client for the LoP and LoR.

Support​

  • Help the Network team with diagnosing performance regression in block production.

Tech debt​

  • Fix property-test failures concerning iterators (#3999 and #4183).

Fostering collaboration​

Risks​

UTxO HD​

  • Moving other parts of the ledger state to disk might require a major redesign of the prototype. For instance, if it turns out that the epoch change rules require access to the full ledger state. If this is the case, we might accept this risk and do the redesign after the initial release of UTxO-HD.
  • Integration with downstream clients might require more work than we anticipate.
  • Access to the benchmarking's team time and resources.
  • Benchmarking results might show significant performance degradation, which will require additional work if such performance degradation is not accepted by other stakeholders.
  • The prototype's performance might not be accepted by other stakeholders. Here we need to clearly communicate that this is necessary to ensure that as the blockchain size grows, the node can operate within reasonable memory constraints.

· 4 min read
Damian Nadales

Consensus Quarterly Update​

2022-09 - 2022-11​

Main achievements​

UTxO HD​

  • As a consequence of the errors observed when running distributed mempool benchmarks, we re-designed the UTxO HD mempool integration, which fixed these errors and lead to a simpler and more maintainable design.

  • We focused on increasing test coverage for the UTxO-HD prototype. In particular, we added property tests for:

    • Backing store (work ongoing)
    • Era transitions

  • The property tests we added uncovered several bugs, which is a great result given the exponential increase in the cost of finding bugs as they are closer to deployment.

  • One of the errors found by our tests required us to work on improvements in the Haskell bindings for LMDB. This work is ongoing.

  • We started working on the mempool property tests that will exercise the new code paths that UTxO HD introduced.

  • We developed, benchmarked and tested an implementation of sequences of differences based on "anti-diffs". Performance results of diff sequence operations show that we achieved a speedup of about 4x across several scenarios. Note: this speedup is taking into account diff sequence operations only, so the consensus-wide speedup is less than 4x.

  • We integrated the "anti-diff" prototype into the UTxO HD feature branch.

Genesis​

  • We wrote a simulator that demonstrates soundness of an abstract implementation of the new chain selection rule.
  • We elaborated a draft specification for the Genesis implementation (currently awaiting feedback from other architects).
  • We elaborated a draft specification for the ChainSync Jumping optimization. In particular, this includes a proof sketch that the latter preserves liveness and safety in all cases.
  • With the Networking team, we co-designed the eclipse avoidance mechanism, specifically its coherence with the Genesis implementation plan's security and its dependence on the new ChainSync Jumping optimization.
  • We implemented a prototype for ChainSync Jumping. Initial benchmarks showed a performance degradation wrt the baseline. Our optimization attempts so far have brought the performance closer to the baseline, but not yet to parity.

Conway era​

  • We did most of the heavy lifting required to integrate the Conway era into the Consensus layer.

Technical debt​

  • We started working on enabling CI nightly tests, which revealed several test failures due to thunks being found it data structures used by the ledger and consensus. We made a lot of progress fixing those thunk errors, but some errors still remain.

  • We elaborated a db-analyser benchmark for the ledger operations. This led us to the identification of high processing time at epoch boundaries, and we could not observe any performance degradation that can be attributed to era changes.

  • We fixed a source of flakiness in the ChainDB QSM test.

  • We clarified a common source of confusion around VRF tie-breaking and cross-era chain selection.

  • We fixed a bug in the maximum-allowed ledger major protocol version.

Fostering collaboration​

  • We spent time making cardano-updates the central source of information for the core teams stakeholders.
  • We went through the Galois gap analysis and extracted actionable points to take on next.
  • Bart and Yogesh continued with their onboarding and stated making substantial contributions to consensus.

Next steps​

UTxO HD​

  • Finish the mempool property tests.
  • Benchmark the latest version of the prototype.
  • Elaborate a document that describes new integration test scenarios and pass it to the SDET team.
  • Bring query UTxO by address command performance on par with the baseline version.

Genesis​

  • Receive and incorporate Duncan's feedback on the first draft specification for the Genesis implementation.
  • Begin prototyping the first genesis implementation, unless the first draft needs major changes.
  • Draft a second revision of the Genesis report.
  • Review the second revision with a wider audience, which includes at least Alexander Russell. That feedback will drive a third and hopefully final revision.
  • Investigate how to mitigate the ~30% slowdown we have observed so far in the ChainSync jumping prototype, and try to mitigate it. In particular, we might need to optimize the existing BlockFetch logic.

Tech debt​

  • Enabling nightly CI tests.

Fostering collaboration​

  • Merge the tutorial document Galois wrote; requires CI integration.
  • Come up with our own documentation improvements, many of which were suggested in the Galois gap analysis.
  • Try to hire a new team member.