32 posts tagged with "consensus"

High level summary​

We have a proposed fix for the mempool forging regression observed in the UTxO-HD branch. We need to confirm this by running system level benchmarks. We are still working on a fall back mechanism for keeping the baseline performance of Cardano node, if the performance of the UTxO-HD is not enough. On the Genesis front, we confirmed with the researchers that the proposed Genesis design is satisfactory for the historical Cardano chain. We also have a proposed fix for the wrong protocol version bug, found in the Sanchonet, after transitioning to Conway.

UTxO-HD​

• We optimized the mempool revalidation process, which in turn ought to solve the regression observed during system-level benchmarks in the in-memory version (349). System level benchmark results are pending.
• Regarding the workaround to keep the node's baseline performance if that of the in-memory backend turns out not to be enough for our stakeholders (344), we are still expanding the legacy block package such that we could at some point run the node with a legacy Cardano block. There are some loose ends to wrap up before we can begin the first test run.
• We also brought the UTxO-HD branch up to date with node version 8.4.0.

Genesis​

• We finished the discussion with the Researchers on how to argue that the proposed Genesis design is satisfactory for the existing historical Cardano chain. We are now drafting the final self-contained argument. (4157)

Support​

• We debugged a bad parameter update on the Babbage to Conway transition in the SanchoNet testnet (339). A superficial patch is within reach and we are in the process of reviewing the PRs related to this fix (340, 354, and 355) However we are investigating a more principled redesign of the epoch transition logic, which required us to revisit the existing interfaces of the ConsensusProtocol type class and the HardForkBlock combinator (345 and 346). This is important to prevent these kind of errors in the future. This is an overdue step in the process of taking full ownership of the HFC: reconsidering original HFC design decisions for which we now have much more context, a few years later.

High level summary​

We were able to successfully run the system-level benchmarks for the UTxO-HD implementation, for the first time. There was an important regression in block forging performance that will have to be addressed before UTxO-HD is released. We also revisited the implementation of our query processing logic, which was needed to address the performance regression found in the query-by-address command. The preliminary performance results show that now the performance of this query is on-par with the Cardano baseline version, but we need further confirmation. On the Genesis front, we presented the grinding-aware safety argument for the proposed historical Cardano Genesis windows to the IOG Researchers. The Consensus release engineer finished his rotation: version 8.3.0-pre of cardano-node is releasing 2023 September 5.

UTxO-HD​

• We ran the first successful system-level benchmarks for UTxO-HD (see #203) using the in-memory backend.
  • We observed a factor 12 regression in the forging performance, which we will have to address. There are strong indications that the regression is due to the backing store accesses that take place when taking a mempool snapshot.
  • After the mempool regression is fixed the benchmarks need to be ran again.
  • System-level UTxO-HD benchmarks with the LMDB are still pending.
• UTxO-HD will eventually be necessary due to the growth of the UTxO set and other ledger state structures that live in memory at the moment. However, we are trying a strategy by which we could preserve the baseline performance of the node, in case SPOs and other node users are not ready to migrate yet (see #344).
• We implemented a new way of processing queries at the hard-fork block level, which resolves the performance regression observed in GetUTxOByAddress (see this comment). Preliminary results are promising.
• Regarding the roll out plan, UTxO-HD requires a significant change in the Consensus codebase. Even though we might be able to hide any potential performance impact in the node by keeping all data in memory (#344), the Consensus component was significantly changed, so we might have to postpone releasing this feature to mitigate any risks of conflicting with the implementation of CIP-1694 and release of Conway.

Tech debt​

• We added tests that Consensus emits valid CBOR (#3099). This helped us detect a couple of serialization bugs. The tests still need to be merged into the main branch (#323).

Support​

• Nick Frisby finished his release engineer rotation; cardano-node 8.3.0-pre is releasing 2023 September 5.
• We helped to investigate a protocol version bug in Sanchonet (see #3491).
• We started to implement the Network interface for bootstrap peer functionality, from which Genesis will benefit as well (see #91.

High level summary​

On the UTxO-HD front we reached another important milestone, where we handed over the implementation for benchmarking and testing. We are working on addressing the first issues that were found during the integration tests. On the Genesis front we are finalizing the argument that a checkpoint is not necessary for the initial release, including supporting to support said argument. We are also working on the new node release, and several improvements to our code-base.

UTxO-HD​

• We addressed the issues that came from the benchmarking and testing phase (see #314).
  • We fixed the operational-certificate update exception (#305).

Tech debt​

• We elaborated an inventory of the symbols used by downstream clients in an effort to clearly define the Consensus API, which will help managing changes and their impact to the downstream clients (see this branch).
• We extracted the SOP modules to a separate package as part of our effort to reduce the Consensus API surface area (see this branch).
• We took the first step towards improving how the Cardano hard-fork block is instantiated when running a node (see #275).
• We decreased the flakiness for our test suite (#284).
• We incorporated and merged the pull-request on query versioning (#273).
• We started working on new performance-based regression tests for mempool operations.

Support​

• Nick Frisby continues with his engineer responsibilities. Notably getting the Consensus release out for node pre-release 8.3. Nick also began integrating it into downstream branches (cardano{-api,-cli,-node}). The Node team has now taken over, since Nick had progressed up to the point where they needed to start making new design decisions. Related PRs:
  • #283
  • #301
  • #304
  • #308

High level summary​

This week the Consensus team worked on integrating the latest node release into the UTxO HD branch. Our ad-hoc benchmarks in which we sync a node from scratch showed that the LMDB backend uses around 8GB of memory. We also released new versions of fs-sim and fs-api, and released strict-checked-vars.

UTxO-HD​

• Rebased UTxO-HD on top of node 8.2 (issue)
• Address new Maybe translations for UTxOs in Conway 1.6. (PR).
• Performed ad-hoc benchmarking syncs with both implementations. The LMDB backend uses around 8GB of memory.
• Fixed an infinite loop on the mempool tracing code that was preventing the node from shutting down gracefully.
• Fixed an infinite loop on the ledger state query for traversing queries that prevented the node from running QueryUTxOByAddress (PR).
• Measured the speed of QueryUTxOByAddress under different queryBatchSize values. There is a significant performance degradation for these queries when using the UTxO-HD version. Using the mainnet UTxO set at slot 90 million, querying an address took 40 seconds using the in-memory backend and 90 seconds using the LMDB backend, whereas the baseline took 7 seconds. We need to investigate if we can improve this situation or whether an external service that runs alongside the node is a better solution.
• Refactored and implemented ledgertable-related classes for the general HardforkBlock.

Genesis​

• Frisby and Esgen continue to engage with the Researchers on grinding against the Genesis design.

Fostering collaboration​

• Drafted a document explaining versioning of local state queries #273.

Support​

• Frisby is the release engineer this cycle.
• Successfully created work-in-progress ouroboros-consensus and cardano-api commits that integrate the 2023 Aug 7 tip of cardano-ledger. This will require a release of ouroboros-consensus, which hasn't happened yet.

High level summary​

The db-sync team confirmed that the adaptor module the Consensus team provided for UTxO-HD integration, shows no performance degradation. Other clients who wish to integrate with UTxO-HD without using a on-disk storage could use that same approach for now. We also focused on completing tracing support for UTxO-HD in cardano-node. Currently we are working on UTxO-HD documentation, both for downstream consumers and for the general public; and generalizing the implementation further.

The team working on Genesis decided on a Genesis-motivated change of the epoch structure with the researchers, wrote a self-contained description of the statistical model for historical Genesis windows and eclipse durations, and merged two small improvements to the ChainSync client as part of the onboarding efforts for the team that will implement Genesis.

The Consensus team also investigated an issue on sanchonet (the Conway testnet) manifesting in long syncing pauses; identified a ledger bug which was promptly fixed by the Ledger team.

Genesis liaising​

Improvements to the ChainSync client:

• #222.
• #233.