ouroboros-consensus-0.1.0.0: Consensus layer for the Ouroboros blockchain protocol
Safe HaskellNone
LanguageHaskell2010

Ouroboros.Consensus.Storage.VolatileDB.Impl

Description

Volatile on-disk database of blocks

Logic

The VolatileDB is a key-value store of blocks indexed by their hashes. It is parameterised by the block type blk.

The "volatile" in the name refers to the fact that the blocks stored in it make up the volatile part of the chain, i.e., the last k blocks of the chain, which can still be rolled back. Not only the last k blocks of the current chain are stored in this database, but also blocks of forks which we have switched from or will switch to.

The VolatileDB appends new blocks sequentially to a file. When volMaxBlocksPerFile are stored in the current file, a new file is started.

The VolatileDB provides four main operations:

  1. Adding blocks with putBlock
  2. Get blocks or information about them with getBlockComponent
  3. Accessing the in-memory indices using getBlockInfo and filterByPredecessor
  4. Garbage collecting blocks older than a given slot using garbageCollect

Garbage collection will only delete a file from the VolatileDB when all blocks in it have a slot older than the one passed to garbageCollect.

Errors

When an exception occurs while modifying the VolatileDB, we close the database as a safety measure, e.g., in case a file could not be written to disk, as we can no longer make sure the in-memory indices match what's stored on the file system. When reopening, we validate the blocks stored in the file system and reconstruct the in-memory indices.

NOTE: this means that when a thread modifying the VolatileDB is killed, the database will be closed. This is an intentional choice to simplify things.

The in-memory indices can always be reconstructed from the file system. This is important, as we must be resilient against unexpected shutdowns, power losses, etc.

We achieve this by only performing basic operations on the VolatileDB: * putBlock only appends a new block to a file. Losing an update means we only lose a block, which is not a problem, it can be redownloaded. * garbageCollect only deletes entire files. * There is no operation that modifies a file in-place. This means we do not have to keep any rollback journals to make sure we are safe in case of unexpected shutdowns.

We only throw VolatileDBError. File-system errors are caught, wrapped in a VolatileDBError, and rethrown. We make sure that all calls to HasFS functions are properly wrapped. This wrapping is automatically done when inside the scope of modifyOpenState and withOpenState. Otherwise, we use wrapFsError.

Concurrency

A single folder should only be used by a single VolatileDB. Naturally, a VolatileDB can be accessed concurrently by multiple threads.

File-system layout:

The on-disk representation is as follows:

dbFolder/
  blocks-0.dat
  blocks-1.dat
  ...

Files not fitting the naming scheme are ignored. The numbering of these files does not correlate to the blocks stored in them.

Each file stores a fixed number of blocks, specified by volMaxBlocksPerFile. When opening the VolatileDB, it will start appending to the file with the highest number that is not yet full. If all are full or none exist, a new file will be created.

There is an implicit ordering of block files, which is NOT alpharithmetic. For example, blocks-20.dat < blocks-100.dat.

Recovery

The VolatileDB will always try to recover to a consistent state even if this means deleting all of its contents. In order to achieve this, it truncates the files containing blocks if some blocks fail to parse, are invalid, or are duplicated.

Synopsis

Opening the database

type VolatileDbSerialiseConstraints blk = (EncodeDisk blk blk, DecodeDisk blk (ByteString → blk), DecodeDiskDep (NestedCtxt Header) blk, HasNestedContent Header blk, HasBinaryBlockInfo blk) Source #

EncodeDisk and DecodeDisk constraints needed for the VolatileDB.

defaultArgsApplicative m ⇒ SomeHasFS m → VolatileDbArgs Defaults m blk Source #

Default arguments

openDB ∷ ∀ m blk ans. (HasCallStack, IOLike m, HasHeader blk, GetPrevHash blk, VolatileDbSerialiseConstraints blk) ⇒ VolatileDbArgs Identity m blk → (∀ st. WithTempRegistry st m (VolatileDB m blk, st) → ans) → ans Source #

Re-exported

data BlockValidationPolicy Source #

When block validation is enabled, the parser checks for each block a number of properties and stops parsing if it finds any invalid blocks.

Constructors

NoValidation 
ValidateAll 

data BlocksPerFile Source #

The maximum number of blocks to store per file.

Instances

Instances details
Show BlocksPerFile Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Generic BlocksPerFile Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Associated Types

type Rep BlocksPerFileTypeType Source #

type Rep BlocksPerFile Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

type Rep BlocksPerFile = D1 ('MetaData "BlocksPerFile" "Ouroboros.Consensus.Storage.VolatileDB.Impl.Types" "ouroboros-consensus-0.1.0.0-inplace" 'True) (C1 ('MetaCons "BlocksPerFile" 'PrefixI 'True) (S1 ('MetaSel ('Just "unBlocksPerFile") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 Word32)))

data ParseError blk Source #

Note that we recover from the error, and thus never throw it as an Exception.

Defined here instead of in the Parser module because TraceEvent depends on it.

Constructors

BlockReadErr ReadIncrementalErr

A block could not be parsed.

BlockCorruptedErr (HeaderHash blk)

A block was corrupted, e.g., checking its signature and/or hash failed.

DuplicatedBlock (HeaderHash blk) FsPath FsPath

A block with the same hash occurred twice in the VolatileDB files.

We include the file in which it occurred first and the file in which it occured the second time. The two files can be the same.

Instances

Instances details
StandardHash blk ⇒ Eq (ParseError blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Methods

(==)ParseError blk → ParseError blk → Bool Source #

(/=)ParseError blk → ParseError blk → Bool Source #

StandardHash blk ⇒ Show (ParseError blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Methods

showsPrecIntParseError blk → ShowS Source #

showParseError blk → String Source #

showList ∷ [ParseError blk] → ShowS Source #

data TraceEvent blk Source #

Instances

Instances details
StandardHash blk ⇒ Eq (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Methods

(==)TraceEvent blk → TraceEvent blk → Bool Source #

(/=)TraceEvent blk → TraceEvent blk → Bool Source #

StandardHash blk ⇒ Show (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Methods

showsPrecIntTraceEvent blk → ShowS Source #

showTraceEvent blk → String Source #

showList ∷ [TraceEvent blk] → ShowS Source #

Generic (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

Associated Types

type Rep (TraceEvent blk) ∷ TypeType Source #

Methods

fromTraceEvent blk → Rep (TraceEvent blk) x Source #

toRep (TraceEvent blk) x → TraceEvent blk Source #

type Rep (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.VolatileDB.Impl.Types

type Rep (TraceEvent blk) = D1 ('MetaData "TraceEvent" "Ouroboros.Consensus.Storage.VolatileDB.Impl.Types" "ouroboros-consensus-0.1.0.0-inplace" 'False) ((C1 ('MetaCons "DBAlreadyClosed" 'PrefixI 'False) (U1TypeType) :+: C1 ('MetaCons "BlockAlreadyHere" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (HeaderHash blk)))) :+: (C1 ('MetaCons "Truncate" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (ParseError blk)) :*: (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 FsPath) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 BlockOffset))) :+: C1 ('MetaCons "InvalidFileNames" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 [FsPath]))))

mkBlocksPerFileWord32BlocksPerFile Source #

Create a BlocksPerFile.

PRECONDITION: the given number must be greater than 0, if not, this function will throw an error.