ouroboros-consensus-0.1.0.0: Consensus layer for the Ouroboros blockchain protocol
Safe HaskellNone
LanguageHaskell2010

Ouroboros.Consensus.Storage.ImmutableDB.Impl

Description

Immutable on-disk database of binary blobs

Internal format

The API of the ImmutableDB uses SlotNo to indicate a location in the chain/immutable database. To distinguish EBBs from regular blocks, the hash is used (together they form a RealPoint). The contents of the database are not stored in one big file that is appended to in eternity, but a separate file is created for each ChunkNo.

Within each ChunkNo, the entries are numbered by RelativeSlots. Each SlotNo can be converted to a combination of an ChunkNo and a RelativeSlot (= ChunkSlot) and vice versa. This conversion depends on the size of the chunks: ChunkSize. This size may not be the same for each chunk. When opening the database, the user must give a ChunkInfo that will be used to find out the size of each chunk.

For example:

Chunks:         <──────── 0 ────────> <────── 1 ──────>
chunk size:               4                   3
                ┌───┬───┬───┬───┬───┐ ┌───┬───┬───┬───┐
                │   │   │   │   │   │ │   │   │   │   │
                └───┴───┴───┴───┴───┘ └───┴───┴───┴───┘
'RelativeSlot':   0   1   2   3   4     0   1   2   3
'SlotNo':        EBB  0   1   2   3    EBB  4   5   6

Not all chunks can contain EBBs; see ChunkInfo for details.

Errors

Whenever an UnexpectedFailure is thrown during an operation, e.g., appendBlock, the database will be automatically closed because we can not guarantee a consistent state in the face of file system errors.

Opening the database

The database can be closed and opened again. In case the database was closed because of an unexpected error. When the database is opened again, invalid data will be truncated from the database until a valid prefix is recovered.

Concurrency

The same database should not be opened multiple times concurrently. This is ensured by the file lock of the ChainDB.

The database can have multiple readers, but should only have one writer.

Layout on disk

The database is structured on disk as follows:

/
  00000.chunk
  00000.primary
  00000.secondary
  ..
  00008.chunk
  00008.primary
  00008.secondary

For each chunk, there are three files on disk:

  • A "chunk file" that stores the actual blocks. But nothing more, so nothing is stored for empty slots.
  • A "secondary index file" that stores information about each block: its hash, the slot number or epoch number in case of an EBB, a checksum of the block, the offset of the block in the chunk file, and more. This index is sparse to save space.
  • A "primary index file" that maps slots to offsets in the secondary index file.
Synopsis

Opening the databse

type ImmutableDbSerialiseConstraints blk = (EncodeDisk blk blk, DecodeDisk blk (ByteString → blk), DecodeDiskDep (NestedCtxt Header) blk, ReconstructNestedCtxt Header blk, HasBinaryBlockInfo blk) Source #

EncodeDisk and DecodeDisk constraints needed for the ImmutableDB.

defaultArgsApplicative m ⇒ SomeHasFS m → ImmutableDbArgs Defaults m blk Source #

Default arguments

openDB ∷ ∀ m blk ans. (IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack) ⇒ ImmutableDbArgs Identity m blk → (∀ st. WithTempRegistry st m (ImmutableDB m blk, st) → ans) → ans Source #

Re-exported

data ChunkFileError blk Source #

Defined here instead of in the Parser module because TraceEvent depends on it.

Constructors

ChunkErrRead ReadIncrementalErr

A block could not be decoded

ChunkErrHashMismatch

The previous hash of a block did not match the hash of the previous block.

Fields

ChunkErrCorrupt (Point blk)

The integrity verification of the block with the given point returned False, indicating that the block got corrupted.

Instances

Instances details
StandardHash blk ⇒ Eq (ChunkFileError blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

StandardHash blk ⇒ Show (ChunkFileError blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

data CacheConfig Source #

Constructors

CacheConfig 

Fields

data TraceChunkValidation blk validateTo Source #

Instances

Instances details
Functor (TraceChunkValidation blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Methods

fmap ∷ (a → b) → TraceChunkValidation blk a → TraceChunkValidation blk b Source #

(<$) ∷ a → TraceChunkValidation blk b → TraceChunkValidation blk a Source #

(StandardHash blk, Eq validateTo) ⇒ Eq (TraceChunkValidation blk validateTo) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Methods

(==)TraceChunkValidation blk validateTo → TraceChunkValidation blk validateTo → Bool Source #

(/=)TraceChunkValidation blk validateTo → TraceChunkValidation blk validateTo → Bool Source #

(StandardHash blk, Show validateTo) ⇒ Show (TraceChunkValidation blk validateTo) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Methods

showsPrecIntTraceChunkValidation blk validateTo → ShowS Source #

showTraceChunkValidation blk validateTo → String Source #

showList ∷ [TraceChunkValidation blk validateTo] → ShowS Source #

Generic (TraceChunkValidation blk validateTo) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Associated Types

type Rep (TraceChunkValidation blk validateTo) ∷ TypeType Source #

Methods

fromTraceChunkValidation blk validateTo → Rep (TraceChunkValidation blk validateTo) x Source #

toRep (TraceChunkValidation blk validateTo) x → TraceChunkValidation blk validateTo Source #

type Rep (TraceChunkValidation blk validateTo) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

type Rep (TraceChunkValidation blk validateTo) = D1 ('MetaData "TraceChunkValidation" "Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types" "ouroboros-consensus-0.1.0.0-inplace" 'False) (((C1 ('MetaCons "StartedValidatingChunk" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 validateTo)) :+: C1 ('MetaCons "ValidatedChunk" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 validateTo))) :+: (C1 ('MetaCons "MissingChunkFile" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo)) :+: (C1 ('MetaCons "InvalidChunkFile" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (ChunkFileError blk))) :+: C1 ('MetaCons "MissingPrimaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo))))) :+: ((C1 ('MetaCons "MissingSecondaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo)) :+: C1 ('MetaCons "InvalidPrimaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo))) :+: (C1 ('MetaCons "InvalidSecondaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo)) :+: (C1 ('MetaCons "RewritePrimaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo)) :+: C1 ('MetaCons "RewriteSecondaryIndex" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo))))))

data TraceEvent blk Source #

Constructors

NoValidLastLocation 
ValidatedLastLocation ChunkNo (Tip blk) 
ChunkValidationEvent (TraceChunkValidation blk ChunkNo) 
ChunkFileDoesntFit (ChainHash blk) (ChainHash blk)

The hash of the last block in the previous epoch doesn't match the previous hash of the first block in the current epoch

Migrating Text

Performing a migration of the on-disk files

DeletingAfter (WithOrigin (Tip blk)) 
DBAlreadyClosed 
DBClosed 
TraceCacheEvent !TraceCacheEvent 

Instances

Instances details
StandardHash blk ⇒ Eq (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Methods

(==)TraceEvent blk → TraceEvent blk → Bool Source #

(/=)TraceEvent blk → TraceEvent blk → Bool Source #

StandardHash blk ⇒ Show (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Methods

showsPrecIntTraceEvent blk → ShowS Source #

showTraceEvent blk → String Source #

showList ∷ [TraceEvent blk] → ShowS Source #

Generic (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Associated Types

type Rep (TraceEvent blk) ∷ TypeType Source #

Methods

fromTraceEvent blk → Rep (TraceEvent blk) x Source #

toRep (TraceEvent blk) x → TraceEvent blk Source #

type Rep (TraceEvent blk) Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

type Rep (TraceEvent blk) = D1 ('MetaData "TraceEvent" "Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types" "ouroboros-consensus-0.1.0.0-inplace" 'False) (((C1 ('MetaCons "NoValidLastLocation" 'PrefixI 'False) (U1TypeType) :+: C1 ('MetaCons "ValidatedLastLocation" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 ChunkNo) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (Tip blk)))) :+: (C1 ('MetaCons "ChunkValidationEvent" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (TraceChunkValidation blk ChunkNo))) :+: C1 ('MetaCons "ChunkFileDoesntFit" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (ChainHash blk)) :*: S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (ChainHash blk))))) :+: ((C1 ('MetaCons "Migrating" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 Text)) :+: C1 ('MetaCons "DeletingAfter" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 (WithOrigin (Tip blk))))) :+: (C1 ('MetaCons "DBAlreadyClosed" 'PrefixI 'False) (U1TypeType) :+: (C1 ('MetaCons "DBClosed" 'PrefixI 'False) (U1TypeType) :+: C1 ('MetaCons "TraceCacheEvent" 'PrefixI 'False) (S1 ('MetaSel ('NothingMaybe Symbol) 'NoSourceUnpackedness 'SourceStrict 'DecidedStrict) (Rec0 TraceCacheEvent))))))

data ValidationPolicy Source #

The validation policy used when opening an ImmutableDB.

The validation policy is used by openDB: the initial opening of the database, either an empty database or a database that was previously closed.

The recovery policy dictates which on-disk files should be validated.

Constructors

ValidateMostRecentChunk

The chunk and index files of the most recent chunk stored on disk will be validated.

Prior chunk and index files are ignored, even their presence will not be checked.

A MissingFileError or an InvalidFileError will be thrown in case of a missing or invalid chunk file, or an invalid index file.

Because not all files are validated, subsequent operations on the database after opening may result in unexpected errors.

ValidateAllChunks

The chunk and index files of all chunks starting from the first one up to the last chunk stored on disk will be validated.

A MissingFileError or an InvalidFileError will be thrown in case of a missing or invalid chunk file, or an invalid index file.

Instances

Instances details
Eq ValidationPolicy Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Show ValidationPolicy Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Generic ValidationPolicy Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

Associated Types

type Rep ValidationPolicyTypeType Source #

type Rep ValidationPolicy Source # 
Instance details

Defined in Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types

type Rep ValidationPolicy = D1 ('MetaData "ValidationPolicy" "Ouroboros.Consensus.Storage.ImmutableDB.Impl.Types" "ouroboros-consensus-0.1.0.0-inplace" 'False) (C1 ('MetaCons "ValidateMostRecentChunk" 'PrefixI 'False) (U1TypeType) :+: C1 ('MetaCons "ValidateAllChunks" 'PrefixI 'False) (U1TypeType))

Internals for testing purposes

data Internal m blk Source #

Constructors

Internal 

Fields

  • deleteAfter_HasCallStackWithOrigin (Tip blk) → m ()

    Delete everything in the database after the specified tip.

    PRECONDITION: The tip must correspond to an existing block or genesis.

    The correctness of open iterators is not guaranteed, they should be closed before calling this operation.

    Throws a ClosedDBError if the database is closed.

deleteAfterHasCallStackInternal m blk → WithOrigin (Tip blk) → m () Source #

Wrapper around deleteAfter_ to ensure HasCallStack constraint

See documentation of deleteAfter_.

openDBInternal ∷ ∀ m blk ans. (IOLike m, GetPrevHash blk, ConvertRawHash blk, ImmutableDbSerialiseConstraints blk, HasCallStack) ⇒ ImmutableDbArgs Identity m blk → (∀ h. WithTempRegistry (OpenState m blk h) m ((ImmutableDB m blk, Internal m blk), OpenState m blk h) → ans) → ans Source #

For testing purposes: exposes internals via Internal