ouroboros-consensus-0.1.0.0: Consensus layer for the Ouroboros blockchain protocol

Ouroboros.Consensus.Util.ResourceRegistry

Synopsis

# Documentation

Attempt to allocate a resource in a registry which is closed

When calling closeRegistry (typically, leaving the scope of withRegistry), all resources in the registry must be released. If a concurrent thread is still allocating resources, we end up with a race between the thread trying to allocate new resources and the registry trying to free them all. To avoid this, before releasing anything, the registry will record itself as closed. Any attempt by a concurrent thread to allocate a new resource will then result in a RegistryClosedException.

It is probably not particularly useful for threads to try and catch this exception (apart from in a generic handler that does local resource cleanup). The thread will anyway soon receive a ThreadKilled exception.

Constructors

 ∀ m.IOLike m ⇒ RegistryClosedException FieldsregistryClosedRegistryContext ∷ !(Context m)The context in which the registry was createdregistryClosedCloseCallStack ∷ !PrettyCallStackCallstack to the call to closeNote that close can only be called from the same thread that created the registry.registryClosedAllocContext ∷ !(Context m)Context of the call resulting in the exception

#### Instances

Instances details
 Source # Instance details Methods Source # Instance details

If this exception is raised, it indicates a bug in the caller.

#### Instances

Instances details
 Source # Instance details Source # Instance details

# Creating and releasing the registry itself

Arguments

 ∷ (IOLike m, HasCallStack) ⇒ (ResourceRegistry m → m a) → (a → m ()) Release the resource → (a → m r) → m r

Create a new private registry for use by a bracketed resource

Use this combinator as a more specific and easier-to-maintain alternative to the following.

'withRegistry' $\rr -> 'bracket' (newFoo rr) closeFoo$ \foo ->
(... rr does not occur in this scope ...)

NB The scoped body can use withRegistry if it also needs its own, separate registry.

Use this combinator to emphasize that the registry is private to (ie only used by and/or via) the bracketed resource and that it thus has nearly the same lifetime. This combinator ensures the following specific invariants regarding lifetimes and order of releases.

o The registry itself is older than the bracketed resource.

o The only registered resources older than the bracketed resource were allocated in the registry by the function that allocated the bracketed resource.

o Because of the older resources, the bracketed resource is itself also registered in the registry; that's the only way we can be sure to release all resources in the right order.

NB Because the registry is private to the resource, the a type could save the handle to registry and safely close the registry if the scoped body calls closeA before the bracket ends. Though we have not used the type system to guarantee that the interface of the a type cannot leak the registry to the body, this combinator does its part to keep the registry private to the bracketed resource.

See documentation of ResourceRegistry for a more general discussion.

The thread that created the registry

withRegistry ∷ (IOLike m, HasCallStack) ⇒ (ResourceRegistry m → m a) → m a Source #

Create a new registry

See documentation of ResourceRegistry for a detailed discussion.

# Allocating and releasing regular resources

data ResourceKey m Source #

Resource key

Resource keys are tied to a particular registry.

#### Instances

Instances details
 Source # Instance details Associated Typestype Rep (ResourceKey m) ∷ Type → Type Source # Methodsfrom ∷ ResourceKey m → Rep (ResourceKey m) x Source #to ∷ Rep (ResourceKey m) x → ResourceKey m Source # Source # Instance details MethodsnoThunks ∷ Context → ResourceKey m → IO (Maybe ThunkInfo) #wNoThunks ∷ Context → ResourceKey m → IO (Maybe ThunkInfo) # type Rep (ResourceKey m) Source # Instance details type Rep (ResourceKey m)

Arguments

 ∷ ∀ m a. (IOLike m, HasCallStack) ⇒ ResourceRegistry m → (ResourceId → m a) → (a → m ()) Release the resource → m (ResourceKey m, a)

Allocate new resource

The allocation function will be run with asynchronous exceptions masked. This means that the resource allocation must either be fast or else interruptible; see "Dealing with Asynchronous Exceptions during Resource Acquisition" http://www.well-typed.com/blog/97/ for details.

Arguments

 ∷ ∀ m e a. (IOLike m, HasCallStack) ⇒ ResourceRegistry m → (ResourceId → m (Either e a)) → (a → m Bool) Release the resource, return True when the resource hasn't been released or closed before. → m (Either e (ResourceKey m, a))

Generalization of allocate for allocation functions that may fail

release ∷ (IOLike m, HasCallStack) ⇒ ResourceKey m → m (Maybe (Context m)) Source #

Release resource

This deallocates the resource and removes it from the registry. It will be the responsibility of the caller to make sure that the resource is no longer used in any thread.

The deallocation function is run with exceptions masked, so that we are guaranteed not to remove the resource from the registry without releasing it.

Releasing an already released resource is a no-op.

When the resource has not been released before, its context is returned.

releaseAll ∷ (IOLike m, HasCallStack) ⇒ ResourceRegistry m → m () Source #

Release all resources in the ResourceRegistry without closing.

See closeRegistry for more details.

unsafeReleaseIOLike m ⇒ ResourceKey m → m (Maybe (Context m)) Source #

Unsafe version of release

The only difference between release and unsafeRelease is that the latter does not insist that it is called from a thread that is known to the registry. This is dangerous, because it implies that there is a thread with access to a resource which may be deallocated before that thread is terminated. Of course, we can't detect all such situations (when the thread merely uses a resource but does not allocate or release we can't tell), but normally when we do detect this we throw an exception.

This function should only be used if the above situation can be ruled out or handled by other means.

unsafeReleaseAll ∷ (IOLike m, HasCallStack) ⇒ ResourceRegistry m → m () Source #

This is to releaseAll what unsafeRelease is to release: we do not insist that this funciton is called from a thread that is known to the registry. See unsafeRelease for why this is dangerous.

This is a synchronous operation: the thread will have terminated when this function returns.

Uses uninterruptibleCancel because that's what withAsync does.

Arguments

 ∷ (IOLike m, HasCallStack) ⇒ ResourceRegistry m → String Label for the thread → m a → m (Thread m a)

This function is just a convenience.

Arguments

 ∷ ∀ m a. (IOLike m, HasCallStack) ⇒ ResourceRegistry m → String Label for the thread → m a → m (Thread m a)

Link specified Thread to the (thread that created) the registry

waitAnyThread ∷ ∀ m a. IOLike m ⇒ [Thread m a] → m a Source #

Lift waitAny to Thread

Wait for thread to terminate and return its result.

If the thread throws an exception, this will rethrow that exception.

NOTE: If A waits on B, and B is linked to the registry, and B throws an exception, then A might either receive the exception thrown by B or the ThreadKilled exception thrown by the registry.

Arguments

 ∷ IOLike m ⇒ ResourceRegistry m → String Label for the thread → m a → (Thread m a → m b) → m b

Bracketed version of forkThread

The analogue of withAsync for the registry.

Scoping thread lifetime using withThread is important when a parent thread wants to link to a child thread /and handle any exceptions arising from the link/:

let handleLinkException :: ExceptionInLinkedThread -> m ()
in handle handleLinkException $withThread registry codeInChild$ \child ->
..

handle handleLinkException $do -- PROBABLY NOT CORRECT! child <- forkThread registry codeInChild .. where the parent may exit the scope of the exception handler before the child terminates. If the lifetime of the child cannot be limited to the lifetime of the parent, the child should probably be linked to the registry instead and the thread that spawned the registry should handle any exceptions. Note that in principle there is no problem in using withAync alongside a registry. After all, in a pattern like withRegistry$ \registry ->
..

# Combinators primarily for testing

closeRegistry ∷ (IOLike m, HasCallStack) ⇒ ResourceRegistry m → m () Source #

Close the registry

This can only be called from the same thread that created the registry. This is a no-op if the registry is already closed.

This entire function runs with exceptions masked, so that we are not interrupted while we release all resources.

Resources will be allocated from young to old, so that resources allocated later can safely refer to resources created earlier.

The release functions are run in the scope of an exception handler, so that if releasing one resource throws an exception, we still attempt to release the other resources. Should we catch an exception whilst we close the registry, we will rethrow it after having attempted to release all resources. If there is more than one, we will pick a random one to rethrow, though we will prioritize asynchronous exceptions over other exceptions. This may be important for exception handlers that catch all-except-asynchronous exceptions.

Number of currently allocated resources

Primarily for the benefit of testing.

Create a new registry

You are strongly encouraged to use withRegistry instead. Exported primarily for the benefit of tests.

# opaque

Resource registry

Note on terminology: when thread A forks thread B, we will say that thread A is the " parent " and thread B is the " child ". No further relationship between the two threads is implied by this terminology. In particular, note that the child may outlive the parent. We will use "fork" and "spawn" interchangeably.

# Motivation

Whenever we allocate resources, we must keep track of them so that we can deallocate them when they are no longer required. The most important tool we have to achieve this is bracket:

bracket allocateResource releaseResource $\r -> .. use r .. Often bracket comes in the guise of a with-style combinator withResource$ \r ->
.. use r ..

Where this pattern is applicable, it should be used and there is no need to use the ResourceRegistry. However, bracket introduces strict lexical scoping: the resource is available inside the scope of the bracket, and will be deallocated once we leave that scope. That pattern is sometimes hard to use.

For example, suppose we have this interface to an SQL server

query :: Query -> IO QueryHandle
close :: QueryHandle -> IO ()
next  :: QueryHandle -> IO Row

and suppose furthermore that we are writing a simple webserver that allows a client to send multiple SQL queries, get rows from any open query, and close queries when no longer required:

server :: IO ()
server = go Map.empty
where
go :: Map QueryId QueryHandle -> IO ()
go handles = getRequest >>= \case
New q -> do
h   <- query q                        -- allocate
qId <- generateQueryId
sendResponse qId
go $Map.insert qId h handles Close qId -> do close (handles ! qId) -- release go$ Map.delete qId handles
Next qId -> do
sendResponse =<< next (handles ! qId)
go handles

The server opens and closes query handles in response to client requests. Restructuring this code to use bracket would be awkward, but as it stands this code does not ensure that resources get deallocated; for example, if the server thread is killed (killThread), resources will be leaked.

Another, perhaps simpler, example is spawning threads. Threads too should be considered to be resources that we should keep track of and deallocate when they are no longer required, primarily because when we deallocate (terminate) those threads they too will have a chance to deallocate their resources. As for other resources, we have a with-style combinator for this

withAsync $\thread -> .. Lexical scoping of threads is often inconvenient, however, more so than for regular resources. The temptation is therefore to simply fork a thread and forget about it, but if we are serious about resource deallocation this is not an acceptable solution. # The resource registry The resource registry is essentially a piece of state tracking which resources have been allocated. The registry itself is allocated with a with-style combinator withRegistry, and when we leave that scope any resources not yet deallocated will be released at that point. Typically the registry is only used as a fall-back, ensuring that resources will deallocated even in the presence of exceptions. For example, here's how we might rewrite the above server example using a registry: server' :: IO () server' = withRegistry$ \registry -> go registry Map.empty
where
go :: ResourceRegistry IO
-> Map QueryId (ResourceKey, QueryHandle)
-> IO ()
go registry handles = getRequest >>= \case
New q -> do
(key, h) <- allocate registry (query q) close  -- allocate
qId      <- generateQueryId
sendResponse qId
go registry $Map.insert qId (key, h) handles Close qId -> do release registry (fst (handles ! qId)) -- release go registry$ Map.delete qId handles
Next qId -> do
sendResponse =<< next (snd (handles ! qId))
go registry handles

We allocate the query with the help of the registry, providing the registry with the means to deallocate the query should that be required. We can /and should/ still manually release resources also: in this particular example, the (lexical) scope of the registry is the entire server thread, so delaying releasing queries until we exit that scope will probably mean we hold on to resources for too long. The registry is only there as a fall-back.

We already observed in the introduction that insisting on lexical scoping for threads is often inconvenient, and that simply using fork is no solution as it means we might leak resources. There is however another problem with fork. Consider this snippet:

withRegistry $\registry -> r <- allocate registry allocateResource releaseResource fork$ .. use r ..

It is easy to see that this code is problematic: we allocate a resource r, then spawn a thread that uses r, and finally leave the scope of withRegistry, thereby deallocating r -- leaving the thread to run with a now deallocated resource.

It is only safe for threads to use a given registry, and/or its registered resources, if the lifetime of those threads is tied to the lifetime of the registry. There would be no problem with the example above if the thread would be terminated when we exit the scope of withRegistry.

The forkThread combinator provided by the registry therefore does two things: it allocates the thread as a resource in the registry, so that it can kill the thread when releasing all resources in the registry. It also records the thread ID in a set of known threads. Whenever the registry is accessed from a thread not in this set, the registry throws a runtime exception, since such a thread might outlive the registry and hence its contents. The intention is that this guards against dangerous patterns like the one above.

When thread A spawns thread B using withAsync, the lifetime of B is tied to the lifetime of A:

withAsync .. $\threadB -> .. After all, when A exits the scope of the withAsync, thread B will be killed. The reverse is however not true: thread B can terminate before thread A. It is often useful for thread A to be able to declare a dependency on thread B: if B somehow fails, that is, terminates with an exception, we want that exception to be rethrown in thread A as well. A can achieve this by linking to B: withAsync ..$ \threadB -> do
..

Linking a parent to a child is however of limited value if the lifetime of the child is not limited by the lifetime of the parent. For example, if A does

threadB <- async \$ ..
link threadB

and A terminates before B does, any exception thrown by B might be send to a thread that no longer exists. This is particularly problematic when we start chaining threads: if A spawns-and-links-to B which spawns-and-links-to C, and C throws an exception, perhaps the intention is that this gets rethrown to B, and then rethrown to A, terminating all three threads; however, if B has terminated before the exception is thrown, C will throw the exception to a non-existent thread and A is never notified.

For this reason, the registry's linkToRegistry combinator does not link the specified thread to the thread calling linkToRegistry, but rather to the thread that created the registry. After all, the lifetime of threads spawned with forkThread can certainly exceed the lifetime of their parent threads, but the lifetime of all threads spawned using the registry will be limited by the scope of that registry, and hence the lifetime of the thread that created it. So, when we call linkToRegistry, the exception will be thrown the thread that created the registry, which (if not caught) will cause that that to exit the scope of withRegistry, thereby terminating all threads in that registry.

# Combining the registry and with-style allocation

It is perfectly possible (indeed, advisable) to use bracket and bracket-like allocation functions alongside the registry, but note that the usual caveats with bracket and forking threads still applies. In particular, spawning threads inside the bracket that make use of the bracketed resource is problematic; this is of course true whether or not a registry is used.

In principle this also includes withAsync; however, since withAsync results in a thread that is not known to the registry, such a thread will not be able to use the registry (the registry would throw an unknown thread exception, as described above). For this purpose we provide withThread; withThread (as opposed to forkThread) should be used when a parent thread wants to handle exceptions in the child thread; see withThread for detailed discussion.

It is also fine to includes nested calls to withRegistry. Since the lifetime of such a registry (and all resources within) is tied to the thread calling withRegistry, which itself is tied to the "parent registry" in which it was created, this creates a hierarchy of registries. It is of course essential for compositionality that we should be able to create local registries, but even if we do have easy access to a parent regisry, creating a local one where possibly is useful as it limits the scope of the resources created within, and hence their maximum lifetimes.

#### Instances

Instances details
 Source # Instance details Associated Typestype Rep (ResourceRegistry m) ∷ Type → Type Source # Methods Source # Instance details MethodsnoThunks ∷ Context → ResourceRegistry m → IO (Maybe ThunkInfo) #wNoThunks ∷ Context → ResourceRegistry m → IO (Maybe ThunkInfo) # type Rep (ResourceRegistry m) Source # Instance details type Rep (ResourceRegistry m)