55 Fully asynchronous LSS

Status: unimplemented proposal. This proposal makes storage I/O asynchronous throughout the LSS, including at its public API, so an LSS does not require a collection of threads which block while I/O is in progress.

Motivation

The LSS already separates most background work into LazyWriter, LazyFlusher, LazyCheckPointer and LazyCleaner. Their clients normally signal that work is needed and continue; they do not need the work to finish before making progress. These components are therefore natural asynchronous tasks rather than reasons to dedicate threads to an LSS.

The current asynchronous task wrappers make invocation asynchronous, but the invoked function can still occupy an io_context thread for the entire duration of a blocking file operation. Instead, an operation should submit I/O, return to the executor and resume when its completion is delivered. This is particularly important on platforms such as WebAssembly where threads may be unavailable or undesirable and storage is naturally exposed through asynchronous interfaces.

Transaction commit remains an in-memory operation

Closing an LSS transaction does not provide per-transaction synchronous durability. It commits the transaction to the in-memory log and need not perform any I/O. This fast path should remain immediate, even when represented by an asynchronous public API: an operation which has no reason to suspend may complete immediately.

The proposed design removes FlushWhenClose(). Waiting for a storage flush on every selected transaction would complicate the normal execution model for a facility that is not central to the intended use of the LSS. Applications interested in persistence should instead observe the asynchronous durability progress described below.

Asynchronous storage interface

An asynchronous counterpart or replacement for IRAS should report the completion of reads, writes, resizing and persistence barriers without blocking the calling thread. The exact C++ representation could be an awaitable or a completion handler; the important property is that the LSS can suspend an operation and resume it on its serialized executor.


struct IAsyncRAS
{
    virtual AsyncResult<RASSize> GetSize() = 0;
    virtual AsyncResult<void> Read(octet_t* buffer, RASOffset offset, ssize_t size) = 0;
    virtual AsyncResult<void> Write(const octet_t* buffer, RASOffset offset, ssize_t size) = 0;
    virtual AsyncResult<void> SetSize(RASSize size) = 0;
    virtual AsyncResult<void> Flush() = 0;
};

This is illustrative rather than a proposed final signature. Buffers passed to an operation must remain alive until its completion. The segment queues already provide much of the ownership needed for outstanding segment writes.

Platform implementations can map the interface to native asynchronous facilities, WebAssembly storage promises, or, as a compatibility fallback, blocking file operations performed by a shared worker facility. The fallback may use threads, but the LSS itself does not require a set of threads per store.

Existing lazy activities

LazyWriter: Submit writes for queued segment ranges, retain each segment until completion, and advance the written log position across the contiguous prefix of successful writes.
LazyFlusher: Periodically request writer progress and, when required by the persistence policy, submit a storage persistence barrier. It should not occupy an executor thread while that barrier is pending.
LazyCheckPointer: Construct check-point information in memory, request that the required log prefix be written, and continue with the root-block-division write after the prerequisite completions have arrived.
LazyCleaner: Perform cleaning incrementally, resuming its reads and writes from completion notifications and yielding between units of work.

These activities can share one serialized LSS executor. Storage operations may be outstanding concurrently, while all changes to logical LSS state are applied in a well-defined order on that executor. This can also replace a significant amount of mutex-based coordination.

Asynchronous public API

The public LSS API should not synchronously wait for I/O. Operations which may need storage include opening and recovery, loading an uncached segment, applying back-pressure when no reusable segment buffer is available, explicit maintenance operations, and closing the store. Such operations should complete asynchronously. Transaction operations which touch only resident memory can complete immediately through the same API.

Closing an LSS must itself be asynchronous. It stops the creation of new work, allows or cancels outstanding operations according to a documented policy, and completes only when no completion can subsequently access the LSS. I/O errors must be reported to the affected operation and retained as store state where they also affect later operations.

Durability notification by TSN

Rather than making a transaction wait for durability, the LSS should notify an application when a prefix of committed transactions is known to be durable. Since transactions are serialized, this can be represented by a monotonically increasing transaction serial number (TSN):


using DurableTxnCallback = std::function<void(TSN durableTsn)>;

void SetDurableTxnCallback(DurableTxnCallback callback);
TSN GetDurableTSN() const;

A notification of TSN n means that every committed transaction with a TSN less than or equal to n is known to be recoverable after the documented class of storage failure. A single notification can therefore cover many transactions. GetDurableTSN() allows a caller to handle the case where the TSN of interest became durable before it registered its interest or processed a notification.

The durable watermark advances only over a contiguous log prefix, even if asynchronous writes complete out of order. Completion of a buffered write is not necessarily evidence of durability under abrupt power loss. The watermark must be advanced according to the persistence guarantee of the underlying RAS, normally after the applicable asynchronous flush or barrier has completed.

A check point is not itself the definition of transaction durability: recovery can scan valid log data following the last valid check point. Check-point publication nevertheless has a strict dependency. All log data and metadata referenced by a new root-block division must have completed the required writes before that division is submitted. Expressing the root-block write as a continuation of those completions makes this ordering explicit without blocking a thread.

Back-pressure and ordering

Asynchronous I/O does not imply unbounded buffering. When the segment cache cannot supply another buffer, the transaction operation requesting one should remain pending until writer progress releases a segment. This applies back-pressure without blocking a thread. Cancellation and shutdown must not release buffers which are still referenced by outstanding I/O.

Writes may be issued concurrently where their logical independence permits it, but completion order must not be confused with publication order. Written and durable log positions, reusable buffers, check-point state and the durable TSN are advanced only when their respective contiguous prerequisites have completed successfully.

Previous54 Implementation status and open issues Next56 Asynchronous loading of serial elements