71.1 RAS Writes

The RAS provides a function that issues an asynchronous write. A write specifies a file offset and passes its data as a pointer and size. The function returns immediately with a 64 bit Write Sequence Number (WSN):

using WSN = uint64;
using WSNCount = uint64;

WSN Write(RASOffset offset, const void* buffer, ssize_t size);

WSNs are consecutive and zero-based. The first write has WSN 0, the second has WSN 1, and so on. They define the order in which writes were issued, independently of the order in which the native write operations complete.

Write() is thread-safe. When calls are made concurrently, the RAS is responsible for safely allocating a unique consecutive WSN and registering each write for progress tracking. Callers do not need to serialise their calls to Write().

The allocation of WSNs does not define an order in which writes must be submitted to, executed by or completed by the storage device, and it does not itself provide a write-ordering or persistence guarantee. Native writes may execute and complete in any order. WSNs provide an ordering only for reporting completed and durable prefixes.

Write progress is reported through a callback with two monotonically increasing watermarks:

void OnWriteProgress(WSNCount numCompleted, WSNCount numDurable);

The RAS must invoke OnWriteProgress() sequentially. Calls to the callback must never overlap, even when native write operations complete concurrently. This requirement does not prescribe which thread invokes the callback, but it ensures that the receiver observes progress notifications in a single, well-defined sequence.

The watermarks use half-open ranges. numCompleted means that all writes with WSNs in [0,numCompleted) have completed. Similarly, numDurable means that all writes with WSNs in [0,numDurable) are durable. Both values are initially zero, so there is no need for a distinguished value representing the absence of a write.

A write with WSN wsn has completed when wsn < numCompleted, and is durable when wsn < numDurable. The following invariants always hold:

numDurable <= numCompleted;
numCompleted <= numWritesIssued;

Completion has two related meanings. Firstly, the RAS no longer accesses the input buffer. The caller must keep the buffer valid and unmodified until it receives an OnWriteProgress() call in which numCompleted is greater than the write's WSN. Secondly, that notification establishes an ordering boundary: an overlapping write issued after the notification must not be reordered ahead of the completed write.

Native writes may complete out of order, but the reported values describe contiguous prefixes. For example, if writes 0, 1, 3 and 4 have completed while write 2 is still pending, then numCompleted is 2. When write 2 completes, the watermark can advance directly to 5. Therefore numCompleted is the length of the completed prefix, not necessarily the total number of individual native writes that have completed.

Write-completion granularity

Each WSN identifies one complete logical RAS write request. From the caller's perspective, that request is either pending or completed. The RAS API does not expose partial progress within a write, such as a number of bytes or a proportion of the request that has been transferred.

A native platform write may transfer fewer bytes than requested. This is an implementation detail of the RAS. The implementation can continue writing the remaining suffix under the same WSN, or treat the short transfer as a fatal write error. It must not allocate additional WSNs for native sub-operations, and it must not report the logical write as completed until its entire requested file range has been written successfully.

The caller must retain the complete input buffer until the write's WSN is less than numCompleted. Completion of part of a native transfer does not give the caller permission to modify, reuse or release the corresponding part of the buffer.

The only notification of write-completion progress is numCompleted in OnWriteProgress(). It reports the number of WSNs in the largest contiguous prefix of completely finished logical writes. The API provides no finer-grained completion notification, either within an individual write or for completed writes beyond a gap in that prefix.

Overlapping writes

A write with offset offset and size size covers the half-open file range [offset,offset+size). Two writes overlap if their ranges contain at least one common byte. Given two write ranges [offsetA,offsetA+sizeA) and [offsetB,offsetB+sizeB), they overlap when:

offsetA < offsetB + sizeB &&
offsetB < offsetA + sizeA;

Adjacent ranges do not overlap. For example, [100,200) and [200,300) have no bytes in common.

Pending overlapping writes are forbidden

It is forbidden for the caller to have two overlapping writes outstanding at the same time. After issuing a write with WSN wsn, the caller must not issue a second overlapping write until it has received an OnWriteProgress() notification for which wsn < numCompleted. The notification, rather than the passage of time or an assumption about native I/O progress, gives the caller permission to reuse the range.

WSN first = ras.Write(offset, firstBuffer, size);

// Wait until OnWriteProgress() reports first < numCompleted.

WSN second = ras.Write(offset, secondBuffer, size);

This is a contract on the caller even though Write() itself is thread-safe. If different threads issue the two writes, the caller is responsible for communicating the progress notification between those threads with the required thread synchronisation.

RAS cannot reorder pending and completed overlapping writes

The RAS must not report a write as completed and subsequently reorder a later overlapping write ahead of it. Once OnWriteProgress() reports wsn < numCompleted, any overlapping write issued afterwards must be applied after the completed write. Consequently, numCompleted does not merely report that input buffers can be released; it also establishes the ordering boundary that makes later reuse of the same file range safe.

For example, an implementation must not copy a buffer into private memory, advance numCompleted merely to release the caller's buffer, and then submit a later overlapping write ahead of the privately buffered write. An implementation may release a buffer early only if it also preserves the required ordering of any overlapping write issued after the completion notification.

This guarantee does not make WSN order a general device-ordering rule. Disjoint writes may still be submitted, executed and completed in any order. Concurrent overlapping writes are forbidden rather than ordered by their WSNs. The ordering guarantee arises only when a caller waits for the earlier write to be reported as completed before issuing the later overlapping write.

Windows I/O completion ports satisfy write-order requirements

The contract on the RAS can be assumed to be met straightforwardly by an implementation based on Windows overlapped I/O and I/O completion ports. The RAS submits each write using WriteFile() with an OVERLAPPED structure. Windows places a completion packet on the I/O completion port when the overlapped write operation has completed. Until then, the operation is pending and its buffer and OVERLAPPED structure must remain valid.

The RAS records each native completion and advances numCompleted only across the contiguous prefix of completed WSNs. Native operations and their completion packets may be processed out of order; this affects only when a gap in the completed prefix is closed. It does not require writes to execute in WSN order.

The caller does not issue a second overlapping write until it receives an OnWriteProgress() notification showing that the first write's WSN is less than numCompleted. At that point the Windows operation for the first write has completed, so the second overlapping WriteFile() call is submitted only after the first operation is no longer outstanding. The required ordering therefore follows naturally from native completion and does not require an additional ordering operation or the serialisation of disjoint writes.

Several I/O completion port worker threads may process native completions concurrently. The RAS must still serialise its calls to OnWriteProgress() and ensure that the reported watermarks only increase. This serialisation applies to progress notification and does not prevent native writes from remaining concurrent.

Linux io_uring satisfies write-order requirements

The same contract can be met straightforwardly by a Linux implementation based on io_uring. The RAS describes each write with an IORING_OP_WRITE submission queue entry (SQE), including its file offset, buffer and size. When the kernel has finished processing the request, it places a corresponding completion queue entry (CQE) in the completion queue. The CQE reports the result that the equivalent write system call would have returned.

The buffer of an IORING_OP_WRITE request must remain valid while the request is in flight. The RAS therefore retains the buffer and operation record until it receives the CQE. It then records the native completion and advances numCompleted only across the contiguous prefix of completed WSNs. Requests may execute and complete out of order, so a CQE for a later WSN does not by itself allow the reported prefix to advance past an earlier pending request.

The caller waits for an OnWriteProgress() notification showing that the first write's WSN is less than numCompleted before issuing a second overlapping write. The SQE for the second write is consequently submitted only after the CQE for the first write has established that the first operation is complete. The two overlapping writes are not simultaneously in flight, so the required ordering follows from normal io_uring completion without linking the requests, serialising disjoint writes or introducing another ordering operation.

Submission and completion queues may be serviced concurrently, and an implementation may process CQEs on more than one thread. As with the Windows implementation, the RAS must combine those native results into monotonic prefix watermarks and invoke OnWriteProgress() sequentially. This affects progress reporting, not the concurrency of independent native writes.

RAS diagnostic check for pending overlapping writes

A RAS implementation can retain the file ranges of outstanding writes and check each new write for overlap in diagnostic builds. A range can be removed from this diagnostic set when its write is reported as part of the completed prefix. The diagnostic should identify both conflicting ranges and their WSNs.

The LSS normally leaves a long interval between writes to the same file range because it writes a log and only occasionally writes a new checkpoint. An overlap detected by the RAS is therefore likely to indicate incorrect checkpoint, segment-reuse or write-management behaviour. Checking this caller contract provides a useful way to help ensure that the LSS is correct.

Durability

Completion and durability are different properties. Completion means that a write has crossed the buffer-release and write-ordering boundary described above. It does not normally mean that the data would survive an operating-system crash or loss of power, because file data may remain in operating-system or device caches after the native write operation has completed.

numDurable reports the contiguous prefix of writes for which the RAS persistence guarantee has been established. A write with WSN wsn is durable when wsn < numDurable. Since durability implies completion, numDurable <= numCompleted always holds. The RAS must not advance numDurable merely because ordinary write completions have been received.

The RAS can make many writes durable with one persistence operation. When that operation succeeds, numDurable can advance directly to the end of the covered write prefix and the RAS reports the new value through OnWriteProgress(). This allows durability notifications to be batched rather than requiring a persistence operation for every write.

Asynchronous Flush()

Durability is requested explicitly with an asynchronous Flush() operation:

// Request durability of every write accepted before this call.
// Returns the exclusive end of the requested WSN prefix.
WSNCount Flush();

Flush() is thread-safe and is linearised with concurrent calls to Write(). It captures every write accepted before its linearisation point and returns the exclusive end of that prefix. If it returns target, the request has been satisfied when a later OnWriteProgress() notification reports target <= numDurable. A target of zero is valid and means that no writes preceded the flush.

The call returns without waiting for outstanding writes or for the storage device. Writes issued after the captured prefix may continue while the flush is pending. They are not required to become durable as part of that request, although a platform persistence operation may incidentally include some of them.

The name Flush() does not mean that the RAS has retained writes in an application buffer and should now submit them, nor does it mean that the RAS should try harder to send subsequent writes to the device. Writes are submitted promptly in the ordinary course of operation. Flush() specifically requests notification when the captured prefix has become durable.

Several pending flush requests may be coalesced. For example, requests with targets 20, 25 and 31 can be satisfied by one persistence operation covering the prefix [0,31). When it succeeds, the RAS can report numDurable == 31, thereby satisfying all three requests. The RAS must not report a prefix beyond the one it can prove durable.

Use by the LSS

The LSS writes complete Log Flush Units (LFUs) to the RAS. These writes are intended to be submitted to storage as quickly as possible rather than retained by the RAS while it waits for additional LFUs to accumulate. Ordinary LSS operation needs completion notifications so that write buffers and file ranges can be reused, but it does not generally need to know when each LFU becomes durable.

A checkpoint is one of the few operations that requires an explicit durability boundary. Before the next root-block division is rewritten, the last LFU on which that division depends must be durable. The checkpoint therefore requests a flush after writing that LFU and waits for the returned prefix to be reported as durable:

WSN lastLfuWsn = WriteLastLfu();
WSNCount flushTarget = ras.Flush();

// Continue asynchronously when OnWriteProgress() reports:
//     flushTarget <= numDurable

WriteRootBlockDivision();

The returned target will be greater than lastLfuWsn, because it is an exclusive prefix endpoint and includes at least that LFU. Waiting for flushTarget <= numDurable makes the dependency explicit without blocking a thread. Later log writes may be issued while the checkpoint flush is in progress.

Windows

On Windows, ordinary overlapped WriteFile() completion does not normally establish durability. The conventional operation for forcing buffered file data to storage is FlushFileBuffers(). In response to Flush(), the RAS first ensures that the writes in the target prefix have completed, invokes FlushFileBuffers() for the file, and advances numDurable only after the platform flush has returned successfully.

FlushFileBuffers() is a synchronous call rather than an overlapped I/O operation reported through an I/O completion port. A fully asynchronous RAS must therefore avoid calling it on a thread that must remain available to process other completions. It can run the flush on a shared blocking-I/O worker and deliver the result back to its normal completion mechanism. This does not consume a thread per write, but a worker thread is occupied while the persistence operation is in progress.

Windows also supports FILE_FLAG_WRITE_THROUGH, optionally combined with FILE_FLAG_NO_BUFFERING. With both flags, Windows requests that each write pass through the system and hardware caches to persistent media, subject to support from the storage hardware. Combining FILE_FLAG_NO_BUFFERING with overlapped I/O can provide high asynchronous throughput, but it imposes buffer, size and file-offset alignment requirements and gives up the benefits of the system file cache.

Write-through operation can allow numDurable to advance with the completed write prefix, whereas explicit flushing permits several writes to share one persistence barrier. The best choice depends on the storage device and workload. Calling FlushFileBuffers() after every small write is potentially expensive; batching writes before a flush normally amortises both the system-call and device-cache flush costs.

Linux

Linux io_uring provides IORING_OP_FSYNC, so a file synchronization request can itself be submitted asynchronously in response to Flush() and reported by a CQE. It can provide normal fsync() behaviour or, with IORING_FSYNC_DATASYNC, the data-oriented behaviour of fdatasync().

An fsync SQE is not automatically ordered after previously submitted write SQEs. If submitted as an independent request, it may execute before one of those writes has reached storage. The RAS must therefore establish the dependency explicitly. It can wait until all writes in the target prefix have completed before submitting IORING_OP_FSYNC, or use the appropriate io_uring linking or drain facility. It advances numDurable only after the synchronization CQE reports success.

The asynchronous interface means that no application thread needs to block while the device performs the synchronization. It does not remove the storage cost: dirty pages and device caches must still be flushed, and subsequent work that depends on durability must wait for that operation. As on Windows, issuing a synchronization request after every small write can significantly reduce throughput. Batching a prefix of writes behind one IORING_OP_FSYNC operation allows one device synchronization to advance numDurable across many WSNs.

Performance implications

Asynchrony prevents persistence latency from blocking an execution thread, but it cannot eliminate that latency or the underlying device work. Durability may require cached data to be written, a device cache to be flushed and storage ordering constraints to be honoured. These operations can be much more expensive than accepting data into a cache.

The principal performance choice is therefore the frequency at which Flush() is called. Flushing every write provides the smallest durability lag but may serialise the workload around device barriers. Flushing groups of writes permits higher throughput and allows numDurable to advance in larger increments, at the cost of a larger interval during which completed writes are not yet durable. The RAS interface should expose accurate progress without forcing a particular persistence frequency or batching policy.