CEDA Log Structured Store

Device / Link	Latency (approx)	Bandwidth (approx)	Latency × Bandwidth (“natural chunk”)
L1 cache	1 ns	1 TB/s	~1 KB
L2 cache	3 ns	500 GB/s	~1.5 KB
L3 cache	10 ns	200 GB/s	~2 KB
Main memory (DRAM)	80 ns	50 GB/s	~4 KB
NVMe SSD	100 µs	3 GB/s	~300 KB
SATA SSD	100 µs	500 MB/s	~50 KB
HDD	10 ms	200 MB/s	~2 MB
1 GbE LAN	0.1 ms	125 MB/s	~12.5 KB
10 GbE LAN	0.1 ms	1.25 GB/s	~125 KB
Typical WAN	50 ms	50 Mb/s	~300 KB

7 Write ordering on storage devices

Persistent data normally passes through several independently buffered layers: an application, operating-system page cache, filesystem, block layer and driver, controller or bridge, and finally a device with its own firmware, volatile RAM, and non-volatile media. RAID controllers, virtual machines, network storage, and USB-to-SATA bridges can add more layers. The order in which an application submits writes is therefore not, by itself, the order in which those writes become persistent.

This chapter describes the contract that a crash-consistent storage protocol needs from that stack. It is primarily about local block devices, but the same reasoning applies to other persistent storage: a higher layer can provide no stronger guarantee than the weakest layer through which its durability request passes.

Four different properties

Discussions of a "completed write" often conflate four independent properties:

Visibility: a subsequent read observes the new data. It may merely be reading a cache.
Durability: the data will survive the relevant failure, particularly abrupt loss of power.
Ordering: write A is guaranteed to be durable before write B can become durable.
Atomicity: after a failure a write is entirely old or entirely new, rather than torn.

A successful buffered write normally establishes visibility, not durability. A cache flush can establish durability for preceding writes, but does not make a large write atomic. Checksums and redundant records can detect or tolerate torn writes, but do not force earlier data to storage. A correct protocol states which property it needs at each step rather than relying on the word "write" to imply all four.

Why volatile write caches exist

HDDs and SSDs commonly acknowledge writes after accepting them into volatile device memory. This reduces latency, permits request merging and reordering, and lets the device schedule media updates efficiently. SSD firmware also performs address translation, garbage collection, wear levelling, and metadata updates whose physical order need not resemble the host's logical write order.

Disabling a device write cache is therefore a costly and usually unnecessary way to obtain a durability guarantee. The normal arrangement on commodity hardware is to leave the cache enabled and issue an explicit cache flush at the points required by the storage protocol. The 2008 Microsoft Research report Enforcing Database Recoverability on Disks that Lack Write-Through describes this approach for IDE disks. It reports that most single-disk IDE drives supported the ATA flush command even though most did not support per-write Force Unit Access, and explains both the performance cost and the unreliability of attempting to disable caching as the general solution.

ATA and SATA: FLUSH CACHE

For an ATA or SATA device with a volatile write cache, the FLUSH CACHE and FLUSH CACHE EXT commands tell the device to complete cached writes to non-volatile media before reporting successful completion. An operating system normally emits these commands on behalf of a filesystem durability operation; applications using files do not send ATA commands directly.

A completed flush provides a boundary for writes accepted before it. A protocol can therefore write a set of dependent records, wait for a flush, and only then issue the record that publishes or makes those records reachable. If the publication record must itself be durable before success is reported, it needs either a subsequent flush or a write with suitable FUA semantics.

Force Unit Access (FUA) is a property of an individual write: completion must not be reported until that write is in non-volatile storage. FUA is not the same as a flush of all prior cached writes. A pre-flush and an FUA publication write can be combined to express "everything before this write is durable first, and this write is durable before completion." Older ATA devices and parts of the SATA software stack have not always provided usable native FUA, so operating systems can implement the same result with cache flushes.

SCSI and NVMe

SCSI provides analogous mechanisms through SYNCHRONIZE CACHE and the FUA bit on suitable write commands. NVMe provides a Flush command and an FUA bit on NVM write commands. When a volatile write cache is present, a successful NVMe Flush makes the relevant earlier writes persistent; an FUA write is not to complete until its data is persistent. These are interface contracts rather than claims about the device's internal physical write sequence. The current contracts are specified by the NVM Express specifications.

NVMe also reports atomic-write units, including a distinct atomic-write unit for power-fail conditions. This distinction is important: a device may promise that a small write is atomic during normal operation while making a weaker promise if power is removed during the write. Atomic-write limits do not replace Flush or FUA, and Flush or FUA do not enlarge an atomic-write limit.

Linux

For ordinary file I/O, fsync() requests that modified file data and the metadata needed to retrieve it be transferred to the storage device, and waits for the device to report completion. fdatasync() may omit metadata that is not needed for subsequent data retrieval. Creating or renaming a file can additionally require an fsync() of its directory. These details are described by the Linux fsync(2) manual page. Direct I/O is not a substitute for a durability operation: bypassing the page cache does not, by itself, force a volatile device cache to non-volatile media.

Within the Linux block layer, filesystems express the device-side requirements using a forced flush and FUA. REQ_PREFLUSH guarantees that previously completed writes are non-volatile before the associated write begins. REQ_FUA guarantees that completion of the marked write is not reported until that write is non-volatile. The block layer can translate FUA into a post-write flush when a device has a write cache but no native FUA support. See the Linux kernel documentation on explicit volatile write-back cache control.

Windows

On Windows, FlushFileBuffers() flushes buffered information for a file to the device. A file opened with FILE_FLAG_WRITE_THROUGH requests write-through behavior; FILE_FLAG_NO_BUFFERING separately controls system caching and imposes alignment requirements. The flags are independent. Microsoft describes their interaction and the fact that support ultimately depends on the hardware in File Buffering.

An explicit flush operation is normally preferable to treating every write as write-through. It preserves batching: many related writes can be submitted efficiently and then covered by one deliberate durability boundary. Write-through may still be useful for an individual write or for environments whose lower-level interface maps it efficiently to FUA.

Abrupt power loss

When power disappears, acknowledged data that exists only in volatile memory is lost. The failure can also interrupt a media update, producing a torn write, or interrupt SSD firmware while it is updating translation metadata. Drives with genuine power-loss protection use stored energy to drain volatile state and finish essential metadata updates. An external uninterruptible power supply is valuable, but is not equivalent: it does not cover a device, cable, controller, kernel, or firmware failure that prevents an orderly flush.

Device behavior under power fault has not always matched the simple model "only unflushed writes are lost." Zheng et al., Reliability Analysis of SSDs Under Power Fault (ACM TOCS, 2016), tested 17 commodity SSDs from six vendors over more than 3000 power-fault injections. Fourteen exhibited at least one surprising behavior, including bit corruption, shorn writes, unserializable writes, metadata corruption, or complete device failure. This result does not mean that every tested device ignored a successfully completed flush, and should not be read as a claim about that narrower case. It does show why software should use integrity checks and why a deployment requiring strong guarantees should validate its actual devices, firmware, controllers, and power-loss-protection claims.

The filesystem is another part of the contract. Pillai et al., All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications (OSDI 2014), found that persistence properties varied across six Linux filesystems and identified 60 crash vulnerabilities in eleven applications. The lesson is not that flushes are futile, but that a protocol must use the documented filesystem operations and must not infer persistence order from incidental behavior observed on one filesystem.

What the guarantee depends upon

On a typical PC with a directly attached SATA HDD or SSD, write caching can normally remain enabled. The application asks the operating system for durability, the filesystem and block layer translate that request into flush or FUA operations, and a conforming device honours them. IDE-era devices also commonly implemented ATA FLUSH CACHE. Thus it is inaccurate to say that most off-the-shelf disks must have their caches disabled to support ordered persistence.

The guarantee is conditional on the entire path. A RAID controller with unprotected write-back RAM, a USB bridge that drops cache-management commands, a hypervisor that acknowledges them too early, or firmware that falsely reports completion breaks the contract. Multiple independent devices also create multiple cache domains: flushing one does not flush another. Storage vendors and system operators therefore commonly qualify support by controller configuration, battery or capacitor health, firmware version, and power-loss-protection capability.

8 Log structured store versus Write ahead logging

Write Ahead Logging and the CEDA LSS make fundamentally different choices about persistent layout and whether non-volatile durability lies on the normal transaction path.

Write Ahead Logging

Most conventional transactional database systems use Write Ahead Logging (WAL), commonly with ARIES or a similar recovery algorithm. WAL allows data pages to be updated in place while supporting transaction atomicity and crash recovery. A change is first represented in the WAL and is later written again as part of its data page.

Write-ahead invariant

The defining invariant is that a dirty data page must not reach persistent storage before every WAL record needed to recover that version of the page is persistent. A page may be changed immediately in the in-memory buffer pool, but before the page is written to storage the WAL must be flushed through the page's log sequence number. "Before" refers to persistence, not merely to the order in which the application issues writes.

For a strictly durable transaction, WAL also puts a flush on the commit path. The transaction's commit record must be written to the WAL and the WAL must be flushed through that record before commit can return successfully. The data pages need not be written at commit, but the thread cannot begin a causally subsequent operation that depends on successful commit until the WAL flush completes.

Performance consequences

With strict ACID durability, a successful WAL commit must wait for a flush that covers its commit record. If transactions are serialised, this puts one completed persistence barrier between successive transactions. Serialization may arise because one thread performs the transactions in sequence, because a later transaction causally depends on the earlier durable result, or because transactions contend on the same pages or other shared state. The achievable transaction rate is then bounded by flush latency, irrespective of the WAL's sequential transfer rate.

Group commit changes this only when the system has multiple transactions concurrently ready to commit and is allowed to hold them at the durability boundary together. It does not improve a genuinely serial stream in which the next transaction cannot proceed until the preceding commit is known to be durable. Asynchronous commit removes the wait only by relaxing the D in ACID. These are direct consequences of the force-at-commit design used by a synchronously durable WAL system.

WAL has further costs:

Every change is written to the WAL and later written again in its data pages. In the basic case this means writing the changes twice, so effective write performance can be roughly halved. Log framing, page granularity, indexes, and check pointing can increase the write amplification further.
The page writer must continually respect the write-ahead invariant, introducing coordination between WAL progress, dirty-page eviction, and check pointing.
In-place page writes can be scattered. Lazy writers and page scheduling reduce the cost, but cannot make them equivalent to a single append stream. For typical HDDs and SSDs, the product of sequential bandwidth and I/O latency is commonly between about 64 kB and several megabytes. This latency-equivalent transfer size is usually much larger than a database page, so writing individual dirty pages tends to be dominated by I/O latency rather than transfer bandwidth. Sorting, batching, and concurrent I/O mitigate the effect, but require additional buffering and coordination and cannot make scattered page writes equivalent to the LSS append stream.
In-place allocation makes variable-sized values awkward, particularly strings, which are extremely common in database records. A page representation must reserve a fixed amount of space, impose a size limit, move records when their strings grow, or store the string data separately. Separate storage introduces indirection and harms clustering, so reading one logical record can require access to data held elsewhere. When an object shrinks, its old allocation can leave unused space within the page. More generally, in-place update favours keeping objects at their previous physical locations, making compaction difficult and causing free space and live data to become fragmented over time.
WAL is principally a recovery mechanism for in-place updates, which is the opposite of the natural requirement of Multi-Version Concurrency Control (MVCC): old versions must remain available while readers can still see them. WAL-based systems can implement MVCC, but must compensate using mechanisms such as undo records, copied tuples, version chains, and vacuum or garbage collection. These add indirection, write amplification, and reclamation work. If MVCC is a primary design goal, an in-place update architecture is therefore a poor starting point.

The durability barriers are implemented using the mechanisms described in Write ordering on storage devices. Volatile drive caches do not normally need to be disabled, but every layer of the storage stack must honour the barrier.

Log structured store

The CEDA LSS writes new and changed serial elements to the end of the log. The log is the persistent representation of the data; there is no separate set of in-place data pages to which every change must later be copied. Writes are large and sequential, serial elements can vary in size, and related values can be clustered together. Write performance is therefore typically limited only by the maximum sequential transfer rate of the storage device. The LSS performance measurements demonstrate this behaviour in practice.

The append model is also naturally compatible with MVCC. Updating an element creates a new rendition without requiring the previous rendition to be overwritten immediately. Readers can continue to use an older rendition while it is visible to them, and obsolete renditions can be reclaimed later by the cleaner. Cleaning copies live renditions out of sparsely used segments and recovers whole segments, so object shrinkage and relocation do not leave permanent holes at old physical locations.

Transaction path

Ordinary LSS transaction close does not flush the log. Transactions are serialised and atomic, and a subsequent transaction on the same thread can begin immediately, without waiting for a storage device. The lazy flusher advances durability independently in large sequential units. A failure can therefore lose a recent suffix of closed transactions, but recovery retains an atomic, internally consistent prefix.

This deliberate separation of transaction close from non-volatile durability is central to LSS performance. It allows the CEDA LSS to process about one million serial transactions per second using one thread. A strictly durable WAL transaction cannot follow the same path: its commit cannot return until its WAL record has crossed a persistence barrier. Comparing ordinary LSS transaction throughput with synchronous WAL commit throughput is therefore also a comparison of their intentionally different durability contracts, not merely their file layouts.

FlushWhenClose() is an exceptional escape hatch for code that must coordinate an LSS transaction with an external durable event. One example is two-phase commit, a blocking protocol that is expensive and impractical, and whose coordination and forced-durability costs make it a poor fit for the normal LSS model. FlushWhenClose() exists to make such exceptional integration possible, not to recommend it. When requested, it necessarily sacrifices the ordinary fast path and waits for the transaction and all preceding transactions to become durable.

Check-point publication

Check pointing is also outside the ordinary per-transaction path. During recovery the LSS validates the segment containing the check point referenced by the newest root-block division. If the segment is not valid through the recorded position, recovery rejects that division and reverts to the other root-block division. This makes an out-of-order root-block write recoverable while the preceding division and its reachable data remain intact.

The preferred publication protocol first makes the new check point's dependent log data durable, then writes and makes durable the root-block division that publishes it. This avoids unnecessary fallback and preserves the intended ordering without putting a flush into every transaction:

Write all data on which the new check point depends.
Issue and wait for a durability barrier covering those writes.
Write the root-block division that publishes the check point.
Issue and wait for a durability operation covering the root-block division.

The current RAS API has no explicit durability operation. Other Proposals proposes one for correct check-point publication and for the exceptional FlushWhenClose() path. CRCs, flush sequence numbers, and alternating root-block divisions allow recovery to reject many torn or inconsistent states, but do not themselves impose persistence order.

9 Check pointing

The LSS internally uses a hierarchical map to track the physical locations of all serial elements in the store. It also maintains information about the current utilisation of all the segments.

All this information is itself written to the root block or the log, but only during a check point.

A check point is performed at the following times

After the store is opened and a recovery scan is required
After writing 128 segments (i.e. 64MB) of data to the end of the log
When the store is closed.

A recovery scan is performed when the store is opened and it was found that it had not previously been closed gracefully. In that case it has to scan all segments in the end of the log since the last check point. This allows it to recover all committed transactions and roll back any uncommitted transactions.

The maximum time for the recovery scan is bounded by the time taken to read the segments at the end of the log since the last check point. The time to read 128 segments only takes a few seconds on a modern hard-disk.

10 Cleaning

A background thread that cleans segments with a poor utilisation is started automatically. This defragments the store.

When the store is created or opened, the cleaner threshold can be specified. This defaults to 85% meaning that all segments that are less than 85% utilised will be cleaned.

The cleaner is provided an ordered list of segments to clean after each check point.

11 Lazy writer

The LSS uses a background thread called the "Lazy writer" to write dirty segments in the segment cache to disk. Therefore the thread that opens a transaction and writes some serial elements will typically only write the byte stream to a buffer in memory, allowing it to complete the transaction very quickly without blocking on I/O.

It is important to understand that ending a transaction implicitly commits it, but only in the sense of defining an atomic unit of work. The actual data is written to disk in the background.

When writing larges amounts of data, the segment cache can become full of dirty segments, and in that case it is possible for the thread performing a transaction to block on I/O.

12 Clustering

The easiest way to achieve good read performance is to partition a very large database into mutually exclusive groups of serial elements, where each group is characterised as follows

The serial elements in a group are all closely related, meaning that when one serial element is read from disk, it is likely that other serial elements in the group will also be read in the near future.
There is a tendency for the serial elements in a group to be written to the same segment. This is achieved by writing batches of related serial elements to the LSS at a time. As a corollary to this requirement, a group of related serial elements can't be so large that it defeats the whole idea of them being "clustered together".
Every serial element in the group shares the same high 32 bits of the Serial element Id (Seid). This allows the LSS to achieve clustering in its internal hierarchical map used to track the physical locations of serial elements.

Over time there is an "increasing entropy" effect where related serial elements become spread around the disk. It can be very beneficial to recluster serial elements, particularly serial elements used to implement directory structures. This is achieved by occasionally rewriting all the relevant serial elements to the LSS in a single "batch".

Interestingly, it is the excellent write performance of the LSS, that makes it economical to recluster related data together. Therefore the LSS can also provide very competitive read performance. Experiments have shown it to significantly outperform NTFS when reading or writing thousands of text files.

13 MSSN

MSSN stands for Missing Shutdown Sequence Number. This is a value stored in the root block of the LSS. It equals the number of times that the store has not been gracefully shutdown over its life. This is zero if the store has always been properly closed (so that the store is check pointed and flushed correctly). A large value indicates that there have been many power failures or the client software is not closing the LSS correctly.

The MSSN is useful for correctly mapping Seids to a larger global address space that encompasses objects stored on many computers. Single user applications of the LSS have no need for the MSSN other than a simple diagnostic.

When the store is first created the MSSN is initialised to zero. The value persists and is only incremented during recovery if it is found that the store wasn't previously shut down gracefully.

14 Reading serial elements

It is not necessary to open a transaction in order to read serial elements. In fact multiple threads can concurrently read serial elements. This corresponds to the shared read access mode often provided by high performance database systems. Furthermore, multiple threads can be reading serial elements concurrently with a running transaction! In other words, the LSS provides no mutex between the thread writing new data to the store, and the threads that are reading existing serial elements.

It is therefore up to the layers above the LSS to make sure concurrent reading and writing is a valid thing to do. For example if there is some sense in which the database is logically partitioned into "separate worlds" then concurrent reading and writing can make sense in certain cases. For example, while one "world" provides exclusive write access, another "world" can support shared read access.

Reading a serial element may block on I/O as the LSS loads a segment from disk. A very large serial element may take up many segments. These are loaded on demand as the serial element is read as a byte stream.

The serial element is read as a non-seekable byte stream. I.e. the entire serial element must be read from start to finish. It provides a reliable "End of stream" indicator.

15 Writing serial elements

To write a serial element, call the following function on a ILssTransaction:


    ICloseableOutputStream* WriteSerialElement(Seid seid);

This function is used to write new serial elements and also to re-write existing serial elements.

It returns an output stream that can be used to write the content as a byte stream. Note that the entire serial element must always be written. Then the output stream must be closed. The interface pointer must not be deleted.

It is very important to close the output stream before calling WriteSerialElement() again to write another serial element, and also before closing the transaction.

Only the thread that opened the transaction is allowed to write a serial element.

Example

The following code either opens an existing store and reads the root serial element or creates a new store if it doesn't exist and writes the root serial element. Then the store is closed. For simplicity error handling is not shown.


#include "Ceda/cxLss/ILogStructuredStore.h"
#include <assert>
#include <iostream>

void WriteRootSerialElementExample()
{
    const char* path = "myfile";

    bool createdNew;
    ceda::LssSettings settings;
    ceda::ILogStructuredStore* lss = ceda::CreateOrOpenLSS(
        path,
        nullptr,
        createdNew,
        ceda::OM_OPEN_ALWAYS,
        settings);
    if (createdNew)
    {
        // Create a 32 bit Seid space for all Seid allocations
        ceda::SeidHigh seidHigh = lss->CreateSeidSpace();

        // Allocate Seid for root object
        ceda::Seid seid = lss->AllocateSeid(seidHigh);
        assert(seid == ceda::ROOT_SEID);

        const char* buffer = "Hello world";

        ceda::ILssTransaction* txn = lss->OpenTransaction();

        // Write root object
        ceda::ICloseableOutputStream* os = txn->WriteSerialElement(seid);
        os->WriteStream(buffer, strlen(buffer)+1);
        os->Close();

        txn->Close();
    }
    else
    {
        ceda::Seid seid = ceda::ROOT_SEID;

        const int BUFSIZE = 100;
        char buffer[BUFSIZE];

        // Read root object
        ceda::ICloseableInputStream* is = lss->ReadSerialElement(seid);
		ceda::ssize_t numBytesRead = is->ReadStream(buffer, BUFSIZE);
        is->Close();

        std::cout << "Read: " << buffer << '\n';
    }

    lss->Close();
}

16 Seid allocation

Serial element Ids

Each serial element is uniquely identified by a 64 bit number called a Seid. A serial element cannot change its Seid over time.

Each time a serial element is written to the LSS, it must be rewritten in its entirety - even if only a small part of its content changes. It will in fact be written to a completely new location within the store - i.e. at the "end of the log".

If that seems particularly wasteful then the serial elements should be finer grained. Although it is intuitively better to have quite small serial elements to minimise the number of bytes to be written to disk when changes are made, it can often be better to use fairly coarse serial elements because that is an easy way to ensure clustering on disk is reasonably good. It also reduces the various space and time overheads associated with a serial element, such as the need to index its physical location. Note finally that the product of transfer rate and seek time for a modern hard-disk is quite large - of the order 512k, so it is not efficient to write lots of small objects to disk if they can become poorly clustered over time.

When a new serial element is to be written to the LSS for the first time, it is necessary to allocate a fresh Seid for it. According to the above section on clustering, the programmer needs to have some concept of partitioning the serial elements into groups that will be clustered on disk. Therefore Seid allocation involves two separate calls.


// Create a new Seid space.  This is the upper 32 bits of the Seids shared by a group of
// related serial elements that should be clustered together on disk.
// Never returns 0.
virtual SeidHigh CreateSeidSpace() = 0;

// Allocate a new, unused Seid within the Seid space associated with the given SeidHigh.
// Never returns a null Seid.
virtual Seid AllocateSeid(SeidHigh seidHigh) = 0;

CreateSeidSpace() is used to create a high 32 bit part of a Seid, giving a "Seid space" allowing for up to 4 billion "related" serial elements. Each serial element within this cluster is allocated a Seid with a call to AllocateSeid().

In practice there shouldn't be anywhere near 4 billion serial elements in a single "cluster" because that would defeat the whole idea! The "sweet spot" for a cluster is of the order of a few mega bytes.

It is permissible to make these calls outside of an LSS transaction! Any number of threads can concurrently allocate Seids because the above Seid allocation functions are thread-safe.

Allocation of affiliate Seids

The following is an alternative Seid allocation function


   bool AllocateAffiliateSeid(Seid& seid);

This is useful for very large sets of serial elements where it is difficult to know a priori how to divide the serial elements into separate Seid spaces.

Before this function can be used to allocate Seids, it is first necessary to "bootstrap" by calling CreateSeidSpace() to allocate a Seid space, then AllocateSeid() to allocate a root serial element in the space. Then, instead of calling AllocateSeid() to allocate additional serial elements, it can be preferable to call AllocateAffiliateSeid(). The Seid passed into the function represents an "affiliate" Seid to which the new Seid returned by the function will be clustered.

Eg the affiliate may be a parent node in a tree of nodes. AllocateAffiliateSeid() does a good job of allocating Seids for trees of nodes which grow over time from any position by adding child nodes.

Boot strapping a store

It is common for a serial element to store (within its byte stream content) the Seids of other serial elements. For example. these could represent the "children" in a whole-part hierarchy of objects.

Typically an application using the LSS will need to write some sort of "root" registry or directory object to the store with a known Seid. This is the starting point for accessing all other objects in the store.

ROOT_SEID is the Seid for this root serial element.

Just after creating a new LSS, it is guaranteed that the first call to CreateSeidSpace() will return the high 32 bits of ROOT_SEID. It is then guaranteed that the first call to AllocateSeid() (passing in the high 32 bits of ROOT_SEID) will return ROOT_SEID.

17 Durability

Durability relates to the 'D' in the so called ACID properties typically expected of OO and relational database systems.

ACID properties are particularly relevant to the management of data that relates directly to real world processes such as airplane reservation systems, or financial systems. In these cases, a transaction on a computer is associated with events in the real world. For example, when a client withdraws cash from an ATM. Clearly it is necessary for the database to correctly record all such withdrawals. This leads to the durability requirement. In practice this means that every transaction must be flushed to disk as part of the commit.

Ordinary storage devices can retain write caching while supporting explicit durability barriers. The operating system translates a file flush into the appropriate device cache flush or FUA operation. The barrier is nevertheless expensive compared with an in-memory transaction: a thread executing serial durable transactions must wait for one before beginning an action that depends on the preceding commit. Write ordering on storage devices describes these mechanisms and the conditions under which they can be trusted.

The same limit applies when transactions are serialised by contention rather than by the application thread—for example, when they repeatedly update the same pages or shared state. A synchronously durable WAL system must complete a flush covering each commit before the next dependent commit can advance. Without a set of concurrent commits to batch, storage-barrier latency directly limits TPS.

On a rotating HDD, a forced flush can wait for cached writes, head movement, and media rotation. The durable transaction rate is bounded by the complete write-and-barrier latency, which varies with the drive and workload. A 10 ms barrier permits at most 100 serial durable transactions per second, before any transaction-processing cost is included. Barrier latencies in this range place a rotating HDD in the same general territory as the 50 to 70 flushes per second historically observed on some 5400 and 7200 RPM IDE drives.

An SSD removes mechanical seek and rotational delays, but does not make a durability barrier free. A synchronous request still crosses the operating system, filesystem, driver, controller, and device firmware, and may need to drain volatile state and complete NAND or mapping-metadata updates. Latency varies greatly between SATA SSDs, ordinary NVMe SSDs, devices with power-loss protection, and specialised low-latency media. Measurements in Asynchronous I/O Stack: A Low-latency Kernel I/O Stack for Ultra-Low Latency SSDs show that even when device latency has fallen to tens of microseconds, operating-system overhead is a material part of a synchronous write()+fsync() path.

The arithmetic illustrates the limit independently of any particular device:

Complete durability-barrier latency	Maximum serial durable transactions per second
10 ms	100
1 ms	1,000
100 microseconds	10,000
10 microseconds	100,000

In practice, a rotating HDD supports only tens to low hundreds of serial forced-durability operations per second. Stock SATA and NVMe SSDs can raise this to hundreds, thousands, or in favourable cases low tens of thousands per second, with large differences between devices, filesystems, workloads, and power-loss-protection arrangements. These are barrier rates rather than complete transaction rates; transaction processing reduces the achievable result further. SSDs therefore raise the ceiling substantially, but a forced flush still destroys the ordinary LSS transaction-performance model.

By contrast the LSS focuses on the management of data that doesn't (in a transactional sense) relate to real world processes. Examples are editing of text documents, spreadsheets, statistical analysis, web browsing, GIS, multimedia databases, source code repositories and CAD. In these cases the durability constraints can be relaxed a little - by only flushing transactions to disk every few seconds. Atomicity is still required to protect the integrity of the data. However, a transaction only "commits" in the sense of defining an atomic unit of work, rather than demanding it go to non-volatile storage as part of the commit.

This of course means that a user may lose some edits on system failure, but losing at most a few seconds of work is fine for the type of data managed by the LSS.

The exceptional FlushWhenClose() mode allows an LSS transaction and all preceding transactions to request durability before close returns. It exists for external coordination protocols such as two-phase commit. Two-phase commit is blocking and imposes coordination and forced-durability costs that make it expensive and impractical. Those costs conflict with the normal LSS performance model; its mention here is not a recommendation. Normal transactions retain the high-throughput path described above. The current RAS interface still needs an explicit non-volatile-storage operation before the exceptional mode can make that guarantee reliably on every supported platform.

18 Backup and hot standby for the LSS

The LSS optionally supports hot standby and incremental backup.

Note however that at present, hot standby has fairly relaxed assumptions about how up to date the standby store must be.

The backup / hot-standby system is compatible with 24x7 operation of a store which continuously reads/writes large amounts of data. The LSS properly supports applications that are write bound for prolonged periods.

When the LSS is created or opened, a path to a directory for the delta files can optionally be provided. LSS delta-files will automatically be written to this directory. These files are named as follows

    nnnnnn.lssdelta

where nnnnnn is a sequence number, called a Check Point Sequence Number (CPSN).

Note that the path to the main LSS file is independent of the path to the directory of delta-files. They could easily be on different hard-disks.

To avoid limiting the write performance of the system, it is recommended that a separate local hard-disk be used for storing the deltas. This will allow the deltas and the LSS file to be written concurrently. It also means either hard-disk can fail without losing data.

Note that with virtual file systems it is easy to have delta files written directly to a remote site. However that may expose the LSS to network outages. A better strategy may be to write deltas to a local hard-drive, and a separate process is responsible for copying these files to a remote site. During network outages the application is able to continue running.

A single delta-file is written for each check point on the LSS. The LSS stores a CPSN in the main LSS file. This helps ensure that deltas are applied in the right sequence. The CPSN directly corresponds to the sequence number used for naming the delta files.

With the default settings the LSS performs a check point after writing 128 segments (or 64 MB). A check point is also performed whenever the store is closed.

Delta files respect check point boundaries. Note in turn that check point boundaries respect both flush unit and transaction boundaries.

Snapshot records in delta files contain timestamps. When deltas are applied, a maximum timestamp can be specified to reconstruct the latest complete transaction at or before that time.

Check point identifiers

The LSS generates a 128 bit GUID called a Check Point Identifier (CPID) for each check point. The current CPID is stored in the root block of the LSS. Each delta-file stores an input CPID and output CPID. A delta file may only be applied if its input CPID matches the current CPID of the store. The store's CPID is then set to the output CPID defined by the delta-file.

Both the CPID and CPSN are used to validate a delta-file (i.e. to see whether it is allowed to be applied to the store). Note that two stores can share a common ancestry, then diverge. The use of CPIDs ensure that delta-files are never applied incorrectly.

Note that the first CPSN to be applied isn't specified on the command line to LssApplyDeltas.exe. Instead the LSS can work this out itself (because it stores the current CPSN in its root block). This, together with the CPID validation makes LssApplyDeltas.exe "idiot proof".

Hot standby configuration

As long as the main application is not running, (and the main LSS file is not opened) it can be copied using the file system. This creates a "level 0 backup". The copy will of course have the same CPID and CPSN, recorded in the root block. However, there are two significant problems with making a complete copy of the store

If the store is large then it can take a long time
The application that reads/writes the store can't be running while the copy is made.

Once a copy has been made the delta-files can be used to very efficiently and safely bring the copy into sync with the main store.

Consider that we have previously created a "standby" store (by making a file system copy). Let the 24x7 application be configured to automatically create the delta-files in the normal way. Let LssApplyDeltas.exe be run repeatedly so it applies deltas to the "standby" as soon as they become available. At quiescence the standby will match the main store.

Note that this process is compatible with 24x7 operation of the main store (because it never needs to be shut down).

It is easy to create any number of "standby" stores in various stages of how up to date they are, because deltas are not consumed when they are applied to a store. Furthermore the standby stores and the deltas can be backed up to tape etc. Therefore this approach provides a great deal of flexibility.

19 LssCompare.exe

This console application can be used to compare two LSS stores to see if they are (logically) equal - i.e. as a mapping from OID to byte stream.

Command line:


    LssCompare path1 path2

where path1 and path2 are paths to two different LSS files to be compared.

This is useful for validating the backup system.

20 LssApplyDeltas.exe

A console application called LssApplyDeltas.exe is able to apply deltas to an existing LSS store, called a "level 0", to bring it more up to date.

On the command line two or three arguments may be specified...


    LssApplyDeltas level0path deltasDirPath [cpsn2]

Argument 1
    level0path      The path to an existing LSS store, called the "level 0"

Argument 2
    deltasDirPath   the path to the directory containing the delta-files

Argument 3 [optional]
    cpsn2           A "one past end" value of the cpsn, to specify what delta files should
                    be applied to the level 0.

                    The half open interval [cpsn1, cpcn2) is applied.  cpsn1 is determined
                    automatically from the level 0 file.  Note that delta files are applied up
                    but not including cpsn2.

LssApplyDeltas can be passed the cpsn2 parameter to limit the number of deltas to be applied. Currently this is only at the coarse granularity of check-point boundaries. [In the future it is expected that it will also be possible to specify a date/time stamp for more precisely controlling what transactions are applied]

If the delta files directory contains the delta files from 0 onwards, and there is no level 0 LSS file, then LssApplyDeltas.exe will actually create a level 0 from the delta files

The LSS always uses the extension "partial" for the current delta file being written. This is renamed with the extension "lssdelta" after the delta file has been completed. It is assumed that this approach is sufficient to ensure that LssApplyDeltas won't apply a partially written delta file.

LssApplyDeltas is idiot proof in that it will never apply an inappropriate delta.

21 CEDA LSS API

The LSS (Log Structured Store) is a persistent store for arbitrary sized binary objects, referred to as serial elements.

The LSS is supported on all flavors of 32 bit and 64 bit Windows from Windows 95 onwards. The store is written to the hard-disk (which could be FAT32 or NTFS) as a single file. This file grows as required to accommodate new data written to the store.

Public header

The API is defined in the header ILogStructuredStore.h, here it is reproduced without all the comments and other distractions and simplified slightly:

Simple example

The serial elements are written to the store within a transaction, and the LSS ensures atomicity of each transaction. i.e. all changes made to the store by a transaction are applied or else none are applied. For example, a transaction could fail to commit because of a power failure. The next time the store is opened, any uncommitted transactions are rolled back. This "recovery scan" is performed automatically whenever the store is opened. The time for a recovery scan is bounded, and on typical hardware will never take longer than a few seconds.

Serial elements are read or written as a byte stream, in a manner similar to the C functions fread() and fwrite(). The store can deal efficiently with very small and very large serial elements. Assuming compacting is good, the overhead is of the order of 20 bytes per object.

The LSS achieves excellent write performance, typically limited only by the maximum transfer rate of the hard-disk. Disk head seeks during writing of data are kept to a minimum by writing new data to the end of the log using large segments. By default segments are 512 kbyte.

When the LSS is opened, a background thread is automatically started that cleans segments with a poor utilisation (i.e. below a preset threshold). The data on a segment to be cleaned is written to the end of the log, allowing the segment to be returned to an internal free segment pool. Because of this, users of the LSS never need to concern themselves with "fragmentation" of the store.

However, a user of the LSS needs to be concerned with clustering related data together, in order to achieve maximum read performance. This is essentially achieved by writing related data close together in time (so the related serial elements tend to be written to the same segments). Note that rewriting individual serial elements over time has the effect of upsetting the clustering. Reclustering simply involves rewriting a collection of related serial elements to the end of the log. The background cleaner thread will automatically defragment the store.

It is important to note that the LSS is not concerned with concurrency control on access to the serial elements. It certainly doesn't provide strict two phase locking, or any other locking protocol to enforce serialisation of transactions. Instead, it assumes that a layer above the LSS is responsible for concurrency control.

Serial elements are identified by a 64 bit Seid (Serial element identifier). The LSS provides a mechanism for allocating new, unused Seids as required. The number of serial elements is actually limited by the maximum size of the store which is about 500 TB, rather than the size of the 64 bit Seid space.

API

The following document the CEDA LSS API:

CreateOrOpenLSS()
LssSettings
ILogStructuredStore
ReadOnlyBuffer
IContiguousSerialElement
ILssTransaction
LssStats
LssDumpSettings

22 CreateOrOpenLSS

Create or open a Log Structured Store, using the given path and LssSettings. Subsequent access to the store is through the ILogStructuredStore interface.

Note that the underlying file is opened with exclusive read/write access. It is not possible for another process (say) to open the same LSS file.

The EOpenMode parameter provides a number of options for opening and creating LSS stores. If errors occur due to a file's existence when it's not expected or absence when it's expected will result in a FileException being thrown.

If lssPath = "memfile", then the LSS will reside in memory.

createdNew returns a flag for whether a new store was created. If the openMode = OM_CREATE_ALWAYS, OM_DELETE_EXISTING or OM_CREATE_NEW any existing files will be deleted and createdNew will be true.

Note that if an existing store is opened and the existing segment size doesn't match the segment size specified in the settings, then the requested segment size will be ignored.

If an error occurs then throws a FileException (see FileException.h) or CorruptLSSException.

After using the store it must be closed by calling the Close() method. The interface pointer must not be deleted.

If deltasDirPath is not nullptr then it specifies the path to a directory in which to create delta files.

CreateOrOpenLSS() never returns nullptr


ILogStructuredStore* CreateOrOpenLSS(
    ConstStringZ lssPath,
    ConstStringZ deltasDirPath,
    bool& createdNew,
    EOpenMode openMode,
    const LssSettings& settings);

Example

The following code either opens an existing store or creates a new one if it doesn't exist. Then the store is closed. For simplicity error handling is not shown.

With the settings below, 4MB segments are used. This is the unit of reading and writing from disk. The segment size cannot be changed for existing stores.

There are 64 segments in the cache. This means the cache takes up 64 x 4MB = 256MB of system memory.


#include "Ceda/cxLss/ILogStructuredStore.h"

void OpenThenCloseLssExample()
{
    const char* path = "myfile";
    bool createdNew;
    LssSettings settings;
    settings.segmentSize = 4*1024*1024;
    settings.maxNumSegmentsInCache = 64;
    settings.flushTimeMilliSec = 2000;
    ILogStructuredStore* lss = CreateOrOpenLSS(
        path,
        nullptr,
        createdNew,
        OM_OPEN_ALWAYS,
        settings);
    if (createdNew)
    {
        std::cout << "A new store was created";
    }
    else
    {
        std::cout << "An existing store was opened";
    }
    lss->Close();
}

23 LssSettings

Various settings that are used to initialise an LSS when it is created or opened with a call to CreateOrOpenLSS().


struct LssSettings
{
    int flushTimeMilliSec;
    double cleanerUtilisationPercent;
    bool enableFileBuffering;
    bool enableWriteThrough;
    int maxNumSegmentsInCache;
    int numSegmentsPerCheckPoint;
    int segmentSize;
    bool forceIncrementMSSN;
    bool validateSUTDuringCheckPoint;
};

`int flushTimeMilliSec`

Maximum time to flush the log after committing a transaction

Default value : 1000

`double cleanerUtilisationPercent`

If utilisation falls below this threshold then the segment is cleaned

Default value : 85.0

`bool enableFileBuffering`

If set then the Win32 file cache will be used. Typically not required because the LSS performs its own buffering, with its segment cache. If enableFileBuffering is false then CreateFile() is called with FILE_FLAG_NO_BUFFERING

Default value : false

`bool enableWriteThrough`

If enableWriteThrough is true then CreateFile() is called with FILE_FLAG_WRITE_THROUGH

Default value : false

`int maxNumSegmentsInCache`

Maximum number of segments in the segment cache. With default values segment cache is 32 x 512kB = 16 MB.

Default value : 32

`int numSegmentsPerCheckPoint`

Sets the "rate" at which the store is check pointed. With the default values a check point is performed after writing 128 x 512kB = 64 MB to the log. This controls the maximum time taken to perform a recovery scan. For a modern hard-disk, it only takes about one second to read 64MB. Performing check points rarely has the advantage of writing less "meta data" to the log, and ensuring that the meta data is well clustered. It also means the root block is written less often.

Default value : 128

`int segmentSize`

Size of each segment - the unit of reading from disk. If too small, then performance becomes dominated by the head seek and rotational delay times of the hard-disk. If too large then performance becomes overly dependent on the clustering of related data. As a very rough guide, should equal the product of the maximum transfer rate of the hard-disk in bytes per second, times the seek time in seconds. Eg for transfer rate = 50 MB/sec, seek = 10 msec then product = 500k See Chunk size for a broader discussion of the relationship between device latency, bandwidth and an appropriate transfer size.

Default value : 524288 (512kB)

`bool forceIncrementMSSN`

Force increment of the MSSN during start up

Default value : false

`bool validateSUTDuringCheckPoint`

For debugging purposes only

Default value : false

24 ILogStructuredStore

Any of the methods in the interface can throw either of the following types of exception

CorruptLSSException: Indicates a serious error with the store. In theory this should never happen unless there has been file corruption.
FileException: Indicates a low level file I/O error.


struct ILogStructuredStore
{
    virtual void Close() = 0;
    virtual void SetMaxNumSegmentsInCache(int n) = 0;
    virtual MSSN GetMissingShutdownSeqNum() const = 0;

    // Serial Element Ids (Seids)
    virtual SeidHigh CreateSeidSpace() = 0;
    virtual Seid AllocateSeid(SeidHigh seidHigh) = 0;
    virtual bool ReserveSeid(Seid seid) = 0;
    virtual SeidLow PeekNextSeidLow(SeidHigh seidHigh) = 0;
    virtual bool AllocateAffiliateSeid(Seid& seid) = 0;
    virtual void GetSeidsInSeidSpace(
        xvector<SeidLow>& seidLows,
        SeidHigh seidHigh) const = 0;

    // Read serial elements
    virtual bool SerialElementExists(Seid seid) const = 0;
    virtual ICloseableInputStream* ReadSerialElement(Seid seid) const = 0;
    virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;

    // Transactions
    virtual ILssTransaction* OpenTransaction() = 0;

    // Throttle control on a "producer"
    virtual void BlockUntilLowWaterMark() const = 0;
    virtual bool ReachedHighWaterMark() const = 0;

    // Diagnostics support
    virtual void GetStats(LssStats& stats) const = 0;
    virtual void DumpLSS(
        LssStats& stats,
        xostream& os,
        const LssDumpSettings& ds) const = 0;
    virtual bool RecurseSeidMap(
        xvector<Seid>& children,
        Seid seid = BEGIN_RECURSE_SEID_MAP,
        bool enableOverflowPackets = false) const = 0;
    virtual bool WriteInfoOnNodeForGivenSeid(Seid seid, xostream& os) const = 0;
};

`void Close()`

The LSS must be explicitly closed, even if exceptions have been thrown by the LSS It is an error to close the LSS while reading a serial element, or a transaction is open.

`void SetMaxNumSegmentsInCache(int n)`

Set the maximum number of segments in the segment cache. Note that the one segment cache serves as both a read and write cache. n must be at least 4.

`MSSN GetMissingShutdownSeqNum() const`

Returns the number of times the store has not been gracefully shut down over its entire life

Serial Element Ids (Seids)

Serial elements in the store are uniquely identified by a Serial Element Identifier (Seid).

`SeidHigh CreateSeidSpace()`

Create a new Seid space. This is the upper 32 bits of the Seids shared by a group of related serial elements that should be clustered together on disk. This function is threadsafe - i.e. multiple threads can safely call this function Never returns 0.

`Seid AllocateSeid(SeidHigh seidHigh)`

Allocate a new, unused Seid within the Seid space associated with the given SeidHigh. This function is threadsafe - i.e. multiple threads can safely call this function Never returns a null Seid.

`bool ReserveSeid(Seid seid)`

Ensures the seids in { Seid(low,high) | high == seid.high_ && low <= seid.low_ } are reserved. Returns false if that range of seids was already reserved.

`SeidLow PeekNextSeidLow(SeidHigh seidHigh)`

Returns the next available SeidLow for the given SeidHigh, without actually performing an allocation. Not compatible with using AllocateAffiliateSeid().

`bool AllocateAffiliateSeid(Seid& seid)`

The Seid passed by reference serves as both an in and out parameter to the function. This function allocates a fresh Seid (the out-parameter) that is "affiliated" with an existing Seid (passed as the in-parameter).

Consider that a new Seid needs to be allocated, and the new serial element should be clustered with some other existing serial element, called the "affiliate". Eg the affiliate may be a parent node in a tree of nodes. AllocateAffiliateSeid() does a good job of allocating Seids no matter the order in which nodes are added to the tree.

This function is threadsafe - i.e. multiple threads can safely call this function

`void GetSeidsInSeidSpace(xvector<SeidLow>& seidLows, SeidHigh seidHigh) const`

Retrieve all the Seids (actually only the low 32 bit part of each Seid) in the Seid space associated with the given SeidHigh. seidHigh must not be zero.

Read serial elements

`bool SerialElementExists(Seid seid) const`

Does a serial element with the given Seid exist? The given Seid must not be null This function is threadsafe - i.e. multiple threads can safely call this function

`ICloseableInputStream* ReadSerialElement(Seid seid) const`

Provides a stream for reading the serial element with the given Seid. The given Seid must not be null

The returned stream must be closed after it is used (including when exceptions are thrown by the LSS).

Returns nullptr if no serial element exists with the given Seid

It is an error to call this function on a serial element that is currently opened for writing (within a transaction), or being deleted using a call to DeleteSerialElement().

Shared reading of serial elements is supported. I.e. any number of threads can independently (and concurrently) read the same serial element, assuming each such thread has made an independent call to ReadSerialElement() - i.e. they don't try to share a returned ICloseableInputStream.

`IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const`

Provides an alternative to ReadSerialElement() for reading a serial element as a contiguous block of memory. Obviously this function shouldn't be called for very large serial elements that don't fit in physical memory and therefore would result in page faulting.

The given Seid must not be null

The returned IContiguousSerialElement must be closed after it is used (including when exceptions are thrown by the LSS).

Returns nullptr if no serial element exists with the given Seid

It is an error to call this function on a serial element that is currently opened for writing (within a transaction), or being deleted using a call to DeleteSerialElement().

Shared reading of serial elements is supported. I.e. any number of threads can independently (and concurrently) read the same serial element

Transactions

`ILssTransaction* OpenTransaction()`

All changes (i.e. mutative work) done on an LSS (apart from Seid allocations) must be done by a thread that has opened a transaction.

A transaction is intended for a single thread. Only the thread that called OpenTransaction() on the LSS is permitted to call the methods on the returned ILssTransaction

It is an error to close the LSS while there is an open transaction. The LSS internally uses a mutex to ensure that only one thread opens a transaction at a time.

There is no concept of aborting or rolling back a transaction.

Throttle control on a "producer"

`void BlockUntilLowWaterMark() const`

Blocks until it is appropriate for the producer to begin writing changes to the LSS again (because the LSS has written enough segments out to disk).

`bool ReachedHighWaterMark() const`

Returns immediately, and indicates whether the producer has written enough changes to the LSS such that it should call BlockUntilLowWaterMark() in order to wait for the LSS lazy writer to "catch up"

Diagnostics support

`void GetStats(LssStats& stats) const`

Retrieve stats about the LSS in an LssStats variable.

`void DumpLSS(LssStats& stats, xostream& os, const LssDumpSettings& ds) const`

Write information about all the segments, flush units and packets in the entire store to the given output stream according to the flags in the given LssDumpSettings. This function should only be called on relatively small stores because it will write many mega bytes of text for a large store.

`bool RecurseSeidMap( xvector<Seid>& children, Seid seid = BEGIN_RECURSE_SEID_MAP, bool enableOverflowPackets = false) const`

Can be used to recurse through all the Seids in the store. This is based on the fact that Seids form an 8 level hierarchical map.

RecurseSeidMap() should first be called with a seid equal to BEGIN_RECURSE_SEID_MAP. This will return the children at the top level. Up to 256 children may be returned. A child seid may then be passed in again to a call to RecurseSeidMap(), to recurse down through the Seid map. After doing this 4 times, a SeidHigh will be obtained. After doing this an extra 4 times the Seids of serial elements will be obtained. ReadSerialElement() may then call called to access their serialised state.

enableOverflowPackets determines whether seids of overflow packets will be returned in the iteration of the seids. It is usually appropriate to call with enableOverflowPackets = false

Returns false if the given Seid is invalid

`bool WriteInfoOnNodeForGivenSeid(Seid seid, xostream& os) const`

For the given Seid, writes diagnostic information to the given stream. This includes the total size of the packet in bytes.

25 ReadOnlyBuffer

A ReadOnlyBuffer allows for direct read access to a contiguous serial element


struct ReadOnlyBuffer
{
    const octet_t* buffer;
    ssize_t size;
};

26 IContiguousSerialElement

Interface IContiguousSerialElement provides access to a single serial element recorded in a ReadOnlyBuffer which represents a contiguous buffer in memory.

A contiguous serial element must be explicitly closed, even if exceptions have been thrown by the LSS. It is an error to close the LSS before the closing all the contiguous serial elements that have been opened for reading.


struct IContiguousSerialElement
{
    virtual void Close() = 0;
    virtual ReadOnlyBuffer GetBuffer() const = 0;
};

27 ILssTransaction

A transaction is associated with mutative work on the LSS - i.e. for creating, rewriting or deleting serial elements.

During a transaction it is possible to delete or write serial elements. Writing a serial element encompasses creation of a new serial element as well as rewriting an existing serial element.

There is no concept of clients aborting a transaction. If a thread begins a transaction then it must eventually commit the transaction. As such there is no concept of roll-back during the normal operation of the LSS. Roll-back only occurs during recovery (i.e. when the store is opened when it was not previously closed gracefully)

Only a single thread can open a transaction on a given LSS at a time. This is enforced by a mutex within the implementation of the LSS. The mutex is locked by OpenTransaction() and unlocked when Close() is called on the transaction.

An exception may occur in the middle of a transaction. For example, a call to ReadSerialElement() may fail because of a low level I/O error, throwing a FileException. It is vital that the client still calls Close() even though an exception occurred. Otherwise the mutex will not be closed, and there could be a subsequent dead-lock - such as when the client tries to close the LSS.

Note that when such an internal I/O error occurs, the LSS will enter an "error" state, preventing any transactions from being propagated to disk, even though the transaction is explicitly closed. In other words, closing a transaction normally but doesn't always mean it is being committed.

The best way to ensure correctness in the face of exceptions is to declare an instance of an AutoCloser<ILssTransaction> on the frame in order to perform a transaction on the LSS within a lexical scope.


struct ILssTransaction
{
    virtual void Close() = 0;
    virtual void FlushWhenClose() = 0;
    virtual ICloseableOutputStream* WriteSerialElement(Seid seid) = 0;
    virtual bool DeleteSerialElement(Seid seid) = 0;
    virtual void DeleteSeidSpace(SeidHigh seidHigh) = 0;
};

`void Close()`

A transaction must be explicitly closed, even if exceptions have been thrown by the LSS. Must only be called by the thread that originally opened this transaction. It is an error to close the transaction before the current serial element being written is closed.

If FlushWhenClose() has previously been called on this transaction then Close() will synchronously flush this and all previous transactions on the LSS.

`void FlushWhenClose()`

Puts this transaction into a mode where it will synchronously flush all data written to the LSS when this transaction is closed.

Must only be called by the thread that originally opened this transaction The subsequent Close() will only return after the transaction (and all previous transactions) have been written to disk - at least according to the Win32 calls. Note that the LSS file is opened with "no write through cache". Despite this hard-disks that have their local cache enabled may defeat the assumption that the data is made durable. This could be a problem for a multi-phase commit protocol (for example).

`ICloseableOutputStream* WriteSerialElement(Seid seid)`

Returns a stream to be used to write the serial element with the given Seid. If the serial element already exists then the previous rendition will be replaced by a new one. The returned stream must be closed after it is used. Furthermore, it must be closed before the next call to WriteSerialElement(), DeleteSerialElement(), DeleteSeidSpace() or Close() on this transaction.

Never returns nullptr.

Must only be called by the thread that originally opened this transaction It is an error to call this function on a serial element that is currently opened for reading

For more details see writing serial elements.

`bool DeleteSerialElement(Seid seid)`

Permanently delete the serial element with the given Seid Must only be called by the thread that originally opened this transaction It is an error to delete a serial element that is currently opened for reading or writing Returns false if there is no serial element with the given Seid.

`void DeleteSeidSpace(SeidHigh seidHigh)`

Delete the Seid space associated with the given SeidHigh. The Seid space must be empty - i.e. by calling DeleteSerialElement() as required to delete all serial elements in the Seid space. Must only be called by the thread that originally opened this transaction

28 LssStats

Records summary statistics over the entire store


struct LssStats
{
    int numCheckPoints;       // Total number of check points that have been performed
    int numRecoveries;        // Number of times that the store has performed a recovery
    int64 numTransactions;    // Total number of transactions
    int segmentSize;          // Size of each segment in bytes
    int64 fileSize;           // The current size of the LSS file in bytes. Total size
                              // in bytes taken up by all live packets in the store
    int64 utilisation;
    int numLiveHeadPackets;
    int numLiveOverflowPackets;
    int numLiveRPMPackets;
    int numObsoletePackets;
};

29 LssDumpSettings

Flags passed to the member function DumpLSS() on ILogStructuredStore


struct LssDumpSettings
{
    bool showRecoveryLog;
    bool showRootBlockInfo;
    bool showSeidAllocInfo;
    bool showSUT;
    bool showFlushUnits;
    bool showSegments;
    bool showLivePackets;
    bool showDeadPackets;
    bool showStats;
    bool showCRC;
};

30 LSS Performance

The CEDA LSS achieves read/write performance unmatched by other database technologies. It has been found to outperform BTrieve by a significant factor and yet Btrieve is supposed to be one of the fastest database systems in the world.

A comparison of the write performance of the CEDA LSS to the Oracle BerkeleyDB C++ has been undertaken. The test involved writing a million (key,value) pairs using 1000 atomic transactions on an Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM with a pair of SSDs in RAID0. In both cases the database used a 512MB cache.

With 4096 byte mapped values BerkeleyDB took 2 minutes 20 seconds while the CEDA LSS took 5 seconds.

BerkeleyDB was not space efficient. It wrote over 12GB to disk (more than 3 times the amount of actual data). By contrast the CEDA space overhead was tiny (about 0.3%).

The following table summaries the results

Measure	BerkeleyDB	CEDA LSS
Time	2 minutes and 20 seconds	4 seconds
Num disk read operations	3266	0
Num disk write operations	1009525	988
Num bytes read from disk	23252282	0
Num bytes written to disk	12672112224	4117327872
Disk space overhead factor	3.22	1.0034
Effective write rate	28 MB/sec	980 MB/sec

NuoDB

NuoDB is a memory centric, ACID compliant distributed SQL database. The NuoDB ingestion rate has been measured. With 4096 byte mapped values NuoDB took 12 minutes and 30 seconds on the same i7-4700MQ laptop.

Kyoto Cabinet

Kyoto Cabinet is a high performance database storage engine. The database is a simple data file organized in either a hash table or a B+ tree containing key value pairs which are variable length byte sequences.

The Kyoto Cabinet ingestion rate has been measured on the same i7-4700MQ laptop.

With 4096 byte mapped values Kyoto Cabinet took 19 seconds with a hash table and 8 minutes with a B+Tree. Unfortunately the SSDs on the machine are now fragmented and performance may have been somewhat better when the SSDs were new.

A comparison has also been performed using a 4GB RAM disk, and for 4096 byte mapped values Kyoto Cabinet took over 8 seconds with a hash table (half the rate of CEDA LSS).

On a RAM disk for 4 byte mapped values CEDA B+Tree was 11x faster at writing and 3x faster at reading than Kyoto HashDB, and 42x faster at writing and 9x faster at reading than Kyoto TreeDB.

For 128 byte mapped values CEDA LSS was 4x faster at writing than Kyoto HashDB, and 22x faster at writing and 13x faster at reading than Kyoto TreeDB.

31 CEDA LSS versus Oracle BerkeleyDB

CEDA has a high performance storage engine which features atomic transactions, automatic crash recovery, incremental backup, slave mirroring, optional transaction durability and concurrent reading with writing. It is well suited to being used as an embedded key-value database, as an alternative to products like the Oracle BerkeleyDB.

The following are the results of a comparison of the write performance of the CEDA LSS against the Oracle BerkeleyDB (C++ version).

The test involved writing a million (key,value) pairs using 1000 atomic transactions (not flushed) on a laptop which is an x64 Windows 7 platform with a pair of SSDs in RAID0 (striped). In both cases the database used a 512MB cache. Note that certainly for the CEDA LSS the file was unbuffered (i.e. not using the Windows file system cache) (see the code used for the test).

Results with 4096 byte mapped values

With 4096 byte mapped values the total amount of data to be written to disk is about 4GB. For both databases this is 8 times the size of the cache, so the test is dominated by the writing of data to non-volatile storage.

BerkeleyDB was not space efficient. It wrote over 12GB to disk (more than 3 times the amount of actual data). By contrast the CEDA space overhead was tiny (about 0.3%).

BerkeleyDB performance was very poor. It took 2 minutes and 20 seconds using over 1 million write operations while CEDA only took 4 seconds using less than 1000 write operations.

BerkeleyDB used over a million write operations while the CEDA LSS used less than a thousand.

These results were discussed on an Oracle BerkeleyDB forum here.

Results for other sizes of the mapped values are tabulated below. Also shown are results for a B+Tree implemented on top of the CEDA LSS.

Across all the mapped value sizes tested, the CEDA LSS completed the workload between 5.7 and 40 times faster than BerkeleyDB. It also used substantially less disk space and required far fewer write operations.

BerkeleyDB

Mapped value size (bytes)	Time (sec)	Database size (bytes)	Log files size (bytes)	__db files size (bytes)	Effective rate (MB/sec)	Disk space wastage factor
4	3.78	30416896	178257920	550969344	3.03	63.30
8	3.80	34021376	188743680	550969344	4.02	48.36
16	3.83	45162496	199229440	550969344	5.98	33.14
32	4.21	72228864	251658240	550969344	9.06	21.87
64	4.44	102146048	314572800	550969344	15.46	13.44
128	5.49	230932480	503316480	550969344	23.62	9.45
256	7.36	421306368	828375040	550969344	34.21	6.82
512	11.9	678215680	1342177280	550969344	41.67	4.94
1024	30.9	1756733440	2967470080	550969344	31.85	5.11
2048	66.4	8226021376	2380267520	550969344	29.53	5.43
4096	140	8226021376	4424990720	550969344	27.96	3.22
8192	212	16418021376	8671723520	550969344	36.89	3.13

CEDA LSS

Mapped value size (bytes)	Time (sec)	Database size (bytes)	Effective rate (MB/sec)	Disk space wastage factor
4	0.66	25690112	17.3	2.1408
8	0.66	29360128	23.1	1.8350
16	0.66	37748736	34.7	1.5729
32	0.67	53477376	56.9	1.3369
64	0.69	85458944	99.5	1.1869
128	0.74	149422080	175.3	1.0987
256	0.81	277348352	310.8	1.0506
512	1.01	533725184	491.0	1.0264
1024	1.09	1045430272	902.9	1.0130
2048	1.64	2069364736	1195.6	1.0065
4096	3.99	4117757952	980.9	1.0034
8192	10.1	8213495808	774.3	1.0016

CEDA B+Tree implemented on top of the LSS

Mapped value size (bytes)	Time (sec)	Database size (bytes)	Effective data rate (MB/sec)	Disk space wastage factor
4	0.26	12582912	44.0	1.0486
8	0.26	16777216	58.7	1.0486
16	0.27	24641536	84.8	1.0267
32	0.30	40370176	127.2	1.0093
64	0.38	72876032	180.7	1.0122
128	0.50	136839168	259.4	1.0062
256	0.76	264765440	331.3	1.0029
512	1.35	520617984	367.3	1.0012
1024	2.24	1032847360	439.4	1.0008
2048	3.86	2057830400	508.0	1.0009
4096	6.87	4106747904	569.7	1.0007
8192	14.7	8217165824	532.0	1.0021

BerkeleyDB Test Code


// Test code minus error handling and timing:
void BerkeleyTest(int objectSize)
{
    const char* environPath = "env";
    const char* dbPath = "my_db.db";
    DbEnv env(0);
    env.open(
        environPath,
        DB_CREATE |
            DB_INIT_LOCK |
            DB_INIT_LOG |
            DB_INIT_MPOOL |
            DB_INIT_TXN,
        0);
    Db database(&env, 0);
    database.open(
        NULL,
        dbPath,
        NULL,
        DB_BTREE,
        DB_CREATE | DB_AUTO_COMMIT,
        0);
    __int64 keyid = 0;
    std::vector buffer(objectSize);

    // note: only this for loop is being timed
    for (int i=0 ; i < 1000 ; ++i)
    {
        DbTxn* txn = NULL;
        env.txn_begin(NULL, &txn, 0);
        for (int j=0 ; j < 1000 ; ++j)
        {
            Dbt key(&keyid, sizeof(keyid));
            Dbt data(buffer.data(),objectSize);
            database.put(
                txn,
                &key,
                &data,
                DB_NOOVERWRITE);
            ++keyid;
        }
        txn->commit(0);
    }

    database.close(0);
    env.close(0);
}

# DB_CONFIG
set_cachesize   0       536870912        0
set_flags       DB_TXN_NOSYNC
set_lg_regionmax        1048576
set_lg_max              10485760
set_lg_bsize            2097152

Comparison of I/O

The total number of I/O operations and total I/O bytes for the process were recorded using the Windows Task Manager. The numbers in both the following tables seem repeatable down to the last digit.

BerkeleyDB

Mapped value size (bytes)	Reads	Writes	Bytes read	Bytes written
4	25	3816	19770	201576287
8	26	4261	19770	212867996
16	27	5631	19770	243405945
32	32	8961	19770	314159978
64	38	12649	19770	406681096
128	56	28481	19770	731210030
256	87	51905	19770	1244327637
512	23979	107405	195341626	2215365406
1024	360135	575797	2947861818	7660111973
2048	3071	1008353	23252282	10624106569
4096	3266	1009525	23252282	12672112224
8192	30287	2038569	241290554	25322162241

These numbers reveal inherent inefficiencies in BDB, particularly for 1024 byte mapped values. It is reading 3x the amount of data it is supposed to be writing!

CEDA LSS

Mapped value size (bytes)	Writes	Bytes written
4	11	25229824
8	12	29229568
16	14	37230080
32	18	53230592
64	26	85231104
128	41	149232128
256	72	277233664
512	133	533237760
1024	255	1045244928
2048	499	2069260288
4096	988	4117327872
8192	1968	8213450240

32 LSS performance measurement code

The following code was timed for various values of mappedValueSize to measure the ingestion rate of the CEDA LSS.


const int NUM_TXN = 1000;
const int NUM_ROWS_PER_TXN = 1000;
int numRecords = NUM_TXN * NUM_ROWS_PER_TXN;
const char* path = "test.lss";
bool createdNew;
LssSettings settings;
settings.maxNumSegmentsInCache = 128;
settings.numSegmentsPerCheckPoint = 512;
settings.segmentSize = 4*1024*1024;
ILogStructuredStore* lss = CreateOrOpenLSS(
    path, nullptr, createdNew, OM_OPEN_ALWAYS, settings);
SeidHigh seidHigh = lss->CreateSeidSpace();
std::vector<octet_t> buffer(mappedValueSize);
for (int i=0 ; i < NUM_TXN ; ++i)
{
    ILssTransaction* txn = lss->OpenTransaction();
    for (int j=0 ; j < NUM_ROWS_PER_TXN ; ++j)
    {
        Seid seid = lss->AllocateSeid(seidHigh);
        ICloseableOutputStream* os = txn->WriteSerialElement(seid);
        os->WriteStream(buffer.data(), buffer.size());
        os->Close();
    }
    txn->Close();
}
lss->Close();

33 NuoDB ingestion rate measurement

NuoDB is a memory centric, ACID compliant distributed SQL database.

This test was written using the C++ API accessing the NuoDB Community Edition Release 2.5.6 for Windows which is a single-host version of the NuoDB distributed database.

The ingestion rate was measured on an Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM, running Windows 10 64 bit, having a pair of SSDs in RAID0 (striped).

Test 1

A single table was created in an otherwise empty database named 'pt' on localhost:48004 using the following command:


create table names (id int primary key, name string)

The following C++ code was used to populate the table with a million records using one thousand transactions, where each record has a 64 bit key and a mapped value of 4kB:


const int NUM_TXN = 1000;
const int NUM_ROWS_PER_TXN = 1000;
const int MAPPED_VALUE_SIZE = 4096;
std::string nameStr(MAPPED_VALUE_SIZE,'x');
const char* name = nameStr.c_str();
PreparedStatement* stmt = connection->prepareStatement(
    "insert into names (id,name) values (?,?)");
try
{
    int id = 0;
    for (int t=0 ; t < NUM_TXN ; ++t)
    {
        for (int r = 0; r < NUM_ROWS_PER_TXN; r++)
        {
            stmt->setInt(1, id++);
            stmt->setString(2, name);
            stmt->addBatch();
        }
        stmt->executeBatch();
        connection->commit();
    }
    stmt->close();
}
catch (SQLException& xcp)
{
    connection->rollback();
    throw;
}

The time taken to run the test was about 13 minutes. The final size of the database files (C:\ProgramData\nuodb\production-archives\pt) was 4.31GB.

Test 2

The previous test was only using 32 bit keys, and using strings for the mapped values. This test instead uses the 'bigint' datatype for the keys which is 64 bit as used in the performance measurements on the other DBMS products that have been tested, and also uses 'varbinary' for the mapped values, in case this is more efficient than 'string'.


create table names (id bigint primary key, name varbinary(8192))

The following C++ code was used to populate the table with a million records using one thousand transactions:


const int NUM_TXN = 1000;
const int NUM_ROWS_PER_TXN = 1000;
const int MAPPED_VALUE_SIZE = 4096;
std::string nameStr(MAPPED_VALUE_SIZE,'x');
const char* name = nameStr.c_str();
PreparedStatement* stmt = connection->prepareStatement(
    "insert into names (id,name) values (?,?)");
try
{
    int64_t id = 0;
    for (int t=0 ; t < NUM_TXN ; ++t)
    {
        for (int r = 0; r < NUM_ROWS_PER_TXN; r++)
        {
            stmt->setLong(1, id++);
            stmt->setBytes(2, MAPPED_VALUE_SIZE, name);
            stmt->addBatch();
        }
        stmt->executeBatch();
        connection->commit();
    }
    stmt->close();
}
catch (SQLException& xcp)
{
    connection->rollback();
    throw;
}

The time taken to run the test was 749 seconds (12.5 minutes).

According to the Windows Task Manager, during this time the following processes were taking up the CPU and memory resources:

Process name	Memory	CPU
MsMpEng.exe	61MB	12%
nuodb.exe	2.4GB	10%

Note that when one of the eight processors is at 100%, the CPU usage is reported as 12.5% in the Windows Task Manager on this machine.

Test 3

This is essentially Test 2 with various mapped value sizes: 4,8,16,32,64,128,...,8192.

For each mapped value size, before the ingestion of a million records the table was dropped but the database was not deleted.

Mapped value size	Time (s)	Effective data rate (MB/s)
4	37.8	0.303
8	37.4	0.408
16	39.2	0.584
32	38.8	0.984
64	40.4	1.70
128	42.0	3.09
256	47.1	5.34
512	51.1	9.70
1024	107	9.24
2048	425	4.61
4096	916	4.27
8192	2070	3.77

Interestingly for 4096 byte mapped values the test took 916 seconds (previously was 749). It appears the database performance degrades over time.

To get an idea of the disk space usage the following information was obtained on the folder C:\ProgramData\nuodb\production-archives\pt:

Size	17.7 GB (19,053,864,558 bytes)
Size on disk	18.7 GB (20,158,586,880 bytes)
Contains	310,182 files 3,129 folders

The average file size is only 60kB.

34 Kyoto Cabinet ingestion rate measurement

The performance measurements have been made on an Intel Core i7-4700MQ 2.4GHz laptop with 16GB RAM, running Windows 10 64 bit.

Writing to the following was measured:

A pair of 250GB SSDs in RAID0 (striped); and
A 4GB RAM disk by DATARAM

The RAM disk is useful for measuring the CPU load.

The time to write 1M, 10M or 100M records (i.e. key-value pairs) with sequentially increasing 64 bit keys has been measured. The size of the mapped value is either 4,8,16,32,..., or 8192 bytes. In the case of the 4GB RAM disk there is a limit imposed on the mapped value size to allow the database to fit on a 4GB drive. The data is written using 1000 transactions.

After writing the store it is closed then reopened, and all the records in the database are read with sequentially increasing keys.

It might be expected that the Kyoto HashDB will perform badly when the amount of data is too large to fit in the memory cache, if the sequential key access is randomised by the hash function.

Building the Kyoto library

Kyoto Cabinet version 1.2.76 was built using Microsoft Visual Studio 2015. The VCmakefile was modified to support an x64 build under VS2015 with dynamic CRT libs (i.e. using the /MD compiler switch instead of /MT):


VCPATH = C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC
SDKPATH = C:\Program Files (x86)\Windows Kits\8.1
WINKITS_10 = C:\Program Files (x86)\Windows Kits\10
WINKITS_VER = 10.0.10240.0

CLFLAGS = /nologo \
  /I "$(VCPATH)\Include" /I "$(VCPATH)\PlatformSDK\Include" /I "$(SDKPATH)\Include" \
  /I "." \
  /DNDEBUG /D_CRT_SECURE_NO_WARNINGS \
  /O2 /EHsc /W3 /wd4244 /wd4351 /wd4800 /MD

LIBFLAGS = /nologo \
  /libpath:"$(VCPATH)\lib\amd64" /libpath:"$(WINKITS_10)\Lib\$(WINKITS_VER)\ucrt\x64" /libpath:"$(SDKPATH)\Lib\winv6.3\um\x64" \
  /libpath:"."
LINKFLAGS = /nologo \
  /libpath:"$(VCPATH)\lib\amd64" /libpath:"$(WINKITS_10)\Lib\$(WINKITS_VER)\ucrt\x64" /libpath:"$(SDKPATH)\Lib\winv6.3\um\x64" \
  /libpath:"."

Test code

HashDB database tuning

The following C++ code has been used to tune the HashDB for a given number of records. Earlier results without tuning suggest this is worth doing to achieve better performance.


void tuneDB(HashDB& db, int numRecords)
{
    db.tune_options(HashDB::TLINEAR);
    db.tune_buckets(2*numRecords);
    db.tune_map(512*1024*1024);
}

TreeDB database tuning

The following C++ code has been used to tune the TreeDB for a given number of records


void tuneDB(TreeDB& db, int numRecords)
{
    db.tune_options(TreeDB::TLINEAR);
    db.tune_buckets(numRecords / 10);
    db.tune_map(512*1024*1024);
}

Code to write key value pairs with sequential keys to a database

The following C++ code has been used to insert key,value pairs ("records") where the keys are sequentially increasing 64 bit numbers. This is achieved using a given number of transactions where each transaction inserts a given number of records. For simplicity the error handling code is not shown.


void write(const char* path, int numTxns, int numRecordsPerTxn, int mappedValueSize)
{
    HashDB db;
    int numRecords = numTxns * numRecordsPerTxn;
    tuneDB(db, numRecords);
    db.open(path, HashDB::OWRITER | HashDB::OCREATE);
    int64 id = 0;
    std::vector<char> buffer(mappedValueSize);
    for (int t=0 ; t < numTxns ; ++t)
    {
        db.begin_transaction();
        for (int r = 0; r < numRecordsPerTxn; ++r)
        {
            db.set( (const char*)&id, sizeof(id), buffer.data(), buffer.size());
            ++id;
        }
        db.end_transaction();
    }
    db.close();
}

Code to read key value pairs with sequential keys from a database

The following C++ code has been used to read all the records with sequentially increasing keys.


void read(const char* path, int numTxns, int numRecordsPerTxn, int mappedValueSize)
{
    HashDB db;
    int numRecords = numTxns * numRecordsPerTxn;
    tuneDB(db, numRecords);
    db.open(path, HashDB::OWRITER);
    std::vector<char> buffer(mappedValueSize);
    for (int64 id=0 ; id < numRecords ; ++id)
    {
        db.get((const char*)&id, sizeof(id), buffer.data(), buffer.size());
    }
    db.close();
}

Results

It would be good to do comparisons for the pair of SSDs in RAID0. Unfortunately it appears they have become heavily fragmented over time, such that the maximum write rate is only about 300-400MB/sec (it was about 850MB/sec originally).

The write and read rate is given both in terms of the records per second (the unit is kops/s which means 1000 records per second) and bytes per second (the unit is MB/s which means 2²⁰ bytes per second).

HashDB with RAM disk

num txns	num records per txn	mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	write data time (s)	read data time (s)	write record rate (kops/s)	read record rate (kops/s)	effective write rate (MB/s)	effective read rate (MB/s)
1000	1000	4	12000000	36589240	24.6	2.22	0.36	450	2815	5.2	32.2
1000	1000	8	16000000	44589240	28.6	2.22	0.37	450	2735	6.9	41.7
1000	1000	16	24000000	52589240	28.6	2.19	0.37	456	2680	10.4	61.3
1000	1000	32	40000000	68589240	28.6	2.20	0.47	455	2123	17.4	81.0
1000	1000	64	72000000	100589240	28.6	2.20	0.48	454	2094	31.2	143.8
1000	1000	128	136000000	164589240	28.6	2.26	0.52	442	1917	57.3	248.6
1000	1000	256	264000000	292589240	28.6	2.35	0.57	425	1752	106.9	441.1
1000	1000	512	520000000	548589240	28.6	2.57	0.78	390	1275	193.2	632.1
1000	1000	1024	1032000000	1060589240	28.6	4.17	3.07	240	326	235.8	320.6
1000	1000	2048	2056000000	2084589240	28.6	5.90	4.56	169	219	332.2	429.6
1000	1000	4096	4104000000	4132589240	28.6	8.27	5.76	121	173	473.4	679.0
1000	10000	4	120000000	382365640	26.2	20.84	3.66	480	2732	5.5	31.3
1000	10000	8	160000000	462365640	30.2	20.97	3.74	477	2674	7.3	40.8
1000	10000	16	240000000	542365640	30.2	21.16	4.20	473	2383	10.8	54.5
1000	10000	32	400000000	702365640	30.2	26.66	18.06	375	554	14.3	21.1
1000	10000	64	720000000	1022365640	30.2	32.91	28.23	304	354	20.9	24.3
1000	10000	128	1360000000	1662365640	30.2	39.39	35.56	254	281	32.9	36.5
1000	10000	256	2640000000	2942365640	30.2	44.47	40.63	225	246	56.6	62.0
1000	100000	4	1200000000	3754364736	25.5	1005.80	542.76	99	184	1.1	2.1

Note that the performance drops when the database size exceeds the cache (512MB). For example for 4 byte mapped values the write rate dropped from 5MB/s to about 1MB/s when writing 100M records.

TreeDB with RAM disk

num txns	num records per txn	mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	write data time (s)	read data time (s)	write record rate (kops/s)	read record rate (kops/s)	effective write rate (MB/s)	effective read rate (MB/s)
1000	1000	4	12000000	37548544	25.6	8.13	1.01	123	987	1.4	11.3
1000	1000	8	16000000	34189056	18.2	7.43	1.00	135	996	2.0	15.2
1000	1000	16	24000000	48890880	24.9	6.41	0.98	156	1020	3.6	23.3
1000	1000	32	40000000	74410752	34.4	8.14	1.10	123	906	4.7	34.5
1000	1000	64	72000000	187328256	115.3	9.18	4.44	109	225	7.5	15.5
1000	1000	128	136000000	295424512	159.4	13.07	6.45	77	155	9.9	20.1
1000	1000	256	264000000	567815680	303.8	17.77	9.73	56	103	14.2	25.9
1000	1000	512	520000000	2054669824	1534.7	30.04	16.09	33	62	16.5	30.8
1000	10000	4	120000000	439591680	32.0	165.68	231.13	60	43	0.7	0.5
1000	10000	8	160000000	335977216	17.6	176.95	232.51	57	43	0.9	0.7
1000	10000	16	240000000	2128638976	188.9	261.68	298.52	38	33	0.9	0.8
1000	10000	32	400000000	999380736	59.9	276.85	305.10	36	33	1.4	1.2
1000	10000	64	720000000	2531212800	181.1	394.35	283.12	25	35	1.7	2.4
1000	50000	4	600000000	2605504768	40.1	2980.22	2522.90	17	20	0.2	0.2

These measurements suggest the TreeDB doesn't perform well when the amount of data exceeds the memory cache.

CEDA LSS with RAM disk

num txns	num records per txn	mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	write data time (s)	read data time (s)	write record rate (kops/s)	read record rate (kops/s)	effective write rate (MB/s)	effective read rate (MB/s)
1000	1000	4	12000000	25690112	13.7	0.46	0.40	2191	2481	25.1	28.4
1000	1000	8	16000000	29360128	13.4	0.46	0.41	2157	2450	32.9	37.4
1000	1000	16	24000000	37748736	13.8	0.47	0.41	2139	2438	49.0	55.8
1000	1000	32	40000000	53477376	13.5	0.51	0.42	1980	2370	75.5	90.4
1000	1000	64	72000000	85458944	13.5	0.53	0.45	1870	2209	128.4	151.7
1000	1000	128	136000000	149422080	13.4	0.60	0.49	1679	2024	217.8	262.4
1000	1000	256	264000000	277348352	13.3	0.81	0.60	1235	1658	311.0	417.5
1000	1000	512	520000000	533725184	13.7	1.16	0.83	862	1200	427.4	595.2
1000	1000	1024	1032000000	1045430272	13.4	1.53	0.96	655	1037	644.3	1020.2
1000	1000	2048	2056000000	2069364736	13.4	2.28	1.22	439	821	861.5	1609.2
1000	1000	4096	4104000000	4117757952	13.8	3.80	1.75	263	572	1029.9	2239.2
1000	10000	4	120000000	251658240	13.2	4.61	4.32	2167	2314	24.8	26.5
1000	10000	8	160000000	292028416	13.2	4.68	4.17	2139	2399	32.6	36.6
1000	10000	16	240000000	371720192	13.2	4.73	4.38	2116	2284	48.4	52.3
1000	10000	32	400000000	531628032	13.2	4.92	4.50	2031	2224	77.5	84.8
1000	10000	64	720000000	851968000	13.2	5.18	4.47	1929	2236	132.5	153.5
1000	10000	128	1360000000	1491599360	13.2	5.63	4.76	1776	2101	230.3	272.5
1000	10000	256	2640000000	2771910656	13.2	7.12	5.43	1404	1842	353.4	463.6
1000	100000	4	1200000000	2515009536	13.2	45.81	41.96	2183	2383	25.0	27.3
1000	100000	8	1600000000	2915041280	13.2	46.38	41.81	2156	2392	32.9	36.5
1000	100000	16	2400000000	3715104768	13.2	46.81	42.20	2136	2370	48.9	54.2

The CEDA LSS scales very well in this test, even though the database size well exceeds the size of the LSS segment cache. For 4, 8 and 16 byte mapped values the write and read rates are independent of the number of records from 1M to 100M records:

num records	4 byte value write rate (MB/s)	4 byte value read rate (MB/s)	8 byte value write rate (MB/s)	8 byte value read rate (MB/s)	16 byte value write rate (MB/s)	16 byte value read rate (MB/s)
1000000	25.1	28.4	32.9	37.4	49.0	55.8
10000000	24.8	26.5	32.6	36.6	48.4	52.3
100000000	25.0	27.3	32.9	36.5	48.9	54.2

CEDA B+Tree with RAM disk

num txns	num records per txn	mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	write data time (s)	read data time (s)	write record rate (kops/s)	read record rate (kops/s)	effective write rate (MB/s)	effective read rate (MB/s)
1000	1000	4	12000000	12582912	0.6	0.20	0.11	5111	9003	58.5	103.0
1000	1000	8	16000000	16252928	0.2	0.20	0.11	5020	8723	76.6	133.1
1000	1000	16	24000000	24641536	0.6	0.21	0.12	4679	8017	107.1	183.5
1000	1000	32	40000000	40370176	0.4	0.25	0.15	4034	6823	153.9	260.3
1000	1000	64	72000000	72351744	0.3	0.33	0.19	3034	5351	208.3	367.4
1000	1000	128	136000000	136314880	0.3	0.45	0.27	2240	3728	290.5	483.5
1000	1000	256	264000000	264241152	0.2	0.68	0.41	1468	2454	369.7	617.8
1000	1000	512	520000000	520617984	0.6	1.15	0.72	870	1382	431.2	685.1
1000	1000	1024	1032000000	1032323072	0.3	2.28	1.29	438	775	431.4	762.4
1000	1000	2048	2056000000	2056257536	0.3	4.42	2.62	226	382	443.4	748.7
1000	1000	4096	4104000000	4104650752	0.7	9.00	6.68	111	150	435.0	585.8
1000	10000	4	120000000	121110528	0.1	2.27	1.49	4409	6722	50.5	76.9
1000	10000	8	160000000	160956416	0.1	2.33	1.53	4292	6537	65.5	99.8
1000	10000	16	240000000	240648192	0.1	2.47	1.62	4047	6182	92.6	141.5
1000	10000	32	400000000	401080320	0.1	2.82	2.06	3548	4844	135.3	184.8
1000	10000	64	720000000	720896000	0.1	3.45	2.19	2897	4556	198.9	312.8
1000	10000	128	1360000000	1361051648	0.1	4.77	2.94	2096	3400	271.8	440.9
1000	10000	256	2640000000	2640838656	0.1	9.03	4.80	1108	2085	278.9	525.0
1000	100000	4	1200000000	1205862400	0.1	29.01	20.78	3447	4812	39.4	55.1
1000	100000	8	1600000000	1605894144	0.1	28.82	21.36	3470	4682	53.0	71.5
1000	100000	16	2400000000	2405957632	0.1	40.68	33.56	2458	2980	56.3	68.2
1000	100000	32	4000000000	4006084608	0.1	40.93	26.43	2443	3784	93.2	144.3
1000	300000	4	3600000000	3617062912	0.1	87.10	55.47	3444	5408	39.4	61.9

Side by side comparisons on RAM disk for 1M records

The following comparisons are between the CEDA LSS, CEDA B+Tree, Kyoto HashDB and Kyoto TreeDB on the same machine, built with the same compiler, for 1000 transactions and 1000000 records, on the 4GB RAM disk.

Note that there is no data for Kyoto TreeDB for mapped values of 1024, 2048 and 4096 bytes because in those cases the 4GB RAM disk wasn't large enough for the database.

Write octets rate

Comparison of the effective write rate in MB/second (i.e. the rate for which real data in the keys and values is written).

Write records rate

Comparison of the rate at which records are written in units of 1000 records/second.

Read octets rate

Comparison of the effective read rate in MB/second.

Read records rate

Comparison of the rate at which records are read in units of 1000 records/second.

Space overhead

These plots show the average space overhead in bytes per key,value pair for different sizes of the mapped value.

Taking out the Kyoto TreeDB allows us to see the space overheads of the other databases more clearly. The space overhead for the CEDA LSS is between 13 and 14 bytes, and less than one byte for the CEDA B+Tree.

35 Kyoto Cabinet without tuning

It has been found tuning can improve the results, so these numbers should not be taken as indicative of the potential performance of the Kyoto HashDB and TreeDB.

HashDB with SSD

Note this is with the default tuning parameters

mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	open database time (ms)	write data time (s)	read data time (s)	close database time (ms)	effective write rate (MB/s)	effective read rate (MB/s)
4	12000000	38297720	26.3	2.5	3.31	0.44	3.4	3.5	26.2
8	16000000	38297720	22.3	0.9	3.35	0.46	3.7	4.6	33.3
16	24000000	46297720	22.3	1.1	3.36	0.44	4.2	6.8	52.1
32	40000000	62297720	22.3	1.0	3.25	0.56	5.3	11.8	67.7
64	72000000	94297720	22.3	1.1	4.38	2.54	5.5	15.7	27.1
128	136000000	166297720	30.3	0.9	6.04	4.48	5.5	21.5	28.9
256	264000000	294297720	30.3	0.8	7.14	5.51	5.6	35.3	45.7
512	520000000	550297720	30.3	0.8	8.32	6.05	5.3	59.6	82.0
1024	1032000000	1062297720	30.3	0.9	10.80	6.84	5.4	91.1	143.9
2048	2056000000	2086297720	30.3	0.9	12.73	7.40	5.5	154.0	264.9
4096	4104000000	4134297720	30.3	1.2	19.04	8.77	5.6	205.6	446.4
8192	8200000000	8230297720	30.3	1.0	34.91	35.99	5.7	224.0	217.3

TreeDB with SSD

Note this is with the default tuning parameters.

mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	open database time (ms)	write data time (s)	read data time (s)	close database time (ms)	effective write rate (MB/s)	effective read rate (MB/s)
4	12000000	37331712	25.3	2.0	11.17	1.10	122.2	1.0	10.4
8	16000000	35577856	19.6	1.6	9.39	1.09	119.3	1.6	14.0
16	24000000	44252416	20.3	3.7	8.59	0.88	132.4	2.7	26.1
32	40000000	74349824	34.3	3.5	10.98	1.17	141.2	3.5	32.7
64	72000000	199813120	127.8	2.1	14.74	7.31	133.9	4.7	9.4
128	136000000	295232256	159.2	3.1	21.96	10.33	84.5	5.9	12.6
256	264000000	568140032	304.1	3.6	34.80	17.32	60.3	7.2	14.5
512	520000000	2054346240	1534.3	1.4	48.66	21.00	63.8	10.2	23.6
1024	1032000000	4284145152	3252.1	1.6	122.29	23.53	136.5	8.0	41.8
2048	2056000000	4935943168	2879.9	1.8	206.73	28.77	283.7	9.5	68.1
4096	4104000000	4374228480	270.2	1.5	474.11	149.67	2596.2	8.3	26.2
8192	8200000000	8470268160	270.3	1.6	1026.33	409.39	22424.5	7.6	19.1

HashDB with RAM disk

Note this is with the default tuning parameters

mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	open database time (ms)	write data time (s)	read data time (s)	close database time (ms)	effective write rate (MB/s)	effective read rate (MB/s)
4	12000000	38297720	26.3	2.2	2.28	0.43	18.8	5.0	26.7
8	16000000	38297720	22.3	4.5	2.26	0.43	19.1	6.7	35.5
16	24000000	46297720	22.3	3.5	2.23	0.43	22.9	10.2	53.3
32	40000000	62297720	22.3	4.8	2.25	0.56	30.2	17.0	68.7
64	72000000	94297720	22.3	3.5	2.99	2.27	32.5	22.9	30.2
128	136000000	166297720	30.3	4.2	4.26	3.99	57.0	30.4	32.5
256	264000000	294297720	30.3	1.7	5.17	5.01	105.9	48.7	50.3
512	520000000	550297720	30.3	3.8	6.12	5.43	151.9	81.1	91.4
1024	1032000000	1062297720	30.3	1.6	7.30	5.93	195.0	134.9	165.8
2048	2056000000	2086297720	30.3	4.3	8.68	6.73	288.8	225.8	291.4
4096	4104000000	4134297720	30.3	4.0	11.45	7.70	356.5	341.9	508.2

TreeDB with RAM disk

Note this is with the default tuning parameters

mapped value size (bytes)	total data (bytes)	database size (bytes)	space overhead per record (bytes)	open database time (ms)	write data time (s)	read data time (s)	close database time (ms)	effective write rate (MB/s)	effective read rate (MB/s)
4	12000000	37331712	25.3	2.2	9.22	1.13	134.7	1.2	10.2
8	16000000	35577856	19.6	4.2	8.76	1.11	136.2	1.7	13.7
16	24000000	44252416	20.3	5.3	7.73	0.92	161.5	3.0	25.0
32	40000000	74349824	34.3	2.2	9.64	1.16	161.8	4.0	32.9
64	72000000	199813120	127.8	2.1	18.95	7.66	154.1	3.6	9.0
128	136000000	295232256	159.2	4.8	19.67	11.02	139.8	6.6	11.8
256	264000000	568140032	304.1	5.7	30.63	17.74	114.5	8.2	14.2
512	520000000	2054346240	1534.3	2.1	65.47	19.90	99.8	7.6	24.9

36 Implementation overview

The implementation is organised around a segment writer and a collection of cooperating background tasks. Transactions append log records to the segment currently being prepared in memory. Completed segments pass through a writer queue and are written and flushed asynchronously. Checkpointing bounds recovery work, while the cleaner reclaims poorly utilised segments.

Principal classes

The following diagram shows the principal implementation classes and their relationships. An orange arrow denotes inheritance, a blue arrow denotes ownership, and a dotted blue line denotes a reference. Classes carrying a mutex are also identified.

The chapters that follow trace the implementation from the storage medium and on-disk structures, through the write and read paths, to persistent maps, checkpointing, recovery and maintenance.

37 Storage medium

The LSS accesses its storage through the IRAS random-access-store interface. An IRAS is a linear byte-addressed store with 64-bit addresses. It supports aligned memory allocation, reading, writing, querying the current size, and truncating the store.

The caller may invoke RAS operations from multiple threads without using an external mutex. The IRAS implementation is responsible for making those calls thread-safe and for serialising them, so two operations on the same RAS do not execute concurrently. As with other explicitly closed interfaces, the caller must ensure that no operation is in progress, and that no new operation can begin, when it calls Close() or destroys the RAS.


using RASAddress = int64;
using RASSize = RASAddress;

struct IRAS
{
    virtual ~IRAS() {}

    virtual int GetDiskSectorSize() const = 0;

    virtual void* AllocateMemoryBlock(int size) = 0;
    virtual void FreeMemoryBlock(void* p) = 0;

    virtual void Close() = 0;

    virtual void Read(void* buffer, RASAddress offset, int numBytes) = 0;
    virtual void Write(const void* buffer, RASAddress offset, int numBytes) = 0;

    virtual RASSize GetSize() const = 0;
    virtual void TruncateSize(RASSize newSize) = 0;

    virtual void StopWriting() = 0;
};

AllocateMemoryBlock() returns a buffer suitable for the implementation's Read() and Write() operations. A write can extend the store. The supplied implementations return zeroes for the portion of a read beyond the current size. StopWriting() prevents subsequent writes.

Source: Ceda/cxLss/IRAS.h

The supplied implementations store the LSS in a file on Windows or Linux, or in a PagedBuffer using MemRAS. The abstraction could represent another linear medium, but raw-partition support has not been implemented. It is recorded as a proposal in Other.

Segments and addressing

The storage medium contains fixed-size segments. The segment size is chosen when an LSS is created and cannot subsequently be changed for that store. The default is 512 kB. A segment is identified by a positive 32-bit SegId; zero is reserved to represent a null segment reference.

For a file-backed LSS, the 64 kB root block occupies the start of the file and the segments follow it. Segment identifiers begin at one, so the byte position of a segment is calculated as:


ROOT_BLOCK_SIZE + (segid - 1) * segmentSize

Writes automatically grow the underlying RAS as necessary. At a checkpoint, the LSS finds the highest utilised segment and truncates the RAS immediately after it. The lazy cleaner does not give special preference to segments at the end of the file: it selects sufficiently underutilised segments by utilisation. The file therefore becomes shorter only when all segments beyond the highest utilised segment have become free.

File fragmentation

Growing a file in many small steps can contribute to filesystem fragmentation. Each extension may be allocated from the free space available at that moment, and allocations for other files can occupy the adjacent space between extensions. The file can consequently be represented by many separate extents rather than a small number of contiguous extents.

The file RAS implementations try to reduce this effect by extending the file in 512 kB increments rather than extending it by only the size of each write. Larger extensions can reduce the number of filesystem allocation and metadata operations and can improve the opportunity for contiguous allocation, but they do not guarantee it.

There is also a distinction between changing a file's logical length and reserving physical storage. In particular, ftruncate() on Linux can extend the logical file without immediately allocating all of its data blocks. Those blocks can still be allocated separately as they are written. A filesystem preallocation operation such as fallocate() expresses the intention to reserve an extent more directly, but LinuxFileRAS does not currently use it.

Fragmentation is most directly harmful on a rotating disk, where accessing separated extents can require additional head movement. It is generally less costly on an SSD because there is no mechanical seek, although a large number of extents can still increase filesystem metadata and I/O overhead. The benefit of the 512 kB extension policy therefore depends on the filesystem, storage device, available free space, and other allocation activity.

Windows file: FileRAS

FileRAS stores the LSS in a single Windows file and uses Win32 operations with 64-bit file positions. File buffering is disabled by default using FILE_FLAG_NO_BUFFERING, because the LSS maintains its own segment cache. Buffers are allocated with VirtualAlloc() so they satisfy the alignment expected by the current implementation. File buffering can instead be enabled through LssSettings.

FILE_FLAG_WRITE_THROUGH can be requested separately. Reads and writes are serialised by a mutex. When a write extends the file, FileRAS rounds the new file size up to the next 512 kB boundary to reduce filesystem fragmentation. This extension granularity is independent of the configurable LSS segment size. A checkpoint can subsequently truncate the file to the exact end of the highest utilised segment. The implementation currently reports a hard-coded 512-byte sector size rather than querying the alignment requirements of the file or volume.

Sources: Ceda/cxLss/src/FileRAS.h and Ceda/cxLss/src/FileRAS.cpp

Linux file: LinuxFileRAS

LinuxFileRAS stores the LSS in a single file using open(), read(), write(), lseek(), and ftruncate(). It uses Linux open-file-description locks to prevent conflicting access by another process. Shared-read mode takes a read lock; other open modes take a write lock. Reads and writes within the process are serialised by a mutex.

As on Windows, a write that extends the file rounds its new size up to the next 512 kB boundary, while a checkpoint can truncate it to the exact end of the highest utilised segment. Although LinuxFileRAS::Open() accepts the file-buffering and write-through settings, it currently does not use them: the file is opened without O_DIRECT or O_SYNC. The implementation therefore uses buffered file I/O while still reporting a hard-coded 512-byte sector size and enforcing 512-byte offsets and transfer lengths.

Sources: Ceda/cxLss/src/LinuxFileRAS.h and Ceda/cxLss/src/LinuxFileRAS.cpp

Memory: MemRAS

MemRAS stores the LSS in a caller-owned PagedBuffer. It is useful for tests and transient stores small enough to keep in memory. Close() has no work to do, and truncation changes the size of the paged buffer. Like the file implementations, access is serialised and reads beyond the current size are completed with zeroes.

For consistency with the file implementations, MemRAS reports a 512-byte sector size and requires buffers, offsets, and transfer sizes to be aligned to 512 bytes. On Windows it allocates buffers with VirtualAlloc(); on other platforms it uses malloc().

Sources: Ceda/cxLss/src/MemRAS.h and Ceda/cxLss/src/MemRAS.cpp

See RootBlock for the implications of the historical alignment assumptions shared by these implementations.

38 RootBlock

When the RAS uses unbuffered or direct file I/O, the operating system or filesystem may require file offsets, transfer lengths, and memory-buffer addresses to satisfy storage-specific alignment constraints. These constraints should be queried for the file or volume rather than inferred from a fixed historical sector size. Modern storage can require 4 kB alignment.

The current cxLss file format places the static and dynamic root-block headers on 1 kB boundaries. The current file RAS also reports a hard-coded 512-byte sector size rather than querying the actual direct-I/O alignment requirements. The 1 kB layout therefore reflects a historical format assumption; it does not by itself support an environment that requires alignment greater than 1 kB.

The total size of the root block is 64k bytes. It consists of the following sections.

Description	Size
Root block header - static part	1k
Root block header - dynamic part	1k
1st root block division	31k
2nd root block division	31k
Total	64k

The static part of the root block stores the following information

The magic string "CEDANET-LSS" used to verify that the file is in fact a Ceda LSS
The total size of the root block header (currently 2k = 2048 bytes)
The total size of the root block (currently 64k = 65536 bytes)
The segment size (currently 512k = 524288 bytes)
The disk sector size (typically 512 bytes)

The remainder is padded with zeros.

The dynamic part of the header starts on a new 1k boundary so it can be written independently of the static part. It contains a single boolean flag - the Graceful Shutdown Flag (GSF) used during startup to test whether the LSS was shut down gracefully the last time it was opened.

The root block divisions contain the check point information and are written in strict alternation using Challis' algorithm and validated with a 32bit CRC. The divisions start and end on 1k boundaries to ensure they can be written independently.

A root block division contains the following fields

Data type	Description
int m_crc	32 bit CRC to validate the division
int m_msn1	For Challis’ algorithm
SUT m_sut	A snapshot of the SUT at the time of the last check point.
RPM m_rpm	A snapshot of the root RPM-node at the time of the last check point
LogRecordPosition m_positionOfLastCheckPoint	Identifies the location of the last check point record in the log
int m_currentOSID
GUID m_checkPointId	Each check point is given a unique identity by creating a guid at the time of the check point. This is used to validate flush units during a recovery scan.
int m_msn2	For Challis’ algorithm

Unbuffered or direct I/O can require aligned file offsets, transfer lengths, and memory buffers. The required alignment depends on the file, filesystem, volume, and storage device, and should be queried from the operating system. The 1 kB boundaries used by this root-block layout are a historical file-format choice, not a general statement about modern storage-sector sizes.

The total size of the root block is 64k bytes. It consists of the following sections

        Description                    Size
     ----------------------------------------------
        root block header
            static part                 1k
            dynamic part                1k
        1st root block division        31k
        2nd root block division        31k
     ----------------------------------------------
            total                      64k

The static part of the root block stores the following information

The magic string "CEDANET-LSS" used to verify that the file is in fact a Ceda LSS
The total size of the root block header (currently 2k = 2048 bytes)
The total size of the root block (currently 64k = 65536 bytes)
The segment size (typically 512k = 524288 bytes)
The disk sector size (typically 512 bytes)

The remainder is padded with zeros.

Code


struct RootBlockHeader
{
    void Init(int diskSectorSize, int segmentSize);
    void Validate(int diskSectorSize);
    void WriteDiagnosticInfo(xostream& os) const;
    static constexpr size_t SizeOnDisk() { return StaticSize() + DynamicSize(); }
    static constexpr size_t StaticSize() { return ROOT_BLOCK_STATIC_HEADER_SIZE; }
    static constexpr size_t StaticUsed() { return sizeof(magicStr_) + sizeof(schema_) + sizeof(rootHeaderSize_) + sizeof(rootBlockSize_) + sizeof(segmentSize_) + sizeof(diskSectorSize_); }
    static constexpr size_t StaticPadding() { return StaticSize() - StaticUsed(); }

    char magicStr_[LSS_MAGIC_STR_SIZE];
    int32 schema_;
    int32 rootHeaderSize_;
    int32 rootBlockSize_;        // Total size of the root block including the header
    int32 segmentSize_;
    int32 diskSectorSize_;

    static constexpr size_t DynamicSize() { return ROOT_BLOCK_DYNAMIC_HEADER_SIZE; }
    static constexpr size_t DynamicUsed() { return sizeof(gracefulShutdown_); }
    static constexpr size_t DynamicPadding() { return DynamicSize() - DynamicUsed(); }
    bool gracefulShutdown_;
};

/*
The root block consists of a header and two divisions.  The divisions are written in strict
alternation using Challis' algorithm.

Each division needs to store the following information (see LSS::SerialiseRootBlockDivisionPayload())

                                                Size
    first magic string                  14 bytes
    schema                              4 bytes
    sut_                                up to 28k bytes will allow for 448 terra byte stores
                                        assuming 512k segments.
    rpm_                                up to about 2K bytes
    positionOfLastCheckPoint_           8 bytes
    currentOidHigh_                     4 bytes
    checkPointId_                       16 bytes
    second magic string                 12 bytes
*/

const int ROOT_BLOCK_DIVISION_SIZE = 31 * 1024;

// A division has [ crc, msn1, payload, msn2 ]
// The space available for the payload is 31732
const int ROOT_BLOCK_DIVISION_PAYLOAD_SIZE = ROOT_BLOCK_DIVISION_SIZE - sizeof(CRC32) - 2*sizeof(ModSeqNum);

// division must be 0 or 1.
// Returns the offset to the start of the division from the start of the root block - i.e. this is the
// offset to the division within the file.
inline ssize_t GetRootBlockDivisionOffset(int division)
{
    cxAssert(division == 0 || division == 1);
    return ROOT_BLOCK_HEADER_SIZE + division * ROOT_BLOCK_DIVISION_SIZE;
}

class RootBlock
{
public:
    RootBlock(LSS& lss);
    ~RootBlock();
    void Read(bool& gracefulShutdown);
    void WriteStaticHeader();
    void WriteDynamicHeader(bool gsf);
    void PrepareNextDivisionInMemory(IInputStream& serialisedRpmRootNode);
    void WriteNextDivisionToDisk();
    void WriteDiagnosticInfo(xostream& os) const;

private:
    LSS& lss_;
    RootBlockHeader header_;
    octet_t* rootBlockBuffer_ = nullptr;
    ModSeqNum msn_ = 0;
};

38.2 Challis algorithm

The Challis algorithm is used for the information in the root block of the LSS which is updated during each check point.

Consider that we need to atomically write some data to a fixed check point location on disk. One approach is to duplicate the data in two different places and write them in strict alternation. Each mirrored copy (or division) begins and ends with a modification sequence number (MSN). This serves two purposes:

On recovery, if the start and end MSNs of a division are different then it indicates that a crash occurred in the middle of writing that division, so it is corrupt and must be ignored. For extra confidence, it is also possible to compute a checksum such as a CRC and store this in each division.
If both divisions are valid, then we pick the one with the largest MSN, since this is the most up to date.

This is called Challis' algorithm.

When writing a division it is vital that error codes are checked to be confident that the division is written correctly. Otherwise it is possible that both divisions become invalid, and that may be disastrous.

Representation of each division on disk

The following grammar defines the layout of a division on disk.


    <uint32> = (little endian binary representation using 4 octets)
    <int32> =  (little endian binary representation using 4 octets)

    <crc32> = <uint32>
    <msn>   = <int32>
    <zero-padding>   = 0x00 *

    <division> = <crc32> <msn> <payload> <zero-padding> <msn>

Note that the representation is the same on all platforms (and involves little endian representation of integers).

It is assumed the total size of the division is statically defined, therefore zero padding of the payload may be required. The trailing MSN always occupies the last 4 octets of the division.

The 32 bit CRC is calculated on all the octets in the division apart from the first 4 octets.

Buffers in system memory

When the LSS root block is read into a buffer in system memory (in a similar manner to using fread), the buffer corresponds exactly to the bytes on disk. So on a big endian machine, the buffer in memory is using little endian format, just as in the file. For example this is the case for the recorded CRC and the leading and trailing MSNs.

This is the same when a buffer is prepared in memory, ready to be written to disk.

No longer using packed structures

Originally the implementation used data structures which exactly match the representation on disk. However that approach has the following disadvantages:

It requires #pragma pack(push,1) directives, and that raises questions about unaligned memory access on some architectures such as ARMv7.
The memory footprint is increased when the disk representation is greater than what is needed in memory - such as typically in the case where there are mirrored copies, and the copies may be padded with zeros
Even if unaligned memory access is supported by the hardware, there may be performance issues
We don't get the little endian versus big endian byte swapping for platform independent representations of ints and floats on disk. We want to allow LSS files to be written on one machine and read on another.

It is typically best to work with the native platform dependent representation in system memory and only use the platform independent representation in the buffers which are read/written from disk.

CRC

The CRC32 on each division is calculated on the buffer in memory which starts with the leading MSN (msn1) and ending with the trailing MSN (msn2). Note that the CRC is inclusive of the leading and trailing MSNs.

Having both the buffer in memory as well as platform dependent data structures representing the same information makes the design more complex to understand. It is better to only use a buffer when it is needed. This happens when:

Reading the rootblock - this involves a single 64kB read.
Writing the rootblock when the store is being created for the first time.
Writing the dynamic header just after the store is opened and just before it is closed, to indicate whether a graceful shutdown has occurred.
Write a Challis division during a check point

The two divisions are identified by an index which can be 0 or 1.

ChallisDivisionInfo

The following data structure holds information about a single division:


struct ChallisDivisionInfo
{
    CRC32 file_crc = 0;     // The CRC recorded in the file
    CRC32 calc_crc = 0;     // The CRC calculated from the data in the file
    ModSeqNum msn1 = 0;     // The leading MSN
    ModSeqNum msn2 = 0;     // The trailing MSN
};

Consider that the LSS has just been opened and the 64kB root block has been read from disk into a buffer in memory. For a given division we can obtain a pointer to the start of the division in system memory.

The following method of ChallisDivisionInfo can be used to initialise all four members from a division loaded into a buffer in system memory.


void ReadFromFile(const octet_t* pDivision, ssize_t divisionSize);

Writing to the LSS

The following methods of interface IRAS are relevant to writing the root block. Note unfortunately that this API implies synchronous I/O and memory copying. We won't be fixing that today! It is important to allocate the buffer with the provided allocation function.


typedef int64 RASAddress;

struct IRAS
{
    // Allocate memory blocks that are suitable for Read() and Write() calls because they are aligned
    // on sector boundaries.
    virtual void* AllocateMemoryBlock(int size) = 0;
    virtual void FreeMemoryBlock(void* p) = 0;

    // Write a buffer to the store.  The size of the file will automatically grow if necessary
    // to include the given range.
    // Throws exceptions on error
    // Note that write caching is not employed.  The data is flushed synchronously by this function
    virtual void Write(const void* buffer, RASAddress offset, int numBytes) = 0;

    ...
};

Root block sections


    const int ROOT_BLOCK_STATIC_HEADER_SIZE = 1024;
    const int ROOT_BLOCK_DYNAMIC_HEADER_SIZE = 1024;
    const int ROOT_BLOCK_HEADER_SIZE = ROOT_BLOCK_STATIC_HEADER_SIZE + ROOT_BLOCK_DYNAMIC_HEADER_SIZE;

    const int ROOT_BLOCK_DIVISION_SIZE = 31 * 1024;

    // The root block consists of the header plus two divisions.  With the above values this is
    // 1k + 1k + 31K + 31k = 64k
    const int ROOT_BLOCK_SIZE = ROOT_BLOCK_HEADER_SIZE + (2 * ROOT_BLOCK_DIVISION_SIZE);

    root block =
    [
        static root header (1kB)
        dynamic root header (1kB)
        root block division 0 (31 kB)
        root block division 1 (31 kB)
    ]

The following function gets the offset into the file for the given division.


// division must be 0 or 1.
inline ssize_t GetDivisionOffset(int division)
{
    cxAssert(division == 0 || division == 1);
    return ROOT_BLOCK_HEADER_SIZE + division * ROOT_BLOCK_DIVISION_SIZE;
}

Writing a new division

It is assumed the two divisions in the root block can be written independently, so writing the next division doesn't upset the previous valid division.

In order to write a new Challis division we need to know which division. This is determined by the least significant bit of the MSN.

With existing compile time settings the root block is 64kB, with a 2kB header. Division 0 is offset 2kB and division 1 is offset 33kB.


// rootBlockBuffer is the pointer to 64kB buffer used for reading and writing the root block or parts thereof
void WriteDivision(IRAS& ras, int32 msn, octet_t* rootBlockBuffer)
{
    int division = (msn & 1);   // 0 or 1
    RASAddress divisionOffset = GetDivisionOffset(division);

    octet_t* buffer = rootBlockBuffer + divisionOffset;

    // Leave space for the CRC
    octet_t* start = buffer + sizeof(CRC32);

    OutputArchive ar(start);
    ar << msn;
    SerialisePayloadOfDivision(ar);
    ar << msn;
    cxAssert(ar-buffer == ROOT_BLOCK_DIVISION_SIZE);

    // Fill in the CRC
    Serialise(buffer, CalculateCRC32(start, ar-start));

    ras.Write(buffer, divisionOffset, ROOT_BLOCK_DIVISION_SIZE);
}

Validating a division

When the LSS is opened we need to check the two divisions to see whether they are valid. If neither is valid we are stuffed. If only one is valid then that is the one we pick. If both are valid we pick the one with the highest MSN.

We want a function which validates a given division, and either returns -1 if invalid, or else the MSN


// rootBlockBuffer is the pointer to 64kB buffer used for reading the root block
int32 ValidateDivision(const octet_t* rootBlockBuffer, int division)
{
    cxAssert(division == 0 || division == 1);
    RASAddress divisionOffset = GetDivisionOffset(division);

    octet_t* buffer = rootBlockBuffer + divisionOffset;
    CRC32 crc;
    int32 msn1, msn2;
    InputArchive ar(buffer);
    ar >> crc >> msn1;
    ar.Skip( ROOT_BLOCK_DIVISION_SIZE - sizeof(CRC32) - 2*sizeof(MSN) );
    ar >> msn2;
    cxAssert(ar-buffer == ROOT_BLOCK_DIVISION_SIZE);

    if (msn1 < 0)
        return -1;  // Invalid MSN

    if (msn1 != msn2)
        return -1;  // Leading and trailing MSNs are unequal

    if (crc != CalculateCRC32(buffer + sizeof(CRC32), ROOT_BLOCK_DIVISION_SIZE - sizeof(CRC32)))
        return -1;  // Invalid because CRC is wrong

    return msn1;
}

References

Challis, M. F. "Database Consistency and Integrity in a Multi-User Environment", Databases: Improving Usability and Responsiveness, Academic Press, pp. 245-270, 1978.

Code


using ModSeqNum = int32;

void WriteChallisDivisionInfo(octet_t* pDivision, ssize_t divisionSize, ModSeqNum msn);

struct ChallisDivisionInfo
{
    bool IsValid() const { return msn1 >= 0 && msn1 == msn2 && file_crc == calc_crc; }
    void WriteDiagnosticInfo(xostream& os) const;
    void ReadFromFile(const octet_t* pDivision, ssize_t divisionSize);
    ModSeqNum GetMsn() const { return IsValid() ? msn1 : -1; }
    static constexpr ssize_t PayLoadSize(ssize_t divisionSize) { return divisionSize - sizeof(CRC32) - 2*sizeof(ModSeqNum); }
    static constexpr ssize_t PayLoadOffset() { return sizeof(CRC32) + sizeof(ModSeqNum); }

    CRC32 file_crc = 0;     // The CRC recorded in the file
    CRC32 calc_crc = 0;     // The CRC calculated from the data in the file
    ModSeqNum msn1 = 0;     // The leading MSN
    ModSeqNum msn2 = 0;     // The trailing MSN
};

struct ChallisDivisionPair
{
    void WriteDiagnosticInfo(xostream& os) const;

    // Returns MSN for the latest valid version, or -1 if neither version is valid
    // if useMostUpToDateDivision is false then try to return the second latest version
    ModSeqNum GetMSNOfLatestValidVersion(bool useMostUpToDateDivision) const;

    ChallisDivisionInfo divisions[2];
};

39 The log

The LSS records changes by appending log records to a sequence of segments. Log records describe the stored packets and the events needed for recovery, while log flush units provide the framing, validation and durability boundaries used when the log is written and recovered.

The following subchapters describe log flush units, the different forms of log record, and the positions used to identify records within the segmented log.

39.1 Log Flush Units

Flush Unit Header

A Flush Unit Header is 32 bytes and contains the following fields:

Field	Description
`CRC32 crc32`	32 bit CRC calculated on all remaining bytes in the flush unit starting with the checkPointId and ending with the payload
`Guid checkPointId`	Used to verify that this is in fact a flush unit header belonging to the LSS
`FlushSeqNumber fsn`	A 32 bit Flush Sequence Number assigned when the flush unit was written to the tail of the log.
`int numBytesInPayload`	32 bit size of the binary payload in bytes associated with this flush unit
`SegId nextSegid`	Identifies the next segment in the forward linked list of segments in the log. Never zero - even for the last segment in the log! To avoid needing to write this value later, we instead make an early decision on what segment will be used next! This choice is marked in the SUT in memory. Since this value is never zero some other means is required to identify the end of the log. This is achieved using a strong validation test on flush units during the recovery scan.

Log Flush Unit

The RAS provides a method that returns the sector size. This will typically be 512 bytes. In that case there will be 1024 sectors per 512k segment.

Data is always written to the end of the log using a Log Flush Unit (LFU). An LFU always begins and ends on disk sector boundaries within a single segment. LFUs are written one after the other until the segment is full.

A flush of the log involves

Preparing the relevant sectors in memory (within the segment in memory to be flushed)
Writing all the sectors to disk.

An LFU begins with a FlushUnitHeader. This takes up the first 32 bytes of the first sector in the LFU. The header stores the CRC calculated on the header + payload, allowing for validity testing of the LFU. In particular it tests whether the LFU was written in its entirety to disk. This avoids many assumptions about the hard-disk such as atomicity of writing disk sectors, or the order in which sectors are written.

After the header is the binary payload. The payload is zero padded to fill up the last sector in the LFU.

Checksum

The first field in a flush unit header is a 32 bit CRC. When a flush unit is closed, the LSS calculates the CRC over the rest of the header, beginning with the check point Id, and the flush unit payload. The CRC field itself and the sector padding after the payload are not included. The recorded CRC is checked when a flush unit is read during recovery and other operations which scan flush units.

Calculating a CRC over every flush unit payload has proved to be a significant overhead to write performance. The source contains a faster additive 32 bit checksum named CalcChecksum32 as a commented alternative, but the current implementation uses CalculateCRC32.

The compile-time macro ENABLE_FLUSH_UNIT_CRC controls the CRC calculation. Setting it to zero causes the calculated value to be zero, effectively disabling validation of the flush unit contents. This option was only added to measure the performance cost of CRC calculation. The intention has never been to use an LSS without checksums.

Checksums are required because a process may fail after only part of a flush unit has been written. Segments are not cleared when they are recycled, so a partial write can leave a mixture of new data and data from the previous use of the segment. The old data may include log records and snapshot records. Recovering those records as though they belonged to the new flush unit could corrupt the recovered state of the store.

The LSS also makes no assumption about the order in which writes reach non-volatile storage. Storage devices are well known for reordering writes, particularly when they have large on-device memory caches. Such reordering is particularly problematic for Write Ahead Logging, which depends on log records reaching durable storage before the data pages they protect. In practice, reliable durability barriers may be unavailable, incorrectly implemented, or sufficiently expensive that they are disabled. The correctness of a WAL system then depends on assumptions that the storage stack does not satisfy. The LSS instead combines the checksum, check point Id and flush sequence number when validating flush units, making it extremely unlikely that a recovery scan will accept invalid data.

Check points should not be performed too frequently, because a check point updates the root block with a new recovery position and check point Id, exposing another boundary at which out-of-order writes must be considered. When the store is opened, the LSS reads the last valid root block division and validates every flush unit from the beginning of the segment containing the check point up to the recorded check point position. The flush units must have valid payload sizes and checksums, and the scan must end exactly at the recorded position. This is intended to detect errors caused by writes reaching non-volatile storage out of order around a check point.

During recovery, the system will find the last valid LFU. All the following must be true for a valid LFU.

The checkPointId is correct
The FSN corresponds to the next assigned sequence number to the LFU
The size of the payload is reasonable (e.g. it must not overflow the segment)
The NextSegId is a valid segment number
The last sector is correctly padded with zeros
The CRC is correct

Note that segments can be recycled without clearing away old data. The system must robustly identify the end of the log. It is assumed that the validity test defined above will have extremely low probability of making a mistake.

The checkPointId is a guid generated each time a check point is performed, and stored in the root block. This avoids one LSS file accidentally recovering LFUs that don't belong to the check point.

Recovery

Recovery begins by reading the root block and finding the last valid check point. The check point has an associated check point Id - a 128 bit guid. This is used to validate flush units in the recovery scan.

Now 2^128 = 3.4E38 is a very large number, so the probability of incorrectly validating a flush unit is very small. In fact, recovering flush units at the rate of 1GHz would take 10^22 years to give a reasonable chance of seeing randomly generated bytes look like the check point Id.

Between check points, it is not possible for segments to be recycled - because of the use of the delta-FSS. Therefore, all segments written with the checkPointId will comprise a single linear list of segments.

After recovery is completed, the store is immediately check pointed. This generates a new check point id, eliminating the chance that previously written flush units will be accidentally validated on a subsequent recovery.

Example scenario :

Store is gracefully shut down. This always involves a check point, and therefore a new check point id will be generated. On startup no recovery scan of the store will be attempted. Also, there is no need to check point the store on startup. Flush units will be written using the last valid check point.
[It will be safer to check point the store anyway - because when the file for the store is copied, we always continue writing flush units with independent GUIDs, avoiding any risk of one store recovering the other's store's flush units]
Store is not gracefully shut down, a large number of segments have been written since the last valid check point. In the worst case, the sectors may have been written to disk out of order. On startup a recovery scan will be performed. This will validate flush units in turn. The recovery scan ends at the first flush unit with an invalid CRC or checkPointId. This may happen "early" if sectors have been written out of order. If the power fails then recovery will be repeated the next time. After recovery a check point is performed. Only when this is completed successfully will the next recovery start the scan from a different position and use a different check point Id. At that point we don't care about all the old flush units because they will have an out of date check point Id in the flush unit headers.
Between check points segments may be cleaned. However they are only marked as free in the delta-FSS, so there is no chance they will be recycled. Therefore we will never write a segment that already contains flush units with the existing check point id.

Code


// Sequence number assigned to flush units in the log.  Starts at one for each check point
typedef int32 FlushSeqNumber;

// 32 bytes
const int FLUSH_UNIT_HEADER_SIZE = sizeof(CRC32) + sizeof(Guid) + sizeof(FlushSeqNumber) + sizeof(int32) + sizeof(SegId);

#pragma pack(push,1)
    struct FlushUnitHeader
    {
        void AssignCRC() { crc32_ = CalcCRC(); }
        bool IsValidCRC() const { return crc32_ == CalcCRC(); }

        enum EStatus
        {
            OK,
            BAD_CRC,
            BAD_CHECKPOINTID,
            BAD_FSN,
            BAD_PAYLOAD_SIZE,
            BAD_NEXT_SEGID,
        };

        // Validate this flush unit, where it is expected to have the given checkPointId and fsn.
        // positionOfFlushUnit is the zero based offset in bytes of the flush unit header from the
        // start of the segment.
        EStatus GetValidityStatus(const Guid& checkPointId, FlushSeqNumber fsn, int positionOfFlushUnit, int lssSegmentSize) const;

        bool IsPayloadSizeValid(int positionOfFlushUnit, int lssSegmentSize) const;

        void WriteToStream(xostream& os) const;

        // Assumes numBytesInPayload_ is initialised correctly
        CRC32 CalcCRC() const;

        //////////////// Member variables
        CRC32 crc32_;
        Guid checkPointId_;
        FlushSeqNumber fsn_;
        int32 numBytesInPayload_;
        SegId nextSegid_;
    };
#pragma pack(pop)

class FlushUnitIterator
{
public:
    FlushUnitIterator() : segment_(nullptr), pos_(0) { }
    FlushUnitIterator(Segment* segment, int pos) { Set(segment,pos); }
    int GetPos() const { return pos_; }
    int GetEndPos() const;
    Segment& GetSegment() const { cxAssert(segment_); return *segment_; }
    void Set(Segment* segment, int pos)
    {
        segment_ = segment;
        pos_ = pos;
        AssertValid();
    }
    FlushUnitHeader& operator*() const
    {
        AssertValid();
        cxAssert(pos_ < segment_->GetSegmentSize());
        return * (FlushUnitHeader*) segment_->GetRawBuffer(pos_);
    }
    FlushUnitHeader* operator->() const
    {
        return &operator*();
    }
    FlushUnitIterator& operator++()           // Prefix
    {
        pos_ = GetEndPos();
        AssertValid();
        return *this;
    }
    bool IsPayloadSizeValid() const
    {
        cxAssert(segment_);
        FlushUnitHeader& header = operator*();
        return header.IsPayloadSizeValid(pos_, segment_->GetSegmentSize());
    }
    int GetPayloadPos1() const
    {
        return pos_ + FLUSH_UNIT_HEADER_SIZE;
    }
    int GetPayloadPos2() const
    {
        FlushUnitHeader& header = operator*();
        return pos_ + FLUSH_UNIT_HEADER_SIZE + header.numBytesInPayload_;
    }
    void WriteToStream(xostream& os) const;

public:
    Segment* segment_;

    // The zero based offset in bytes of the flush unit header from the start of the segment.
    // Must be a multiple of the disk sector size.
    int pos_;
};

struct IFlushUnitVisitor
{
    virtual void VisitFlushUnit(int positionOfFlushUnit, const FlushUnitHeader& h) = 0;
};

const FlushSeqNumber DONT_CHECK_INITIAL_FSN = -1;

// Visit all the valid flush units within the range [i1,i2) of the given segment where i1 and i2
// are multiples of the disk sector size.
// Returns the end position of the scan, which will be a multiple of the disk sector size.  This
// may be i2, or less than i2 if an invalid flush unit was encountered.

int VisitValidFlushUnitsInSegment(
    Segment& segment,
    int i1, int i2,                 // Range [i1,i2) where i1 and i2 are on disk sector boundaries
    FlushSeqNumber& fsn,            // Can equal DONT_CHECK_INITIAL_FSN
    const Guid& checkPointId,
    IFlushUnitVisitor& visitor);

39.2 LogRecord

Log records are written to flush units within segments. Obviously it is not permissible for a log record to overflow a FlushUnit or Segment.

The physical position of a log record is specified with a LogRecordPosition. This identifies the segment using a 32 bit SegId and a 32 bit integer offset within the segment.


struct LogRecordPosition
{
    SegId segid_;
    int32 offset_;
};

Given the physical position, it is possible to efficiently seek directly to that location and read the log record.

Kinds of log records

There is an enum corresponding to the 5 kinds of log records:


enum ELogRecordType
{
    LR_PACKET,
    LR_DELETE_PACKET,
    LR_SNAPSHOT,
    LR_TIMESTAMPED_SNAPSHOT,
    LR_NEXT_SEID_HIGH
};

Record	Description
`LR_PACKET`	A packet contains a contiguous blob of binary data and is uniquely identified by a `Seid`. There are two kinds of packets: An RPM packet records the serialised state of a single RPM node at level 0 through to level 6. Dirty RPM nodes are serialised as RPM packet log records in bottom up order during a check point. Data packets are used for recording serial elements. Serial elements can be very large (many giga bytes) therefore they may be split into a forward linked list of data packets. These are sometimes called packet chains. We call the first packet in the chain the head packet. Each packet in the chain is identified by a Seid. A data packet records the seid of the next data packet in the chain if any. The RPM records the locations of all the data packets in a chain.
`LR_DELETE_PACKET`	Records the deletion of a single serial element with a given `Seid`. The `Seid` must identify the head packet of a packet chain.
`LR_SNAPSHOT`	Commits previously written packet records. Used in version 1 stores, otherwise no longer used.
`LR_TIMESTAMPED_SNAPSHOT`	Like LR_SNAPSHOT but adds both a transaction sequence number and a timestamp for when the transaction was committed. Used after version 1 stores.
`LR_NEXT_SEID_HIGH`	Records a `SeidHigh` to log a change to the seid high allocator in the RPM (i.e. the member nextSeidHigh_ of RPM7).

A log record begins with a single octet which gives its type:


using LogRecordType = octet_t;

Since there are only 5 kinds of log records, we can use the low 4 bits of LogRecordType for recording the ELogRecordType and the high 4 bits provide 4 additional flags, which are used by LR_PACKET log records.


const int NEXT_SEID_PRESENT_BITPOS = 7;
const int HEAD_PACKET_BITPOS       = 6;
const int RPM_PACKET_BITPOS        = 5;
const int CLEANING_PACKET_BITPOS   = 4;

const LogRecordType NEXT_SEID_PRESENT_MASK = (LogRecordType) (1 << NEXT_SEID_PRESENT_BITPOS);
const LogRecordType HEAD_PACKET_MASK       = (LogRecordType) (1 << HEAD_PACKET_BITPOS);
const LogRecordType RPM_PACKET_MASK        = (LogRecordType) (1 << RPM_PACKET_BITPOS);
const LogRecordType CLEANING_PACKET_MASK   = (LogRecordType) (1 << CLEANING_PACKET_BITPOS);

Bit	Description
`NEXT_SEID_PRESENT_BITPOS`	Set for a data packet record which is not at the end of the packet chain. In that case the serialised packet finishes with the `Seid` of the next packet in the chain.
`HEAD_PACKET_BITPOS`	Set for a data packet which is at the head of a packet chain recording a serial element (i.e. the first packet in the packet chain). For robustness, it is preferable to distinguish between packets at the head of their chain from the remaining overflow packets. This allows the LSS to detect when an attempt is made to read/write or delete an invalid Seid - i.e. that doesn't correspond to the head packet of a serial element.
`RPM_PACKET_BITPOS`	Set for an RPM packet. Otherwise it is a data packet.
`CLEANING_PACKET_BITPOS`

Serialised format of log records

The serialised format of a log record is described by the following (binary) grammar


    <octet> = 8 bit unsigned integer

    <uint32> = <octet> <octet> <octet> <octet>
               (little endian binary representation of a 32 bit unsigned integer)

    <uint64> = <octet> <octet> <octet> <octet> <octet> <octet> <octet> <octet>
               (little endian binary representation of a 64 bit unsigned integer)

    <LogRecordType> = <octet>

    <SeidHigh> = <uint32>

    <seid> = <uint64>

    <next-seid> = <seid>

    <data-size> = <uint32>

    <data> = <octet>*

    <TxnSeqNumber> = <uint64>

    <HPTime> = <uint64>

    <PacketLogRecord> =
        <LogRecordType>
        <seid>
        <data-size>
        <data>
        [ <next-seid> ]

    <DeletePacketLogRecord> =
        <LogRecordType>
        <seid>

    <SnapShotLogRecord> =
        <LogRecordType>

    <TimeStampedSnapShotLogRecord> =
        <LogRecordType>
        <TxnSeqNumber>
        <HPTime>

    <NextSeidHighLogRecord> =
        <LogRecordType>
        <SeidHigh>

    <LogRecord> =
        <PacketLogRecord> |
        <DeletePacketLogRecord> |
        <SnapShotLogRecord> |
        <TimeStampedSnapShotLogRecord> |
        <NextSeidHighLogRecord>

Data alignment

Log records can appear at almost any offset position within a segment because log records are written consecutively within a flush unit and log records can be almost any size when serialised.

Therefore is not possible to use data structures for reading and writing the parameters of log records on architectures such as ARMv7 which don't support unaligned memory access for 32 bit and 64 bit values.

Code


enum ELogRecordType
{
    LR_PACKET,
    LR_DELETE_PACKET,
    LR_SNAPSHOT,
    LR_TIMESTAMPED_SNAPSHOT,
    LR_NEXT_SEID_HIGH
};

const int NEXT_SEID_PRESENT_BITPOS = 7;
const int HEAD_PACKET_BITPOS       = 6;
const int RPM_PACKET_BITPOS        = 5;
const int CLEANING_PACKET_BITPOS   = 4;

using LogRecordType = octet_t;

const LogRecordType NEXT_SEID_PRESENT_MASK = (LogRecordType) (1 << NEXT_SEID_PRESENT_BITPOS);
const LogRecordType HEAD_PACKET_MASK       = (LogRecordType) (1 << HEAD_PACKET_BITPOS);
const LogRecordType RPM_PACKET_MASK        = (LogRecordType) (1 << RPM_PACKET_BITPOS);
const LogRecordType CLEANING_PACKET_MASK   = (LogRecordType) (1 << CLEANING_PACKET_BITPOS);

// The size of a packet header is 13 octets:
//
//      LogRecordType   type        1 octet
//      Seid            seid        8 octets
//      int32           bufferSize  4 octets
const size_t PACKET_HEADER_SIZE = sizeof(LogRecordType) + sizeof(Seid) + sizeof(int32);

// Sequence number assigned to snapshot records in the log
using TxnSeqNumber = int64;
const size_t TIMESTAMPED_SNAPSHOT_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(TxnSeqNumber) + sizeof(HPTime);

const size_t NEXT_SEID_HIGH_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(SeidHigh);

const size_t DELETE_PACKET_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(Seid);

struct PacketInfo
{
    LogRecordType type_ = 0;           // Low two bits equals LR_PACKET
    Seid seid_;
    int32 bufferSize_ = 0;
    LogRecordPosition position_;       // Current position of the packet
    octet_t* buffer_ = nullptr;        // Points at the data of the packet (i.e. offset by PACKET_HEADER_SIZE)
    Seid nextSeid_;
    SegmentUnreserver segmentUnreserver_;      // Unreserves segid when destructs
    SegmentAccessor segmentAccessor_;          // Releases the segment when destructs
};

struct ILogRecordVisitor
{
    virtual void VisitPacketRecord(const PacketInfo& pi) = 0;
    virtual void VisitSnapshotRecord(LogRecordPosition nextPos, TxnSeqNumber txsn, HPTime timeStamp) = 0;
    virtual void VisitDeletePacketRecord(Seid seid) = 0;
    virtual void VisitNextSeidHighLogRecord(SeidHigh seidHigh) = 0;
};

void VisitLogRecordsInBuffer(octet_t* buffer, int segid, int s1, int s2, bool onlyVisitPackets, ILogRecordVisitor& visitor);

// Visit all log records within the given segment in the range [s1,s2) where s1 and s2 are integer
// offset positions into the buffer
// 'onlyVisitPackets' indicates whether the client is only interested in packet log records
void VisitLogRecordsInBuffer(Segment& segment, int s1, int s2, bool onlyVisitPackets, ILogRecordVisitor& visitor);

void VisitAllLogRecordsInSegment(Segment& segment, bool onlyVisitPackets, ILogRecordVisitor& visitor);

///////////////////////////////////////////////////////////////////////////////////////////////////

/*
RetrievePacketBufferAtGivenPosition() provides the basis for reading any given packet in the LSS.
It is suitabe for both data and RPM packets.

Typically the given LogRecordPosition will have been recorded by the RPM.

This function is fully threadsafe (any number of threads can access existing packet buffers).

The implementation calls GetSegment() on the segment cache which may block on I/O or on eviction
of a segment from the cache.  The corresponding ReleaseSegment() in performed when the PacketInfo
provided by the caller destructs (see PacketInfo::segmentAccessor_).

The segment is not reserved (in the SUT) by this function.
*/
void RetrievePacketBufferAtGivenPosition(LSS& lss, LogRecordPosition p, PacketInfo* pi);

bool RemoveChain(LSS& lss, Seid seid);

40 LssTxn


class LssTxn : public ILssTransaction
{
public:
    LssTxn(LSS& lss);
    void Open();
    virtual void Close();
    virtual void FlushWhenClose();
    virtual ICloseableOutputStream* WriteSerialElement(Seid seid);
    virtual bool DeleteSerialElement(Seid seid);
    virtual void DeleteSeidSpace(SeidHigh seidHigh);

private:
    LSS& lss_;
    SerialElementWriter serialElementWriter_;
    bool flushWhenClose_;
    bool haveWrittenPackets_;
    bool haveDeletedPackets_;
};

41 SegmentWriter

Each log flush unit (LFU) stores a check point id and a 32 bit Flush Sequence Number (FSN). The check point is a guid that is generated each time a check point is performed. Consider the following depiction of the log broken up into LFUs.

Log flush units showing check point identities and sequence numbers; the units written by check point cpid2 retain cpid1 until the check point completes

Note the following

An LFU stores the identity of the last completed check point. The LFUs written while check point cpid2 is being made therefore still contain cpid1.
After the last check-point LFU is closed, the LSS records the boundary shown by the dotted line as positionOfLastCheckPoint_. This LogRecordPosition is written into the new root-block division.
The open, empty LFU immediately after that position is then changed to cpid2, and its FSN is reset to 1. Later LFUs increment the FSN.
If the preceding LFU exactly fills a segment, the stored position is the end of that segment; logically it is still the dotted boundary before the first cpid2 LFU.

When to start a new segment

The Segment Writer is responsible for breaking up serial elements into packets.

Serial elements that overflow without good reason will reduce performance. A small serial element that spans two segments reduces localisation which degrades read performance.

Criteria...

Let nFree be the number of bytes remaining in the segment.

Let nSize be the remaining size of the serial element.

If nSize ≤ nFree then we should write the remainder of the serial element as a single packet. If nSize > nFree then the object will overflow. We choose to write part of the serial element to the current segment if nFree > T1 and nSize > T2 where T1 and T2 are suitable fixed thresholds

SegmentWriter

Takes on two responsibilities

manages the writing of Log Record Sets to segments being prepared in memory
manages a lazy writer that uses posted tasks to write segments to disk.

Writing a Log Record Set

Only one thread can write a Log Record Set (LRS) at a time. This is enforced by using a mutex, and providing the following two methods


    void BeginLRS(bool doingCheckPoint);
    void EndLRS();

It is crucial that any thread that calls BeginLRS() will (soon) call EndLRS(), otherwise other threads will be prevented from writing an LRS.

After calling BeginLRS(), a thread will typically call GetSegmentBeingWrittenInMemory() to gain access to the current segment being prepared in memory. It can be assumed that a flush unit has already been opened, and the client simply writes log records at the current write position.

A client will need to check whether the next log record will fit by calling SegmentBeingWrittenInMemory::GetNumFreeBytes(). If there is insufficient space in the current segment, then the client should call


    void StartNewSegment()

in order to start a fresh segment in memory. Again it can be assumed that a flush unit has already been opened, and the client simply writes log records at the current write position. The previous segment will be pushed onto the back of the LazyWriterQueue (LWQ) allowing it to be written to disk by the LazyWriter.

There is no advantage in flushing only part of an LRS. Therefore a client writing an LRS will not be interested in breaking up a segment into smaller flush units. Therefore the client should not call the following methods on SegmentBeingWrittenInMemory


    void StartFlushUnitOnNewSegment(Segment* s, SegId nextSegid);
    bool StartAnotherFlushUnit(LogRecordPosition& flushPosition, bool& needToStartANewSegment);

Managing the lazy writer

All that is required is to start and stop it. Also, a method is provided to allow the log to be flushed.

Support for check pointing

The thread performing a check point must open an LRS in order to write the dirty RPM nodes to the log. BeginLRS() takes a boolean flag to indicate whether the thread is performing a check point.

After writing all the dirty RPM nodes during a check point, the following function is called.


    LogRecordPosition MakeLRSFlushableForCheckPoint(const Guid& checkPointId);

This makes the LRS flushable by closing the current flush unit and opening a new fresh one. The given check point id uniquely identifies the check point. This is written to the freshly opened flush unit and all the subsequent flush unit headers to validate flush units read by a recovery scan. Also the freshly opened flush unit has its FSN reset back to 1.

The returned LogRecordPosition points at the end of the log (for the check point). The position will be aligned on a disk sector boundary and may equal the LSS segment size. It is the appropriate start position for a recovery scan, and therefore will be stored in the root block as part of the check point information.

After the LRS has been closed by calling EndLRS(), the log must be flushed. This is guaranteed to flush the last LRS because the call to MakeLRSFlushableForCheckPoint() made it flushable.

Determining when to perform a check point

The SegmentWriter is responsible for determining when check points should be performed. It simply counts the number of segments that have been prepared in memory since the last check point. When this exceeds a threshold (currently 128, corresponding to 64MB for 512k segments), the following method is called on the LSS.


    LSS::SignalNeedToCheckPoint()

This tells the LSS that another check point needs to be performed.

Code


class SegmentWriter
{
public:
	SegmentWriter(LSS& lss);
	~SegmentWriter();
    void Start(RecoveryScanInfo* rsi);
    void Stop();
    void Flush(bool flushForCheckPoint);
    void FlushLastFlushableLRSForCheckPoint();
    void BeginLRS(bool doingCheckPoint);
    void EndLRS();
    LogRecordPosition UpdateFlushPos();
    SegmentBeingWrittenInMemory& GetSegmentBeingWrittenInMemory() { return segmentBeingWrittenInMemory_; }
    void StartNewSegment();

private:
    LSS& lss_;
    bool isStarted_;
    LazyWriter lazyWriter_;
    TxnMutex txnMutex_;
    bool doingCheckPoint_;
    SegmentBeingWrittenInMemory segmentBeingWrittenInMemory_;
    int numSegmentsSinceLastCheckPoint_;
};

42 LazyCleaner

The cleaner basically needs to clean segments with the lowest utilisation first.

It would be very expensive to maintain a list of (segment Id, utilisation) pairs ordered by utilisation. For a one terabyte store, there are 2 million segments. To continually remove elements and move them to a new position would be quite expensive.

Consider that the cleaner only occasionally scans the SUT in order to calculate a list of segments to be cleaned. This can be done at startup, and after each check point. This could involve the following approach:

Perform a linear scan of the SUT. For each segment id, add it to a "bin" according to the utilisation. The bins correspond to utilisation ranges [0%,10%), [10%,20%), .... , [90%,100%). Note that binning gives us reasonable ordering without the need for an O(nlogn) sorting algorithm
From the bins, determine what segments need to be cleaned

The third-party literature includes research on cleaning algorithms for log-structured stores.

Running the cleaner in the background at regular intervals

Previously

The LazyCleaner used to use a TimeOutTask. The TimeOutTask has a dedicated std::thread which (effectively) sits in a loop waiting for 1000 milliseconds then cleaning segments for up to 200 milliseconds. So the cleaner has an upper bound of about 20% usage of one CPU processor.

The LSS has a LazyCleaner directly in a member variable, so their lifetimes are tied. Start/stop on an LSS calls Start/Stop on the LazyCleaner. This used to in turn call Start/Stop on the TimeOutTask which starts/stops the std::thread:


class LSS
{
    void Start() { lazyCleaner_.Start();  }
    void Stop()  { lazyCleaner_.Stop();   }

    LazyCleaner lazyCleaner_;
};

class LazyCleaner
{
    void Start() { timeOutTask_.Start(...); }
    void Stop()  { timeOutTask_.Stop();     }

    TimeOutTask timeOutTask_;
};

class TimeOutTask
{
    void Start() { thread_ = std::thread(...); }
    void Stop()  { thread_.join(); }

    std::thread thread_;
};

Now

The LazyCleaner now uses an AsyncRepeatedTask.

In other words the LazyCleaner doesn't have a std::thread, instead it uses a boost::asio::io_context from an IoContextPool and a dead-line timer. That means we have no counterpart to calling join() on a std::thread to wait until the thread has finished running the lambda.

An AsyncRepeatedTask is similar to a TimeOutTask in that it repeatedly runs a given task and it provides a Stop() function that synchronously guarantees that the task is no longer executed.


#include "Ceda/cxThread/AsyncRepeatedTask.h"

class LazyCleaner
{
public:
    LazyCleaner(LSS& lss)
        lss_(lss)
    {
    }

    void Start()
    {
        asyncRepeatedTask_ = MakeAsyncRepeatedTask(lss_.GetAnIoContext());
        asyncRepeatedTask_->Start(1000,
            [this](std::atomic<bool>& abort)
            {
                CleanSomeSegments(abort);
            });
    }

    void Stop()
    {
        asyncRepeatedTask_->Stop();
    }

private:
    void CleanSomeSegments(std::atomic<bool>& abort);

    LSS lss_;
    std::shared_ptr<AsyncRepeatedTask> asyncRepeatedTask_;
};

Cleaning

The cleaner is able to relocate objects without loading them into C++ objects (i.e. it simply moves the bytes around). It works at the level of the packets. Therefore it doesn't need a CedaLock, improving concurrency.

The cleaner uses a LRS-op. It transfers live packets within one or more segments to the end of the log, to allow the segments to be freed (ie returned to the FSS). There is no need to flush the log after cleaning a segment.

It is important to understand that cleaning a segment doesn't write to the segment - either in memory or on disk. On the contrary, it is important that it be left alone! This is because the segment is still needed in case of power failure because it is reachable from the last valid check point. In fact it is not permissible to return a cleaned segment back to the FSS, because that may mean that the segment gets reallocated (saying for writing more data to the end of the log). Instead, it is necessary to wait for the end of the next check point. Only then is the segment ready to be re-used. This gives us the concept of the delta-FSS to track the segments that have been freed since the last valid check point. The delta-FSS is only transferred to the FSS at the end of a check point.

Steps to clean a segment

Choose the next segment to be cleaned
Access the segment in the SC. This will load the segment if required
Open a commit phase on the LSS. This will gain a write lock on the log
Scan the packets in the segment, and interrogate the RPM to find out which packets are live. Write the live packets to the end of the log
Close the commit phase. This will write a snapshot record, update the SUT, RPM and release the write lock on the log
Add the SegId of the cleaned segment to the delta-FSS. (Actually this may be implicit - because the utilisation of the segment should have fallen to zero and the SUT detects this transition)

Note that the cleaner only interrogates the RPM during the commit phase. The cleaner sees a stable version of an up to date RPM at this time, so it can't make mistakes about what packets are live.

The cleaner is only permitted to clean segments that come before the last check point. In these segments, only live packet records are useful. Snapshot records have already been taken into account by the last valid check point and are obsolete.

The cleaner can't work on segments that haven't been flushed to disk because the cleaner may only work on segments that come before the last valid check point, and a check point flushes the log.

Choosing the segment to clean

There are significant advantages in prioritising segments to clean, according to the utilisation and the assumed likelihood that the segment will undergo further changes. Cleaning is performed by a low priority background thread that spends most of its time asleep. It wakes up occasionally and checks whether the utilisation of any segments has fallen below a threshold. If so those segments are cleaned.

In practise some packets are "hot" and are changed repeatedly, while other packets are "cold" and are only rarely changed.

A segment is as hot as the hottest live packet remaining within it. It is assumed that coldness of a live packet is proportional to its age. The age of a segment is defined to be the age of the youngest live packet remaining within it (ie the minimum of all the ages).

A hot segment should be cleaned at a lower utilisation threshold than a cold segment because it is likely that the utilisation of a hot segment will continue to fall - and the longer we wait the less unnecessary work that is performed by the cleaner. A particularly hot segment should not be cleaned because its utilisation may fall rapidly to zero, so the cleaner only needs to remove it from the log.

The Sprite log structured storage uses the following measure to prioritise segment cleaning

quality = benefit/cost = free space generated * age / cost = (1-u) * age / (1+u)

where u is the utilisation (from 0 to 1) of the segment. The "benefit" is proportional to the age because a long age suggests that the cleaning will be long lasting.

When a transaction commits, all packets marked as dirty as part of that transaction are stamped with the current system clock. We record the timestamp of the youngest live object within each segment in the SUT. When a transaction commits, for each dirty object (ie live object that becomes obsolete in a segment), we compare its modification time stamp in the PBT to the time stamp stored in the SUT. The SUT is adjusted if necessary so it continues to provide the modification timestamp of the youngest live object remaining in the segment.

When the SUT is check pointed (ie written to the root block) there is probably no advantage in recording the timestamps. This will help reduce the size of the root block.

When segments are first loaded into memory, the modification timestamp is initialised to the current time. This represents very hot segments that are less likely to be cleaned.

Code



class LazyCleaner
{
public:
    LazyCleaner(LSS& lss);

    void Start();
    void Stop();
    void CalcSegmentsToClean();

private:
    void CleanSegment(SegId segid);
    void CleanSomeSegments(std::atomic<bool>& abort);

private:
    LSS& lss_;
    std::shared_ptr<AsyncRepeatedTask> asyncRepeatedTask_;
    std::atomic<bool> haveSegmentsToClean_;
    std::deque<SegId> segmentsToClean_;
};

43 SegmentCache

SegmentCache owns the set of Segment objects resident in memory and provides pinned access to them by SegId. It is both a read cache for segments loaded from the Random Access Store (RAS) and a write cache for segments prepared by the SegmentWriter and awaiting output by the lazy writer.

A successful call to GetSegment() or AllocateFreeSegment() returns a segment with one acquired access, called a pin in this chapter. The caller must eventually release that pin. A pinned segment remains resident and cannot be evicted or recycled. Code normally uses SegmentAccessor to release a pin automatically.

Public operations

GetSegment(segid): Returns a pinned segment, loading it from the RAS if it is not already resident. It may block while another thread loads the same segment, or while waiting for an unpinned cache entry that can be evicted.
AllocateFreeSegment(segid): Returns a pinned, empty in-memory segment for a SegId which the caller has already allocated from the SUT free-space machinery. It does not read the former contents of that physical segment. It may create a new cache object, recycle an unpinned cache object, or reuse an unpinned object with the requested SegId which is still resident.
ReleaseSegment(segment): Releases one pin. When the access count becomes zero, the segment is inserted into the Segment Eviction Queue (SEQ) and becomes eligible for eviction or recycling.
SetMaxNumSegmentsInCache(n): Sets the cache-size limit, which must be at least four, and immediately attempts to evict enough unpinned segments to meet it. Shrinking is best effort when segments are pinned.
GetStats(stats): Returns a synchronized snapshot of cache counters, the configured limit, and the number of resident segments.
Clear(): Deletes every resident segment. Although access to the cache structures is synchronized, the caller must ensure that every pin has already been released and no cache operation remains in progress. The SegmentCache destructor calls this operation.

Clients and read path

The ordinary packet-read path first uses the RPM to map a logical packet Seid to a LogRecordPosition. The position supplies the physical SegId and byte offset. RetrievePacketBufferAtGivenPosition() then calls GetSegment() and addresses the packet at that offset in the pinned segment buffer. Packet chains repeat this process for their remaining packets.

The lazy cleaner is another important client. It pins an entire source segment with a SegmentAccessor while scanning its log records and identifying live packets. Dumping, checkpointing and recovery scanning also use scoped accessors when examining resident segments.

Owned state and representation

The cache owns a map from SegId to resident Segment*, and owns every segment in that map. It also owns the SEQ, the configured capacity, statistics and the mutex which protects their combined state.

The following diagram shows a snapshot of this state. Every entry in map_ points to one resident segment. The clients above the cache hold pins on the top three segments, whose positive access counts equal their incoming pin arrows. The bottom three segments have zero access counts and form the intrusive doubly linked SEQ; the SEQ object points to its first and last members.

SegmentCache containing a SegId-to-Segment map, three client-pinned segments and three zero-access segments in an intrusive doubly linked eviction queue

Part of the cache representation is stored intrusively in each Segment:


int accessCount_;
Segment* prevInSEQ_;
Segment* nextInSEQ_;

These fields are physically members of Segment, but semantically belong to the SegmentCache module. They are private and accessible only to SegmentCache and SEQ. This intrusive representation permits constant-time removal from the eviction queue without a separate node allocation or lookup.

Representation invariant

For every segment resident in the cache:

If its access count is positive, it is pinned and is not in the SEQ.
If its access count is zero, it occurs exactly once in the SEQ.
Every segment in the SEQ is owned by the cache and is present in the cache map.
Only code holding the SegmentCache mutex changes map membership, access counts or SEQ links.

The cache mutex therefore protects state distributed across three classes:


SegmentCache::maxNumSegmentsInCache_
SegmentCache::map_
SegmentCache::stats_
SEQ::first_
SEQ::last_
Segment::accessCount_
Segment::prevInSEQ_
Segment::nextInSEQ_

The following operations access this protected state:


SEQ::Clear()
SEQ::PushFront()
SEQ::PushBack()
SEQ::Remove()
SEQ::TryPopFront()

SegmentCache::Clear()
SegmentCache::GetStats()
SegmentCache::SetMaxNumSegmentsInCache()
SegmentCache::EvictExcessSegments_()
SegmentCache::TryEvictSegment_()
SegmentCache::TryRecycleSegment_()
SegmentCache::TryAddSegment_()
SegmentCache::_TryGetSegment()
SegmentCache::TryAllocateFreeSegment_()
SegmentCache::ReleaseSegment()

The public SegmentCache operations in this list acquire the mutex. The private helpers and SEQ operations are called only when the SegmentCache mutex is already held. This gives the complete set of related state one synchronization boundary even though that state is distributed across SegmentCache, SEQ and Segment.

None of these operations waits for cache capacity or performs RAS I/O while holding the mutex. GetSegment() releases the mutex before loading a segment or waiting on an event, and AllocateFreeSegment() releases it before waiting for the SEQ to become nonempty. This avoids lock-order dependencies between the cache mutex, I/O and blocking waits, and prevents unrelated segment operations from being held behind them.

Cache hits and concurrent loading

Internally, a lookup produces one of three results:


LOADING_OR_AVAILABLE
NEED_TO_LOAD
SEGMENT_CACHE_FULL

For a cache hit, the access count is incremented under the mutex. If it was previously zero, the segment is first removed from the SEQ. The returned pin prevents the segment from being evicted while the client reads it.

For a miss, one thread installs a pinned segment in the map and becomes responsible for loading it. It sets isLoading_, resets the segment's loadedEvent_, releases the cache mutex and calls Segment::ReadFromRAS(). Other threads requesting the same SegId acquire their own pins and wait on that event. Completion is published with a release store to isLoading_; waiting threads use an acquire load before accessing the buffer.

Loading is coordinated per segment, not globally. The cache permits different segments to be loaded concurrently because it does not retain its mutex across RAS I/O. This establishes what SegmentCache permits; it does not by itself establish that every RAS implementation supports those overlapping calls correctly.

Capacity, eviction and recycling

The default LssSettings::maxNumSegmentsInCache is 32. With the historical default segment size of 512 KiB this represents 16 MiB of segment buffers. The setting limits the number of resident segments, so the corresponding byte capacity changes with the configured segment size.

There are no separately sized read and write caches. Segments loaded for reading and segments in the write pipeline share the complete capacity. This allows an unpinned read-cache entry to be recycled instead of blocking a writer merely because a fixed write-cache partition is full, and conversely allows recently written segments to remain useful as cached read data.

If a missing segment is requested while the number of resident segments is below the configured limit, the cache creates a new Segment. At the limit it instead pops an unpinned segment from the front of the SEQ, removes its old map entry, resets it and inserts it under the requested SegId. Recycling avoids a deallocation and allocation of the segment and its buffer.

If the cache is full and the SEQ is empty, every resident segment is pinned. The caller waits on the SEQ's manual-reset event. Releasing the final pin on a segment inserts it into the SEQ, signals the event and allows a waiting request to try again.

Eviction policy

The eviction policy is thread-local and defaults to LSS_LRU_EVICTION. Under the LRU policy, the final release pushes a segment onto the back of the SEQ and eviction pops the segment which has been inactive longest from the front. Under LSS_MRU_EVICTION, the final release pushes the segment onto the front, making a recently released segment the next candidate. The public LssEvictionPolicySetter provides scoped changes to this thread-local policy.

Resizing

Reducing the configured limit immediately evicts available SEQ entries. If all remaining segments are pinned, the attempt stops and the actual cache size may temporarily exceed the new limit. In the current implementation, a later ReleaseSegment() adds an entry to the SEQ but does not itself resume EvictExcessSegments_(); convergence to the lower limit therefore depends on subsequent cache activity. This behaviour should be considered if prompt shrinking is required.

Writable segments

The SegmentWriter calls AllocateFreeSegment() to obtain an empty pinned segment. The pin is handed from the SegmentWriter to the lazy writer along with the completed segment. The lazy writer releases it after writing it to disk. Consequently, segments in the write pipeline cannot be evicted even when the cache is under pressure.

All packets in a segment being prepared in memory can become obsolete before the lazy writer writes that segment. The segment must nevertheless continue through the write pipeline: it forms part of the forward log sequence, including its flush-unit sequence numbers and recovery framing. Its pin prevents cache pressure from discarding it merely because its live-packet utilisation has fallen to zero.

The current log-tail segment is mutable while the SegmentWriter prepares it. Once the SegmentWriter has finished a segment and handed it to the lazy writer, readers may share it while it remains pinned for output; the lazy writer does not modify its recorded contents.

Relationship with segment utilisation and free space

The SegmentCache does not decide whether a physical SegId is logically free. The SUT, reservations, FSS and delta-FSS establish when a SegId may safely be allocated for new log data. The SegmentCache answers a different question: whether a resident in-memory Segment object is currently unpinned and may be evicted or repurposed.

The cache does not inspect utilisation when a pin is released, and it does not add segments to the delta-FSS. A segment with zero live packet utilisation may remain resident like any other cached segment. When the SUT later supplies its SegId to AllocateFreeSegment(), the old cache object can be reset and reused if it is unpinned.

For the ordinary data-read path, this separation permits cleaning to coexist with readers. A cleaner can relocate a live packet and atomically redirect the RPM while an earlier reader continues to use the old packet through its pinned segment. The SUT reservation held by that reader prevents the old physical segment from entering the delta-FSS until the read has released both its cache pin and reservation. The cleaner and reservations chapters describe the wider relocation and reuse protocol.

An allocated SegId already resident in the cache

A segment can remain cached across the checkpoint which makes its SegId reusable. If AllocateFreeSegment() finds that SegId in the map, the current implementation requires its access count to be zero. It removes the segment from the SEQ, sets its access count to one, clears its recorded size and returns it as the new writable segment.

A positive access count in this case is an assertion failure, not a defined blocking path. The ordinary data-read path supports this assertion by pairing the cache pin with a SUT reservation: the RPM lookup reserves the physical segment before the packet buffer is obtained, and PacketInfo destroys its SegmentAccessor before its SegmentUnreserver. A contiguous zero-copy read likewise releases its segment before unreserving the SegId. This proves the required ordering for those read paths: a long-running reader prevents a zero-utilisation segment from reaching the delta-FSS and, ultimately, the reusable FSS. It has not yet been established here that every other cache client which can overlap checkpoint publication preserves an equivalent lifetime rule.

Error handling

If ReadFromRAS() throws a FileException, the loading thread records the error in the LSS file-exception status, clears isLoading_, signals waiting threads, releases its pin and rethrows. The cache contains no per-segment load-result state. A waiting caller therefore wakes after the event is signalled without directly receiving the loader's exception from SegmentCache.

The wider LSS error state may prevent useful work after such a failure, but this dependence is not expressed by the SegmentCache interface. The failure path should be reviewed to ensure that every waiting caller observes the failed load before using the segment buffer.

Open questions

The following points are not resolved by the current review and should not be treated as established properties of the implementation:

All pin and reservation lifetimes. The ordinary packet and contiguous zero-copy read paths release their SegmentCache pin before their SUT reservation. A complete audit is still needed for recovery, checkpointing, dumping, cleaning, the writer pipeline and any other client that can overlap checkpoint publication. Until that audit is complete, the assertion in TryAllocateFreeSegment_() that an already-resident free SegId has access count zero is supported for normal readers but not proved globally.
Concurrent RAS reads. SegmentCache deliberately permits different segments to execute ReadFromRAS() concurrently. Each concrete RAS backend must be checked for the required thread safety and for any hidden serialization or shared file-position state.
Failure observed by waiting loaders. When the responsible loading thread fails, it signals the same completion event used for success. A waiting caller does not receive a stored per-segment exception. It remains to be established that the global LSS error state is checked on every such path before the caller can use an invalid buffer, or else the cache needs an explicit failed-load state.
Convergence after shrinking. If SetMaxNumSegmentsInCache() cannot reach the new limit because all candidates are pinned, later releases only place candidates in the SEQ. It is unresolved whether delayed convergence is intentional or whether final release should resume eviction until the configured limit is met.
Redundant segments in the writer pipeline. The implementation keeps a completed segment pinned until the lazy writer writes it, even if its live-packet utilisation has fallen to zero. The historical rationale is preservation of the forward log and flush-unit recovery sequence. That rationale should be checked against the current recovery format to determine whether writing every such redundant segment remains required or is merely a consequence of the current pipeline.

Implementation components

The following subchapters describe the principal representation components owned by this module:

Segment contains the buffer, write size, loading coordination and intrusive cache state.
SegmentAccessor provides scoped release of a pinned segment.
Segment Eviction Queue (SEQ) holds unpinned eviction candidates.

Code


// Available implementations of the map
#define LSS_SC_STD_MAP              0
#define LSS_SC_STD_UNORDERED_MAP    1
#define LSS_SC_BOOST_UNORDERED_MAP  2
#define LSS_SC_BOOST_FLAT_MAP       3

class SegmentCache
{
public:
    SegmentCache(LSS& lss, int maxNumSegmentsInCache);
    ~SegmentCache();
    void GetStats(SegmentCacheStats& stats);
    void SetMaxNumSegmentsInCache(int n);
    void Clear();
    Segment* AllocateFreeSegment(SegId segid);
    Segment* GetSegment(SegId segid);
    void ReleaseSegment(Segment* s);

private:
    void EvictExcessSegments_();
    Segment* TryEvictSegment_();
    Segment* TryRecycleSegment_(int segid);
    Segment* TryAddSegment_(int segid);

    enum ETryGetSegment
    {
        LOADING_OR_AVAILABLE,
        NEED_TO_LOAD,
        SEGMENT_CACHE_FULL
    };

    std::pair<ETryGetSegment,Segment*> _TryGetSegment(SegId segid);
    Segment* TryAllocateFreeSegment_(SegId segid);

private:
    LSS& lss_;
    mutable std::mutex mutex_;
    int maxNumSegmentsInCache_;
    SEQ seq_;
    SEGMENT_CACHE_MAP map_;
    SegmentCacheStats stats_;
};

44 Recoverable Packet Map (RPM)

Logical contents

The Recoverable Packet Map (RPM) is a persistent map from Seid to LogRecordPosition. It records every data packet in a packet chain, not only the head packet. The head and each overflow packet have their own Seid and their own entry in the RPM. The RPM also records the locations of RPM packets. For each entry, the LogRecordPosition identifies the physical location in the log of the latest version of that packet. There is one RPM for the LSS.

The RPM also maintains the state used to allocate Seids. It records the next available SeidHigh and, for each SeidHigh, the next available SeidLow.

Structure

The RPM is implemented as an eight-level radix tree keyed by the eight bytes of a 64-bit Seid.

Each RPM node in the radix tree stores an array of 256 LogRecordPosition values, using an index in the range [0,255]. The storage space required is at most 256 x 8 = 2 KiB. The levels are labelled from 0 to 7 where 0 is at the "bottom" and 7 is at the "top" of the radix tree. There is exactly one RPM-node at level 7. This node is stored in the root block of the LSS. All the other RPM-nodes are stored in the log as indivisible (packet) log records (indivisible means that the RPM-node can't be broken up into a chain of packets, as done for serial elements).

The level 7 RPM-node in the root block stores the current locations of the RPM-nodes at level 6. The RPM-nodes at level 6 store the current locations of the RPM-nodes at level 5. This continues until we get to the RPM-nodes at level 0. These store the current locations of the packets that contain actual data.

Seid allocations increment the next SeidLow or SeidHigh as ordinary unsigned integers. This fills RPM-nodes in index order at level 0 (for SeidLow) or level 4 (for SeidHigh). When a level 0 RPM-node is filled, we create a new level 0 RPM-node under the parent (level 1) RPM-node. If this level 1 RPM-node is full then we create a new level 1 RPM-node under the parent (level 2) RPM-node. This "create RPM-node on overflow" concept continues up through the levels, corresponding to normal base-256 integer counting. Therefore the fan-out at a given node will often be much less than 256. When an RPM-node is serialised to disk, only the portion of the array of LogRecordPosition that is in use is written to the archive.

In-memory and on-disk state

The on-disk RPM is a snapshot taken at the last checkpoint. It is updated only by a checkpoint and does not change as individual transactions write serial elements. The SUT follows the same model: its in-memory state is current, while its on-disk state is updated at checkpoints.

The in-memory RPM represents the current logical state, but only a subset of its nodes need to be resident. Nodes from the checkpointed RPM are faulted into memory on demand. A clean resident node contains no changes relative to its on-disk representation and may be evicted and loaded again later.

As the Segment Writer writes serial elements as packet chains into segments in memory, it updates the in-memory RPM with the locations of the new data packets. The affected RPM nodes are marked as dirty. Collectively, the dirty nodes record the delta between the current in-memory RPM and the checkpointed RPM on disk. A dirty node cannot be evicted because its changes do not yet exist in the on-disk RPM.

Dirtiness is propagated eagerly up through the ancestors all the way to the root node at level 7. This traversal up through the ancestors can terminate as soon as a node is encountered that has already been marked as dirty.

Checkpoints

When a checkpoint is performed, the dirty RPM nodes at level 0 are written to the log as packet records. This updates the positions of those level 0 nodes recorded in their dirty level 1 parents. The dirty level 1 nodes are then written, followed by each successive dirty level through level 6. The RPM must therefore be written in bottom-up order, beginning with level 0 and ending with level 6.

Finally the level 7 RPM node, which has also been marked as dirty, is written to the root block.

Recovery

The on-disk RPM and SUT are brought up to date at checkpoints rather than being rewritten for every transaction. A checkpoint writes the dirty RPM nodes in bottom-up order, writes the level 7 node to the root block, and records the corresponding SUT state. Together these structures describe the store at the checkpoint.

Recovery begins from the last valid checkpoint and scans the subsequent log records. Data-packet records are self-describing: they identify the Seid and physical packet location and provide the information needed to reconstruct subsequent changes to both the RPM and SUT. Delete records and other recovery-relevant records are handled in the same scan. These changes are not applied merely because their records were encountered; they become part of the recovered state only when the scan reaches the snapshot log record that commits their transaction. Records after the final valid snapshot are ignored.

The recovery scan therefore starts with the checkpointed RPM and SUT and deduces every committed change made since that checkpoint. Transactions do not need to contain explicit records for each corresponding RPM or SUT update. This permits checkpoints to be relatively infrequent while still allowing the current state to be recovered.

Packet relocation and SUT accounting

RPM-nodes that are written to the log have a Seid. Like any other packets they can be copied to the end of the log as a segment is cleaned.

During a check point, dirty RPM-nodes are written to the log. This renders previous versions of the RPM-nodes obsolete - in fact an obsolete packet will no longer be referenced by the RPM. It is necessary to subtract the total size of the obsolete RPM packet from the utilisation in the SUT.

Level 0 RPM nodes do not store data-packet sizes. Updating the SUT for an existing serial element requires processing its packet chain through the Segment Read Cache to obtain each packet's size and next Seid. Caching only the head packet's size in the RPM would therefore provide little benefit, while caching the complete chain metadata would increase the size and complexity of the RPM.

Seid encoding

The 64-bit space of valid Seid values is partitioned into four mutually exclusive sets:

Description	Minimum Seid	Maximum Seid
Null Seid	`0x0000000000000000`	`0x0000000000000000`
Seids used for data packets	`0x0000000100000000`	`0xfeffffffffffffff`
Seids used for RPM packets	`0xff00000000000000`	`0xfffffffffffffffe`
Level 7 RPM-node identifier	`0xffffffffffffffff`	`0xffffffffffffffff`

A Seid used for a data packet has the byte representation c6.c5.c4.c3 c2.c1.c0.p, ordered from most significant to least significant. The four-byte SeidHigh value c6.c5.c4.c3 is in the range 0x00000001 through 0xfeffffff, while the four-byte SeidLow value c2.c1.c0.p may have any value from 0x00000000 through 0xffffffff. Consequently, c6 is less than 0xff; when c6 is zero, at least one of c5, c4 and c3 is nonzero. The range used for data packets is contiguous across both SeidHigh and SeidLow.

        SeidHigh :  00000001 to feffffff
        SeidLow  :  00000000 to ffffffff for each SeidHigh

This provides approximately 1.837 x 10¹⁹ data Seids. More importantly, SeidLow is an ordinary contiguous 32-bit allocation space. Allocating the next Seid normally requires only an integer increment, and ranges of related Seids remain numerically contiguous. This makes allocation, range storage, Seid compression and affiliation substantially simpler.

RPM nodes occupy the separate region whose most significant byte is 0xff. For levels 0 through 6, one or more leading 0xff bytes identify an RPM packet, and the number of leading 0xff bytes identifies its level. The prefix is unambiguous because a Seid used for a data packet cannot begin with 0xff.

        Level 6 RPM packet          ff.ff.ff.ff. ff.ff.ff.c6
        Level 5 RPM packet          ff.ff.ff.ff. ff.ff.c6.c5
        Level 4 RPM packet          ff.ff.ff.ff. ff.c6.c5.c4
        Level 3 RPM packet          ff.ff.ff.ff. c6.c5.c4.c3
        Level 2 RPM packet          ff.ff.ff.c6. c5.c4.c3.c2
        Level 1 RPM packet          ff.ff.c6.c5. c4.c3.c2.c1
        Level 0 RPM packet          ff.c6.c5.c4. c3.c2.c1.c0
        Data packet                 c6.c5.c4.c3  c2.c1.c0.p
                                    <-----------------------  increasing significance

The all-ones value 0xffffffffffffffff identifies the single level 7 RPM node. That node is stored in the root block and is not written as an RPM packet.

Testing the most significant byte distinguishes Seids used for data packets from RPM-node identifiers. Testing the complete Seid against zero detects the null Seid. No reserved values or gaps need to be skipped while allocating within a SeidLow range.

Traversing the RPM

The eight bytes of a data Seid are consumed from most significant to least significant while traversing the RPM. The byte c6 indexes the level 7 root to select a level 6 node, c5 selects a level 5 node, and so on. The final byte p indexes a level 0 node to select the data packet's LogRecordPosition.

Masking and shifting expresses the traversal independently of the host machine's byte order:


uint64 value = seid.value_;
for (int level = 7; level >= 0; --level)
{
    int index = (value >> (level * 8)) & 0xff;
    // Use index at this level of the RPM.
}

Every index is in [0,255], so each RPM node has a uniform fan-out of 256. For an RPM-packet Seid, the leading 0xff bytes identify the node level and the remaining bytes identify its path.

Eviction policy for RPM nodes

RPM nodes are faulted into memory from disk on demand. For a very large store the memory footprint of the RPM nodes can become excessive. Therefore an eviction policy is required.

Navigation to an RPM node always proceeds downwards from the root node. This suggests that parent nodes are evicted if and only if there are no descendants resident in memory.

As a result we only worry about an LRU eviction policy on the level 0 nodes.

Eviction of RPM nodes is performed occasionally by the lazy cleaner thread (only when it is not cleaning). This involves a depth first traversal of the entire RPM tree currently resident in memory. This traversal can automatically evict level 1-6 nodes that have no child nodes resident in memory.

Note that it is important to only evict clean RPM nodes.

Seids must be allocated independently of writing serial elements

Otherwise it would be impossible to create two objects that referenced each other by Seid! This means that Seid allocation must be able to run in a separate phase from writing the serial elements.

Unfortunately this means that we must support creation of whole trees of RPM nodes that don't actually have any descendent data packets. Therefore lifetime management of RPM nodes can't simply depend on whether there exist descendent data packets.

The RPM should be marked as dirty from Seid allocations

It is conceivable that the implementation would only mark the RPM as dirty when the mapping from Seid to LogRecordPosition changes, but not when Seid allocation information changes. However there are two good reasons to additionally mark the RPM as dirty when allocation information changes.

This avoids the chance that eviction of RPM nodes leads to recycled seid allocations
Closing and opening the store (gracefully) won't cause recycled seid allocations - which could surprise the application programmer.

Seid allocations

In order to support independent creation of objects at different sites, each process has its own independent "Seid space", uniquely identified by a 128 bit GUID called an OidSpaceGuid. Each process independently maps OidSpaceGuids to local 32 bit SeidHigh values. This mapping is 1-1.

When a Seid is sent over the wire, it is necessary to map the remote SeidHigh to a local SeidHigh. The OidSpaceGuids are used to initialise the SeidHigh conversion map correctly. This can be done on demand during the course of a session - the sender simply sends (OidSpaceGuid,SeidHigh) pairs before sending new SeidHighs for that session.

Each site will still want to cluster related Seids. Note that inevitably Seids created by different sites can't be clustered because they will always use a different SeidHigh.

Within a Seid space (i.e. for a given SeidHigh), the LSS supports allocation of affiliate Seids. This is based on trying to use the largest possible prefix in the Seidlow. Note that the SeidHigh is disregarded! Related Seids will naturally cluster even though they may have been affiliated via Seids generated on a different site!

We require a map from SeidHigh to OidSpaceGuid and vice versa. The former could be achieved by storing OidSpaceGuids directly in the RPM. The latter would required a separate data structure. However we choose to put this is a layer above the LSS, to keep the LSS simple.

Consider that in a single CedaLock many thousands of objects are created. We want these to affiliate correctly. An autoincrement approach may be reasonably good, but perhaps not as good as using the affiliation hints provided by the process. It is possible to imagine pathological cases where the time order of creating objects doesn't relate very well to their affiliation.

When a Seid is allocated we *expect* that in the near future it will really be used. Therefore there seems little disadvantage in eagerly allocating the RPM nodes.

The only remaining issue is to correctly clip RPM nodes from the tree.

Consider the following scenario :

Create objects
Check into repository.
Delete all objects
Create more objects.
Check into repository.

It would be bad to recycle Seids in step 4. Note that there was no non-graceful shutdown so we can't assume a new Seid space will prevent recycled Seids. The conclusion is that it is not reasonable to recycle Seids.

Non-affiliated Seid Allocations

An individual non-affiliated SeidLow allocation does not append an explicit allocation record to the log. The checkpointed RPM records nextSeidLow_ in each RPM3, while every committed data-packet record written after the checkpoint contains the Seid that was allocated for that packet. The log also has an LR_NEXT_SEID_HIGH record for changes to the SeidHigh allocator.

During recovery, replaying a data-packet record is therefore also evidence that its Seid has already been allocated. If its SeidLow is greater than or equal to the checkpointed nextSeidLow_, the RPM advances nextSeidLow_ to the value immediately after the recovered SeidLow. The same high-water-mark rule is applied to nextSeidHigh_ when required, in addition to replaying LR_NEXT_SEID_HIGH records. Because non-affiliated allocation is sequential and the counters only move forwards, advancing each counter beyond every recovered Seid is sufficient to prevent a recovered Seid from being allocated again.

To support recovery correctly, the RPM needs to be able to create nodes on demand as requests are made to add entries.

For an RPM3 that uses only non-affiliated allocation, nextSeidLow_ records the complete allocation history. Empty descendant RPM nodes therefore contain no allocation state and can be removed without allowing Seids to be reused. When affiliated allocation is used, descendant nodes may contain allocation counters and overflow links that must be retained even when they contain no live data-packet positions. The RPM3 itself must be retained in either case because it stores nextSeidLow_.

Writing dirty RPM nodes

Dirty RPM nodes must be written to the log during a check point.

RPMNode defines a virtual WriteDirtyRPMNodes() method. This base class implementation is suitable for RPM0. If the node is dirty it serialises the RPM packet to the given writer.

WriteDirtyRPMNodes() is overridden by RPMi to *first* recurse into the (dirty) child nodes, before processing that node. Note that dirty nodes are always resident in memory - there is no need to load child nodes into memory. By processing child nodes first, we ensure that dirty nodes are written to the log in bottom up order.

WriteDirtyRPMNodesToLogAndRPM7ToArchive() is implemented by RPM7. This needs to lock the RPM mutex. Also, only the descendent nodes are written to the log. The RPM7 node itself is written to a separate archive that will end up being written to the root block as part of the check point.

Note that RPM dirty does not imply there are nodes to write to the log!

Consider that only a Seid space has been allocated. This only marks the RPM7 as dirty, and therefore there are no dirty nodes to be written to the log.

Bad idea : Using the RPM to track which data packets have been loaded

RPM-nodes for levels 1 to 7 keep track of what RPM-nodes at the next (lower) level have been brought into memory. In fact they directly cache an array of 256 memory pointers to RPM-nodes. By analogy, it would seem reasonable for the level 0 RPM-nodes to keep track of what data packets have been brought into memory.

Instead, the Segment Read Cache (SRC) keeps track of the segments that have been faulted into memory, and there is no concept of eagerly breaking up a segment into independent packets. For that reason we don’t have a C++ class that represents a single packet, and the level 0 RPM-nodes have no idea of what data packets have already been loaded into memory by the SRC.

Interestingly, if packets were indeed represented as C++ objects, and the RPM was used to track which packets have been brought into memory, we would have a significant problem : Segment loading would be unbounded. When a segment is loaded we may read a packet with an OID that requires other segments to be loaded in order to retrieve the relevant RPM nodes (in order to indicate the fact that the packet has been loaded in the RPM in memory). These segments can in turn read packets that require yet more packets to be loaded. In the worst case, a chain reaction could end up loading the entire store into memory!

As a general rule, we favour a design that keeps the (complicated) RPM as simple as possible. This will hopefully reduce the amount of meta-data that needs to be written to the store. For example, the RPM won’t be able to tell us how big an existing packet is for the purposes of updating the segment utilisation in the SUT. Only the SRC can give us access to information stored in the packet, such as the total packet size or the next OID in the chain.

Indexing into the RPM

Given a Seid c6.c5.c4.c3 c2.c1.c0.p used for a data packet, the RPM consumes the bytes in that order. Each byte is used directly as an index in [0,255]. The first seven indexes select the path from the root to a level 0 node; p selects the data packet position stored in that node. No byte value acts as a terminator or requires special treatment.

RPM responsibilities

The RPM is responsible for the following:

Provides a mapping from Seid ---> LogRecordPosition, for every packet in the system, i.e. for both data and RPM packets.
Takes care of marking nodes as dirty, and ensure they are all written to the log in the right order during a check point.
Provide functions for adding and removing entries, either by the LRSWriterForCheckPoint, or during recovery
Provides functions for allocating Seids

Class inheritance hierarchy

Use of `LSS&` in RPM nodes

Every RPMNode stores an LSS&. The reference is accepted by the base-class constructor and is propagated into every node created at every level of the RPM:


class RPMNode
{
public:
    RPMNode(LSS& lss, RPMNode* parent, int index, int level, Seid seid);

protected:
    LSS& lss_;
};

The reference does not represent ownership of the LSS. The LSS owns the RPM, which in turn owns its resident nodes. It is an upward reference from each node to the aggregate which ultimately owns it, and is used as a route from a node to other LSS modules and store-wide state.

Faulting child RPM nodes into memory

Only some of the RPM tree is resident in memory. An internal RPM node records the LogRecordPosition of each non-resident child. When traversal needs such a child, the RPM uses the LSS reference to reach the segment-reading facilities, locate the packet containing the child RPM node, access its bytes and deserialise a new child node. The LSS reference is then passed to the child constructor so the same process can continue at lower levels.

Segment access and reservation

Resolving an RPM entry ultimately produces the physical position of a packet in a segment. RPM operations can use LSS& to reach the segment cache and, where requested by the operation, reserve or acquire access to the corresponding segment. RPM navigation is therefore coupled to both the logical Seid-to-position mapping and the mechanism used to make the containing segment accessible.

SUT utilisation accounting

RPM packets themselves occupy space in log segments. When an RPM packet is replaced, relocated or made obsolete, the utilisation recorded for the affected segments must be adjusted. The RPM uses its route through the LSS to reach the SUT and its reservation/accounting operations. This means an operation which appears to update an RPM node may also update store-wide segment utilisation.

Dirty-node and checkpoint coordination

Changing a packet position or Seid-allocation state marks an RPM node and its ancestors as dirty. Dirty nodes must remain resident and are later written in bottom-up order during a checkpoint. The implementation uses LSS-level RPM and checkpoint state when maintaining this bookkeeping, including store-wide dirty-node counts and the decision that enough RPM work has accumulated to require a checkpoint.

Consequences of the reference

The single LSS& therefore conceals several different dependencies: loading RPM packets, accessing or reserving segments, updating SUT utilisation, maintaining RPM-wide state and coordinating checkpoint work. A node constructor does not reveal which of these facilities a particular node operation can use. Because every descendant receives the reference, every node is also coupled to the lifetime and complete private interface of the LSS even when most node operations only manipulate the node's local arrays and metadata.

Code


class RPMNode
{
public:
    RPMNode(LSS& lss,RPMNode* parent,int index,int level,Seid seid);
    virtual ~RPMNode() {}
    virtual void Clear();
    virtual bool EvictUnusedDescendentNodes(int minSeqNum) = 0;
    int GetLevel() const { return level_; }
    virtual void Serialise(Archive& ar) const;
    virtual void Deserialise(InputArchive& ar);
    virtual void WriteDirtyRPMNodes(LRSWriterForCheckPoint& writer);
    virtual void DumpAllocatedSeids(xostream& os) const {}
    virtual bool RemovePacket(int index);
    virtual bool HaveChild(int index) const;
    virtual void WriteInfo(xostream& os) const;
    LogRecordPosition GetPacketPosition(int index) const { return positions_[index]; }
    void SetPacketPosition(int index, LogRecordPosition pos);
    void SetThisPacketPositionAndSize(LogRecordPosition pos,int totalPacketSize);
    Seid GetSeidOfChild(int index) const;
    void GetSeidsOfChildren(xvector<Seid>& children, bool enableOverflowPackets) const;
    Seid GetSeid() const { return seid_; }
    SeidLow GetSeidLow() const { return seid_.low_; }
    bool IsEmpty() const { return numChildren_ == 0; }
    bool IsFull() const { return numChildren_ == 256; }
protected:
    void MarkAsDirty();

protected:
    LSS& lss_;
    RPMNode* parent_;
    int index_;
    int level_;
    Seid seid_;
    mutable bool isDirty_;
    int totalPacketSize_;
    LogRecordPosition positions_[256];
    int numChildren_;
    SeidLow forward_;
    int numLocalAffiliateAllocs_;
};

class RPM0 : public RPMNode
{
public:
    RPM0(LSS& lss, RPMNode* parent, int index, Seid seid);
    virtual void Clear();
    virtual bool EvictUnusedDescendentNodes(int minSeqNum);
    SeidLow AllocateSeidLow();

    private:
    int lastUsedSeqNum_;
    friend class RPM7;
};

class RPMi : public RPMNode
{
public:
    RPMi(LSS& lss,RPMNode* parent,int index,int level, Seid seid);
    virtual ~RPMi();
    virtual void Clear();
    virtual bool EvictUnusedDescendentNodes(int minSeqNum);
    virtual void WriteDirtyRPMNodes(LRSWriterForCheckPoint& writer);
    virtual void DumpAllocatedSeids(xostream& os) const;
    virtual bool RemovePacket(int index);
    virtual bool HaveChild(int index) const;
    RPMNode* CreateChildNode(int index);
    RPMNode* AllocateChildNode();
    RPMNode* GetChildNode(int index);
    const RPMNode* GetChildNode(int index) const { return const_cast<RPMi*>(this)->GetChildNode(index); }
    RPMNode* AlwaysGetChildNode(int index);
    void WriteDirtyChildRPMNodes(LRSWriterForCheckPoint& writer);
    void CreateChildNodeAndDeserialiseFromPacket(int index, const void* buffer, int size, int totalPacketSize);
    RPM0* AllocateChildLevel0();
    RPMi* AllocateChildLevel1();
    RPMi* AllocateChildLevel2();

private:
    RPMNode* nodes_[256];
};

class RPM3 : public RPMi
{
public:
    RPM3(LSS& lss,RPMNode* parent,int index,Seid seid);
    virtual void Clear();
    virtual void DumpAllocatedSeids(xostream& os) const;
    virtual void Serialise(Archive& ar) const;
    virtual void Deserialise(InputArchive& ar);
    virtual void WriteInfo(xostream& os) const;
    bool AllowForSeidLow(SeidLow seidLow);
    SeidLow AllocateSeidLow();
    SeidLow PeekNextSeidLow() const;
    void GetSeidsInSeidSpace(xvector<SeidLow>& seidLows) const;
    RPM0* BindLevel0(SeidLow x);
    RPM1* BindLevel1(SeidLow x);
    RPM2* BindLevel2(SeidLow x);
    std::pair<SeidLow,bool> AllocateAffiliateSeidLow(SeidLow x);

private:
    SeidLow nextSeidLow_;
};

class RPM7 : private RPMi
{
public:
	RPM7(LSS& lss);
    virtual void Clear();
    virtual void Serialise(Archive& ar) const;
    virtual void Deserialise(InputArchive& ar);
    void WriteDirtyRPMNodesToLogAndRPM7ToArchive(LRSWriterForCheckPoint& writer, Archive& arRootNode);
    LogRecordPosition GetPacketPosition(Seid seid, bool reserveSegment);
    void SetPacketPosition(Seid seid, LogRecordPosition pos);
    bool RemoveDataPacket(Seid seid);
    Seid AllocateSeid(SeidHigh seidHigh);
    bool ReserveSeid(Seid seid);
    SeidLow PeekNextSeidLow(SeidHigh seidHigh);
    Seid AllocateOverflowPacketSeid(SeidHigh seidHigh);
    bool AllocateAffiliateSeid(Seid& seid);
    SeidHigh AllocateSeidHigh();
    void DumpSeidAllocationInfo(xostream& os) const;
    bool NeedCheckPoint() const;
    void GetSeidsInSeidSpace(xvector<SeidLow>& seidLows, SeidHigh seidHigh) const;
    void DeleteSeidSpace(SeidHigh seidHigh);
    void EvictUnusedNodes();
    bool RecurseSeidMap(xvector<Seid>& children, Seid seid, bool enableOverflowPackets) const;
    bool WriteInfoOnNodeForGivenSeid(Seid seid, xostream& os) const;
    bool IsDirty() const { return isDirty_; }
    void IncrementNumDirtyLevel0Nodes() { ++numDirtyLevel0Nodes_; }
    std::optional<SeidHigh> GetDirtyNextSeidHigh();
    void UpdateNextSeidHigh(SeidHigh nextSeidHigh);

private:
    void PrivateSerialise(Archive& ar) const;
    SeidHigh PrivateAllocateSeidHigh();
    void TouchRPM0(RPM0* node);
    virtual void WriteInfo(xostream& os) const;
    RPMNode* GetNodeForGivenSeid(Seid seid);
    const RPMNode* GetNodeForGivenSeid(Seid seid) const { return const_cast<RPM7*>(this)->GetNodeForGivenSeid(seid); }
    void PrivateAllowForSeidHigh(SeidHigh seidHigh);
    RPMNode* GetParentNodeAndChildIndex(Seid seid, int& index);
    RPMNode* AlwaysGetParentNodeAndChildIndex(Seid seid, int& index);
    RPM3* AlwaysGetLevel3Node(SeidHigh seidHigh);
    const RPM3* AlwaysGetLevel3Node(SeidHigh seidHigh) const { return const_cast<RPM7*>(this)->AlwaysGetLevel3Node(seidHigh); }
    RPM3* GetLevel3Node(SeidHigh seidHigh);

private:
    mutable std::mutex mutex_;
    SeidHigh nextSeidHigh_;
    bool nextSeidHighIsDirty_;
    int timeSeqNum_;
    mutable int numDirtyLevel0Nodes_;
};

45 Segment Utilisation Table (SUT)

The Segment Utilisation Table (SUT) describes the utilisation and availability of the segments in an LSS. The cleaner uses it to choose segments worth cleaning, and the Segment Writer uses it to allocate segments that may safely be overwritten.

Logical state

Logically, the SUT consists of the following state:

State	Meaning
Segment size	The fixed size in bytes of every segment in the LSS.
Number of segment entries	The exclusive upper bound of the allocated SegId address space.
Utilisation map	A 32-bit utilisation value for every allocated SegId.
Total utilisation	The sum of all values in the utilisation map, recorded as a 64-bit integer.
FSS	SegIds that may be allocated and overwritten immediately.
Delta-FSS	SegIds that have become free but cannot be reused until a checkpoint makes that safe.
Reservation counts	The number of active uses temporarily preventing each SegId from becoming free.

SegId zero is not a usable segment. If the number of segment entries is N, the allocated segments have SegIds in the range 1 through N - 1, and N is the next SegId that will be created if no existing segment can be reused. Each utilisation must be non-negative and less than the segment size. The total utilisation permits the amount of live packet data in the LSS to be obtained without scanning the utilisation map.

Utilisation

The utilisation of a segment is the total encoded size of all live data and RPM packets in that segment. The encoded size of a packet includes its 13-byte packet header, its payload, and the optional trailing 8-byte Seid identifying the next packet in the chain.

Utilisation does not include snapshot records, delete-packet records, flush unit headers, or padding added at the end of a flush unit to reach a disk-sector boundary. Snapshot and delete-packet records contain recovery information but are not live packets that must be retained or relocated by the cleaner.

Every segment containing packets has at least one 32-byte flush unit header. Consequently, for a 512 KiB segment, the utilisation is strictly less than the segment size and cannot exceed 512 KiB - 32 bytes = 524256 bytes.

Writing a new packet increases the utilisation of its destination segment. Removing or superseding a packet decreases the utilisation of its former segment. Recovery replays these changes so that the in-memory SUT is brought forward from the last valid checkpoint.

Segment availability and reservations

A utilisation of zero means that the segment contains no live data or RPM packet bytes. It does not by itself mean that the segment can be overwritten. Recovery may still need the segment, a reader may still be accessing an obsolete packet, the Segment Writer may be preparing the segment in memory, or the segment may hold an internal SUT section whose bytes are not counted as packet utilisation.

For each segment, the SUT therefore also maintains a reservation count. A reservation denotes a current use that prevents the segment from becoming free, independently of its utilisation. More than one user may reserve the same segment, so the count rather than a Boolean value is logically significant. A segment cannot become eligible for reuse until both its utilisation and reservation count are zero.

The SUT owns two sets of zero-utilisation, unreserved segments:

The Free Segment Stack (FSS) contains segments that may be allocated and overwritten immediately.
The delta-FSS contains segments that have become free since the last valid checkpoint. They cannot yet be overwritten because recovery from that checkpoint may still need their contents.

When a segment's utilisation falls to zero, it is added to the delta-FSS if its reservation count is also zero. If reservations remain, it is added when the final reservation is released, provided its utilisation is still zero. A successful checkpoint makes the segments accumulated in its delta-FSS safe to overwrite, so they are transferred to the FSS after the new checkpoint has been published.

Allocation removes a SegId from the FSS and immediately reserves it. If the FSS is empty, allocation extends the segment address space by creating a new SegId and reserves that instead. The reservation protects a newly allocated segment while its utilisation may still be zero. These rules are described further in Free Segment Stack (FSS) and Reservations.

In-memory and on-disk state

The SUT in memory represents the current logical state. Its utilisation values, total utilisation, segment address-space extent, FSS, delta-FSS and reservations change as the Segment Writer writes packets, packets become obsolete, segments are allocated, and users acquire or release reservations. The on-disk SUT remains the snapshot from the last valid checkpoint; these changes are not written to it individually.

Only part of the in-memory state is persistent. A checkpoint records the total utilisation, the number of segment entries and every segment's utilisation. The FSS is derived from this information at startup by finding zero-utilisation segments that are not occupied by persistent internal structures. The delta-FSS is empty at the state represented by a checkpoint, and reservations for ordinary runtime users are transient. Any reservations required for persistent internal structures are reconstructed when those structures are loaded.

During recovery, the SUT is first reconstructed from the last valid checkpoint. The recovery scan then applies the effects of subsequently committed packet writes, deletions and segment allocations. The resulting in-memory SUT again represents the current logical state, even though those later changes are not yet part of an on-disk SUT checkpoint.

Implementation structure

The SUT class owns the complete state described above and the mutex that protects it. It delegates storage of the segment count and individual utilisation values through the pure abstract interface ISUT. This permits it to select either SmallSUT or LargeSUT without changing the higher-level utilisation, allocation, reservation or checkpoint rules.

SmallSUT stores all utilisations directly in the root block. When that representation grows beyond its configured capacity, the SUT changes permanently to the two-level LargeSUT representation. LargeSUT writes dirty SUT sections using shadow paging and stores references to those sections in the root block. Until the new root-block division is committed, the representation belonging to the previous checkpoint remains valid.

Thread safety

The SUT class serializes access to utilisation, allocation, free-segment and reservation state with one mutex. SmallSUT and LargeSUT are used through that class and do not independently provide the complete concurrency policy.

Why utilisation is not stored at 16-bit resolution

It might appear worthwhile to store each utilisation as a 16-bit value. This would halve the storage required for the utilisation entries in both SmallSUT and LargeSUT, allowing approximately twice as many entries in a root-block division or SUT-section. A lower-resolution value would also seem adequate for comparing segments when selecting profitable candidates for cleaning.

Utilisation is not merely a cleaning estimate, however. It is exact accounting of the encoded bytes belonging to live data and RPM packets. When a packet is written, its exact encoded size is added; when that packet is superseded or deleted, the same exact size is subtracted. The exact result is needed to determine when a segment has no live packets and, subject to its reservation count, may be added to the delta-FSS.

Rounding utilisation to 16-bit resolution loses information needed by later subtractions. Two different exact byte counts can have the same rounded representation. Subtracting the same obsolete packet size can then leave one of those segments empty and the other nonempty, even though the stored values are indistinguishable. Rounding each individual packet contribution does not solve the problem: rounding errors accumulate across packets, small packets may receive inappropriate charges, and the sum of independently rounded contributions may exceed the available 16-bit range.

Segments commonly reach zero without being processed by the cleaner. A segment may contain only a few serial elements, or temporal clustering may place a related tree of objects together so that deleting the tree makes all its packets obsolete. Exact accounting allows these ordinary updates to recognize an empty segment immediately. With only a rounded value, the LSS would instead need an additional exact per-segment counter, a reverse index, or a scan of the segment combined with RPM lookups before it could safely release the segment. Those mechanisms remove the simplicity and much of the storage benefit of the proposed representation.

Consequently, a rounded 16-bit utilisation could serve as a separate cleaning heuristic, but it cannot replace the exact utilisation value while retaining the current zero-detection and free-segment safety guarantees. The SUT therefore stores exact 32-bit utilisations.

Code


class SUT
{
public:
    SUT(LSS& lss);
    ~SUT();
    void SetSegmentSize(int segmentSize);
    void Clear();
    int GetTotalNumSegments() const;
    int GetUtilisation(SegId segid) const;
    void OffsetUtilisation(SegId segid, int deltaUtilisation);
    SegId AllocateSegId();
    void AllocateGivenSegId(SegId segid);
    void DumpSUT(xostream& os) const;
    void WriteDirtySUTSections();
    void Serialise(Archive& ar) const;
    void Deserialise(InputArchive& ar);
    void RetrieveDeltaFss(SegIdStack& deltaFss);
    void AddToFss(const SegIdStack& fss);
    int GetLastUtilisedSegment() const;
    void ReserveSegment(SegId segid);
    void UnreserveSegment(SegId segid);

private:
    int PrivateGetTotalNumSegments() const { return current_->GetTotalNumSegments(); }
    int PrivateGetUtilisation(SegId segid) const { return current_->GetUtilisation(segid); }
    SegId PrivateAllocateSegId();
    void PrivateReserveSegment(SegId segid);
    void PrivateUnreserveSegment(SegId segid);
    void ValidateFSSDuringCheckPoint();
    void IncrementTotalNumSegments() { current_->IncrementTotalNumSegments(); }
    void SetUtilisation(SegId segid, int u) { current_->SetUtilisation(segid,u); }
    bool IsSegmentReserved(SegId segid) const { return reservations_.IsSegmentReserved(segid); }
    void InitialiseFSSAtStartup();

private:
    LSS& lss_;
    mutable std::mutex mutex_;
    int segmentSize_;
    int64 totalUtilisation_;
    ISUT* current_;
    SmallSUT small_;
    LargeSUT large_;
    Reservations reservations_;
    SegIdStack fss_;
    SegIdStack deltaFss_;
};

Segment size	Number of segments	Store size
512 KiB	5,114	2,557.0625 MiB (approximately 2.497 GiB)
1 MiB	5,114	5,114.0625 MiB (approximately 4.994 GiB)
2 MiB	5,114	10,228.0625 MiB (approximately 9.988 GiB)
4 MiB	5,114	20,456.0625 MiB (approximately 19.977 GiB)

46 CheckPoint

A check point is performed at regular intervals to reduce the time required for crash recovery. The last valid check point divides the log at a position called the last valid check point position. This position is represented by a LogRecordPosition, which identifies a segment and an offset within that segment. It divides the log into two sections:

The records before the last valid check point are referred to as the check pointed records.
The records on or after the last valid check point are referred to as the recovery records. These are the records that are scanned to perform a recovery

The recovery segments are the segments that contain recovery records.

Check point identity and log position

Each log flush unit (LFU) contains a check point identity (CPID) and a 32-bit Flush Sequence Number (FSN). The CPID identifies the last completed check point from which recovery can begin. The FSN increases through the LFUs belonging to that recovery sequence.

The diagram shows check point cpid2 being made. Its check-point data, including dirty RPM nodes, is appended in the two highlighted LFUs with FSNs 22 and 23. Those LFUs still contain cpid1, because cpid1 remains the last completed check point while the new check point is being constructed.

After the final check-point LFU has been closed, SegmentWriter::UpdateFlushPos() returns a LogRecordPosition at the end of that LFU. The LSS assigns this value to positionOfLastCheckPoint_. It is exactly the position represented by the dotted line in the diagram: the boundary after the LFUs written by the check point and before the first LFU belonging to the new recovery sequence.

The LSS then calls SegmentBeingWrittenInMemory::SetCheckPointId(). This changes the already open but still empty LFU immediately to the right of the dotted line to cpid2 and resets its FSN to 1. Subsequent LFUs retain cpid2 and increment the FSN.

If the LFU immediately before the dotted line exactly fills a segment, the stored LogRecordPosition is represented as the end of that segment rather than offset zero of the next segment. It still denotes the same logical boundary in the LFU sequence.

Publication in the root block

The next root-block division is prepared with a coherent snapshot of the SUT root, RPM root, positionOfLastCheckPoint_, the new CPID, and the other check-point state. LSS::SerialiseRootBlockDivisionPayload() serialises positionOfLastCheckPoint_ directly into the division as a LogRecordPosition.

The checkpoint LFUs are made flushable and the log is flushed before RootBlock::WriteNextDivisionToDisk() writes the prepared division. Once that division is valid on disk, it publishes cpid2 as the newest check point and records the dotted-line position as the place from which its recovery scan begins.

The relevant source path is:

SegmentWriter::UpdateFlushPos() closes the last check-point LFU and returns the boundary position.
LSS::CheckPointWithGivenCpid() stores the position and switches the empty next LFU to the new CPID and FSN 1.
RootBlock::PrepareNextDivisionInMemory() serialises the check-point state into the next division.
RootBlock::WriteNextDivisionToDisk() writes that division after the log flush.

A checkpoint writes a coherent, up to date version of the RPM and SUT to the log and root block. Locking of the log ensures that no other thread is making changes to the RPM and SUT as they are check pointed.

Check pointing involves the following steps

Begin an LRS-op
Write any dirty RPM-nodes to the log
Write any dirty SUT-nodes to the log
Set lastCheckPoint to point at the end of the log
End the LRS-op
Flush the log
Write the SUT, root RPM-node and the position of the check point record to the root block.

After the check point is completed, the cleaner may be allowed to process more segments because the location of the last check point has advanced.

It is vital that the log be flushed to perform a check point. It is not allowable for the root block to reference log records that haven't been flushed yet.

A check point is not performed if the last LRS-op was a check point.

Check pointing is performed by a thread that sits in an infinite loop, performing check point operations then sleeping for a configurable time (say 3 minutes). The check point reduces the amount of recovery work after a crash.

We should flush the log more regularly than we check point to reduce the potential for significant data loss. Flushing every 20 seconds may be suitable.

The following code might be suitable for the check pointing thread


while(1)
{
    for (int i=0 ; i < numFlushesPerCheckPoint ; ++i)
    {
        FlushLog();
        Sleep(tmFlushInterval);
    }
    CheckPoint();
    Sleep(tmFlushInterval);
}

By sleeping for long intervals, check pointing represents a relatively small I/O and CPU load on the system.

TODO: It would be better to perform a check point after a configurable number of bytes has been written to the log since the last check point - because this relates to the time taken for recovery. During quiet periods check points will be far apart.

Thread safety

The LSS uses a mutex to make sure that only one thread does a check point at a given time.


// Used to write a single LRS for the purposes of a check point
class LRSWriterForCheckPoint
{
public:
    LRSWriterForCheckPoint(LSS& lss);
    ~LRSWriterForCheckPoint();

    // Allow for writing an RPM packet.
    // Used by the RPM to write its dirty nodes during a check point.
    /*gives*/ IOutputStream* WriteRPMPacket(Seid seid, RPMNode* node);

private:
    LSS& lss_;
};

47 Recovery

The recovery scan uses a RecoveryLogRecordVisitor.

Recovery scan

Conceptually, recovery involves two separate phases

Initialise the store according to the last valid check point.
Scan forwards through the daisy chained segments from the last valid check point, "replaying history" but only when snapshot log records are encountered. Note that this scan typically begins midway through the first segment to be scanned.

The "replay history" concept forces certain design decisions. For example, at the end of phase 1 the LSS should be in a valid state, and corresponding to the time when the last valid check point was performed. This makes phase 2 optional apart from a desire to recover as much data as possible.

Phase 1 begins by reading the root block and finding the last valid check point. The check point has an associated Check Point Id (CPID) - a 128 bit guid. This is used to validate flush units in the recovery scan.

Now 2^128 = 3.4 x 10^38 is a very large number, so the probability of incorrectly validating a flush unit is very small. In fact, recovering flush units at the rate of 1GHz would take 10^22 years to give a reasonable chance of seeing randomly generates bytes look like the check point Id.

During recovery, the system will find the last valid Log Flush Unit (LFU). All the following must be true for a valid LFU.

The CPID is correct
The FSN corresponds to the next assigned sequence number to the LFU
The size of the payload is reasonable (e.g. it must not overflow the segment)
The NextSegId is a valid segment number
The last disk sector is correctly padded with zeros
The CRC is correct

Note that segments can be recycled (from the free segment stack) without clearing away old data. The system must robustly identify the end of the log - it certainly must not inadvertently recover garbage. It is assumed that the above validity test will have extremely low probability of making a mistake.

Between check points, it is not possible for segments to be recycled - because of the use of the delta-FSS. Therefore, all segments written with the CPID will comprise a single linear list of segments.

Example scenario :

Store is gracefully shut down. This always involves a check point, and therefore a new CPID will be generated. On startup no recovery scan of the store will be attempted. Also, there is no need to check point the store on startup. Flush units will be written using the last valid check point.
Consider that the store is copied after it has been gracefully shutdown. Unfortunately both stores will use the same CPID. The downside is that the file system may recycle disk sectors as one file is deleted or truncated, and the other file grows, allowing for erroneous yet valid looking flush units to be seen in the growing file.
Store is not gracefully shut down, a large number of segments have been written since the last valid check point. In the worst case, the sectors may have been written to disk out of order. On startup a recovery scan will be performed. This will validate flush units in turn. The recovery scan ends at the first flush unit with an invalid CRC or CPID. This may happen "early" if sectors have been written out of order. If the power fails then recovery will be repeated the next time. After recovery a check point is performed. Only when this is completed successfully will the next recovery start the scan from a different position and use a different CPID. At that point we don't care about all the old flush units because they will have an out of date CPID in the flush unit headers.

Between check points segments may be cleaned. However they are only marked as free in the delta-FSS, so there is no chance they will be recycled. Therefore we will never write a segment that already contains flush units with the existing check point id.

Replaying history

Conceptually recovery should be compared to replaying history. During the scan numerous function calls must be made to bring the RPM, SUT, FSS, delta-FSS etc up to date. These should match the same calls that were made when the data was originally written to the log.

Cleaning packets	Some data packets seen in the recovery scan may correspond to packets that were moved to the end of the log by the cleaner. It is necessary to update the RPM (to track the new location of the packet) and SUT (to update the utilisation of the source and destination segments).
RPM packets	These only represent secondary (i.e. indirect) information about changes to the store. Therefore for the purposes of recovery they can be ignored.
Data packets	If a head of a packet chain, then the old serial element chain must be removed (updating both the RPM and SUT). The new location of the packet must be recorded in the RPM, and the utilisation of the segment must be updated in the SUT.

Note that during this "replay", the utilisation of segments may fall to zero and therefore they will automatically be saved in the delta-FSS.

To ensure transactions are atomic, these changes should only be applied when the snapshot log records are encountered.

Replay must include the calls to UnreserveSegment() and AllocateGivenSegId() on the SUT.

Truncation of the log

The recovery scan must avoid making any changes to the flush units. Attempting to make a change could cause data loss because writing flush units is not guaranteed to be atomic - particularly if the flush unit is larger than a disk sector. I.e. trying to modify a flush unit can leave it in a partially written state (if there is a power failure during the recovery), and its CRC is no longer valid.

As a result of this, we only allow the log to be truncated at flush unit boundaries. Therefore the recovered flush units may contain log records that aren't actually committed by a snapshot record.

The log is always truncated at the end of the last valid flush unit containing a snapshot record.

Regeneration of the last delta file

Note that different hard-disks can have different amounts of write cache, and therefore it can't be assumed that on power failure the data actually written to the disks is consistent with the order in which write operations were issued by the software.

During recovery it is necessary to regenerate the delta file - because it may have been written on a different hard-disk and less was written to the delta file because that disk happens to use a large write cache.

Recovery can come across RPM log records. In fact there could be any amount of such records without ever encountering a snapshot record. In that case we assume there is nothing to recover and therefore no check point will be performed at the end of the recovery. This aligns well with the idea to use dirtiness of the RPM as a test for whether an attempted check point is done.

It seems possible that during recovery we write flush units to a delta file, but it is eventually found that there was no "real" data to recover. Testing the RPM is the best indicator. In that case the delta file should be rewritten from scratch!

Note that only valid flush units (e.g. valid CRC) are recovered, and written to the delta file. Therefore barring disk failures, an invalid flush unit should never been seen in a delta file. However, we must allow for flush units that contain log records that are never actually committed by a snapshot log record.

The recovery only iterates over the flush units a single time. Therefore it will write flush units to the delta file without actually knowing whether they will end up being committed by a subsequent snapshot log record. At the end of the recovery it may be necessary to truncate the delta file.

To achieve this we need to calculate the length of the delta-file. This is done as follows


int64 committedLength = 0;
int64 deltaLength = 0;

int committedfsn = 0;

int fsn = 1;
for each flush unit
{
    < process flush unit >

    deltaLength += size of flush unit;

    if (flush unit contained snapshot record)
    {
        committedLength += deltaLength;
        deltaLength = 0;

        committedfsn = fsn;

        endOfLog = position at end of flush unit
    }

    ++fsn;
}

Recovery of the FSN

During the recovery scan, each time we read a snapshot log record we should record the FSN of the flush unit containing the snapshot record. This gives us the last flush unit, and is used to initialise the FSN in the SegmentWriter.

Recovery

An initial implementation of recovery can simply truncate the log at the position of the last valid check point. However this may cause excessive data loss on failure.

A more advanced recovery algorithm scans the log from the last valid check point, because a number of valid snapshots of committed transactions may be present. These must be applied to the RPM and SUT.

It is assumed that the recovery effort is not great, and there is no need to log the recovery process itself. If the system crashes before recovery is complete, then the same work must be repeated the next time the recovery is performed.

It is not allowable for transactions to run while recovery is in progress – because the LSS can’t reliably return the most up to date version of a serial element until recovery has brought the RPM up to date. Therefore recovery is completed before the LSS returns the ILSS interface pointer, preventing clients from calling any ILSS methods while recovery is in progress.

Truncation of the log

When the recovery scan reaches the end of the log, it is possible that a number of packet records have been scanned without the subsequent snapshot record (perhaps because of a power failure before all data was flushed to the log). Before the log can be used, it is necessary to truncate the log – or otherwise these packet records may wrongly be realised as part of a subsequent snap shot in another recovery.

If truncation is required, it is performed synchronously at the end of recovery by

Rewrite the segment trailer for each segment being removed from the log. Set the nextSegId, size, and SSN to zero. This must be done in reverse order – ie start at the last segment in the log and work backwards.
Rewrite the size in the segment trailer of the segment containing the last snapshot or check point record.

Recovery algorithm

Recovery involves a scan of the log records beginning from the last valid check point.

Let the following structure be used to store information about a data packet

struct PacketInfo

{

LogRecordPosition m_position; // The position of this data packet (ie SegId and offset)

OID m_oid; // The OID assigned to the packet

int m_size; // The size of the packet in bytes

}

Let the PacketList (PL) be a std::deque<PacketInfo>, used to store information about all the data packets that have been encountered during the recovery scan.

Let the DeletionList (DL) be a std::deque<OID> used to accumulate all the LR_DELETE_PACKET records during the recovery scan.

Steps

Read the root block, according to Challis’ algorithm. Load the SUT, root RPM-node and the segment and offset of the last valid check point
Initialise the truncate-log-position to null
Initialise the PL and DL to empty
Initialise a vector S SegId to empty
Let SSN equal the SSN of the segment containing the last valid check point.
Scan the log records from just after the last valid check point. For each encountered segment add its SegId to S.
1. If we come across an RPM-packet record then goto 7.
2. Store each data packet record in the PL
3. If reach a snapshot record

SW.SSN += S.size()

Clear S
For each packet in the PL, add entry to the RPM and apply change to the SUT (ie account for obsoleted packet, if any and the new packet).

For each OID in the DL, remove the entry from the RPM. Decrease the utilisation of the segment containing the packet by the size of the packet with the given OID.

Clear the PL and DL
Set the truncate-log-position to just after the snapshot record

If the truncate-log-position is not null and it doesn’t match the end of the log then truncate the log to that position. S contains the list of segments that need to be removed from the log.

Code


struct RecoveryScanInfo
{
    RecoveryScanInfo();
    Guid checkPointId_;
    FlushSeqNumber fsn_;
    LogRecordPosition endOfLog_;
    SegId nextSegid_;
};

48 ReadSerialElement

ReadSerialElement.cpp

Source: Ceda/cxLss/src/ReadSerialElement.cpp

49 DeltaFile

DeltaFileWriter is used to write LSS delta files. These provide a basis for backup of an LSS.

Delta files

     CPSN   0                    1                  2                       3
     CPID   {0000...}            {FC37...}          {4FB9...}               {EE28...}

 LSS   :    +--------------------+------------------+-----------------------+----------------

 Deltas:    <-------------------> <----------------> <---------------------> <---------------
                 delta 0               delta 1              delta 2               delta 3

            |                    |                  |                       |
            |                    |                  |                       |
          check                check              check                   check
          point                point              point                   point
           #0                   #1                 #2                      #3

Note the following

Delta files always begin at 0, and are sequenced by the CPSN (Check Point Sequence Number)
It is assumed that the initial, empty store is check pointed with CPSN=0 and CPID={00000000-0000-0000-0000-000000000000}
When all delta files are present (i.e. starting from 0), it is possible to create the initial level 0 by simply creating an empty store. I.e. there is no need to make a copy of the original empty store. This is because all empty stores are equivalent.
Delta files respect check point boundaries. Note in turn that check point boundaries respect both flush unit and transaction boundaries.
Each check point involves writing dirty RPM nodes to the end of the log and the delta file. So a delta file tends to finish with RPM packets. These are ignored when a delta file is applied to a level 0.
When LSS::SynchronousCheckPoint() is called, it (normally) tests whether the root RPM node is dirty. If the RPM is clean then there have been no changes to the store since the last check point so no check point is performed.
Data packet log records written by the cleaner don't represent a logical change to the store, and therefore should not be applied to the level 0. This is not merely a question of efficiency! The cleaner works at the level of packets, not whole serial elements, and there is no assumption of consistency between how the main LSS and a level 0 break up serial elements into packets.
It is possible that between two check points nothing but cleaning has occurred. So there are no "logical" changes to the store. Therefore when applying the delta to the level 0, it may be found that nothing needs to be done. After applying a delta to the level 0 and a checkpoint is performed, the RPM may be clean, yet the check point needs to be done anyway, just to make sure the level 0 root block records the new CPSN and CPID.
For correctness, the existing delta file (as well as the log) must be flushed before updating the root block. Otherwise, if a power failure occurs a delta file could be incomplete, making it impossible to bring the level 0 up to date.
To help ensure this we open the delta files with write-through cache.
The LSS may be on a different disk from the delta files. This makes it difficult to know for certain whether one is ahead of the other.
On recovery we always use the main LSS store for the recovery scan (not the delta file), and rewrite the last delta file from scratch (overwriting any previous data).
We are careful to initialise the segment writer so it continues using the last CPID and respects the FSN's. In any case, after recovery we immediately check point, (unless the RPM is clean).
During recovery only valid flush units may be written to the delta file. Flush units are always written in their entirety, even if recovery only actually made partial use of the flush unit. This may happen because recovery depends on finding snapshot log records.
Therefore to respect transaction atomicity, when a delta file is applied to a level 0, it is necessary to only apply changes when snapshot records are encountered. The delta file format definition must allow for some final log records that aren't committed by a snapshot record.

Writing delta file issues

It is not reasonable to break up delta files in an arbitrary way. For example, delta files should respect transaction boundaries or else applying delta files will be complicated by the need for transaction atomicity. Furthermore validation of adjacent delta files is made easier if they respect check point boundaries, for then we can think of each delta file as mapping the LSS from an input CPID to an output CPID. A Check Point Id (CPID) is a guid generated at each check point.

During a check point the LSS maximises concurrency by flushing the log (which can block on I/O) without blocking other threads from performing additional transactions. I.e. it releases the check point LRS mutex (used to write the dirty RPM nodes) *before* flushing the log. This can be a significant advantage, because check points are typically performed in the background by the lazy check pointer thread, so it makes sense for this thread to block while flushing the log without holding up threads that want to continue writing data to the segment cache.

Unfortunately this complicates the writing of the delta files. Conceptually during each check point the existing delta file should have its footer written and be closed, and the next delta file should be opened. However, once the check point releases the LRS mutex, another thread may write new data to the log. Furthermore there is nothing to stop the lazy writer waking up and needing to write the new data to a fresh delta file. This may happen concurrently with the check pointer which is synchronously flushing the log and old delta file.

Only one thread is able to write to the log and the delta files at a time. (Usually this is the lazy writer thread. Less often it is the thread that is flushing the log - either a client of the LSS, or else the thread doing a check point). It therefore makes sense for the thread writing to the delta files to itself work out when the old delta file needs to be closed and the next delta file opened (rather than the thread doing a check point). With this approach the delta file writer has no need to be thread-safe, because delta-files are only opened, written and closed by one thread at a time.

As the delta file writer writes sections of segments to disk, they can be scanned as a sequence of flush units. The flush unit headers tell the delta file writer precisely where the check point boundaries are. Note that check point boundaries always respect both flush unit boundaries and transaction boundaries.

A synchronous flush call is made to flush the log as part of a check point. This blocks out the lazy writer - because the mutex LazyWriter::writeSegmentsToDiskMutex_ ensures that only one thread is able to write segments to disk at a time. This calls


DeltaFileWriter::CloseDeltaFileIfOpen(int cpsn, const Guid& cpid)

The argument cpsn identifies the delta file to be closed. There is only a need to close the delta file if it hasn't been closed already.

Power failure while applying a delta

When a delta is applied to a level 0 we validate the CPID, CPSN and TXSN. Then we scan the delta file and apply each transaction in turn. During this process check points on the level 0 are not performed, because the flag LSS::forBackUp_ is set to true.

If there is a power failure then only some of the transactions may have been applied to the level 0. However, because no check points have been performed, in fact it is more a case that *none* of the delta file has been applied to the level 0. In fact the next time we try to apply the delta file, all transactions will need to be applied from scratch.

The TXSN in the root block makes it possible (in a future version of the LSS) to partially apply a delta to the LSS, and to later correctly apply the remainder of the delta file.

Code


class SFile
{
    cxNotCloneable(SFile)
public:
    SFile();
    ~SFile();

    void Open(ConstStringZ filename, bool read);
    bool IsOpen() const;
    void WriteBuffer(const void* buffer, int numBytes);
    void SetSize(int size);
    void Close();
private:
    FILE* fp_;
    xstring filename_;
    int numBytesWritten_;
};

class DeltaFileWriter
{
public:
    DeltaFileWriter(LSS& lss, ConstStringZ deltasDirPath);

    bool EnableWriteDeltaFiles() const { return enableWriteDeltaFiles_; }

    void SetCpsn(int cpsn, const Guid& cpid) { cpsn_ = cpsn; cpid_ = cpid; }

    // If writing to delta files is enabled then
    //      1. Opens the delta file (if not already opened)
    //      2. writes the given buffer to the delta file
    // Does nothing if writing to delta files is not enabled.
    // May throw FileException
    void Write(const void* buffer, int numBytes);

    // Called during a check point of the LSS.  If the delta file with given cpsn is still open
    // then write its footer.
    // May throw a FileException
    void CloseDeltaFileIfOpen(int cpsn, const Guid& cpid);

    void AbortDeltaFile();

    void TruncateDeltaFile(int fsn, int length);

private:
    bool DeltaFileIsOpen() const { return file_.IsOpen(); }

    // Throws FileException if there was an error writing the file
    void WriteFooterAndClose(const Guid& checkPointId);

    // Throws FileException if file couldn't be opened
    void OpenDeltaFile();

    // Throws FileException if there was an error writing the file
    void WriteBuffer(const octet_t* p1, const octet_t* p2);

    // Throws FileException if there was an error closing the file
    void CloseDeltaFile();

private:
    LSS& lss_;
    bool enableWriteDeltaFiles_;
    xstring deltasDirPath_;

    int fsn_;
    Guid cpid_;
    int cpsn_;

    SFile file_;
};

50 CopyStore

An entire Log Structured Store file can be copied with a call to LssCopy().

This has the effect of compacting the store. All the live serial elements are written to a new LSS using a single transaction. There is no space wasted on redundant serial elements, RPM packets, snapshot log records and SUT sections.

CopyStore.cpp

Source: Ceda/cxLss/src/CopyStore.cpp

51 DumpLss

DumpLss.cpp

Source: Ceda/cxLss/src/DumpLss.cpp

52 Component source reference

The implementation chapters identify the principal files in the cxLss source tree. Complete source listings are not duplicated in this document because they would quickly become stale; the source tree remains authoritative for exact interfaces and current code.

Area	Principal source components
Transactions and serial elements	`LssTxn`, `SerialElementWriter`, `ReadSerialElement`
Segments	`Segment`, `SegmentAccessor`, `SegmentQueue`, `SegmentUnreserver`, `SegIdStack`
Recovery	`RecoveryScanner`, `RecoveryLogRecordVisitor`, `CheckPoint`
Segment utilisation	`ISUT`, `SmallSUT`, `LargeSUT`, `SUTSection`, `SUTSectionRef`, `Reservations`
Administration	`CopyStore`, `DumpLss`, `DeltaFile`

The explanatory chapters name the relevant header and implementation files at the point where each component is discussed. The source tree remains the authority for exact interfaces and current code.

53 Third-party literature

The following independent research is relevant to the design of log-structured stores.

Name	Date	Description
Database Consistency and Integrity in a Multi-User Environment	1978	M. F. Challis, in Databases: Improving Usability and Responsiveness, Academic Press, pp. 245–270.
The Design and Implementation of a Log-Structured File System	1992	Mendel Rosenblum and John K. Ousterhout, ACM Transactions on Computer Systems, vol. 10, no. 1, pp. 26–52.
A Log-Structured Persistent Store	1996	David Hulse and Alan Dearle, in Proceedings of the 19th Australasian Computer Science Conference, pp. 563–572.
Efficiently Reclaiming Space in a Log Structured Store	April 2020	Research by David Lomet and Chen Luo on prioritising segment cleaning in log-structured stores. Published at IEEE ICDE 2021.

54 Implementation status and open issues

This chapter preserves the status notes formerly maintained on the cxLss implementation index. The open items are possible future work rather than descriptions of the current implementation or commitments, and some may have been superseded by later work.

Open issues and proposals

Investigate the deadlock issue discussed in Deadlock bug, February 2022, including whether the lazy flusher can avoid locking the segment writer to update the flush position.
Clip redundant RPM nodes from the tree and consider an alternative serialisation for sparse RPM nodes.
Scale the txLss test for high-core-count machines and reduce its output verbosity.
Use a platform-independent 32-bit type for MSSN rather than int.
Complete support for asynchronous serial-element loading and MVCC.
Validate and refine RPM-node eviction.
Develop a more advanced policy for choosing when to clean segments.
Allow Linux permissions to be passed to LinuxFileRAS::Open().
Replace an assertion in VisitLogRecordsInBuffer() with an appropriate run-time check.
Clarify assumptions about file-sector and page sizes.
Provide a function that validates an entire LSS.
Expand diagnostics, including store statistics, segment utilisation, packet listings and RPM browsing.

Recorded completed work

Dedicated threads were replaced by IoContextPool.
LazyWriter and LazyCheckPointer use AsyncSignaledTask.
LazyFlusher uses AsyncPeriodicTimer.
LazyCleaner uses AsyncRepeatedTask.
An LSS can be supplied with an IoContextPool, or use the default pool provided by cxThread.

55 Fully asynchronous LSS

Status: unimplemented proposal. This proposal makes storage I/O asynchronous throughout the LSS, including at its public API, so an LSS does not require a collection of threads which block while I/O is in progress.

Motivation

The LSS already separates most background work into LazyWriter, LazyFlusher, LazyCheckPointer and LazyCleaner. Their clients normally signal that work is needed and continue; they do not need the work to finish before making progress. These components are therefore natural asynchronous tasks rather than reasons to dedicate threads to an LSS.

The current asynchronous task wrappers make invocation asynchronous, but the invoked function can still occupy an io_context thread for the entire duration of a blocking file operation. Instead, an operation should submit I/O, return to the executor and resume when its completion is delivered. This is particularly important on platforms such as WebAssembly where threads may be unavailable or undesirable and storage is naturally exposed through asynchronous interfaces.

Transaction commit remains an in-memory operation

Closing an LSS transaction does not provide per-transaction synchronous durability. It commits the transaction to the in-memory log and need not perform any I/O. This fast path should remain immediate, even when represented by an asynchronous public API: an operation which has no reason to suspend may complete immediately.

The proposed design removes FlushWhenClose(). Waiting for a storage flush on every selected transaction would complicate the normal execution model for a facility that is not central to the intended use of the LSS. Applications interested in persistence should instead observe the asynchronous durability progress described below.

Asynchronous storage interface

An asynchronous counterpart or replacement for IRAS should report the completion of reads, writes, resizing and persistence barriers without blocking the calling thread. The exact C++ representation could be an awaitable or a completion handler; the important property is that the LSS can suspend an operation and resume it on its serialized executor.


struct IAsyncRAS
{
    virtual AsyncResult<RASSize> GetSize() = 0;
    virtual AsyncResult<void> Read(octet_t* buffer, RASOffset offset, ssize_t size) = 0;
    virtual AsyncResult<void> Write(const octet_t* buffer, RASOffset offset, ssize_t size) = 0;
    virtual AsyncResult<void> SetSize(RASSize size) = 0;
    virtual AsyncResult<void> Flush() = 0;
};

This is illustrative rather than a proposed final signature. Buffers passed to an operation must remain alive until its completion. The segment queues already provide much of the ownership needed for outstanding segment writes.

Platform implementations can map the interface to native asynchronous facilities, WebAssembly storage promises, or, as a compatibility fallback, blocking file operations performed by a shared worker facility. The fallback may use threads, but the LSS itself does not require a set of threads per store.

Existing lazy activities

LazyWriter: Submit writes for queued segment ranges, retain each segment until completion, and advance the written log position across the contiguous prefix of successful writes.
LazyFlusher: Periodically request writer progress and, when required by the persistence policy, submit a storage persistence barrier. It should not occupy an executor thread while that barrier is pending.
LazyCheckPointer: Construct check-point information in memory, request that the required log prefix be written, and continue with the root-block-division write after the prerequisite completions have arrived.
LazyCleaner: Perform cleaning incrementally, resuming its reads and writes from completion notifications and yielding between units of work.

These activities can share one serialized LSS executor. Storage operations may be outstanding concurrently, while all changes to logical LSS state are applied in a well-defined order on that executor. This can also replace a significant amount of mutex-based coordination.

Asynchronous public API

The public LSS API should not synchronously wait for I/O. Operations which may need storage include opening and recovery, loading an uncached segment, applying back-pressure when no reusable segment buffer is available, explicit maintenance operations, and closing the store. Such operations should complete asynchronously. Transaction operations which touch only resident memory can complete immediately through the same API.

Closing an LSS must itself be asynchronous. It stops the creation of new work, allows or cancels outstanding operations according to a documented policy, and completes only when no completion can subsequently access the LSS. I/O errors must be reported to the affected operation and retained as store state where they also affect later operations.

Durability notification by TSN

Rather than making a transaction wait for durability, the LSS should notify an application when a prefix of committed transactions is known to be durable. Since transactions are serialized, this can be represented by a monotonically increasing transaction serial number (TSN):


using DurableTxnCallback = std::function<void(TSN durableTsn)>;

void SetDurableTxnCallback(DurableTxnCallback callback);
TSN GetDurableTSN() const;

A notification of TSN n means that every committed transaction with a TSN less than or equal to n is known to be recoverable after the documented class of storage failure. A single notification can therefore cover many transactions. GetDurableTSN() allows a caller to handle the case where the TSN of interest became durable before it registered its interest or processed a notification.

The durable watermark advances only over a contiguous log prefix, even if asynchronous writes complete out of order. Completion of a buffered write is not necessarily evidence of durability under abrupt power loss. The watermark must be advanced according to the persistence guarantee of the underlying RAS, normally after the applicable asynchronous flush or barrier has completed.

A check point is not itself the definition of transaction durability: recovery can scan valid log data following the last valid check point. Check-point publication nevertheless has a strict dependency. All log data and metadata referenced by a new root-block division must have completed the required writes before that division is submitted. Expressing the root-block write as a continuation of those completions makes this ordering explicit without blocking a thread.

Back-pressure and ordering

Asynchronous I/O does not imply unbounded buffering. When the segment cache cannot supply another buffer, the transaction operation requesting one should remain pending until writer progress releases a segment. This applies back-pressure without blocking a thread. Cancellation and shutdown must not release buffers which are still referenced by outstanding I/O.

Writes may be issued concurrently where their logical independence permits it, but completion order must not be confused with publication order. Written and durable log positions, reusable buffers, check-point state and the durable TSN are advanced only when their respective contiguous prerequisites have completed successfully.

56 Asynchronous loading of serial elements

Status: partially superseded proposal. The worker-task design discussed below has since evolved, but asynchronous reading of contiguous serial elements is not part of the established LSS interface.

At the time of this proposal, each opened instance of the LSS created four worker threads for the following:

Lazy check pointer
Lazy flusher
Lazy writer
Lazy cleaner

The proposal considered using a shared worker facility for these tasks and, in addition, supporting asynchronous loading of contiguous serial elements. This is needed to implement asynchronous loading of objects in a PSpace.

Perhaps better still, when the LSS is opened, it is provided with some kind of thread-pool it can use? That way, if an application needs to create many LSS instances, it could avoid creating so many threads. In fact a boost asio context pool might be the way to go, because it provides a boost::asio::deadline_timer , which is exactly what we need to run jobs periodically without having threads call a Sleep() function.

The LSS has a Lazy check pointer etc as follows:


class LazyWriter
{
    LazyFlusher lazyFlusher_;
};

class SegmentWriter
{
    LazyWriter lazyWriter_;
};

class LSS
{
    SegmentWriter segmentWriter_;
    LazyCleaner lazyCleaner_;
    LazyCheckPointer lazyCheckPointer_;
};

Interface to synchronously read contiguous serial elements

The following structs represent contiguous serial elements:


struct ReadOnlyBuffer
{
    const octet_t* buffer;
    ssize_t size;
};

// Provides access to a serial element recorded as a contiguous buffer in memory.
struct IContiguousSerialElement
{
    // A contiguous serial element must be explicitly closed, even if exceptions have been
    // thrown by the LSS.
    // It is an error to close the LSS before the closing all the contiguous serial elements
    // that have been opened for reading.
    virtual void Close() = 0;

    virtual ReadOnlyBuffer GetBuffer() const = 0;
};

Synchronous loading of contiguous serial elements from the LSS involves calling the following method of ILogStructuredStore:


/*
Provides an alternative to ReadSerialElement() for reading a serial element as a contiguous
block of memory.  Obviously this function shouldn't be called for very large serial elements
that don't fit in physical memory and therefore would result in page faulting.

The given Seid must not be null

The returned IContiguousSerialElement must be closed after it is used (including when
exceptions are thrown by the LSS).

Returns nullptr if no serial element exists with the given Seid

It is an error to call this function on a serial element that is currently opened for
writing (within a transaction), or being deleted using a call to DeleteSerialElement().

Shared reading of serial elements is supported. I.e. any number of threads can independently
(and concurrently) read the same serial element
*/
virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;

Asynchronous API

It is proposed that the following method be added to interface ILogStructuredStore:


virtual void AsyncReadContiguousSerialElement(
    Seid seid,
    std::function<std::error_code ec, const octet_t* buffer, ssze_t size> handler) const = 0;

'handler' is a callback function which is called when the serial element has been loaded, or else an error occurred. If an error occurred then the handler is called with buffer==nullptr and size==0.

The std::function is copied by value into what could be regarded as a job queue in the LSS.

Note that in C++11 passing by value is recommended if a copy is needed - it often produces more efficient code (see Want Speed? Pass by Value by Dave Abrahams). Therefore the std::function is passed by value since it will need to be copied to put it in a job queue.

A std::function value may contain a std::shared_ptr, allowing it to hold objects alive while the async I/O is pending. For example the caller may use this technique:


class MyClass : std::enable_shared_from_this<MyClass>
{
    void do_read(Seid seid)
    {
        auto self(shared_from_this());
        lss_.AsyncReadContiguousSerialElement(seid,
            [this, self](std::error_code ec, const octet_t* buffer, ssze_t size)
            {
                if (!ec)
                {
                    // handle async load of the object
                }
            });
    }
    ILogStructuredStore& lss_;
};

If the LSS is closed then all the pending handlers are called with an error code that indicates the operation was aborted.

Review of existing support for asynchronous loads from the LSS

See AsyncObjectLoader. Also AsyncJobQueueOnThread.

Implementation

It might be appropriate to tailor the API to support the requirements of PSpaces:

It might make sense for each PSpace to use a different io_context for loading serial elements, to provide better concurrency. Note that the LSS supports concurrent loading of serial elements from its segment cache.
When a PSpace is closed there is a need to ensure all the async loading for that PSpace is stopped synchronously.
The PSpace ROT only needs a single call-back function to handle all async load serial element requests, we don't need a separate std::function instance for each OID.
All AsyncBinds in a PSpace are protected by the CSpace mutex. Therefore the "producer" of the sequence of OIDs to be loaded is effectively single threaded. For performance it would be good if most AsyncBinds simply push an OID onto a std::deque which is protected by the CSpace mutex.
All the loading of serial elements in a PSpace uses a single io_context which is run by a single thread. Therefore we have a single threaded "consumer".
For fairness to other clients that happen to use the same io_context, it shouldn't process too many OIDs in one batch.

Let a PSpace record a std::deque<OID> for the OIDs that need to be loaded asynchronously. This std::deque<OID> is protected by the CSpace mutex. PSpace::AsyncBind(OID oid) may cause an oid to be pushed onto this deque. If the deque transitions from empty to non-empty then a task to process the deque is posted.

For a given PSpace there is a single AsyncObjectLoader which uses a single io_context to load serial elements from the LSS. AsyncObjectLoader has a std::deque<OID> member which is only accessed by the single threaded io_context. When the AsyncObjectLoader deque is empty the io_context thread can lock the PSpace and swap the two deques.

57 Proposed MVCC support in the LSS

Status: proposal. This chapter describes intended behaviour, not the current LSS implementation.

This chapter describes the intended changes required to make the LSS support multiversion concurrency control (MVCC). It is a proposal, not a description of the current implementation. The existing LSS has one mutable Recoverable Packet Map (RPM), and clients must prevent a serial element from being rewritten or deleted while it is open for reading. The proposed design replaces that restriction with stable read snapshots. A reader will continue to see the same committed rendition of every serial element for the lifetime of its read transaction, even while the writer publishes later renditions.

The proposal retains an important simplifying property of the LSS: there is one serial writer. MVCC is used to allow any number of readers to run concurrently with that writer; it is not intended to introduce concurrently mutating LSS transactions. Write transactions remain totally ordered, so the LSS does not require deadlock detection, general write-write conflict detection, or a multi-writer commit protocol.

At a high level, a snapshot is formalised as a map from Seids to serial elements. A consistent snapshot means that this map remains unchanged for the lifetime of the read transaction, regardless of writers creating new serial elements, deleting existing serial elements, or rewriting existing serial elements. A write transaction starts from the state left by preceding write transactions and updates its own map without changing the maps used by existing readers. Close() completes the transaction, and subsequent write transactions see its changes, but existing and new readers continue to use the most recently published LSS snapshot. The old map and every packet needed through it remain available until its last reader has finished.

LSS writer transactions are serialised. Each is assigned a unique, monotonically increasing 64-bit transaction sequence number from the history of the LSS, going back to when its empty file was first created. Only a subset of writer transactions are required to publish snapshots. In the following example, writers 100 and 102 decay directly into already-pinned readers at the instant they call CloseAndPublishSnapshot(), whereas writer 101 closes without publishing. The writer transactions need not be contiguous in time, and long-lived readers can overlap idle periods, later writers, and later readers.

Timeline showing writers 100 and 102 decaying into overlapping readers while writer 101 closes without publishing

LSS API changes

The API must express snapshot lifetime explicitly. It is not sufficient for each call to ReadSerialElement() to select the latest mapping independently: two such calls made by one logical operation could straddle publication of an LSS snapshot and observe an inconsistent mixture of states. A read transaction therefore owns a snapshot, and every read performed through that transaction resolves its Seid through the same RPM root.

Read transactions

The LSS will provide the following explicit read-transaction handle:


struct ILssReadTransaction
{
    virtual bool SerialElementExists(Seid seid) const = 0;
    virtual ICloseableInputStream* ReadSerialElement(Seid seid) const = 0;
    virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;
    virtual void Close() = 0;
};

struct ILogStructuredStore
{
    virtual ILssReadTransaction* OpenReadTransaction() const = 0;
    // ...
};

Putting the read operations on the transaction handle ensures that every read in the transaction uses the same snapshot, preventing an operation from observing an inconsistent mixture of states. The handle is both the authority to read and the object that pins the selected root. This design rejects passing a separate snapshot handle to read methods on ILogStructuredStore, because putting the operations on ILssReadTransaction expresses their required snapshot directly. There is no implicit selection of a new root during a transaction.

Opening a read transaction selects and retains the most recently published snapshot. Closing the transaction releases that snapshot. The selected snapshot may already be stale when the read transaction begins: it does not include writer transactions completed since the most recent publication. A read transaction is read-only; it cannot be promoted into a write transaction. It may be used for an arbitrary number of serial elements and for serial elements discovered by following information read earlier in the same transaction.

The following rules form part of the API contract:

Every operation on a read transaction observes exactly one committed snapshot.
The snapshot's map from Seids to serial elements remains unchanged until the transaction is closed, regardless of writers creating, deleting, or rewriting serial elements.
Every serial element in that map remains readable regardless of whether its RPM nodes or serial-element data are currently resident in memory.
A deletion committed after the snapshot was opened does not remove the element from that snapshot.
An element created after the snapshot was opened does not appear in that snapshot.
Closing an LSS while read transactions or streams are open remains an error.
A read transaction must be closed even when a read operation throws an exception.
An ILssReadTransaction can serve concurrent immutable reads on several threads. Each returned stream is confined to the thread using that stream.

An input stream inherits the snapshot of its transaction. Closing the transaction while one of its streams remains open is an error, because the stream may still need RPM nodes or data packets retained by the snapshot. Closing a stream does not close the transaction; this permits several sequential or simultaneous streams to share one consistent view.

The owner of a read transaction closes it only after all readers and streams using that snapshot have released it. Concurrent read support does not require concurrent mutation of the RPM; it requires thread-safe immutable traversal, lazy loading, and segment cache access.

Rejected `allowStale` alternative

An alternative API could make OpenReadTransaction() take a Boolean allowStale argument. When true, it would immediately pin the most recently published root. When false, it would have to synchronise with the single writer, wait behind any write transaction that already owns the writer lock, publish the latest completed writer frontier, and only then return a read transaction. The resulting snapshot would include every write transaction completed before that publication point, but opening the read transaction could block for the duration of an active write transaction.

This design rejects that freshness flag as the mechanism for associating a read snapshot with a particular writer view. The LSS instead provides ILssWriteTransaction::CloseAndPublishSnapshot(), which completes the writer transaction, publishes its exact working frontier, and returns an ILssReadTransaction pinned to the resulting root. This directly transfers a mutable writer view into its immutable read view without a second operation racing to identify what "current" means.

Write transactions

The existing ILssTransaction is replaced by ILssWriteTransaction. There is still at most one open write transaction. It owns a committed base root and a private working root. Reads made through the write transaction observe the base snapshot plus that transaction's own writes and deletions.


struct ILssWriteTransaction
{
    virtual bool SerialElementExists(Seid seid) const = 0;
    virtual ICloseableInputStream* ReadSerialElement(Seid seid) const = 0;
    virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;
    virtual ICloseableOutputStream* WriteSerialElement(Seid seid) = 0;
    virtual bool DeleteSerialElement(Seid seid) = 0;
    virtual void FlushWhenClose() = 0;
    virtual void Close() = 0;
    virtual ILssReadTransaction* CloseAndPublishSnapshot() = 0;
};

Read-your-writes behaviour is important. After replacing an element, a read through the same write transaction returns the replacement. After deleting it, existence tests and reads report that it is absent. Other read transactions continue to see the previously committed rendition until the write frontier is published at an LSS persistence boundary; read transactions already open continue to see their existing snapshot.

Both streaming and contiguous reads resolve through the write transaction's private working root. Any output stream for the element must be closed before it is read, and every returned input stream or contiguous serial element must be closed before the write transaction is closed or published.

There is no concept of aborting an LSS transaction. Close() completes the transaction but does not require creation of a reader-visible RPM snapshot. Its changes become part of the completed writer frontier from which the next write transaction proceeds. CloseAndPublishSnapshot() completes the transaction, atomically publishes that frontier as an immutable RPM root, consumes the write transaction, and returns an ILssReadTransaction pinned to exactly that root.

Snapshot identity and diagnostics

A transaction sequence number is useful as a diagnostic snapshot identity. It can be exposed for tracing and testing, but callers must not use it to locate packets or mutate retention state. The snapshot handle, rather than a bare sequence number supplied by a caller, proves that the corresponding root is still pinned.

Statistics should report the number of active read transactions, the oldest active snapshot, the age of that snapshot, the number of deferred root releases awaiting the writer, the oldest deferred release, the number of retained RPM nodes, and the bytes and segments retained solely because of old or deferred snapshots. These values are essential when diagnosing either a slow reader that prevents reclamation or an idle store whose writer has not yet drained completed releases.

Implementation changes

Separate committed, working and checkpoint roots

The current LSS owns one mutable RPMRoot. MVCC requires at least three distinct roles:

Published root: the newest completed LSS root offered to readers.
Working root: the writer-private frontier containing completed unpublished transactions and the active transaction's changes.
Checkpoint root: a root pinned while its state is made durable and installed in the root block.

These roles may temporarily refer to the same root, but the implementation must not assume that they do. A reader can hold an older published root while many newer write transactions complete without publication. A checkpoint can serialise one completed root while the writer proceeds to construct and later publish another root at an LSS persistence boundary.

Copy-on-write RPM

The RPM is an eight-level radix tree selected by the eight bytes of a Seid. That structure is naturally suited to copy-on-write. To change one level-zero entry, the writer copies the nodes on the path from the root to that entry and shares every unchanged subtree. A transaction changing nearby Seids can reuse paths it has already copied.

old root R100 ── A ── B ── old leaf
                 └──────── many unchanged subtrees

new root R101 ── A'── B'── new leaf
                 └──────── the same unchanged subtrees

Published nodes are logically immutable. A writer must never alter a node reachable from a published, checkpoint or reader-pinned root. It first ensures that the path belongs exclusively to its working view, copying nodes where necessary, and then changes the private copy.

The present RPM representation has parent pointers, mutable child arrays, dirty flags, and eviction state. These concerns need to be separated. A node in a DAG can have more than one parent, so a single parent pointer is not meaningful. Dirty and construction state belongs to the writer or checkpoint machinery, not to a shared immutable logical node. Physical cache state may remain mutable only where it does not alter the logical contents of the snapshot.

Deferred single-writer reference counting

RPM nodes use intrusive reference counts, but those counts need not be atomic. The one LSS writer has exclusive authority to change every logical RPM-node reference count. Each retained logical root owns a reference to its top node, and each internal node owns references to its logical children. Copying a path, installing a child, publishing a root, releasing a writer base, checkpoint retention, and the recursive destruction of an unreachable subtree are all performed by the writer. Readers only traverse immutable nodes; they never increment or decrement references within the RPM DAG.

This deliberately differs from a graph built from shared_ptr. Atomic reference counts on every shared RPM edge would introduce read-modify-write operations and memory barriers throughout a large, frequently shared radix-tree DAG. The LSS already serialises mutation, so paying for concurrent ownership mutation at every node would discard an important benefit of its single-writer design. Centralising logical reference-count changes in the writer permits ordinary integer counts and makes the ordering of root retirement, packet retirement, SUT accounting and node destruction explicit.

Closing a read transaction

Closing an ILssReadTransaction does not immediately decrement the associated immutable RPM root. It makes the transaction unusable and submits a deferred root-release record to the LSS. The root remains physically retained until the LSS writer drains that record. Delayed release is safe: it can retain memory and packets for longer than necessary, but it cannot allow a reader-visible node or packet to be reclaimed too early.

last reference to an LSS read transaction is released
    └── closes the ILssReadTransaction
            └── enqueue deferred release of RPM root R100

next LSS writer obtains exclusive ownership
    └── drain deferred releases
            └── decref R100 and recursively release newly unreachable nodes

The deferred-release record must keep the root identity valid until it is consumed. Closing may occur on any reader thread, so transferring the record to the writer requires one coarse-grained synchronisation operation, such as a mutex-protected list, an MPSC queue, or a per-thread retirement list handed to the writer. This is one synchronisation event per retired LSS snapshot, not one atomic operation per reader, serial-element read, RPM node, or shared edge. Even the root-level event is expected to be relatively infrequent.

Draining deferred releases

After acquiring exclusive writer access, a write transaction drains all pending root releases before it begins modifying the working RPM. For each released root the writer decrements its ordinary reference count. A count reaching zero causes the writer to release the node's logical children, which may recursively make whole unshared paths unreachable. The same pass performs or records the corresponding packet-retirement and segment-utilisation changes. No reader can race with these count updates: a reader with a live transaction still retains its root, while a closed transaction is no longer permitted to traverse it.


void DrainDeferredRootReleases()
{
    for (RPMNode* root : TakeDeferredRootReleases())
        Decref(root);
}

void Decref(RPMNode* node)
{
    assert(HasExclusiveWriterAccess());
    assert(node->refCount > 0);

    if (--node->refCount == 0)
    {
        for (RPMNode* child : node->logicalChildren)
            if (child) Decref(child);
        RetireNodeMappingsAndDelete(node);
    }
}

The code is illustrative: a lazily unloaded child may be represented by a packet position rather than a resident pointer, and destruction must update the ownership representation actually used by the RPM. The invariant is that logical child ownership is changed only under exclusive writer access. A physical cache reference used to keep a lazily loaded node in memory is a separate concern and must not silently become another concurrently modified logical RPM reference count.

Creation and acquisition of snapshot handles

CloseAndPublishSnapshot() is the normal writer-to-reader acquisition path and requires no concurrent increment. While it already has exclusive writer access, the LSS installs the immutable root's ownership and constructs an ILssReadTransaction for it. Readers retain that transaction handle and never touch RPM reference counts.

The exceptional OpenReadTransaction() operation still needs a safe way to acquire the currently published root. The simplest initial implementation serialises this short acquisition with root publication and retirement, increments the root using the same exclusive ownership discipline, and then releases the lock before any serial-element I/O. If strict writer-thread-only count mutation is required, the acquisition can instead be routed through the writer; this additional machinery is introduced only if concurrent direct LSS clients require it.

Idle stores, maintenance and shutdown

If no later write transaction opens, deferred releases may otherwise remain pending indefinitely. That does not compromise correctness, but it delays memory and segment reclamation. The queue must also be drained at maintenance boundaries that already obtain exclusive LSS access, including checkpoint or cleaning preparation where appropriate, and while closing the LSS after all read transactions have closed. An implementation may additionally drain when the queue or retained-byte estimate crosses a threshold. A dedicated reclamation thread is unnecessary unless bounded release latency becomes an actual requirement.

Closing the LSS while read transactions remain open is still an error. Once they have all closed, shutdown drains the final deferred releases before destroying the RPM and verifies that every root reference is accounted for. Diagnostics must distinguish live reader-held roots from closed roots awaiting writer reclamation; otherwise a quiet store can appear to leak snapshots even though their release records are pending.

Lazy RPM loading and eviction

RPM nodes can currently be loaded lazily from their packet positions and evicted to limit memory use. MVCC must preserve that facility without modifying snapshot meaning. An immutable node may contain a thread-safe cache slot that changes from an unloaded packet position to a loaded immutable child. That is a physical caching transition, not a logical mapping change: every thread must obtain an equivalent child regardless of which thread performs the load.

Eviction similarly removes only a cached in-memory representation. It cannot remove or rewrite a logical child of an immutable snapshot. Loading, publication, reference release and eviction can run on different threads, so the ownership of a loaded child and the synchronisation of each cache slot must be specified carefully. It should be possible to prove that a node cannot be destroyed while a thread is loading or traversing one of its children.

Building a write transaction

Opening a write transaction continues from the latest completed writer frontier and creates a working-root reference. Writing or deleting a serial element appends the appropriate log records as it does now, but updates only the working RPM. It must not subtract the old packet from global live-byte accounting merely because the working view supersedes it; a published reader may still use the old mapping.

The working mapping must not expose a partially written serial element. The serial-element writer can accumulate the new packet chain and install its first mapping only after the output stream closes successfully. If writing fails, the incomplete records remain unreachable. A replacement should be represented as one logical mapping change from the old chain to the completed new chain.

Deletes are tombstones in the working view rather than destructive changes to older roots. Deleting and then rewriting the same Seid within one transaction must have well-defined read-your-writes behaviour and must produce only the final mapping when the root is published.

Atomic completion and root publication

Close() completes a log transaction and advances the unpublished writer frontier without changing which root future readers select. CloseAndPublishSnapshot() additionally converts that frontier into an immutable published root and returns a read handle pinned to it. Its required ordering is approximately:

Close every serial-element output stream and finish its packet chain.
Finish the working RPM and all transaction-local liveness changes.
Append the timestamped snapshot log record that marks the transaction complete for recovery.
Advance the committed transaction sequence number.
Make the root immutable and transfer one of the writer's references to a new ILssReadTransaction, thereby pinning exactly that root.
Publish the already-pinned root with release semantics.
Consume the write transaction and release its references to the old base and transaction-private state.
Perform a synchronous flush if requested by FlushWhenClose().

Ownership is continuous across this transition. The write transaction owns the working root from its creation, and CloseAndPublishSnapshot() establishes the returned read transaction's ownership before making the immutable root visible. There is no interval in which the root has been published but a reader must race to acquire a reference to it. This is both safer and cheaper than separate Close(), publish, and OpenReadTransaction() operations.

The returned read transaction owns the newly published LSS root. Publication is a non-failing in-memory step after all allocations and fallible preparation have completed. A mutex can initially protect selection and publication of the root and the exceptional acquisition performed by OpenReadTransaction(). It is never held while reading serial-element data. Readers with an existing transaction do not participate in this acquisition protocol.

Packet lifetime and segment utilisation

Packet retention is the most significant change outside the RPM. In the current implementation, a packet becomes dead when the one RPM no longer points to it. Under MVCC a packet is live while it is reachable from any reader-pinned root, the published root, the working root, or a root required by checkpoint and recovery processing.

reclaimable(packet) =
    not reachable from any retained logical root
    and not required by the recovery boundary
    and not reserved or actively accessed by the segment machinery

The SUT must therefore stop treating replacement in the newest mapping as immediate physical death. There are two promising accounting strategies:

Reference accounting associates ownership with immutable level-zero RPM nodes or packet mappings. When the last node containing a mapping is destroyed, the corresponding packet chain loses its final logical reference and its bytes can be subtracted from segment utilisation.
Epoch reclamation records the commit at which a packet was superseded. Because commits are totally ordered, it can be reclaimed when every active snapshot is newer than that retirement point and no checkpoint or recovery constraint retains it.

Deferred single-writer reference accounting follows the RPM DAG directly and permits node and packet ownership counts to remain non-atomic. Closing a read transaction merely queues its root release; the next writer can release batches of retired nodes and packet mappings while it has exclusive access. An epoch scheme remains a possible alternative if measurement shows that recursive reference accounting is too expensive, but it must preserve the same conservative lifetime guarantees. Whichever representation is chosen must account for every packet in a chained serial element, including chains spanning several segments.

Retirement and physical reuse are different events. A packet can cease to be visible to every logical root yet remain unavailable for overwrite until checkpoint, segment-cache access, reservations and the delta-FSS rules also permit reuse. MVCC adds a retention condition; it does not replace the existing crash-safety conditions.

Snapshot-aware cleaning

The cleaner currently determines liveness by comparing a packet with the position selected by the current RPM. It must instead respect all retained snapshots. A packet used only by an old reader is still live, and the containing segment cannot be returned to the FSS.

The simplest correct first design is for the cleaner to relocate packets selected by the current committed view while leaving old-snapshot-only packets in place. An old root contains an old physical position, so copying that packet elsewhere does not help the old reader unless the root can also be changed—which would violate its immutability. The source segment remains pinned until those readers finish.

A later design could introduce stable physical indirection so relocation updates a shared location cell without changing the logical rendition selected by a snapshot. That adds another concurrent data structure and recovery concern and should not be required for initial MVCC support. Long-running read transactions pinning sparsely used segments are an acceptable and diagnosable first trade-off.

Cleaner selection should take retained bytes into account. A segment with little current-state data but much snapshot-pinned data is not a useful cleaning candidate. Statistics should distinguish current live bytes, old-snapshot live bytes, reservations, and bytes that are logically retired but awaiting a checkpoint boundary.

Segment cache and open streams

Snapshot retention protects a packet's logical and on-disk lifetime. The segment cache continues to protect the memory containing a packet while an input stream accesses it. These are complementary: a snapshot may retain a packet for a long time without keeping its segment resident, and the segment may be loaded on demand when a later read through that snapshot reaches it.

Consequently, recycling checks must consider both MVCC reachability and the existing segment access and reservation counts. The transition to zero logical references must use the same synchronisation domain as the decision to enter the delta-FSS, so a reader cannot resolve a retained packet just as its segment becomes reusable.

Checkpoints

A checkpoint pins one committed RPM root and writes its dirty nodes bottom-up before installing its root in the distinguished root-block area. Writer transactions may continue to create later roots, provided all packets and RPM nodes needed by the checkpoint remain retained until installation is complete.

The checkpoint machinery must no longer use mutable dirty flags embedded in nodes shared with readers. It needs a way to identify which immutable nodes already have valid durable packet positions and which new nodes must be serialised. A newly constructed node can record immutable provenance or checkpoint metadata outside its logical contents. Once serialised, the durable position may be installed in a thread-safe physical metadata field or in checkpoint-owned tables.

The published root and checkpoint root advance independently. Publication provides in-process visibility; checkpointing shortens recovery and advances the boundary after which eligible segments can be reused. A graceful close must prevent new transactions, wait for readers or reject the close, finish required background work, and checkpoint an appropriate final committed root.

Recovery

Recovery still begins from the last valid checkpoint root and scans subsequent log records. A timestamped snapshot record identifies the end of a complete transaction. Changes after the last complete snapshot record are ignored. Replaying each complete transaction constructs the next committed RPM root, although recovery need retain only the newest root because no pre-crash read transactions survive process termination.

The on-disk format need not store all historical in-memory roots. MVCC history is retained for active runtime snapshots, not as a promise to reopen an arbitrary old snapshot after restart. If copy-on-write RPM packets change the checkpoint encoding or node metadata, the root-block schema must be versioned and old stores must either remain readable or receive an explicit migration tool.

Memory ordering and locking

The single writer serialises logical mapping changes, but several shorter-lived synchronisation problems remain. Readers concurrently retain the published root, traverse and lazily load RPM nodes, access the segment cache, and release old roots. Background checkpoint and cleaner tasks also retain roots and update physical bookkeeping.

The design should use a small publication lock or a proven atomic-reference scheme to prevent a root from being destroyed between loading its pointer and retaining it. After a reader has retained a root, ordinary logical RPM traversal should require no global writer lock. Locks for lazy loading, the segment cache, SUT and FSS should have a documented order, and root destruction must not call into those subsystems while holding a lock that creates an inverse ordering.

Resource limits and operational policy

MVCC replaces blocking between readers and the writer with retention pressure. A reader that remains open indefinitely can prevent old packets and poorly utilised segments from being reclaimed. The LSS must make this cost visible and may provide configurable warnings or limits for snapshot age and retained bytes.

Forcibly invalidating a snapshot would break the primary API guarantee and should not be the default. If an installation requires hard resource bounds, expiration must be an explicit policy under which later reads fail with a specific exception. Normal operation should instead identify the owner and age of long-lived transactions through tracing and statistics.

Validation and tests

Correct reclamation is more difficult than retaining versions, so implementation should proceed in stages. The first working version can conservatively retain every superseded packet. Once snapshot selection and atomic publication are thoroughly tested, packet retirement, SUT changes and cleaning can be enabled incrementally.

Required tests include:

A reader continues reading an old multi-segment rendition while the writer replaces it.
Several Seids read through one transaction always come from one committed root.
A writer observes its own creations, replacements and deletions.
Readers opened before and after CloseAndPublishSnapshot() observe the expected different renditions.
Delete followed by recreation of the same Seid does not confuse an older snapshot.
Releasing the last old snapshot makes its retired packets eligible for reclamation.
Closing a read transaction queues a root release without changing any RPM-node reference count on the reader thread.
The next writer drains queued releases and recursively decrements RPM nodes using only its exclusive access.
Many reader threads can close snapshots concurrently without losing or processing a deferred release twice.
A store with no subsequent writes drains deferred releases during maintenance or orderly shutdown.
No snapshot-reachable segment enters the FSS, including during cleaner and checkpoint activity.
A snapshot can reload an evicted RPM node and data segment after later commits.
Failure at every log-write, snapshot-record, publication and flush boundary recovers a valid committed state.
Many readers can share a root without copying the RPM or holding the writer lock during I/O.
Diagnostics correctly attribute retained nodes, packets and segments to old snapshots.

Stress testing should combine long and short readers, large packet chains, repeated replacement of hot Seids, deletions, RPM eviction, segment-cache pressure, cleaning, checkpoints and injected I/O failures. The validator should be extended to calculate packet reachability from every retained root and compare it with SUT accounting. Assertions should favour retaining too much storage over reclaiming a packet whose reachability is uncertain.

Suggested implementation sequence

Specify API lifetime, visibility, thread-affinity and error semantics in executable tests.
Implement an in-memory copy-on-write RPM with retained immutable roots.
Add explicit read transactions and make write transactions use private working roots.
Implement CloseAndPublishSnapshot() as the atomic conversion from a write frontier to a pinned immutable read root.
Add deferred root-release submission and drain it under exclusive writer access.
Use non-atomic intrusive counts for logical RPM-node ownership and validate every increment and decrement as writer-owned.
Keep superseded packets conservatively live while validating snapshot behaviour.
Add snapshot-aware packet retirement and SUT accounting.
Make the cleaner and FSS rules aware of retained snapshots.
Adapt RPM eviction, checkpoints and recovery to immutable shared nodes.
Add operational statistics, leak detection and long-snapshot diagnostics.
Run correctness, crash-injection and performance tests before removing the old access-clash restrictions.

The log-structured layout already preserves old packet bytes until they are cleaned or their segments are reused. MVCC therefore does not require a second value store. The essential change is to retain the old mappings that identify those bytes and to make every reclamation path respect those mappings. Copy-on-write RPM roots provide stable logical snapshots; snapshot-aware SUT, cleaner, checkpoint and segment-reuse rules keep their physical packets alive for exactly as long as required.

58 Proposed MVCC support in cxPersistStore

Status: proposal. This chapter describes intended behaviour, not the current cxPersistStore implementation.

This proposal builds on the proposed MVCC support in the LSS. The LSS provides stable snapshots of persisted serial elements; cxPersistStore composes each of those bases with an immutable per-PSpace object delta so application readers can observe stable states that have not yet been written to the LSS.

Layered overview

cxPersistStore partitions the objects in one LSS into PSpaces. Each PSpace has its own mutex, mutable Resident Object Table (ROT), dirty-object state, and immutable ROT snapshots. A user transaction is declared on a PSpace, not on the whole PersistStore. The PSpace is therefore the unit of logical mutation and snapshot isolation above the LSS.

cxPersistStore
    ├── PSpace A: mutex + mutable ROT + immutable ROT snapshots
    ├── PSpace B: mutex + mutable ROT + immutable ROT snapshots
    └── PSpace C: mutex + mutable ROT + immutable ROT snapshots
             │
             └── one shared LSS: concurrent readers + one serial writer

The LSS and a PSpace require snapshots for different reasons and at very different rates. An LSS snapshot preserves a map from Seids to serial elements already applied to the LSS. A PSpace snapshot preserves the object state visible to an application operation, including edits that have not yet been applied to the LSS. Consequently, an immutable ROT root does not always correspond to a newly published RPM root.

A PSpace snapshot is composed of an immutable ROT root and a retained LSS base snapshot. The ROT embodies an in-memory delta for that PSpace relative to the base:

PSpace snapshot S = LSS base snapshot B + immutable PSpace ROT delta D

Several successive snapshots of one PSpace can share the same LSS base while representing different in-memory deltas. Different PSpaces can publish independently. This is necessary for interactive applications. Moving an object with a mouse may produce updates at approximately 50 Hz, and asynchronous calculations need frequent stable inputs so that readers do not block writers and independent calculations can run in parallel. Writing each of those application states to the LSS would be wasteful. A PSpace can instead publish cheap copy-on-write ROT snapshots at the required interactive rate, while cxPersistStore applies accumulated deltas from one or more PSpaces to the shared LSS every few seconds.

Where concurrency comes from

The mutex of PSpace A does not prevent a transaction from mutating PSpace B. Writers in well partitioned PSpaces can therefore run concurrently in memory. Immutable ROT snapshots allow readers and asynchronous calculations to continue without taking the PSpace writer mutex. On a ROT cache miss, the LSS already permits different threads to read different serial elements concurrently. The single LSS writer serialises only the eventual physical application of PSpace deltas; it does not serialise ordinary PSpace mutation or immutable reads.

This means the database can support considerably more concurrency than the single LSS writer might initially suggest. Its scalability depends on placing independently updated object graphs in separate PSpaces and avoiding unnecessary shared mutable state. The design must preserve concurrent LSS reads: an immutable snapshot read must not acquire the LSS writer lock or a PersistStore-wide lock.

Three independent cadences

The design separates three events which must not be conflated:

Publishing a PSpace snapshot. That PSpace's mutable ROT root is frozen and a new mutable root is created by copy-on-write. Each PSpace can do this independently and frequently enough to feed its user-interface readers and asynchronous calculations.
Applying PSpace deltas to the LSS. At a lower rate, normally every few seconds, cxPersistStore selects stable deltas from one or more dirty PSpaces and serialises them through one LSS write transaction. CloseAndPublishSnapshot() turns the resulting global RPM state into a new retained LSS base snapshot.
Flushing completed LSS transactions to durable storage. This is independent of both snapshot cadences. A transaction can request a synchronous flush on close through FlushWhenClose(); the same force-flush facility is available to user transactions declared on a PSpace through cxPersistStore. Without a forced flush, the LSS LazyFlusher periodically flushes completed work according to LssSettings::flushTimeMilliSec, which defaults to 1000 milliseconds.

The first cadence controls how quickly readers and calculated results observe a stable application state. The second controls how much in-memory change can accumulate before it is represented in the LSS. The third controls the amount of completed LSS work that can be lost in a process or machine failure. Relaxed durability permits the common editing case to avoid a disk flush for every mouse movement, while an operation with an external durability requirement can explicitly force a flush.

The writer-to-reader transition is especially appropriate when MVCC is exposed by layers above the LSS. When cxPersistStore applies selected immutable PSpace deltas to the LSS, it uses the read transaction returned by CloseAndPublishSnapshot() as the new persisted base for subsequent PSpace snapshots. This pins exactly the LSS state produced from that persistence batch. A call to OpenReadTransaction(false) would instead publish whichever writer state happened to be current, independently of the selected PSpace cutoffs. It is therefore the wrong composition mechanism. The Boolean API is not part of this design; it can be reconsidered separately if direct LSS clients establish a concrete requirement for it.

PSpace snapshots over an LSS base

Each immutable ROT root represents one snapshot of one PSpace. It retains an LSS base snapshot and records that PSpace's object-level delta relative to the base. Many immutable ROT roots, belonging to one or several PSpaces, can share one base. The retained base owns the ILssReadTransaction that pins its global RPM root; an individual ROT root retains the base rather than necessarily owning a distinct LSS read transaction.

When cxPersistStore opens, it obtains the initial base with OpenReadTransaction(). A later persistence batch obtains its replacement base from CloseAndPublishSnapshot(). Replacing a PSpace's current base does not alter its existing ROT snapshots: an old base remains pinned until the last ROT root, reader, or calculation in any PSpace that depends on it is released.

                         ┌── PSpace A snapshot A101 with delta DA101
LSS base snapshot B100 ──├── PSpace A snapshot A102 with delta DA102
                         ├── PSpace B snapshot B51  with delta DB51
                         └── PSpace C snapshot C7   with delta DC7

Partition independence and global LSS bases

An RPM root is global to the LSS file, whereas a ROT root concerns one PSpace. PSpaces can therefore retain global LSS roots with different transaction sequence numbers. This is safe for ordinary object access because each PSpace reads and writes its own OID/Seid domain. A persistence batch that changes PSpace A need not force unchanged PSpace B to publish a new ROT snapshot merely because the global RPM root advanced.

PersistStore-wide metadata, including the PSpace map and any serial elements not owned by one PSpace, requires an explicit base-selection policy. Such metadata must not be interpreted through an arbitrary PSpace snapshot. Similarly, an operation that reads several PSpaces holds a collection of PSpace snapshots. Ordinary PSpace transactions do not imply that this collection is one atomic PersistStore-wide snapshot; a requirement for cross-PSpace consistency needs explicit coordination.

A lookup first consults the immutable ROT snapshot. A resident object entry supplies the object rendition and a tombstone reports that the object does not exist. If there is no entry, the object is loaded through the LSS read transaction retained by the base. Every lazy load therefore uses the same base as that PSpace's ROT delta and cannot accidentally advance to a newer RPM snapshot. The dirty state of the ROT path and entry records whether the object is part of the delta or is merely a clean cached object loaded from the base.


Read(snapshot, oid)
{
    if (auto entry = snapshot.rot.Find(oid))
    {
        if (entry.IsTombstone())
            return DoesNotExist;
        return entry.Object();
    }

    return snapshot.lssBase->Read(oid);
}

Delta lifetime and eviction

Dirty ROT nodes and the dirty object renditions or tombstones reachable through them embody the delta pending application to the LSS. The delta contains objects created since the base, rewritten renditions of existing objects, and tombstones for deleted objects. Freezing a mutable ROT does not make those entries clean; it makes that particular delta immutable. An absent ROT entry means "consult the LSS base" and therefore cannot also represent deletion. A tombstone must remain in the delta for as long as the snapshot exists.

An object rendition that forms part of the delta cannot be evicted if eviction would leave only the older rendition in the LSS base. Reloading it from that base would make the snapshot revert. The initial implementation therefore strongly retains every new or rewritten object rendition and every deletion tombstone required by a live snapshot. Clean objects whose values come entirely from the LSS base can be evicted and reloaded through the retained LSS read transaction.

Later implementations can replace a retained C++ object with another snapshot-stable representation, such as an immutable serialised byte sequence, a compact delta record, or a record in a spill file. The invariant is not that every delta object must remain materialised as a C++ object; it is that the snapshot must retain enough information to reproduce exactly the same Seid-to-serial-element and OID-to-object mappings without consulting a newer state or falling back to an older base rendition.

Cheap high-frequency ROT snapshots

Publishing a PSpace snapshot freezes that PSpace's current mutable ROT root and publishes it as immutable. A new mutable root initially shares the frozen nodes and object renditions. The next write under that PSpace's mutex copies only the changed object and affected ROT path. Snapshot creation must not scan the complete ROT or serialise every dirty object; delta membership is maintained incrementally as objects are created, rewritten, and deleted.

Publishing a new latest snapshot releases the latest-reference held on the preceding snapshot. If no reader or calculation retained the old root, it is destroyed immediately and its unshared ROT paths and delta objects are reclaimed. A high publication rate therefore does not by itself retain a long history. Memory grows only for snapshots still used by readers or asynchronous work, plus COW nodes and object renditions shared with those snapshots.

Asynchronous calculations retain the immutable PSpace ROT root that defines their input. Calculations for different snapshots or different PSpaces can run concurrently without blocking PSpace writers and without observing changing inputs. Each result records the identity of its input PSpace snapshot. When it completes, cxPersistStore installs it only where it is valid; a slow result for an older snapshot must not overwrite a result for a newer snapshot. Work scheduling can cancel obsolete calculations or coalesce pending requests so that a 50 Hz edit stream does not require every intermediate calculation to run to completion.

Periodically applying PSpace deltas to the LSS

At the persistence cadence, cxPersistStore selects an immutable ROT snapshot as the cutoff for each dirty PSpace included in a batch. Because each cutoff is immutable, serialization can proceed without allowing later edits to change what is being written. The selected deltas are applied through one serialized LSS write transaction. Calling CloseAndPublishSnapshot() at the end of that transaction publishes the resulting global RPM root and returns the already-pinned LSS read transaction for the new shared base.

Selecting a cutoff requires the PSpace mutex only long enough to retain the immutable root and record its generation. Serialization and LSS writing occur after releasing that mutex, while new user transactions continue against the PSpace's newer mutable ROT root. The persistence worker can collect cutoffs from several PSpaces without holding all of their mutexes throughout object serialization or LSS I/O.

selected cutoffs:  PSpace A snapshot A104, PSpace C snapshot C28

before applying A104:  current A = B100 + DA104 + changes after A104
before applying C28:   current C = B100 + DC28  + changes after C28

apply DA104 and DC28 in one LSS transaction; call CloseAndPublishSnapshot()
                                      ↓
new global LSS base B105:             B100 + DA104 + DC28
current A after rebasing:              B105 + changes after A104
current C after rebasing:              B105 + changes after C28

Earlier PSpace snapshots remain unchanged as their old base plus their own delta. New snapshots of A and C use B105 as their base. A PSpace not included in the batch can retain B100. Rebasing mutable state must not simply clear dirty flags: an object written from a cutoff may have been edited again while serialization was in progress. Generation-tagged delta entries or comparison with the immutable cutoff identifies which changes were included in B105 and which later changes must remain dirty.

If PSpace B later persists a delta still based on B100, its LSS write transaction nevertheless starts from the latest global LSS state, B105, and applies only B's own Seid changes. It must not reconstruct and publish the whole older B100 map, which would erase the persisted changes from A and C. Disjoint PSpace Seid domains make this merge straightforward; updates to PersistStore-wide metadata require the explicit coordination described above.

Visibility and durability remain separate. CloseAndPublishSnapshot() makes B105 available as an LSS base but does not by itself require a synchronous disk flush. If the declaring PSpace transaction requests forced durability through cxPersistStore, that request propagates to FlushWhenClose() on the LSS transaction. Otherwise the LazyFlusher flushes according to LssSettings::flushTimeMilliSec, whose default is 1000 milliseconds.

A PSpace transaction that requests forced durability cannot merely leave its delta for the normal multi-second persistence cadence. Closing that transaction waits until a persistence batch containing at least its cutoff has completed and the corresponding LSS flush has returned. Other dirty PSpaces can be included in the same batch, and the flush also makes all preceding completed LSS transactions durable.

After the LSS API is changed, each PSpace must maintain immutable ROT deltas and retain their LSS bases. cxPersistStore must batch selected PSpace deltas into LSS transactions and rebase each participating PSpace's later mutable state without losing edits made after its cutoff.

Preparing objects in a temporary PSpace

cxPersistStore can create and populate objects in a temporary PSpace and later transfer them into a parent PSpace. Most construction then occurs while holding only the temporary PSpace's mutex. The parent mutex is needed only for the transfer, reducing contention on a heavily used parent and preventing readers of the parent from observing partially prepared objects.

The transfer acquires write access to both PSpaces in the defined lock order and transfers object ownership, dirty-object state, deletion state, and resident ROT entries as one operation. The parent snapshot immediately before the transfer contains none of the transferred objects; the first parent snapshot published afterward contains the complete transferred state. An already-retained snapshot of the temporary PSpace remains immutable and readable for its lifetime even though the mutable temporary PSpace has transferred its objects.

The current transfer machinery must be audited for in-place changes to object or CSpace ownership metadata. It must not mutate an object rendition reachable from an older immutable temporary-PSpace snapshot. Where ownership metadata is observable through the snapshot, transfer creates the parent rendition by copy-on-write and leaves the old rendition intact. ROT and dirty-set transfer likewise updates only the new mutable roots, not retained snapshot roots.

Transferred dirty objects remain pending application to the LSS under the parent PSpace. A later persistence batch serialises them with the parent's selected cutoff. Transfer therefore shortens the parent's logical critical section without requiring construction-time mouse movements, imports, or derived work to become separate LSS transactions.

Snapshot identity and diagnostics

A PSpace snapshot identity is distinct from the LSS transaction sequence number and is scoped to its PSpace. Several ROT snapshots in one or more PSpaces can share one LSS base sequence while carrying different deltas. Asynchronous results and ROT-version comparisons therefore use the identity of the relevant PSpace snapshot, not merely the sequence number of the retained LSS base.

Resource limits and operational policy

cxPersistStore must report these values per PSpace: the number and age of retained ROT snapshots, the LSS base shared by each snapshot, bytes retained in immutable deltas, and asynchronous calculations holding old roots. Aggregate statistics must show which PSpaces and LSS bases account for retention. A rapid publication rate is harmless when superseded roots are unused and released immediately. Sustained growth indicates real outstanding readers or calculations. The scheduler can apply backpressure, cancellation, or latest-value coalescing to calculations without invalidating a snapshot already in use.

Validation and tests

Required cxPersistStore tests include:

Many immutable ROT snapshots from one or several PSpaces share one LSS base while returning their distinct new, rewritten, and deleted object states.
Evicting clean ROT cache entries reloads through the retained base, while delta renditions and tombstones cannot revert to that base.
Replacing the latest ROT snapshot immediately reclaims its predecessor when no reader or calculation retained it.
A slow calculation over an old ROT snapshot observes stable inputs and cannot overwrite a result belonging to a newer snapshot.
Transactions mutate different PSpaces concurrently while mutation within each PSpace remains serialised by its mutex.
Readers in different PSpaces concurrently load different serial elements without acquiring the LSS writer lock.
Applying immutable cutoffs from several PSpaces in one LSS batch does not clear later edits from any participating mutable delta.
A PSpace omitted from a persistence batch continues using its existing base and snapshots correctly.
A multi-PSpace reader observes the explicitly selected component snapshots and is not accidentally presented as a globally atomic snapshot.
Objects constructed in a temporary PSpace become visible in the parent as one complete transfer while older retained snapshots remain unchanged.
Forced durability flushes the selected cutoff synchronously, while ordinary updates remain eligible for periodic lazy flushing.

Suggested implementation sequence

Represent each PSpace snapshot as an immutable ROT delta over a retained LSS base, allowing snapshots from several PSpaces to share that base.
Strongly retain delta renditions and tombstones until another snapshot-stable representation exists.
Publish and reclaim high-frequency COW ROT snapshots independently in each PSpace and independently of the lower-frequency LSS persistence cadence.
Batch immutable cutoffs from several dirty PSpaces into one LSS transaction and rebase each participating mutable state without clearing later changes.
Associate asynchronous calculated results with their input PSpace snapshots and prevent obsolete results from replacing newer results.
Preserve temporary-to-parent PSpace transfer as a short, atomic parent update under ordered PSpace locks.
Propagate forced durability from PSpace transactions declared through cxPersistStore while retaining LazyFlusher as the default durability path.

The resulting design separates logical publication from physical persistence. A PSpace can cheaply publish immutable application snapshots as often as its readers and calculations require, while cxPersistStore periodically batches stable deltas through the single LSS writer. Retained LSS bases, ROT deltas, and tombstones together preserve each PSpace snapshot until its final user releases it.

59 Backup enhancements

Status: unimplemented proposals. The existing delta-file and hot-standby facilities are described in Backup and hot standby for the LSS. This chapter records possible extensions to those facilities.

Apply a delta while it is being written

The LSS could open each delta file for exclusive write access while allowing shared read access. LssApplyDeltas could then apply complete transactions to a standby before the delta file is closed, reducing replication latency.

Coalescing delta files

A utility could merge a sequence of delta files into one composite delta covering a range of checkpoint sequence numbers. It would verify that the output checkpoint identifier of each input matches the input checkpoint identifier of the next. The composite would inherit its first input identifier and its final output identifier, and could itself participate in a later merge.

A composite delta file could be named:

    nnnnnn-mmmmmm.lssdelta

where nnnnnn and mmmmmm are the first and last checkpoint sequence numbers. LssApplyDeltas could select an efficient combination of individual and composite deltas from the same directory.

Merging could omit intermediate versions of a serial element and deletions that supersede earlier writes. The result could be smaller than its inputs, but would no longer provide every intermediate snapshot. Repeated coalescing could retain fine-grained snapshots for the recent past and progressively coarser snapshots for older periods.

60 RPM improvements

Status: unimplemented proposals. The current Recoverable Packet Map is described in Recoverable Packet Map (RPM). This chapter collects possible changes to its representation, allocation interface, and maintenance.

Remove redundant nodes

An RPM node with no descendant data packets cannot always be deleted because it can preserve Seid allocation state and thereby prevent reuse of previously allocated Seids. Most Seid spaces are not used for local allocation, however, and do not need to preserve that state.

The LSS could be told which Seid spaces are eligible for allocation. It could then recursively remove empty RPM nodes from all other spaces. The upper levels do not need to preserve SeidHigh allocation state because that state is held by the non-evictable RPM7 root. A related optimisation could remove redundant nodes more aggressively from an RPM3 that uses only non-affiliated allocation.

Pin a Seid-allocation node

The LSS could give a client a handle to a Seid space used for allocation. Keeping the corresponding level-three RPM node resident would avoid traversing the upper four levels for every allocation. The handle would need to be closed explicitly so the node could again become eligible for eviction.

Replace byte-wise Seid access with shifting and masking

The contiguous-Seid implementations of GetParentNodeAndChildIndex() and AlwaysGetParentNodeAndChildIndex() take the address of a Seid and walk its bytes from most to least significant. They could instead operate directly on the 64-bit value, extracting each path component with shifts and an 0xff mask. This would express the numeric layout directly, remove the byte-pointer arithmetic and associated dependence on host byte order, and make register-only evaluation easier for the compiler to recognise.

This should not be assumed to improve performance without measurement. An optimising compiler may already replace the source-level byte accesses with register operations, while the dependent RPM node lookups are likely to dominate the cost. Compare optimised assembly and benchmark the complete lookup path before and after the change. Any replacement should preserve handling of leading 0xff bytes, the mapping from packet level to child index, missing-node returns, and the TouchRPM0() call for data packets.

Use a variable radix with wider bottom-level nodes

The RPM currently consumes one byte of a Seid at each level, giving every node a radix of eight bits and up to 256 children. A candidate variable-radix layout, listed from the root towards the data packet, is 10, 10, 12, 10, 10, 12. This covers all 64 bits in six levels and splits cleanly into identical 10, 10, 12 decompositions of SeidHigh and SeidLow. Each component can be obtained with constant shifts and masks.

The 12-bit bottom radix is important because the vast majority of RPM nodes are at the bottom of the tree. A bottom node would contain up to 4096 packet positions rather than 256. For dense, sequentially allocated Seids, one larger bottom node replaces approximately sixteen current bottom nodes. The number of heap allocations for RPM nodes could therefore approach a sixteen-fold reduction, together with similar reductions in per-node construction, destruction, metadata, parent-child links, eviction-list maintenance, and dirty-node bookkeeping. A 10-bit parent of these nodes spans 1024 * 4096, or 4,194,304, data-packet Seids.

This arrangement puts the larger fan-out where consecutive Seids naturally fill it, while using 10-bit, 1024-entry nodes elsewhere. Assuming eight bytes per entry, a dense 10-bit array occupies about 8 KiB and a dense 12-bit array about 32 KiB. Across a densely allocated range, sixteen 256-entry arrays and one 4096-entry array contain the same number of entries; consolidation saves the repeated RPM-node and allocation overhead rather than increasing the total entry storage. Reducing the depth from eight levels to six also removes two dependent node lookups, index operations, and existence branches from a full traversal.

The trade-off is coarser granularity. A sparsely occupied 12-bit leaf can waste more space, and loading, dirtying, serialising, or evicting a 32 KiB node for a small number of entries may cause read or write amplification. A dense-prefix representation could avoid allocating unused trailing entries in memory, but persistence and eviction granularity would still need consideration. The existing scheme for encoding RPM packet levels with leading 0xff bytes also assumes an eight-bit radix and would need to be redesigned. Measurements should compare node-allocation rate, resident memory, cache behaviour, serialised I/O, eviction behaviour, and complete lookup latency.

Sparse nodes

Each RPM node currently reserves a full array of 256 log-record positions even when few entries are present. Alternative in-memory and serialised representations could reduce the memory and storage cost of sparsely populated or formerly full nodes. Any compact representation would trade space against lookup cost and mutation complexity.

Further investigation

Review thread safety and possible deadlocks.
Measure eviction behaviour and memory use.
Verify that Seids cannot be recycled unexpectedly.
Test the RPM independently of the rest of the LSS.
Make the external contracts and internal invariants explicit in the code.

The separate Arena allocation for RPM nodes proposal considers pooling resident nodes and replacing in-memory pointers with compact handles.

61 Compacting the store

Status: unimplemented proposal. This chapter describes a possible iterative in-place compaction operation, not functionality provided by the current LSS implementation.

The user may want to compact the store to make it as small as possible. This can be useful before transferring, archiving, or deploying the store. In-place compaction avoids requiring enough additional storage for a complete second copy.

The LazyCleaner relocates live packets from segments with low utilisation to the end of the log, allowing the cleaned segments to become free. This reclaims space for reuse within the LSS, but does not by itself reduce the size of the file. A file can be truncated only when its final segments are free. Cleaning a segment in the middle of the file merely creates reusable space in the middle; the file becomes shorter only if cleaning and subsequent allocation happen to leave unused segments at the end. The proposed compaction operation is intended to arrange the live packets and free segments deliberately so the unused tail can be removed.

The LssCopy utility program can already be used to compact a store manually. It copies every live serial element from the source LSS into a newly created LSS, leaving obsolete packets and unused space behind. The resulting file is compact, but this is an out-of-place operation: it needs a second file and the user must replace the original store with the copy when appropriate. The proposal below concerns a distinct in-place compaction operation.

Iterative method

The proposed operation would repeat these steps:

Disable the background cleaner and checkpointer.
Clean every segment with utilisation below 100%.
Stop if no segments were cleaned.
Perform a checkpoint and return to step 2.

Segments that are cleaned become available so compaction can be done efficiently in-place.

In the first round of cleaning, all the obsolete packets for RPM-sections and serial elements will be removed. However, the subsequent check point may create new obsolete packets (some of the RPM-sections). These could be anywhere so a second round of cleaning may be required.

In the second round, only segments containing obsolete RPM-sections will be cleaned. This may still cause many objects to be moved to the end of the log. Then another checkpoint is performed. The previous checkpoint may contain RPM-sections that have now become obsolete. The process nevertheless converges because each pass groups the live RPM-sections into fewer, more densely utilised segments.

Schematic example

Each seven-character group below represents one segment. Within a segment:

- represents an obsolete or unused packet position.
D represents a live serial-element data packet.
P represents a live RPM-section packet.
C represents a checkpoint packet.

The table is schematic: after the initial row it shows the compacted live region, rather than preserving the original segment numbers or displaying obsolete segments that can be reused.

Stage	Segment contents	Effect
Initially	`---D---` `-DDP---` `DDDDDDD` `-------` `PPD----`	Live data and RPM packets are scattered through five segments.
Clean 1	`DDDPDDD` `DDDDPPD`	The live packets are copied into two full segments.
Checkpoint 1	`DDD-DDD` `DDDD--D` `PPPC---`	The checkpoint writes replacement RPM sections and a checkpoint packet, making the old RPM packets obsolete.
Clean 2	`DDDDDDD` `DDDDPPP`	The remaining live data and RPM packets are packed together again.
Checkpoint 2	`DDDDDDD` `DDDD---` `PPPC---`	New RPM sections replace the RPM sections in the second segment.
Clean 3	`DDDDDDD` `DDDDPPP`	The live RPM sections are packed back into the second segment.
Checkpoint 3	`DDDDDDD` `DDDDPPP` `C------`	The data and RPM segments remain full; only the new checkpoint packet is appended.
Clean 4	Nothing to clean	No partially utilised segment containing relocatable live packets remains, so the operation stops.

Size reduction of the store

If many objects are deleted from the store, it may be useful for the file size to drop back to a reasonable level. The proposal would repeatedly remove the final segment from the file, cleaning it first when necessary.

62 Arena allocation for RPM nodes

Status: unimplemented proposal. This chapter considers allocating resident RPM nodes from arenas and representing references between them with arena indexes.

Current representation

RPM nodes are currently allocated individually with new and released with delete. Each node has a pointer to its parent, and each resident internal node contains an array of 256 child pointers. On a 64-bit build the child-pointer array alone occupies 2 kB.

The RPM loads nodes on demand. It can also evict individual clean nodes that have not been used recently, so an alternative allocator must continue to support reclamation of individual nodes.

Arena allocation

Allocating RPM nodes from an arena could reduce general-purpose heap allocation overhead and heap fragmentation. Placing nodes together could also improve locality when traversing the RPM. Destroying the whole RPM would not require a separate heap operation for every resident node.

A purely monotonic arena would not be suitable because evicted nodes would continue to occupy memory. The arena would need to reuse released slots, probably by maintaining a free list. RPM node types have different sizes and behaviour, so separate arenas for RPM0, RPMi, and RPM3 would avoid sizing every slot for the largest node. The RPM7 root can remain owned directly by the LSS.

Indexes instead of pointers

Parent and child references could be represented by indexes into the appropriate arena. A reserved index, such as zero, would represent a null reference. Using 32-bit indexes would reduce an internal node's 256-entry child array from 2 kB to 1 kB on a 64-bit build. It would also allow arena storage to move without rewriting every reference.

An index should be an in-memory handle rather than part of the persistent LSS format. If a released slot can be reused while an old handle might still exist, the handle should include a generation or the design must otherwise guarantee that stale indexes cannot be dereferenced.

Code can resolve an index to a pointer while operating on a node, but must not retain that pointer across an operation that can relocate arena storage. Alternatively, arena storage can be divided into non-moving blocks, preserving pointer stability while retaining compact indexes for stored references.

Possible implementation sequence

Introduce type-specific pools or arenas while retaining pointers between RPM nodes.
Measure allocation cost, resident memory, and RPM traversal performance.
If pointer storage remains significant, replace parent and child pointers with 32-bit handles.
Preserve the existing ability to evict and reuse individual clean nodes.

Arena allocation and indexed references are related but independent changes. The arena principally addresses allocation cost, fragmentation, and locality. Indexed references principally reduce the size of internal nodes and make relocation possible.

63 Other Proposals

This chapter collects smaller proposals that do not warrant a separate chapter. The proposals describe possible future changes and are not descriptions of the current implementation.

Avoid references from LSS modules back to `LSS`

The LSS object should be the composition root and coordinator of the storage engine. A module owned by it should not normally retain an LSS& through which it can reach the entire implementation. Such upward references reverse the desired dependency direction: the parent knows its children, but each child can also know and call any other child through the parent.

The current implementation uses this pattern extensively. Examples include SegmentCache, SegmentBase, SegmentWriter, SUT, LargeSUT, RPMNode, RootBlock, LazyWriter, LazyCleaner and LazyCheckPointer. The reference is convenient, but it gives each class an effectively unbounded private API and conceals its real dependencies.

Problems caused by the upward reference

The header or implementation usually needs visibility of LSS, increasing compile-time coupling and making dependency cycles easy to introduce.
A class declaration does not reveal which services it requires. A constructor taking LSS& could use the RAS, SUT, RPM, SegmentCache, error state, settings, checkpointing or any combination of them.
The class can bypass the intended API of a neighbouring module and manipulate a lower-level mechanism directly. This makes module invariants and locking responsibilities harder to identify.
The class is difficult to instantiate and test independently because a complete or heavily mocked LSS must be provided even when only one narrow service is required.
Changes to the internal organization of LSS propagate into modules which should depend only on stable lower-level contracts.

This is principally an architectural problem, not an objection to storing references or using dependency injection. A module should retain references to the specific lower-level services it needs. It should not use its owning aggregate as a service locator.

Replace `LSS&` with narrow capabilities

For each module, list every operation currently invoked through lss_ and group those operations into coherent lower-level capabilities. Constructor parameters should then state those requirements explicitly. The capabilities need not use virtual dispatch: references to concrete module APIs or small value/context objects can preserve the present performance characteristics.

For example, SegmentBase currently uses LSS& to obtain the RAS, segment size and file position of a segment. These constitute a segment-I/O capability rather than a need for the whole LSS:


class SegmentStorage
{
public:
    int GetSegmentSize() const;
    int GetDiskSectorSize() const;
    octet_t* AllocateBuffer();
    void FreeBuffer(octet_t* buffer);
    void ReadSegment(SegId segid, octet_t* buffer);
    void WriteSegment(SegId segid, const octet_t* buffer);
    void WriteSegmentSection(SegId segid, int begin, int end, const octet_t* buffer);
};

class SegmentBase
{
public:
    SegmentBase(SegmentStorage& storage, SegId segid);

private:
    SegmentStorage& storage_;
};

Similarly, SegmentCache needs segment storage plus a place to report file errors. A Segment needs segment storage and a way to release itself to its owning cache; it does not need to navigate through LSS::GetSegmentCache(). The cache can pass itself or a narrow release callback when constructing the segment.

Background scheduling helpers are particularly simple cases. LazyCheckPointer needs an operation which performs a synchronous checkpoint, while LazyFlusher needs an operation which flushes the log. They can receive those operations or the corresponding narrow module references without retaining the entire LSS.

Desired dependency direction

The intended shape is downward and explicit:

LSS coordinator
    - constructs modules
    - supplies their declared dependencies
    - coordinates operations spanning modules

high-level protocols
    checkpoint, recovery, cleaning, transactions
        |
        v
storage mechanisms
    RPM, SUT, SegmentWriter, SegmentCache
        |
        v
physical storage
    segment I/O, RootBlock I/O, IRAS

An operation which genuinely spans sibling modules should normally be coordinated above them rather than implemented by one sibling reaching through LSS into another. This keeps the sibling contracts independently understandable and makes the cross-module ordering visible in the coordinator.

Avoid replacing one service locator with many trivial interfaces

The proposal is not to create an interface for every method or to divide the implementation into arbitrarily small classes. A useful capability should correspond to a coherent service with a stable invariant, such as segment storage or pinned segment access. If every class receives a large bundle of unrelated one-method interfaces, the original coupling has merely been made more verbose.

Nor must every short-lived algorithm object be treated as an architectural module. A visitor local to recovery or dumping may reasonably receive an operation context containing the resources for that operation. The important rule is that long-lived owned modules should expose their required dependencies and should not gain unrestricted access to their parent aggregate.

Migration

Inventory every class storing LSS& and every method it calls through that reference.
Identify the existing module which should own each required operation. Where no clear owner exists, treat that as evidence of an unresolved module boundary rather than immediately creating an interface.
Start with relatively independent lower-level cases such as SegmentBase and SegmentCache. Replace their upward references with explicit storage, error-reporting and ownership capabilities.
Move cross-module workflows into an LSS-level coordinator or an explicit operation object. Preserve the current lock scopes and ordering while doing so.
Add focused tests which instantiate each extracted module with its direct dependencies. The ability to do this without constructing an LSS is one test of whether the new boundary is useful.
Once a module has been migrated, prevent its source subtree from including LSS.h or calling back into LSS, except through an explicitly documented transitional adapter.

Unresolved boundaries

Some dependencies cannot yet be removed mechanically because they expose real architectural questions:

RPM updates also change SUT utilisation and acquire SUT reservations. It must be decided whether that coordination belongs in a higher-level packet-placement operation or in a narrow utilisation capability supplied to the RPM.
LargeSUT stores its own sections in segments allocated and cached by the same storage engine whose allocation state it represents. The self-hosting and checkpoint protocol must remain atomic while its dependencies are made explicit.
SegmentWriter jointly uses the SUT allocator, SegmentCache, log/checkpoint thresholds and checkpoint notification. Separating these dependencies must not split the invariant governing the current and next log segments.
RootBlock publication serializes state belonging to several modules. The correct boundary between root-block I/O and checkpoint coordination needs to be established before removing its LSS&.
The common file-error state is currently reached through LSS. It remains to be decided whether failures should be returned directly, recorded through a narrow error sink, or place the whole store into an explicit failed state.

These points should remain recorded as unresolved design work. Removing LSS& is successful only when it reveals a coherent dependency structure; replacing the reference while preserving the same hidden cycles would be a cosmetic change.

Rename the Free Segment Stack as the Free Segment Set

The FSS should be renamed from Free Segment Stack to Free Segment Set in the implementation and documentation. The acronym can remain FSS, so the new name is more accurate without introducing a second abbreviation for the same concept.

The FSS is conceptually the set of SegIds that are currently safe to allocate and overwrite. Stack semantics are not part of its contract. In particular, the current SegIdStack type does not implement LIFO behaviour: Push() appends at the back of its underlying std::deque, while Pop() removes from the front. Furthermore, when entries from the delta-FSS become reusable, the implementation sorts the combined FSS by SegId. Allocation therefore normally selects the lowest available SegId rather than the most recently freed SegId.

The ordering is an allocation policy layered on top of set membership. Preferring low SegIds helps cluster consecutive writes in nearby parts of the file and favours reuse near the front, making it more likely that unused segments accumulate at the end where the file could eventually be truncated. Calling the structure a stack obscures both its membership semantics and this deliberate ordering policy. Calling it a queue would also be inaccurate because checkpoint transfer can reorder all its entries.

The code change should rename terminology such as SegIdStack, comments, trace output and identifiers where they imply stack semantics. The replacement container name should avoid promising a particular implementation strategy: for example, a name such as FreeSegmentSet can expose insertion, removal and membership operations while keeping lowest-SegId-first allocation as an explicit policy. The delta-FSS should use correspondingly neutral terminology because it too is a collection of segment identifiers rather than a LIFO stack.

The documentation should consistently expand FSS as Free Segment Set while continuing to explain the ordered allocation policy separately. This is principally a source and documentation rename; the words and C++ type names are not part of the persistent LSS representation, so the proposal should not require an on-disk format change. Any externally visible diagnostic text or source-level API names should nevertheless be reviewed for compatibility before completing the rename.

Support a raw disk partition

The persistent LSS implementations currently store the LSS in a file on Windows or Linux. A future implementation could also use a raw disk partition as the linear storage medium behind the IRAS abstraction.

IRAS would be implemented on a partition in the same manner as on a fixed-size file.

Represent a log-record position as a linear offset

The current LogRecordPosition stores a segment identifier and an offset within that segment:


struct LogRecordPosition
{
    SegId segid_;
    int32 offset_;
};

It could instead store a single uint64 offset measured from the start of the first segment in the LSS file:


struct LogRecordPosition
{
    uint64 offset_;
};

Because the segment size is a power of two, the segment identifier and the offset within the segment can be recovered efficiently. If segmentSize == 1 << segmentSizeLog2, the segment identifier is offset_ >> segmentSizeLog2 and the offset within the segment is offset_ & (segmentSize - 1). Converting the value to an absolute file offset additionally requires adding the file position of the first segment.

This would give a log-record position a simple linear representation while retaining constant-time access to its two logical components. Its principal advantage is that a log-record position would no longer be limited by a 32-bit segment identifier. A 64-bit byte offset can address up to 16 EB, whereas the current documentation quotes a maximum store size of about 500 TB.

The 500 TB figure assumes 512 kB segments and arises from the current two-level Segment Utilisation Table (SUT). One 512 kB SUT-section contains 128k four-byte utilisation entries and therefore covers 64 GB of the store. The SUT-root is an array of four-byte segment identifiers stored in one division of the 64 kB root block. Approximately 8,000 such references can fit in the division, allowing the SUT to describe roughly 500 TB.

Changing LogRecordPosition alone would not raise the store-size limit. The SUT, free segment stack, segment cache, file-address calculations, and other structures also use the 32-bit SegId. They would need to be changed as part of the larger-store design. Because log-record positions are stored in persistent LSS structures, adopting the representation also requires an on-disk format change or a compatibility mechanism for existing stores.

Ensure that `FlushWhenClose()` provides durability

Ordinary LSS transaction processing deliberately does not flush the log on every close. This is central to its serial transaction throughput. FlushWhenClose() is an exceptional operation for cases that must coordinate with an external durability protocol, such as two-phase commit. Two-phase commit is a blocking protocol whose coordination and durability barriers make it expensive and impractical and are contrary to the normal LSS fast path; supporting it is not an endorsement of using it.

When that exceptional operation is requested, it is intended to make the transaction, and every transaction preceding it, durable before ILssTransaction::Close() returns. The current implementation closes the current flush unit and calls LSS::Flush() while retaining the transaction's log-record sequence lock. This drains the relevant work from the lazy writer and writes it through the RAS. It waits for the RAS writes to complete, but the RAS interface has no operation that asks the operating system and device to make those writes durable. Data may therefore remain in an operating-system or volatile device cache after Close() returns.

The RAS should provide an explicit durability operation. Its name should make the stronger semantics clear; for example:

struct IRAS
{
    // Write data without requiring an immediate durability barrier.
    virtual void Write(const void* buffer, RASAddress offset, int numBytes) = 0;

    // Make every preceding write durable on non-volatile storage.
    // Return only after the storage stack reports completion; throw on failure.
    virtual void FlushToNonVolatileStorage() = 0;
};

Durability should be separate from Write(). Requiring every write to be durable would prevent the lazy writer from batching sequential writes and would impose a storage barrier for every flush unit. An explicit operation allows many writes and transactions to be covered by one barrier. Opening a file in a mode that forces every write to stable storage can remain an optional configuration, but it should not be required for correct implementation of FlushWhenClose().

Required ordering

When a transaction marked with FlushWhenClose() is closed, the LSS should perform the following operations in order:

Finish adding the transaction's log records and close its current flush unit while retaining transaction serialisation.
Drain the lazy-writer queue through that transaction, waiting for all preceding RAS writes to complete.
Call IRAS::FlushToNonVolatileStorage().
Release the transaction's log-record sequence and return from ILssTransaction::Close() only after the durability operation succeeds.

Retaining transaction serialisation through the barrier prevents a later transaction from entering the log before the required durability point has been established. The barrier may cover more data than strictly required, but it must never cover less. Any error while writing or flushing must be reported to the caller; a failed durability operation must not be presented as a successful durable close.

Check pointing requires the same primitive. Before a new root-block division publishes a check point, all log and snapshot data referenced by that division must be made durable. After writing the new division, a second durability barrier is required before the check point can be considered durable. The required order is therefore:

write referenced log data
flush to non-volatile storage
write the new root-block division
flush to non-volatile storage

Linux mapping

For a file-backed RAS on Linux, the conservative implementation is fsync(fd). It flushes modified file data and the metadata needed to make the file state persistent. fdatasync(fd) may be sufficient when only file data and metadata required for subsequent retrieval need to be durable, but that choice must be checked against file growth, truncation, preallocation, and every supported filesystem. Using O_DIRECT does not remove the need for a durability operation, and using O_SYNC or O_DSYNC for every write is not required by this design.

If creating, renaming, or replacing the LSS file is part of the durability promise, the containing directory may also need to be opened and passed to fsync(). That is a file-lifecycle concern and need not be repeated for every transaction in an already established file.

Windows mapping

For a file-backed RAS on Windows, the explicit operation should call FlushFileBuffers(hFile) and report failure using the existing file exception mechanism. This supports batching when the file is opened without FILE_FLAG_WRITE_THROUGH. Write-through may remain available as an optional setting, but enabling it should not be necessary for FlushWhenClose() to request a durability barrier.

Neither operating system can compensate for a storage device or virtualisation layer that falsely reports completion before data is safe. The contract is therefore that the LSS issues the correct platform durability request, waits for it, and propagates errors—not that it can overcome dishonest or faulty hardware.

Verification

The change should be tested using an instrumented RAS that distinguishes completed writes from durable writes. Simulated power failure should discard completed but non-durable writes. Tests should verify that:

a transaction closed with FlushWhenClose() survives the simulated failure;
all preceding transactions survive the same failure;
transactions not covered by a completed barrier are allowed to disappear;
several transactions can be covered by one barrier;
write and durability failures are propagated by Close(); and
check point data is durable before the root-block division that references it is made durable.

Delay root-block division publication until its dependencies are durable

The check point protocol must be reviewed specifically for failures in which writes reach non-volatile storage out of order. A new root-block division can be internally valid—its sequence values and CRC may all be correct—while some log, snapshot, SUT, or RPM data referenced by that division has not yet become persistent. After a power failure, recovery may then select the new division but be unable to validate or reconstruct the state it describes.

The primary fix is to delay writing the new root-block division. The division is the publication record for the check point and must be the last check point state written. It must not be issued until all data it will reference has been written and an explicit durability barrier for that data has completed. Once the division is written, a second durability barrier is needed before the check point can be reported as complete.

This is a dependency-based delay, not a time delay. Sleeping, waiting for a periodic flusher, or merely waiting for preceding IRAS::Write() calls to return does not establish persistence or prevent reordering. The first IRAS::FlushToNonVolatileStorage() call is the event that permits the delayed root-block write.

The alternating root-block divisions protect against a torn write of a division itself. They do not, by themselves, order that write after writes to other parts of the LSS file. The recovery algorithm does, however, validate the segment containing the newest division's check point through the recorded position. If that validation fails, it rejects the newest division and reverts to the other division. This makes the current LSS more robust than a design that selects a root-block division solely from its sequence values and CRC.

The investigation should establish and document the check point publication invariant:

A root-block division must not be written until every item of persistent state needed to recover from it has been made durable. It must not be accepted as a completed check point until the division itself has subsequently been made durable.

The current implementation should be traced from creation of the check point records through SegmentWriter, LazyWriter, IRAS::Write(), and RootBlock::WriteNextDivisionToDisk(). The analysis must identify:

every log, snapshot, SUT, RPM, and file-size write on which the new division depends;
which writes have merely completed at the API boundary and which have reached non-volatile storage;
whether the operating system, filesystem, controller, or device may persist the new division first;
when the previous root-block division and the segments needed by it cease to be a valid fallback; and
whether segment recycling can overwrite data still needed to recover from the previous division.

Delayed publication protocol

The current implementation already places RootBlock::WriteNextDivisionToDisk() after SegmentWriter::FlushLastFlushableLRSForCheckPoint(). This is the right high-level order, but the latter operation waits only for RAS writes to complete; it does not establish that they are durable. If the explicit RAS durability operation proposed above is adopted, the delayed publication protocol should be:

Write all log, snapshot, allocation, and other persistent state that the new check point will reference.
Drain those writes and call IRAS::FlushToNonVolatileStorage().
Only after that barrier succeeds, write the next root-block division.
Call IRAS::FlushToNonVolatileStorage() again.
Only then report the check point as durable and permit retirement or reuse of state needed solely by the previous check point.

The first barrier prevents the root-block division from overtaking the data it references. The second barrier establishes that the division itself is durable before later operations rely on its publication. This protocol deliberately batches the dependent writes; it does not require every RAS write to be synchronously durable.

Recovery fallback

Recovery already rejects the newest root-block division and retries the previous division when the segment containing the new check point fails validation. This is an important existing defence against out-of-order persistence. The investigation should verify its scope: in particular, which other referenced structures are validated before the new division is accepted, and that all data needed by the previous division remains intact until the new check point is safely established.

Correct publication ordering remains desirable even with this fallback. It avoids reverting to an older check point unnecessarily, reduces the correctness burden placed on retention and validation of the previous state, and protects dependencies that may not be covered by validation of the check-point segment alone.

Failure testing

An instrumented RAS should model completed writes, durable writes, torn writes, and arbitrary persistence order. Tests should introduce a simulated power failure before and after every write and durability barrier in the check point sequence. For every reachable persisted state, reopening the store must produce one of the explicitly permitted outcomes:

recover from the newly published check point;
recover from the preceding check point when publication of the new one was incomplete; or
report an underlying I/O failure that made neither state recoverable.

A valid root-block division that refers to non-durable data must not cause silent loss, acceptance of inconsistent state, or an avoidable corrupt-store result. The tests should cover Windows and Linux file-RAS behaviour, buffered and direct I/O where supported, device write caching, repeated failure during recovery, and segment cleaning around the check point boundary.

Replace `GetDiskSectorSize()` with explicit I/O requirements

IRAS::GetDiskSectorSize() is poorly named. An RAS may be backed by a file, raw partition, virtual device, memory store, HDD, SSD, or NVMe device, so it is not necessarily a disk. The term sector size is also ambiguous: it can refer to a logical block, physical block, atomic-write unit, or direct-I/O alignment. The LSS does not assume that a write of this size is atomic.

The current value is principally used to constrain RAS file offsets, transfer lengths, and allocated buffer addresses. A minimal improvement would therefore be:

virtual int GetIoAlignment() const = 0;

Related source names such as diskSectorSize_ should become ioAlignment_, and the documentation should describe flush units as I/O-aligned rather than sector-aligned. This would clarify the present intent without suggesting sector atomicity.

A single alignment value may nevertheless be insufficient. Depending on the operating system, filesystem, and device, the required alignment of a file offset, transfer length, and memory-buffer address can differ. A more robust interface would return the requirements separately:

struct IoRequirements
{
    int offsetAlignment;
    int lengthAlignment;
    int bufferAlignment;
};

virtual IoRequirements GetIoRequirements() const = 0;

The file RAS should query these requirements using the facilities of the host operating system rather than returning the current hard-coded 512-byte value. The memory RAS can report the natural requirements of its implementation. Buffer allocation, segment sizing, root-block layout validation, flush-unit padding, and every RAS read and write should use the appropriate member rather than a single assumed sector size.

Logical block size, physical block size, and atomic-write size should not be folded into IoRequirements. They are different device properties and should be exposed separately only if an LSS algorithm has a defined use for them. In particular, an alignment value must never be interpreted as an atomicity guarantee.

The current root-block header stores the value under the historical meaning of disk sector size. Renaming the source-level field and clarifying its interpretation need not change the binary file format if it remains the same integer. Supporting alignments incompatible with the current 1 kB root-block boundaries is a separate format issue: the implementation must either reject such a store with a clear diagnostic or introduce a new root-block layout and compatibility mechanism.

Investigate a more efficient checksum

Investigate checksum algorithms that are significantly faster than CRC32 while still providing appropriate protection against corrupted or partially written data. The selected checksum would be used both for the divisions of the root block and for log flush units.

One possibility is to retain the existing CRC32 but calculate it using carry-less multiplication. Implementations can fold multiple blocks of input in parallel using instructions such as x86 PCLMULQDQ or VPCLMULQDQ, or Arm PMULL. If the implementation uses exactly the same polynomial and initial and final processing, this could improve performance without changing the on-disk format. A proven optimized implementation, such as one provided by a storage-oriented library, should be considered before writing a new implementation.

Another candidate is CRC32C. Modern x86 and Arm processors commonly provide dedicated CRC32C instructions. These are checksum instructions rather than SIMD instructions, but can nevertheless be very fast. CRC32C uses a different polynomial from the conventional IEEE CRC32, so adopting it would normally require an on-disk format change or a compatibility mechanism for existing stores.

XXH3 is also worth investigating. It is a fast non-cryptographic hash with implementations using SSE2, AVX2, AVX-512, NEON and VSX, and provides 64-bit and 128-bit results. A 64-bit result would provide a much lower probability of an accidental collision than a 32-bit result, but would enlarge the checksum fields in the root block and flush unit headers and therefore change the on-disk format. Truncating it to 32 bits would avoid enlarging the fields but would discard that advantage.

XXH3 does not provide the formally defined burst-error detection properties of a CRC. Those properties may not be essential for the LSS because the checksum is primarily intended to reject partially written flush units, stale data in recycled segments, and inconsistent writes rather than to protect a communication channel. This should be decided explicitly from the required failure model rather than inferred from throughput measurements alone.

The investigation should:

Implement or obtain an optimized calculation of the existing CRC32 and determine whether it preserves the current on-disk format.
Benchmark hardware-accelerated CRC32C.
Benchmark XXH3-64 as a possible checksum for a new on-disk format.
Measure small and large flush units separately, because setup and final-reduction costs may be significant for small inputs.
Assess the corruption patterns detected by each candidate, as well as its throughput.

Any implementation using processor-specific instructions would need runtime dispatch and a portable fallback unless the supported processor baseline already guarantees those instructions. This is a proposal to replace CRC32 with a more efficient checksum, not to allow the LSS to operate without checksums.

64 Historical documents

Development of the LSS began in 2002. This part preserves historical LSS documents separately from the description of the current system. Terminology, interfaces and implementation details in these documents may be obsolete.

Name	Date	Description
LSS Design	August 2005	Early description of the LSS design.
Backup and hot standby	January 2006	Initial design for LSS backup and hot standby.
Backup and hot standby, revision 2	March 2006	Second revision of the backup and hot-standby design.
Backup and hot standby, revision 3	September 2008	Third revision of the backup and hot-standby design.
CEDA Log Structured Store	March 2010	An overview of the LSS architecture and behaviour at that time.
Why is CEDA LSS such a great storage engine?	May 2015	Presentation of the design goals and advantages of the LSS.
LSS design notes	October 2016	Design notes covering the LSS implementation.
LSS interface	May 2021	Historical description of the LSS programming interface.
LSS diagrams	June 2021	Diagrams illustrating LSS structures and operations.

Editable source documents, presentation material and model files are stored alongside these PDFs in documents/lss/docs/implementation.

65 Legacy Seid encoding

Status: historical. This chapter records the legacy base-255 Seid encoding.

The Recoverable Packet Map (RPM) is a persistent 8 level hierarchical map used to locate the latest versions of recoverable packets given the Seid. These 8 levels correspond to the 8 bytes in a 64 bit Seid. There is one RPM for the LSS.

Each RPM-node in the search tree stores an array of 255 LogRecordPosition, using an index in the range [1,255]. The storage space required is at most 255 x 8 ~ 2k bytes. The levels are labelled from 0 to 7 where 0 is at the "bottom" and 7 is at the "top" of the search tree. There is exactly one RPM-node at level 7. This node is stored in the root block of the LSS. All the other RPM-nodes are stored in the log as indivisible (packet) log records (indivisible means that the RPM-node can't be broken up into a chain of packets, as done for serial elements).

Seid allocations involve incrementing the next Seidlow or Seidhigh, so that RPM-nodes are filled in an ordered manner at level 0 (for Seidlow) or level 4 (for Seidhigh). When a level 0 RPM-node is filled, we create a new level 0 RPM-node under the parent (level 1) RPM-node. If this level 1 RPM-node is full then we create a new level 1 RPM-node under the parent (level 2) RPM-node. This "create RPM-node on overflow" concept continues up through the levels, and relates to counting in base 255. Therefore the fan-out at a given node will often be much less than 255. When an RPM-node is serialised to disk, only the portion of the array of LogRecordPosition that is in use is written to the archive.

There are two types of packets

Data packets
RPM packets

Seids to identity RPM packets

Let b3.b2.b1.b0 represent a 32 bit value made of bytes bi, where b3 is the most significant byte. On little endian (such as x86) architectures, b3 has the highest memory address.

Let a Seid have Seidhigh = c6.c5.c4.c3 and Seidlow = c2.c1.c0.p. Let p > 0, c0 > 0, ... c6 > 0

The Seid space is reserved as follows

                                     SeidHigh      SeidLow
       ------------------------------------------------------
        Null Seid                   0 . 0. 0. 0   0. 0. 0. 0
        Level 6 RPM packet          c6. 0. 0. 0   0. 0. 0. 0
        Level 5 RPM packet          c6.c5. 0. 0   0. 0. 0. 0
        Level 4 RPM packet          c6.c5.c4. 0   0. 0. 0. 0
        Level 3 RPM packet          c6.c5.c4.c3   0 .0. 0. 0
        Level 2 RPM packet          c6.c5.c4.c3  c2. 0. 0. 0
        Level 1 RPM packet          c6.c5.c4.c3  c2.c1. 0. 0
        Level 0 RPM packet          c6.c5.c4.c3  c2.c1.c0. 0
        Data packet                 c6.c5.c4.c3  c2.c1.c0. p

                                    <-----------------------  increasing memory addresses

Number data packets supported = 255^8 ~ 1.788e+19

Note that we don't have the option of reversing the meaning of the bytes because incrementing SeidLow needs to step through p values first.

Accessing the bytes in the required order

Let a 32 bit integer be written as a.b.c.d where a is the most significant byte and d is the least significant byte.

The following code can be used to extract the bytes from 32 bit integer v


d = v & 0xFF;
v >>= 8;
c = v & 0xFF;
v >>= 8;
b = v & 0xFF;
v >>= 8;
a = v & 0xFF;

Note that the bytes are obtained in the order from least significant to most significant. Unfortunately for the purposes of traversing down the RPM, we actually want to extract the most significant byte first.

The following retrieves the bytes in the required order:-


a = v >> 24;
b = (v >> 16) & 0xFF;
c = (v >> 8) & 0xFF;
d = v & 0xFF;


c = v >> 8;
b = c >> 8;
a = b >> 8;
    < use a >
b &= 0xFF;
    < use b >
c &= 0xFF;
    < use c >
d = v & 0xFF;
    < use d >


octet_t* p = ((octet_t*) &seid) + 7;
for (int i=0 ; i < 8 ; ++i)
{
    int index = *p--;
}

66 LSS - old documentation from 2005

Status: historical. This chapter preserves documentation written in 2005. Terminology, interfaces and implementation choices may have changed.

Log structured store (LSS)

Summary

Ceda uses a Log Structured Store (LSS) as the underlying persistence mechanism. Advantages are

The store is simpler that the classical page based systems, like ARIES.
No particular hardware or OS support is required --> platform independence
Write performance is optimised – because all objects are written to the end of the log
Space utilisation is excellent because objects of all sizes are written “crunched up” to the end of the log
Recovery is simple because the log is self describing. A simple scan of the log allows for recovery of indexing information.
Because recovery is fast and efficient, check points can be done infrequently. This greatly reduces the amount of indexing information that is written to disk
It is easy to see how compacting + cleaning algorithms can be devised that recover space and recluster the segments to optimise read performance.
Cleaning eliminates fragmentation.
The persistent store has been decoupled from concurrency, simplifying the design. Rather than all of ACID, it only promises atomicity.

Ceda uses a persistent history buffer to represent all changes to persistent objects – even individual character insertions and deletions in a text document, or resizing of a slider control. This reflects the capability for synchronous editing. Therefore, good write performance is important. A log structured store seems ideal for this purpose.

Overview

A Log-Structured Store (LSS) unifies the object database with the log. ie objects are written to the log, not to a separate area.

As new versions of objects are written to the end of the log, previous versions are rendered obsolete. Therefore the log requires periodic garbage collection (called cleaning). This occurs in parallel with the normal operation of the store.

The log is stored as a linked list of segments. A segment is fairly large to reduce the impact of disk seek operations. Over time, more and more objects within a previously written segment become obsolete. Cleaning involves the relocation of live objects, providing an opportunity to re-cluster objects. It also facilitates application level garbage collection.

The LSS is a persistent heap, and only manages opaque blobs of binary data. These are called serial elements. A layer above the LSS called the Persistent Object Store (POS) associates a serial element with the serialised state of a persistent object. There is no size limit on a serial element. A serial element is identified by a 64 bit Object Identifier (OID). This is assigned when the serial element is first created, and kept for life.

Internally the LSS breaks up large serial elements into smaller pieces called packets. Clients of the store are insulated from this implementation detail.

A client of the LSS reads or writes a serial element as a stream of bytes (through the IFile interface). Neither the LSS nor clients of the LSS need to store a serial element as a single contiguous block in main memory.

Object Identifiers (OIDs)

An OID is a 64 bit number that uniquely identifies a packet or serial element on a given LSS. It is composed from two 32 bit values called OIDhigh and OIDlow.

Note that an OID doesn’t point directly at a serial element in the log. An OID is assigned to a serial element forever, even though the serial element may be written to the end of the log many times. The OID is like a handle and the Recoverable Packet Map (RPM) maps the OID to the current position of the serial element in the log, providing the extra level of indirection to allow the serial element to move around and yet still be found using the same OID.

No steal policy

A steal policy is where dirty objects or pages are written to a data store even though they have not been committed yet. This could be important for long running transactions that modify many objects, or for page based systems where a page always contains uncommitted updates due to fine-granularity locking and overlapping transactions’ updates to that page.

However, it is assumed that in Ceda transactions are short lived, and will only modify a small number of objects. Therefore we assume a no-steal policy, so objects will only be written to disk when a transaction commits. This implies that no undo work is required on recovery.

External interface provided by the LSS

The ILSS interface contains the following functions

Function	Description
void Close()	It is crucial that the LSS be explicitly closed - otherwise there will be data loss
bool ObjectExists(OID oid) const	Does a serial element with the given OID exist?
IFile* Read(OID oid)	Get a file stream for reading a serial element
void Serialise(Archive& ar)	Serialise the entire LSS
void Flush()	Flush all changes.
OSID GetCurrentOsid() const	Get the current OID high used for OID allocations
OID AllocateOID()	Allocate a new and unused OID for a new serial element
OSID AllocateOsid()
ILSSCommitPhase* CreateCommitPhase()	Create a commit phase which is used to write serial elements to the store, or permanently delete serial elements. The returned commit phase has been allocated on the heap and must be deleted. Failing to do so prevents further commit phases from being opened. Multiple threads can call this function, but only one thread can open a commit phase at a time. A thread will block until a thread that has opened a commit phase deletes the commit phase. It is an error to close or delete the LSS while there is an open commit phase.

All changes (ie mutative work) done on an LSS must be done by a thread that has opened a commit phase. A commit phase is intended for a single thread. Only the thread that called CreateCommitPhase() on the LSS is permitted to call the functions in this interface, or delete the commit phase. A commit phase must be deleted after it has been used.

Function	Description
void Delete(OID oid)	Permanently delete a serial element
IFile* Write(OID oid)	Get a file stream for writing a serial element

The Persistent Object Store (POS)

The Persistent Object Store (POS) is the client of the LSS. The implementation of the POS is described in detail later in this documentation. To help understand the LSS it is useful to have an overview of the POS.

The LSS is responsible for managing persistent serial elements, keyed by OID. The POS is responsible for managing the deserialised form of those serial elements - ie the IPersistable objects in memory. For this purpose the POS makes use of the following data structures

A Resident Object Table (ROT) which is a map from OID to pointer location in memory, to keep track of the set of the IPersistable objects that are reachable from the persistent store root and are currently resident in memory
A Dirty Object Set (DOS) which keeps track of the persistent objects that have been marked as dirty and need to be written back to the LSS.

The POS assumes that IPersistable objects use pref<T> smart pointers to reference other persistable objects. A pref<T> stores an OID and this is used to “fault in” persistable objects on demand (ie when the pref<T> is first dereferenced). The POS first tests whether the object is already in memory by looking up the OID in the ROT. If not then the POS makes a request for the serial element in the LSS with the given OID, allowing the IPersistable object to be created and deserialised.

It is assumed that all access to persistable objects is done in the scope of a CedaLock. Therefore there can be at most one thread that makes a read request on the LSS at a given time.

The DOS isn’t processed at the end of every CedaLock transaction. Rather, changes to the C++ objects in memory are allowed to “accumulate”. This is very important for performance. For example, when a user makes hundreds of independent changes to an image (say over a period of 2 seconds) it is likely that the image is only marked as dirty and added to the DOS once. When the DOS is eventually processed, only the final rendition of the image will be written to disk.

A worker thread is responsible for processing the DOS. It is coded as follows

class DirtyObjectSetWriter

{

void DirtyObjectSetWriter::SignalWriteDirtyObjectSet()

{

m_wakeThread.Signal();

}

void WorkerThreadFn()

{

const int PROCESS_DIRTY_OBJECT_SET_TIMEOUT_MSEC = 2000;

while(!m_requestShutDown)

{

m_wakeThread.Wait(PROCESS_DIRTY_OBJECT_SET_TIMEOUT_MSEC);

if (m_requestShutDown) return;

// Get a CedaLock, in order to be able to serialise the objects in the DirtyObjectSet

CedaLock lock(m_abortThread);

if (m_requestShutDown) return;

ASSERT(lock);

m_ps.WriteDirtyObjectsToLSS();

lock.Commit();

}

This ensures that when objects are added to the DOS, at most 2 seconds will elapse before a worker thread processes (or at least tries to process) the DOS - ie writes all the dirty objects to the LSS and clears the DOS. Note that the DOS is processed in the scope of a CedaLock to ensure that no other thread is able to make changes to the persistent objects that are being serialised to the LSS. Therefore at most one thread has read or write access to the LSS at a given time.

When the DOS is processed, we say that a commit phase is performed on the LSS. This involves the following steps

ILSS::CreateCommitPhase() is called to obtain an ILSSCommitPhase interface pointer.
ILSSCommitPhase::Write() is called for all the dirty objects
ILSSCommitPhase::Delete() is called for all the objects to be permanently deleted
The ILSSCommitPhase is deleted.

The LSS uses a large segment cache, so often a commit phase will be able to write all its data to the LSS without blocking on I/O. It is important to note that a commit phase doesn’t flush the log. A separate function ILSS::Flush() is provided for that purpose, but performance is degraded if it is called unnecessarily.

By parenthesising the changes made by a CedaLock transaction, the LSS is able to ensure atomicity, in case the system crashes part way through writing data. On recovery the LSS starts in a state where no transaction has only been partially applied.

Note that a transaction doesn’t enter its commit phase unless all fallible work has been completed. The commit phase is meant to be infallible – in the sense that it can only fail due to a hardware or power failure. For that reason there is never a need to explicitly abort the commit phase in software.

Durability

TODO for doco

Provide a simple diagram that compares central server versus peer-peer with data replication. State the basic differences
Explain that data replication and operational transform immediately leads to the idea that data is for only one process - dramatically easing the need for concurrency control using pessimistic locking.

Assumptions on the hard-disk

The LSS implementation goes to a number of lengths to avoid certain assumptions about the hard-disk. Many data bases assume one or more of the following

Disk sectors (typically 512 bytes) are written atomically
Clusters (often 4k or 8k) are written atomically
Data is written to the platter in the same order that the write commands are issued in software
When a file is created with the FILE_FLAG_WRITE_THROUGH option, WriteFile() doesn’t return until the data has gone to non-volatile storage.

Ceda avoids making these assumptions.

OID allocations

As a transaction runs, new objects may become reachable and therefore need to be assigned an OID. This happens when container objects are marked as dirty. The POS needs to assign OIDs to the objects that become reachable for the first time. Therefore it is necessary to support OID allocation in ILSS rather than ILSSCommitPhase. An insignificant downside is that when a transaction aborts we have some wasted OIDs.

Assumed access pattern for reading serial elements

When the Object Store reads a serial element, the stream of bytes is deserialised into a C++ object (and typically won’t be contiguous in memory). A Resident Object Table (ROT) keeps track of objects that are resident in memory, so there will rarely be a need to read the same serial element twice.

Note that the LSS typically reads a large number of serial elements at a time – because of the coarse granularity of segments. The LSS assumes that locality within a segment is good and therefore it buffers segments that have been read from disk.

Localisation

When objects are written to disk at around the same time, it is likely that they will be written to the same segment. Therefore the LSS tends to localise objects in a segment according to localisation of mutative changes in time. Objects that were previously together will be drawn apart if one object is modified but not the other.

When a user creates large sub-trees of objects in short periods of time there will be good localisation. If a user tends to make changes all over the place there will be poor localisation.

Cleaning techniques that improve localisation are discussed in the chapter on contexts.

State variables in memory to manage the LSS

The following variables are defined in main memory to manage the LSS.

SC	The Segment Cache. Provides a cache of a subset of the segments that have been loaded from disk. Also used for segments that are prepared in memory ready to be written to disk. A maximum number of segments in the cache can be specified. A typical value would be 32 to provide a 16Mbyte cache. An LRU policy is used to choose segments to be evicted from the cache
SW	The Segment Writer. Implements the logic for a background thread called the lazy writer that is used to write segments in the SC to disk.
SUT	The Segment Utilisation Table indicates which segments are free, and the utilisations of segments.
RPM	The Recoverable Packet Map. Maps from OID to {SegId,offset} for every packet in the log. This data structure is only partially loaded into memory.
lastCheckPoint	A LogRecordPosition that points at the last valid check point position in the log.

Efficient serialisation of the entire LSS

It is possible to serialise the entire LSS to a single stream that is suitable for transmission between computers without the overhead of meta-data like the SUT and the RPM. This simply involves serialising all serial elements. Each serial element is serialised by writing the following

The OID assigned to the serial element
The size of the serial element
The serial data of the serial element

The order in which they are sent doesn’t matter apart from maintaining locality.

De-serialisation at the destination involves starting with an empty store and writing the serial elements to segments (breaking them up into packets) and recreating the SUT and RPM at the destination. This can be achieved using one big transaction followed by a check point.

Better performance is gained if the stream is also passed through a compression algorithm.

Excellent transaction throughput

We call information in the SUT and RPM meta-data to distinguish it from the “real” data stored in the serial elements.

Meta-data is redundant and this allows the LSS to only write meta-data during check points, and yet still recover information written to the log after the last check point. By performing fewer check points, the overhead of writing the meta-data can be reduced so that efficiency of the LSS is very high – because most of the time it is writing real data. Furthermore, data is always written to a growing log without delays from disk head seeking. This allows the LSS to support a very high transaction throughput.

Daisy chaining of segments

A check point involves writing data to the log to reduce the time taken for a recovery. A recovery involves a scan through the log records in sequence starting from the last valid check point until reaching the end of the log. There is never a need to scan log records in sequence before the last valid check point. Therefore, only the segments on or after the last check point need to be daisy chained. These segments are never freed by the cleaner (because the cleaner is only allowed to work on segments before the last check point).

We regard the segments after the last check point as forming a log in the normal sense, whereas the segments before the last check point are no longer ordered, and behave more like a read only random access data store. This simplifies the cleaner because it doesn’t need to worry about a linked list as it frees segments.

Segments are only ever added at the tail of the log.

Concurrency notes

Between check points, only free segments can be written to. Therefore we can regard segments as supporting shared read access. As such there is no need to lock segments (say to limit concurrency between a CedaLock thread and the cleaner). However we need to be careful between the cleaner and the check pointer. Perhaps the same thread should be used for both. It seems nice for the LSS to hide both of these concepts. This makes the ILSS interface small and simple.

Log Record Set (LRS)

A Log Record Set (LRS) is a set of log records that are written to the log as a unit during an LRS-op. An LRS consists of zero or more packet records followed by a snap shot record.

There are three types of LRS-op

When the POS commits a transaction
When one or more segments are cleaned
When dirty RPM-sections are written to the log, prior to a check point

In all cases, an exclusive write lock is gained on the SegmentWriter, so that the LRS can be written without overlapping with another LRS.

Concurrency issues

An LRS-op gets an exclusive lock on the SegmentWriter for the life of the operation. Therefore it is not possible for these operations to overlap.

The SUT and RPM are only modified by LRS-ops. Therefore an LRS-op can’t see intermediate states of the SUT or RPM. In particular, a check point will see a consistent SUT and RPM without locking them for an extended period. This ensures that a check point writes a coherent and up to date version of the SUT and RPM to the root block and log. Also, the cleaner will get a coherent and up to date picture of what packets are live and obsolete as it cleans a segment.

The root block and the SUT are only read or written by LRS-ops, so therefore there is no need to protect them with mutexes.

The RPM is only written by LRS-ops, but it can be interrogated outside of an LRS-op. Therefore, it uses a mutex in order to be thread-safe. Locks on the RPM are only granted for short periods.

Packets

Support for large serial elements using packet chains

The LSS supports arbitrarily large serial elements. Large serial elements are split into smaller variable size pieces called packets. A packet is written as a packet log record to a segment. A packet can’t overflow a segment.

The packets of a given serial element are daisy chained. Each packet is assigned its own OID. A packet stores the OID of the next packet in the chain, or a null OID to indicate the tail packet. No! There is a bit in the first byte of a packet log record that indicates whether there is a next packet in the chain.

The OID of a serial element matches the OID of the head packet in the chain for that serial element.

The RPM and the cleaner work at the level of packets without any regard for how they daisy chain into serial elements.

OID management

Detecting dangling references

OIDs are never reused. ie if a serial element is destroyed then its OID may never be assigned to another serial element. This allows the LSS to reliably identify dangling references to persistent objects.

Allocating OIDs to new objects

A client that is assigned a value of OIDhigh can efficiently allocate OIDs in its private address space by simply incrementing OIDlow for each serial element that is created. The RPM stores the next available OIDlow for each value of OIDhigh

Snapshots

The LSS moves atomically from one valid snapshot to the next. This is achieved by writing snapshot records to the log. Packet records are only committed when the subsequent snapshot record is found. If no snapshot record is found on crash recovery, the log is truncated.

A transaction gains exclusive access to the log in its commit phase

Given the no-steal policy, we may as well assume that each transaction gains exclusive access to the log in its commit phase so it can write all of its modified objects, and the commit record in one go without interleaving log records with other transactions or a check point. With sufficient write caching, a transaction will be able to quickly write data to the cache and return, and avoid being blocked by I/O. There doesn’t seem to be any benefit in having multiple transactions all writing to the log at the same time.

Advantages

There is less risk of data loss, because a transaction is given the freedom to write all its data in consecutive log records. As an extreme example, if lots of transactions write their data in parallel and a crash occurs, it is likely that none will have succeeded in writing all the changes to the log.
We avoid mixing the data of unrelated transactions, giving us better data localisation – which will improve read performance.
It simplifies crash recovery. It can be assumed that log records written by a transaction never span a check point, so there is no need to scan log records before the last valid check point during recovery.

Performing a commit phase

When a transaction of the POS enters the commit phase, there is every intention of writing the snapshot record. In fact only hardware or power failure can prevent this from happening. Therefore the in-memory RPM and SUT can be updated as packets are written by the SW to the SC.

The commit phase involves the following steps

Get an exclusive lock on the log
Initialise a temporary vector X of OIDs to empty, used to keep track of the set of packets to be deleted
For each serial element to be written (with given OID)
1. Use the SC to find the current chain C_prev of packets used by the serial element (if any). For each packet in C_prev
  1. Add the OID to X
  2. Subtract the size of the packet from the SUT utilisation for the segment containing the packet

Remove the entry for the packet from the RPM.

Write a chain C_new of packets to the log. For each packet in C_new.
1. Add the size of the packet to the SUT utilisation for the segment containing the packet
2. Add an entry to the RPM
For each serial element to be deleted (with given OID)
1. Use the SC to find the current chain C_prev of packets used by the serial element (if any). For each packet in C_prev
  1. Add the OID to X
  2. Subtract the size of the packet from the SUT utilisation for the segment containing the packet

Remove the entry for the packet from the RPM

Write a snapshot log record. This record contains the list X of OIDs of packets to be permanently deleted as part of the snapshot.
Release lock on the log

Note that the log is not flushed.

Supporting overflow of the root block

Given the idea of the FSS, we can “pinch” some segments for the purpose of supporting overflow of the root block. It is important to write and flush the free segments first, then write the root block. In general we must always write the root block last of all because this is what takes us atomically from one check point to the next.

Implementation notes

The entire functionality of the LSS is implemented in the folder Ceda/Sys/LSS.

Currently we don’t support recovery scanning of the log. Instead, recovery simply involves going back to the last valid check point.

We cater for Challis algorithm in a generic way, so we get code reuse for the root block and the segment trailer.

A lower layer (RAS) represents the underlying file, and the ability to read and write parts of it. This abstraction is useful because it could be replaced by a raw disk partition. It supports a 64 bit address space to allow for more than 2Gig of data.

The next layer breaks this address space up into the two root blocks and the segments. The ability to add new segments will be provided. Read/write functions to the root block and to a given segment will be provided. Also the ability to allocate new segments by growing the underlying file.

A class to encapsulate the SUT has been written. This is implemented as a vector of records. The SUT expands when extra segments are added to the file. Note that the system allows for the SUT and the size of the file to be out of sync – typically the file could have grown then the system crashed before the SUT has grown and been written to a valid check point. Growing of the file is not logged.

The class LSS encapsulates all functionality.

IDEA: Atomic append using checksums

Rather than store a size field in the segment trailer, and need to rewrite the segment trailer every time data is flushed to the log, consider that a flush log record is always written to the log after each flush operation. When scanning the log on recovery, we only trust data that is bounded by a valid flush log record. A flush log record can’t simply store some fixed “magic” values, because segments that are reused already contain previously written flush log records and so there is a risk that data will be trusted by mistake. A flush log record can contain a sequence number and/or checksum over the data in the segment. Note that this approach requires that the recovery scan be resilient to corrupt log records.

67 Introduction

This part serves as a design specification for a new version of the CEDA LSS. It will attempt to incorporate the proposals in Part VII. In particular, the aim is to make the LSS fully asynchronous and support multi-version concurrency control (MVCC).

Version 2 is not required to be compatible with Version 1. Its APIs, in-memory structures and persistent file format may change where that produces a better design.

The aim is to implement the LSS as a pure event-driven state machine, without using coroutines. An event causes the LSS to update its state and may synchronously produce further work or notifications. Operations that require I/O return to the caller and continue later when the RAS reports completion.

For example, the RAS may call back into the LSS with OnReadComplete(). The LSS can use that event to make a newly loaded segment available in the segment cache. Making the segment available may in turn allow pending reads of serial elements to continue: the LSS copies data into their destination buffers with memcpy() and, when those reads are complete, notifies applications built on top of the LSS. In this way, a native I/O completion can drive a sequence of state transitions and application notifications without suspending a coroutine or blocking a thread.

Although the intention is for LSS Version 2 to support asynchronous I/O, it may be better for the initial implementation to use synchronous I/O because that is simpler. Moving to asynchronous I/O can reasonably be treated as a later task. It is better to keep the initial problem simple enough to reason about and obtain good solutions than to introduce so much complexity that progress is slow and mistakes are made.

Although Version 2 is not being implemented in Rust, it is useful to confirm that the design has a straightforward mapping to a borrow checker. Borrow checking often forces ownership and lifetime relationships to be expressed clearly and therefore encourages good design.

The aim is for the Version 2 specification to be sufficiently precise and complete for AI code generators to implement it. If the design, interfaces, ownership rules and persistent formats are specified clearly enough, the resulting implementation will probably be both efficient and correct.

The specification must define the binary representation of the LSS file unambiguously.

Performance principles

The current LSS is already fast, and profiling has shown that a significant proportion of its time can be spent in memcpy(). Version 2 changes how the LSS waits for work, but must not make the existing data path less efficient. In particular, asynchrony must not introduce additional payload copies merely to simplify ownership or completion handling.

RAS reads target their final segment-cache buffers directly. The RAS fills the buffer supplied in a RASReadRequest, and completion transfers that buffer back to the LSS without an intermediate RAS or event buffer. Similarly, an LFU is constructed in the buffer from which the RAS writes it. The WSN completion contract keeps that original buffer alive until the native write no longer needs it, so asynchronous writing does not require a staging copy.

Events carry lightweight operation identities or references to existing state, not copies of payload data. A read-completion event can refer to its RASReadRequest, a segment-cache event can refer to its segment, and write progress is represented by two counters. Internal event dispatch must not move segment or serial-element contents between buffers.

When an application asks the LSS to fill an application-owned buffer, copying data from cached segments into that buffer remains necessary. The implementation should preserve large contiguous memcpy() operations where the packet layout permits them. Existing interfaces that safely provide a direct read-only view of resident data should remain zero-copy and must not be routed through a copied result merely for uniformity with asynchronous operations.

Native I/O may usefully overlap CPU work. For example, the device can load one segment while the LSS copies data from another resident segment. This does not imply that several CPU threads should perform payload copies concurrently. Parallel copies can compete for memory bandwidth and disrupt cache locality, so that form of concurrency should be introduced only when profiling demonstrates a benefit.

For equivalent operations, Version 2 should perform no more payload copies than the current LSS solely because it is asynchronous. Performance tests should compare the two versions and record at least the number of payload copies, bytes copied per application byte, average copy size, heap allocations, event dispatches, cross-thread hand-offs and overall throughput. RAS staging copies and payload copies inside events should both be zero.

The specification does not mandate a particular event queue, object pool, I/O queue depth, batching threshold or thread on which copying occurs. Those are implementation choices to be guided by measurement. The architectural requirement is that asynchronous coordination remains lightweight and does not compromise the efficient buffer usage of the existing LSS.

68 LSS API

This chapter defines the Version 2 LSS API. Value types are structs or aliases, operations which do not belong to an object are free functions, and runtime polymorphism is expressed using pure abstract interfaces. The API does not expose implementation classes.

An LSS is a persistent heap of variable-length serial elements organised in a hierarchical structure.

The logical data hierarchy is:

Lss
    Partition
        Space
            SerialElement

Lss is associated with one LSS file and its lifetime.
Partition is the transaction, MVCC snapshot and mutation-concurrency boundary. At most one mutative transaction is active on a partition, while different partitions may mutate concurrently.
Space is a 32-bit Seid address space containing serial elements.
SerialElement is a variable-length binary value identified by a Seid (pronounced “see-id”), a contraction of serial element identifier.

An LSS can have up to approximately four billion partitions, each partition can have up to approximately four billion spaces, and each space can have up to approximately four billion serial elements.

Identifiers


using PartitionId = uint32;
using SpaceId = uint32;
using Seid = uint32;

inline constexpr PartitionId PRIMARY_PARTITION_ID = 1;
inline constexpr Seid NULL_SEID = 0;

struct PartitionSeid
{
    SpaceId spaceId;
    Seid seid;
};

An Partition is logically a set of Spaces indexed by 32-bit SpaceId. A Space is logically a set of serial elements indexed by 32-bit Seid. Seid zero is null and does not identify a serial element.

Serial-element input


struct ReadOnlyBuffer
{
    const octet_t* data;
    std::size_t size;
};

struct IContiguousSerialElement
{
    virtual ReadOnlyBuffer GetBuffer() const = 0;
    virtual void Close() = 0;
};

An IContiguousSerialElement pins the storage containing its buffer until Close() is called. It is not deleted by the caller.

Views


struct ISpaceView
{
    virtual SpaceId GetSpaceId() const = 0;
    virtual bool SerialElementExists(Seid seid) const = 0;
    virtual ICloseableInputStream* ReadSerialElement(Seid seid) const = 0;
    virtual IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const = 0;
};

struct IPartitionView
{
    virtual PartitionId GetPartitionId() const = 0;
    virtual const ISpaceView* FindSpace(SpaceId spaceId) const = 0;
    virtual void Close() = 0;
};

FindSpace() returns null when the selected partition snapshot does not contain the given Space. A returned Space view is owned by the partition view and remains valid until that partition view is closed.

An IPartitionView pins one immutable MVCC snapshot of the partition's SpaceDirectory. That snapshot entails the RPM snapshot of every Space it contains. Every read through the transaction therefore observes the same completed partition transaction boundary.

Every input stream and contiguous serial element obtained from a partition view must be closed before the view is closed. A partition view may be used concurrently by multiple reader threads, but an individual returned stream is not shared between threads.

Mutative transactions


struct ISpaceTransaction : ISpaceView
{
    virtual Seid AllocateSeids(uint32 count) = 0;
    virtual ICloseableOutputStream* WriteSerialElement(Seid seid) = 0;
    virtual bool DeleteSerialElement(Seid seid) = 0;
};

struct IPartitionTransaction
{
    virtual PartitionId GetPartitionId() const = 0;
    virtual ISpaceTransaction* FindSpace(SpaceId spaceId) = 0;
    virtual ISpaceTransaction* CreateSpace() = 0;
    virtual void FlushWhenClose() = 0;
    virtual void Close() = 0;
    virtual IPartitionView* CloseAndPublishSnapshot() = 0;
};

AllocateSeids(count) returns the first Seid in a newly allocated contiguous range [first, first + count). The count may be zero. In that case no Seids are allocated and the function returns the next Seid, allowing the caller to determine how many Seids remain available in the Space.

An IPartitionTransaction holds the mutex of its containing Partition. At most one such transaction is open on a partition. Transactions on different partitions are independent and may be open concurrently. A transaction never spans partitions.

The Space transaction views are owned by the partition transaction. They become invalid when the partition transaction is closed. Seid allocation and all serial-element mutation occur through these views. A returned output stream must be closed before another mutative operation is performed through the transaction.

There is no transaction abort. Close() completes the transaction without publishing a reader snapshot. CloseAndPublishSnapshot() completes the transaction, publishes its immutable SpaceDirectory snapshot and returns a partition view pinned to that exact snapshot. Both functions consume the mutative transaction.

Calling FlushWhenClose() requests that closing the transaction wait until that transaction and preceding transactions on the same partition have been flushed.

Partitions and stores


struct IPartition
{
    virtual PartitionId GetPartitionId() const = 0;
    virtual IPartitionView* OpenView() const = 0;
    virtual IPartitionTransaction* OpenTransaction() = 0;
};

struct ILss
{
    virtual ~ILss() = default;
    virtual IPartition* GetPartition(PartitionId partitionId) = 0;
    virtual void Close() = 0;
};

struct LssSettings;
struct IRAS;

std::unique_ptr<ILss> CreateOrOpenLss(
    std::unique_ptr<IRAS> ras,
    const LssSettings& settings);

Partition ID zero is never used. PRIMARY_PARTITION_ID is one. GetPartition() returns null if the LSS does not contain the given partition.

The initial implementation contains only the primary partition and does not require a PartitionDirectory. The API for creating and deleting additional partitions will be added when the PartitionDirectory design is selected.

The store owns the RAS passed to CreateOrOpenLss(). The returned unique_ptr owns the LSS interface. All transactions, Space views, streams and contiguous serial elements must be closed before the LSS is closed.

69 Seids

Local 32-bit Seids

Version 2 changes the way Seids are understood. A Seid is a 32-bit identifier which is local to a Space. An LSS supports approximately 2³² partitions, each partition supports approximately 2³² Spaces, and each Space supports approximately 2³² Seids. A globally meaningful packet identity within an LSS is therefore the triple:


(PartitionId, SpaceId, Seid)

Most client data structures do not need to store that pair with every reference. They operate within one address space and store only the local 32-bit Seid. The address-space identity is supplied once by the context in which the structure is accessed.

A limit of approximately 2³² Seids in one address space is workable in practice. A single allocation domain containing more objects than that would already be exceptionally large and should be divided into multiple Spaces.

References which cross an address-space boundary must explicitly carry or otherwise resolve the destination address-space identity. A bare 32-bit Seid has no meaning outside its associated address space. APIs and persistent types should distinguish local Seids from globally qualified identities so that a local Seid cannot accidentally be interpreted in the wrong space.

Locality and contention

Supporting many address spaces provides better locality between separate uses of one LSS. Each use can allocate Seids within its own address space, so its identifiers and RPM paths are clustered independently rather than mixed with unrelated data. Better locality is an additional benefit of the address-space design, independently of the reduction from 64-bit to 32-bit local identifiers.

Separate address spaces can also reduce contention. Independent uses of the LSS operate on different four-level RPM roots, Seid-allocation state and lazily loaded RPM nodes instead of repeatedly touching the same upper levels of one global RPM. Concurrent readers working in different Spaces are less likely to contend on the same RPM cache state or cache lines.


using Seid = uint32;
using SpaceId = uint32;

struct PartitionSeid
{
    SpaceId spaceId;
    Seid seid;
};

An PartitionSeid is meaningful within one Partition. Qualifying it with an PartitionId produces an identity which is unambiguous within the LSS. The important distinction is between the small local Seid stored frequently inside a Space and the qualified identity used when the Space and partition are not already known.

Address-space handles

The LSS hands a client an opaque handle to a Seid address space. Internally the handle is essentially a pointer to the root of that address space's RPM. Operations which repeatedly access one Space retain this handle and use it with local Seids:


class SeidAddressSpace;

LogRecordPosition GetPacketPosition(
    const SeidAddressSpace& addressSpace,
    Seid seid);

The public handle should remain opaque even if its implementation is pointer-like. This permits the LSS to enforce its lifetime and snapshot rules without adding an address-space lookup to every Seid operation.

Four-level RPMs

Each address space has its own four-level Recoverable Packet Map. The four bytes of a 32-bit Seid are used as successive indexes in a radix tree with fan-out 256. The handle gives direct access to the root of this tree, which is equivalent to starting at the old RPM3 node rather than traversing the upper four levels of one global eight-level RPM.

The LSS maintains a sparse directory from SpaceId to address-space RPM roots. This directory is consulted when an address-space handle is obtained, not on every local Seid lookup. It must not be represented by a dense array of 2³² entries.

Space and performance advantages

Using 32-bit local Seids substantially reduces the size of data structures containing many object references. For example, a B+Tree non-leaf node containing 1024 Seids uses 4 KiB less storage than it would with 64-bit Seids. The smaller representation improves cache density and allows more entries to fit in a node or memory page.

An RPM lookup traverses four radix levels rather than eight. This can substantially reduce the CPU cost of an in-memory lookup, although the overall improvement must be measured because cache misses, segment access and other work may also contribute to lookup time. Performance benchmarks should compare the complete lookup operation rather than assume that halving the radix depth exactly doubles throughput.

The four-level tree also reduces the cost of RPM copy-on-write. Updating an RPM leaf requires copying at most the path through its four-level address-space RPM, rather than copying ancestors through an eight-level global tree. Transactions which update nearby Seids may continue to reuse paths already copied into their working RPM.

MVCC and handle lifetime

An address-space handle used by a partition view identifies the immutable four-level RPM root for that address space in the view's snapshot. A writer may create a new root using copy-on-write, while an older reader continues to use its original handle and root. The handle does not select the latest root again for each lookup.

An address-space handle must not outlive the partition view or transaction which supplies it. Removing an address space from a newer snapshot does not invalidate a handle retained by an older reader; the MVCC retention mechanism keeps the old root and its reachable packets alive until that snapshot is released.

Persistent and external identities

Some parts of the LSS operate without an already selected Space. Log scanning, cleaning, checkpointing, diagnostic tools and messages crossing address-space boundaries must be able to identify the relevant address space. Records used in those contexts carry the composite PartitionId, SpaceId and 32-bit Seid, or another encoding which is unambiguously equivalent.

Keeping a composite identity where it is required does not negate the main saving. Frequently stored local references in B+Trees, object graphs and other data structures remain 32-bit, while the address-space identity is carried once by their surrounding context.

69.1 Multiple Address Spaces in a Working Set

This subchapter discusses PSpaces and working sets only to explain the higher-layer motivation for the LSS Space design. They are not part of the LSS Version 2 specification. This material will probably move to the PersistStore document.

Operational transformation and independent object creation

Higher layers of CEDA use operational transformation (OT) to support conflict-free branching and merging of working sets in PSpaces. A working set is a tree of objects together with operations on that tree. Different sites or users can work independently, exchange operations and objects, and merge their changes without requiring a central allocator for every object identifier.

It is important that several sites can create objects in the same logical working set while they are disconnected. Object creation must not allocate the same identity at two sites, and reconnecting the sites must not require either site to rewrite every identifier it has already stored.

Allocation spaces in Version 1

In Version 1, independent sites allocate different SeidHigh values. Each allocation space is also assigned a 128-bit UUID. The UUID is the stable identity of the allocation space across sites; the numerical SeidHigh is only its local representation in one store.

When two sites connect over TCP and exchange operations or objects, they use the UUIDs to establish a translation between their local SeidHigh values. If a sender transmits a 64-bit Seid, the receiver retains its low 32 bits and translates its high 32 bits to the possibly different SeidHigh used locally for the same UUID. Once a connection has established this mapping, many identifiers can be translated without repeatedly sending their UUIDs.

Distinguished private address space

Every PSpace has a distinguished private address space for its internal persistent structures. This address space has no UUID because it does not represent an allocation domain shared between sites. Its Seids are local implementation identifiers and are never sent over the wire.

The PSpace uses 32-bit Seids from this private address space for structures such as B+Tree nodes and links between those nodes. For example, the child references in a B+Tree non-leaf node can be stored as 32-bit Seids:


struct BTreeInternalEntry
{
    Key separator;
    Seid childNode;       // PSpace private address space
};

This is the common path for internal PSpace data structures and obtains the storage, cache-locality and four-level RPM benefits of 32-bit Seids. A B+Tree leaf may still contain 64-bit OIDs when its payload refers to objects in a distributed working-set tree, but the B+Tree's own structure does not need 64-bit identifiers.

The private address space is stored separately from the vector of UUID-bearing object allocation spaces. Keeping it out of that vector prevents a private Seid from accidentally being interpreted as a transmissible working-set OID.


struct PSpaceAddressSpaces
{
    AddressSpaceId privateAddressSpaceId;
    LssAddressSpaceHandle privateAddressSpaceHandle;
};

Object address-space table

Version 2 retains the distributed object-allocation concept while composing a working set from the LSS's 32-bit Seid address spaces. Each working set records its own vector of object allocation spaces whose entries associate:

a local LSS AddressSpaceId;
the 128-bit UUID which globally identifies that allocation space; and
an LssAddressSpaceHandle bound for efficient access.


struct WorkingSetAddressSpaceEntry
{
    AddressSpaceId addressSpaceId;
    UUID uuid;
    LssAddressSpaceHandle handle;
};

std::vector<WorkingSetAddressSpaceEntry> addressSpaces;

This representation is illustrative. In particular, the persistent fields and transient handle may be stored separately. Logically, however, a working-set entry associates all three identities.

OID representation within the working set

Within the tree of objects in a multi-site working set, object references remain 64-bit OIDs. The high 32 bits are an index into the working set's object-address-space vector and the low 32 bits are a Seid local to the selected LSS address space:


struct Oid
{
    uint32 addressSpaceIndex;   // OIDHigh
    uint32 seid;                // OIDLow
};

An OID is therefore not itself a global 64-bit LSS Seid. It is a working-set object reference composed at the working-set layer:

    working-set-local address-space index + address-space-local LSS Seid

Resolving an OID first indexes the working set's object-address-space vector. The resulting handle gives direct access to the four-level RPM for that allocation space, and the low 32-bit Seid is looked up in that RPM. This requires only a vector indexing operation before the local RPM lookup; it does not traverse the upper four levels of a global eight-level RPM.

The three forms of address-space identity

The design deliberately uses three related but distinct forms of identity:

UUID: The globally stable identity of an allocation space. It is used to recognise the same allocation space at different sites and establish translation mappings.
AddressSpaceId: The persistent identity assigned to the address space by one local LSS. Different LSS files may use different values for the same UUID.
Working-set vector index: The compact value stored as OIDHigh in one replica of a working set. Different sites may assign different vector indexes to the same UUID.

The LssAddressSpaceHandle is not another persistent identity. It is a runtime capability giving efficient access to the address space selected by the other identities.

The working set is the identity boundary

A working set has its own persistent identity, vector time and OT history. All replicas of that working set share the same working-set identity, and every operation is tied to it. The meaning of an OID is therefore scoped by the working set:

    WorkingSetId + OID

A bare OID must not be resolved through a PSpace-wide object-address-space table. Two working sets in the same PSpace can use the same numerical OID for unrelated objects. Incoming operations first identify and validate their target working set, and only then interpret OIDHigh through that working set's vector.


struct WorkingSet
{
    WorkingSetId id;
    VectorTime vectorTime;
    std::vector<WorkingSetAddressSpaceEntry> addressSpaces;
};

The PSpace hosts and persists working sets, but it does not combine their UUID-to-address-space mappings. Even if two working sets mention the same allocation-space UUID, their mappings remain logically separate because their object identities and OT histories are separate.

Translation when sites connect

A connection establishes a mapping from the sender's address-space indexes for a particular working set to the corresponding indexes in the receiver's replica of that same working set. For each sender index encountered, the sender communicates the corresponding UUID. The receiver looks up that UUID in the local vector belonging to the identified working set. If it is already present, the receiver uses the existing vector index. Otherwise it creates a local LSS address space and appends a corresponding entry to that working set's vector.

    sender OID
        (sender vector index, Seid)
                    |
                    v
             allocation UUID
                    |
                    v
    receiver OID
        (receiver vector index, Seid)

The low 32-bit Seid does not change. Only the working-set-local high part is translated. The connection can cache the resulting index-to-index mapping, so UUID resolution is required when an allocation space is introduced to the session rather than for every OID sent over the wire.

Each working-set vector and its indexes are local storage metadata, not replicated global numbering. Two replicas can independently assign different vector indexes and different LSS AddressSpaceId values to the same UUID. This is expected and is the reason UUID-based translation is required.

Creating objects at independent sites

Each site which independently creates objects uses an allocation space identified by its own UUID. It allocates 32-bit Seids from the corresponding local LSS address space. Because independently created allocation spaces have different UUIDs, two disconnected sites cannot create the same composite object identity even if they allocate identical 32-bit Seid values.

After operations are exchanged, an object created remotely is represented locally using the vector index assigned to the remote allocation-space UUID together with the unchanged 32-bit Seid. OT can therefore merge operations and object trees without a central Seid allocator and without renumbering the objects created by either site.

Objects do not retain identity when transferred between working sets

An object must not be moved to another working set while retaining its OID. A delayed operation for the source working set could otherwise resolve to the transferred object and be applied outside the OT history and vector-time domain in which the operation was created. That would violate the assumption that every operation targets an object in one identified working set.

Transferring data between distinct working sets is instead an export and import, analogous to moving content between distinct Git repositories. The destination allocates new OIDs, copies the selected object tree and rewrites its internal object references. Later operations in the source working set continue to address only the source objects; they are not redirected to the imported copies.

Branches and replicas of the same working set are different: they share the working-set identity and OT history and use UUID translation to preserve object identities while changes are exchanged.

Persistent and transient state

The PSpace persists the AddressSpaceId of its distinguished private address space. Each working set persists its own identity, vector time and enough information to reconstruct its object-address-space vector, including each entry's UUID and local LSS AddressSpaceId. The pointer-like LssAddressSpaceHandle is transient. It is recreated by binding the persistent entry to the appropriate LSS transaction or snapshot when the PSpace is opened.


struct PersistentWorkingSetAddressSpaceEntry
{
    AddressSpaceId addressSpaceId;
    UUID uuid;
};

struct BoundWorkingSetAddressSpaceEntry
{
    const PersistentWorkingSetAddressSpaceEntry* persistent;
    LssAddressSpaceHandle handle;
};

Separating these representations prevents a process pointer or snapshot-specific capability from being mistaken for durable state while retaining a direct handle on the lookup path.

MVCC snapshot binding

An address-space handle used by a partition view refers to the immutable four-level RPM root for that address space in the view's snapshot. Every bound working-set vector for that view must therefore use snapshot-specific handles. It must not contain a process-wide handle which silently changes to mean the latest RPM root.

A writer can use copy-on-write to create a new RPM root for one address space while readers continue to use older handles. Removing an address-space entry from a newer working-set snapshot cannot invalidate the entry, root or packets retained by an older reader. Their lifetime follows the same deferred release rules as other immutable RPM roots.

Vector-index invariants

The object-address-space vector index is stored in every local OID using that allocation space, so an entry must not be reordered while such OIDs exist. Reusing an index for a different UUID would cause old OIDs to resolve through the wrong RPM. The simplest rule is that entries are append-only and indexes are never reused within the lifetime of a working set.

There can be at most one vector entry for a given UUID in one working set. When a connection introduces a UUID, lookup by UUID must occur before an entry is appended. Each working set therefore needs both indexed access for normal OID resolution and an efficient UUID-to-index map for session establishment.

Relationship to Predica PSpace boundaries

Multiple object allocation spaces within one working set do not permit OID-based references into a different working set or Seid-based references between different PSpaces. The 64-bit OID described here is peculiar to the tree of objects in one working set and supports independent sites creating objects in that same tree. It is not the universal identifier for internal PSpace structures. Predica relationships which cross a PSpace boundary must still use keys defined in the Predica schema. The OIDs and Seids remain hidden persistence identifiers which are not exposed as schema-defined application keys.

Resulting layering

The responsibilities are divided as follows:

The LSS provides many independent 32-bit Seid address spaces and efficient handles to their four-level RPM roots.
Every PSpace has a distinguished UUID-less address space containing internal structures such as B+Trees, whose internal links use 32-bit Seids.
Each working set owns its identity, vector time, OT history and mapping from compact local OIDHigh indexes to UUID-bearing LSS object address spaces.
The PSpace hosts the working sets and provides persistence services without combining their object identity domains.
The OT and messaging layers validate the working-set identity and use UUIDs to translate address-space indexes between replicas of that working set.
Predica uses schema-defined keys, rather than persistence identifiers, for relationships crossing PSpace boundaries.

This retains the distributed-allocation property of 64-bit OIDs where it is needed, while allowing the LSS and local data structures to benefit from compact 32-bit Seids and four-level RPMs.

69.4 Recoverable Packet Map (RPM)

Logically, the RPM maps a 32-bit Seid to a LogRecordPosition, except that Seid 0x00000000 is reserved as the null Seid.

The RPM will no longer store an LSS& in every RPM node.

The RPM is a four-level radix map which supports MVCC using reference-counted nodes and copy-on-write (COW). Publishing produces immutable RPM root nodes which are used by readers.

`RPM` and its root node

Each Space has a level 3 RPM root. Level 3 nodes record the positions of level 2 nodes, and level 0 nodes record the positions of data packets.

The RPM is a map object which holds a reference to its root node. The root is an ordinary RpmInternalNode, no different in type from the other internal nodes. Version 2 does not use an RPM7 subclass for the root.


class RPM
{
    RpmInternalNode* root;
};

Map-level operations such as lookup, copy-on-write mutation and publication belong to RPM. An RpmInternalNode only represents one node in the radix tree. The root is special only because the RPM owns a reference to it.

The writer starts with the current RPM root and shallow-copies only the nodes on paths modified by the transaction. Unchanged subtrees are shared. When the new root is published, the new nodes become immutable and own references to their shared children.

Only the writer changes logical RPM-node reference counts. Readers retain an immutable root and traverse it without changing reference counts within the RPM. Releasing a reader's root is deferred to the writer, as described in the MVCC chapter. The node reference counts therefore do not need to be atomic.

Persistent RPM roots

The RPM roots cannot all be stored directly in the fixed-size LSS root block because an LSS can contain many Spaces. The checkpoint state of each Partition must locate the RPM roots belonging to its Spaces. The exact representation remains to be specified.

Meaning of a read-only snapshot

A read-only snapshot is a logically immutable RPM mapping. It is not the set of RPM nodes which happen to be resident in memory when the snapshot is published. Some child nodes may be represented by their packet positions and loaded later when a reader traverses the RPM.

Readers may therefore grow the in-memory materialisation of a snapshot by loading RPM nodes. Changing a cache entry from an unloaded packet position to a loaded immutable node does not change the logical mapping represented by the snapshot.

Atomic publication of loaded child nodes

An internal RPM node uses separate arrays for the immutable child positions and the mutable atomic pointers to loaded children:


struct RpmInternalNode
{
    uint32 refCount;
    std::array<LogRecordPosition, 256> positions;
    mutable std::array<std::atomic<const RPMNode*>, 256> children;
};

The LogRecordPosition array defines the logical child relationships, while the pointer array records their in-memory materialisation. Keeping the arrays separate gives denser pointer cache lines for resident traversal, where the positions are not needed. A null pointer means that the child has not been loaded. A reader loads and fully constructs the immutable child node, then atomically changes the pointer from null to the child pointer. Release ordering when publishing the pointer and acquire ordering when reading it ensure that another thread which obtains the pointer sees the fully constructed child node.

Publishing a loaded child node does not change any existing logical RPM-node reference count.

On x86/x64, the reader's acquire load normally compiles to an ordinary mov instruction and does not require a separate fence instruction. Reader traversal therefore has excellent performance on these processors.


std::unique_ptr<const RPMNode> LoadChildNode(LogRecordPosition position);

const RPMNode* RpmInternalNode::GetChild(int index) const
{
    auto& childSlot = children[index];
    if (const RPMNode* child = childSlot.load(std::memory_order_acquire))
        return child;

    auto candidate = LoadChildNode(positions[index]);
    const RPMNode* expected = nullptr;

    if (childSlot.compare_exchange_strong(
            expected,
            candidate.get(),
            std::memory_order_release,
            std::memory_order_acquire))
        return candidate.release();

    return expected;
}

LoadChildNode() returns an owned reference using unique_ptr. If the compare-and-exchange succeeds, release() transfers ownership to the published snapshot. If another reader has already published the child, the losing reader's unique_ptr deletes its unpublished node. There is no memory leak.

The SegmentCache serialises access to a segment, so racing readers do not cause duplicate segment I/O. They can both allocate and deserialise a child node from the same RPM packet in the loaded segment before one wins the compare-and-exchange. The probability of this race and the amount of duplicate work are small, so no additional guard against it is required.

This approach does not allow readers to evict loaded nodes from a read-only snapshot. Clearing an atomic child pointer and deleting the node could race with a reader which has already obtained the pointer. Loaded nodes in a read-only snapshot therefore remain resident. Eviction is instead performed on the mutable RPM tree.

Strengths

Readers only perform atomic pointer publication.
Published RPM nodes remain logically immutable.
Release/acquire ordering safely publishes fully constructed child nodes.
Logical RPM-node reference counts remain non-atomic and writer-owned.
Reader traversal does not require a global RPM lock.

Weaknesses

Once a reader loads a node into an immutable snapshot, that node remains resident until the snapshot is released. A long-lived reader can therefore materialise and retain a substantial part of its RPM.
Eviction from the mutable RPM is limited to nodes and cache slots exclusively owned by that mutable tree. It cannot clear child pointers in an immutable node shared with a published snapshot.
Concurrent readers can both allocate and deserialise the same child before one wins the compare-and-exchange, although they do not perform duplicate segment I/O.

Optional later use of RCU for eviction

Status: possible later enhancement.

Read-copy-update (RCU) or epoch-based reclamation could allow loaded nodes to be evicted from read-only RPM snapshots. A reader would enter one read-side critical section for an RPM lookup and could then traverse raw child pointers without incrementing and decrementing a reference count at each radix level.

Eviction would atomically change a cached child pointer back to null and put the removed node on a retired list. The node would be deleted after all readers which could have obtained its pointer had left their read-side critical sections.

The additional overheads include:

entering and leaving an RCU read-side critical section for each RPM lookup;
accessing per-thread epoch state;
registering reader threads with the RCU domain;
maintaining retired-node lists and detecting completed grace periods; and
retaining retired nodes until all relevant readers have finished.

An RPM cache miss may block while a segment is loaded. The reader must not remain in an RCU critical section across that blocking I/O; it would leave the critical section before loading and then retry or publish the child safely.

The initial design does not require RCU. It favours the simpler atomic-publication approach in which nodes loaded into a read-only snapshot remain resident until the snapshot is released. RCU may be introduced later if measurements show that retained RPM nodes create significant memory pressure.

Validation

Deterministic randomised tests should compare the RPM with a simple reference map while retaining multiple older roots. After each generated update, every retained root must continue to produce the mapping that existed when that root was published.

70 `cxSerialise`

cxSerialise will be a new header-only library factored out of cxUtils. It will provide the existing serialisation and deserialisation support, including archives such as InputArchive and the variable-length serialisation functions.

The LSS will use cxSerialise generally for its persistent file format so LSS files are platform-independent. An LSS file created on a little-endian machine can be read on a big-endian machine, and an LSS file created on a big-endian machine can be read on a little-endian machine.

Scope

The scope of cxSerialise is not yet fixed. It may be limited to serialisation to and from binary blobs held in memory. This narrow scope would cover bounded and unbounded memory archives, the binary wire format and variable-length serialisation, while leaving stream and paged-storage facilities in cxUtils.

Alternatively, cxSerialise may also include the stream interfaces currently provided by cxUtils, together with PagedBuffer and its archive/stream adaptors. This broader scope would allow serialisation over contiguous memory, streams and paged buffers to be provided by one library. The choice must preserve the requirement that cxSerialise has no dependency on cxUtils, Boost or any other external library; any included stream or paged-buffer code would therefore need to be factored into cxSerialise with those dependencies removed.

The broader scope might also encompass high-performance checksum support. Checksums are needed by the LSS to detect corruption in persisted data, and placing checksum support alongside the binary representation can allow data to be checksummed efficiently as it is serialised, copied or consumed. The same facility could also be useful for messaging over the wire, where messages may require integrity checks independently of the transport. The checksum API should work with contiguous binary blobs and, if streams and PagedBuffer are included, with those sources and destinations as well.

Dependencies

cxSerialise has no dependencies on any other CEDA library or on any third-party library, including Boost. In particular, it does not depend on cxUtils. The dependency is in the other direction: cxUtils may depend on and use cxSerialise. Code factored out of cxUtils must therefore be reorganised to remove its existing cxUtils and third-party dependencies.

Relevant `cxUtils` files

The core serialisation files to be factored out are:

Archive.h
ArchiveCollection.h
MemoryAccess.h (the relevant little-endian memory access support)
VariableLengthSerialise.h

The following files provide cxUtils-specific serialisation support or use the archive interfaces. They remain in cxUtils, which can use cxSerialise, but their includes and dependencies must be updated as part of the factoring:

ArchiveError.h
ArchiveHPTime.h
ArchiveMagic.h
BPlusTree.h
Crc32.h
File.h
Guid.h
HalfOpenInterval.h
Hex.h
IMsgOrientedInputStream.h
MD5.h
PagedBuffer.h
PagedBufferAsArchive.h
Range.h
RelocatableMap.h
SessionValueCache.h
xostream2.h

Selectable safe and fast archives

The library should provide two implementations of archive types such as InputArchive. An externally defined preprocessor flag selects between them when the library headers are included:


// Defined consistently by the consuming project's build configuration.
#define CX_SERIALISE_SAFE_ARCHIVES 1
#include <Ceda/cxSerialise/Archive.h>

The fast implementation preserves the existing representation and behaviour. It performs no bounds checks and is suitable when the input has already been validated or maximum throughput is required.
The safe implementation is constructed from a bounded sequence of octets, preferably a std::span<const octet_t>, and checks every read, skip and variable-length decode before advancing. Truncated or malformed input produces a well-defined error rather than an out-of-bounds memory access.

Safe `InputArchive` representation

The safe InputArchive can contain a std::span<const octet_t> representing all unread input. As bytes are consumed, the archive advances the start of the span by replacing it with a trailing subspan. The span's data pointer is therefore the current read position and its size is the number of octets remaining:


class SafeInputArchive
{
public:
    explicit SafeInputArchive(std::span<const octet_t> input) : remaining_(input) {}

    std::span<const octet_t> Read(std::size_t n)
    {
        if (n > remaining_.size())
            throw EndOfStreamException();

        auto result = remaining_.first(n);
        remaining_ = remaining_.subspan(n);
        return result;
    }

private:
    std::span<const octet_t> remaining_;
};

The archive should contain the span rather than publicly inherit from it or be an alias for it. This preserves the archive abstraction, prevents client code from bypassing its checked operations, and allows the representation to change without changing the public serialisation interface.

Using std::span does not by itself make deserialisation safe because unchecked operator[] can still access past the end. Each operation must check its complete range before calling first() or subspan(). This representation avoids maintaining a separate current pointer and end pointer and avoids unchecked pointer addition when advancing.

Variable-length integers need both input-bound checks and validation of their maximum encoded length and terminal byte. A compound decode should work on a local copy of the remaining span and only commit the advanced span to the archive after the entire value is valid. An error then leaves the archive position unchanged rather than partially consuming malformed input.

Source compatibility

Both implementations expose the same public type names and normal serialisation interface. Client code continues to use InputArchive, OutputArchive and the existing Serialise()/Deserialise() functions. Flipping the project-wide flag should therefore require virtually no source changes. Code which already knows the buffer size can use a span constructor in either mode:


InputArchive ar(std::span(buffer));
Deserialise(ar, value);

For migration, the pointer-only constructor can remain available in fast mode. Safe mode cannot make that constructor safe because no bound is supplied; it should either be unavailable in safe mode or require a pointer and size. Making unsafe pointer-only construction a compile-time error in safe mode is preferable, because it identifies the relatively small number of call sites that must be changed to provide the buffer extent. After those boundary call sites use spans, application serialisation code can switch modes without changes.

Implementation selection

The two implementations can have internal names such as UnsafeInputArchive and SafeInputArchive, with InputArchive defined as an alias selected by the flag. Shared serialisation algorithms should be templates over the archive interface so they are not duplicated. The flag should have a documented default, while build systems set it explicitly for each consuming project.

The selection must be consistent across every translation unit and linked library in a program. Changing the flag can change archive layout, inline function definitions and instantiated template types; mixing modes can therefore cause one-definition-rule and ABI errors. The selected mode should be part of the exported build configuration, and a link-time configuration symbol or equivalent guard should be used where practical to make mismatches fail clearly. Data written in either mode must use exactly the same wire format, so the choice affects validation and performance but not persistent or exchanged data.

Platform-independent byte order

cxSerialise must be platform-independent with respect to little-endian and big-endian processor architectures. The existing cxUtils serialisation format already specifies little-endian byte order on the wire: Archive.h uses the SetUnalignedLE() and GetUnalignedLE() operations provided by MemoryAccess.h. cxSerialise must preserve this established little-endian wire format rather than introduce a new byte order.

The wire order does not depend on the native byte order of the machine producing it. A little-endian machine must be able to serialise data that is then deserialised correctly by a big-endian machine, and a big-endian machine must be able to serialise data that is then deserialised correctly by a little-endian machine.

Serialisation and deserialisation of every multi-octet primitive must perform the necessary native to wire and wire to native byte-order conversion. This applies to integral and floating-point values, fixed-size fields and any length or control fields used by the archive format. The safe and fast archive implementations must use exactly the same platform-independent wire representation. Cross-endian test vectors should verify byte-for-byte output and round-trip compatibility in both directions.

Floating-point serialisation assumes the IEEE 754 interchange formats: binary32 for 32-bit floating-point values and binary64 for 64-bit floating-point values. The IEEE 754 bit pattern is encoded using the byte order defined by the wire format; it must not be written using the machine's native byte order. Implementations should verify the required floating-point representation at compile time. A platform that does not provide compatible IEEE 754 binary32 and binary64 types is not directly supported unless it explicitly converts its native representation to and from these wire formats.

Scope and verification

The same pattern can be applied where bounds matter to output archives and other cursor-like archive types. Safe output archives must check remaining capacity before writing; sizing archives do not access a buffer but must still detect size arithmetic overflow. Tests should run the same serialisation suite in both configurations, verify byte-for-byte identical output, and exercise the safe mode with truncated input, malformed variable-length integers, zero-length buffers and sizes near arithmetic limits. Benchmarks should quantify the cost of safe mode rather than allowing the safe checks to affect the fast implementation.

Documentation and performance benchmarks

cxSerialise will have its own document describing the library, its wire format, public API, safe and unsafe configurations, error handling, portability guarantees and recommended usage. That document will include performance benchmarks which give users a good idea of the performance difference between the safe and unsafe versions.

The benchmarks should exercise representative primitive values, arrays, collections, variable-length integers and larger structured values. They should report serialisation and deserialisation throughput, latency where useful, and the relative overhead of bounds checking. Both versions must be measured using the same data, compiler, optimisation settings and hardware so the comparison is meaningful. The document should record this test environment and enough methodology for the results to be reproduced.

71 Random Access Store (RAS)

Design criteria

An asynchronous version of IRAS should satisfy the following criteria:

Be compatible with optimal performance on both Windows and Linux.
Be elegant and simple.
Be fully asynchronous when reading and writing serial elements and the root block.
Be safe.
Promote concurrency if and only if concurrency makes sense.

Native foundation

Windows I/O completion ports

Windows supports asynchronous file operations through overlapped I/O. A file is opened with FILE_FLAG_OVERLAPPED, and each operation is submitted with an OVERLAPPED structure that identifies the operation and its file offset. The operation may complete immediately or remain pending while the calling thread continues to do other work.

An I/O completion port (IOCP) provides a completion queue to which file handles can be associated. When an overlapped operation finishes, Windows places a completion packet on the port. One or more threads can retrieve packets from the port, with the configured concurrency limit controlling how many associated threads may run at once. A single completion port can service operations for many file handles, allowing a bounded set of threads to process a large number of outstanding operations.

A completion packet reports the completed operation, its status and the number of bytes transferred. Completion order is not necessarily submission order. The application must retain the OVERLAPPED structure and all buffers referenced by the operation until completion has been reported.

Linux `io_uring`

Linux io_uring uses a pair of shared-memory ring buffers. The application places requests in the submission queue and the kernel places their results in the completion queue. This arrangement reduces the number of system calls and permits requests and completions to be processed in batches.

Each submission queue entry describes an operation and contains a user-defined value that is returned with its completion queue entry. Operations include file reads, writes, synchronization and many other forms of I/O. Requests are normally independent and may execute or complete out of order. Linked requests can be used when a sequence of operations has an explicit dependency.

io_uring can register files and buffers in advance, reducing per-operation setup and memory-management overhead. Buffers used by outstanding operations must remain valid until the corresponding completion entries have been processed.

Creation and shared I/O resources

A native asynchronous RAS needs an environment that submits I/O and dispatches its completions. On Windows this includes an I/O completion port and threads that retrieve completion packets. On Linux it includes one or more io_uring instances and code that submits SQEs and drains CQEs. Other implementations may use an executor, an I/O context, a platform event loop or a shared worker facility for operations that have no native asynchronous equivalent.

These resources should not normally be created separately for every RAS or LSS. Many Windows file handles can share one I/O completion port, and a Linux I/O service can process operations for many file descriptors. A process-level service can therefore use a bounded collection of threads and I/O contexts for many open stores:


struct IAsyncIoService;

std::unique_ptr<IRAS> CreateFileRAS(
    IAsyncIoService& ioService,
    ConstStringZ path,
    const RASSettings& settings);

This signature is illustrative. The important design point is that the platform RAS implementation, or a factory that creates it, receives the shared completion environment. Platform concepts such as IOCP handles, io_uring rings and completion threads remain behind that abstraction and are not exposed to the LSS.

An already constructed RAS is passed into the LSS when the store is created or opened:


std::unique_ptr<ILss> CreateOrOpenLss(
    std::unique_ptr<IRAS> ras,
    const LssSettings& settings);

This makes the storage implementation an explicit dependency of the LSS. It also permits the same LSS implementation to use a native file RAS, an in-memory RAS, a test or fault-injection RAS, a WebAssembly storage implementation or another future backend without giving the LSS knowledge of the backend's scheduling mechanism.

A convenience API that accepts a filename and constructs the RAS internally can instead receive an environment or RAS factory that is already bound to the shared I/O service. It should not require platform-specific completion resources to be threaded through the logical LSS implementation.

Lifetime

The shared I/O service must outlive every RAS that uses it, and a RAS and its callback receiver must outlive all native operations submitted through that RAS. Closing an LSS stops new logical work, drains or cancels its outstanding RAS operations, and ensures that no further callback can reach the LSS before destroying the RAS. The shared service remains available to other stores and is stopped only after all dependent RAS instances have closed.

The LSS event-driven state machine does not necessarily require its own dedicated executor or thread. RAS completions and application requests can be admitted to a thread-safe LSS event queue whose events are processed sequentially by a run-to-completion drain owner. A separate executor may be provided when application notifications require thread affinity, but it is not an inherent requirement of the RAS or LSS design.

71.1 RAS Writes

The RAS provides a function that issues an asynchronous write. A write specifies a file offset and passes its data as a pointer and size. The function returns immediately with a 64 bit Write Sequence Number (WSN):


using WSN = uint64;
using WSNCount = uint64;

WSN Write(RASOffset offset, const void* buffer, ssize_t size);

WSNs are consecutive and zero-based. The first write has WSN 0, the second has WSN 1, and so on. They define the order in which writes were issued, independently of the order in which the native write operations complete.

Write() is thread-safe. When calls are made concurrently, the RAS is responsible for safely allocating a unique consecutive WSN and registering each write for progress tracking. Callers do not need to serialise their calls to Write().

The allocation of WSNs does not define an order in which writes must be submitted to, executed by or completed by the storage device, and it does not itself provide a write-ordering or persistence guarantee. Native writes may execute and complete in any order. WSNs provide an ordering only for reporting completed and durable prefixes.

Write progress is reported through a callback with two monotonically increasing watermarks:


void OnWriteProgress(WSNCount numCompleted, WSNCount numDurable);

The RAS must invoke OnWriteProgress() sequentially. Calls to the callback must never overlap, even when native write operations complete concurrently. This requirement does not prescribe which thread invokes the callback, but it ensures that the receiver observes progress notifications in a single, well-defined sequence.

The watermarks use half-open ranges. numCompleted means that all writes with WSNs in [0,numCompleted) have completed. Similarly, numDurable means that all writes with WSNs in [0,numDurable) are durable. Both values are initially zero, so there is no need for a distinguished value representing the absence of a write.

A write with WSN wsn has completed when wsn < numCompleted, and is durable when wsn < numDurable. The following invariants always hold:


numDurable <= numCompleted;
numCompleted <= numWritesIssued;

Completion has two related meanings. Firstly, the RAS no longer accesses the input buffer. The caller must keep the buffer valid and unmodified until it receives an OnWriteProgress() call in which numCompleted is greater than the write's WSN. Secondly, that notification establishes an ordering boundary: an overlapping write issued after the notification must not be reordered ahead of the completed write.

Native writes may complete out of order, but the reported values describe contiguous prefixes. For example, if writes 0, 1, 3 and 4 have completed while write 2 is still pending, then numCompleted is 2. When write 2 completes, the watermark can advance directly to 5. Therefore numCompleted is the length of the completed prefix, not necessarily the total number of individual native writes that have completed.

Write-completion granularity

Each WSN identifies one complete logical RAS write request. From the caller's perspective, that request is either pending or completed. The RAS API does not expose partial progress within a write, such as a number of bytes or a proportion of the request that has been transferred.

A native platform write may transfer fewer bytes than requested. This is an implementation detail of the RAS. The implementation can continue writing the remaining suffix under the same WSN, or treat the short transfer as a fatal write error. It must not allocate additional WSNs for native sub-operations, and it must not report the logical write as completed until its entire requested file range has been written successfully.

The caller must retain the complete input buffer until the write's WSN is less than numCompleted. Completion of part of a native transfer does not give the caller permission to modify, reuse or release the corresponding part of the buffer.

The only notification of write-completion progress is numCompleted in OnWriteProgress(). It reports the number of WSNs in the largest contiguous prefix of completely finished logical writes. The API provides no finer-grained completion notification, either within an individual write or for completed writes beyond a gap in that prefix.

Overlapping writes

A write with offset offset and size size covers the half-open file range [offset,offset+size). Two writes overlap if their ranges contain at least one common byte. Given two write ranges [offsetA,offsetA+sizeA) and [offsetB,offsetB+sizeB), they overlap when:


offsetA < offsetB + sizeB &&
offsetB < offsetA + sizeA;

Adjacent ranges do not overlap. For example, [100,200) and [200,300) have no bytes in common.

Pending overlapping writes are forbidden

It is forbidden for the caller to have two overlapping writes outstanding at the same time. After issuing a write with WSN wsn, the caller must not issue a second overlapping write until it has received an OnWriteProgress() notification for which wsn < numCompleted. The notification, rather than the passage of time or an assumption about native I/O progress, gives the caller permission to reuse the range.


WSN first = ras.Write(offset, firstBuffer, size);

// Wait until OnWriteProgress() reports first < numCompleted.

WSN second = ras.Write(offset, secondBuffer, size);

This is a contract on the caller even though Write() itself is thread-safe. If different threads issue the two writes, the caller is responsible for communicating the progress notification between those threads with the required thread synchronisation.

RAS cannot reorder pending and completed overlapping writes

The RAS must not report a write as completed and subsequently reorder a later overlapping write ahead of it. Once OnWriteProgress() reports wsn < numCompleted, any overlapping write issued afterwards must be applied after the completed write. Consequently, numCompleted does not merely report that input buffers can be released; it also establishes the ordering boundary that makes later reuse of the same file range safe.

For example, an implementation must not copy a buffer into private memory, advance numCompleted merely to release the caller's buffer, and then submit a later overlapping write ahead of the privately buffered write. An implementation may release a buffer early only if it also preserves the required ordering of any overlapping write issued after the completion notification.

This guarantee does not make WSN order a general device-ordering rule. Disjoint writes may still be submitted, executed and completed in any order. Concurrent overlapping writes are forbidden rather than ordered by their WSNs. The ordering guarantee arises only when a caller waits for the earlier write to be reported as completed before issuing the later overlapping write.

Windows I/O completion ports satisfy write-order requirements

The contract on the RAS can be assumed to be met straightforwardly by an implementation based on Windows overlapped I/O and I/O completion ports. The RAS submits each write using WriteFile() with an OVERLAPPED structure. Windows places a completion packet on the I/O completion port when the overlapped write operation has completed. Until then, the operation is pending and its buffer and OVERLAPPED structure must remain valid.

The RAS records each native completion and advances numCompleted only across the contiguous prefix of completed WSNs. Native operations and their completion packets may be processed out of order; this affects only when a gap in the completed prefix is closed. It does not require writes to execute in WSN order.

The caller does not issue a second overlapping write until it receives an OnWriteProgress() notification showing that the first write's WSN is less than numCompleted. At that point the Windows operation for the first write has completed, so the second overlapping WriteFile() call is submitted only after the first operation is no longer outstanding. The required ordering therefore follows naturally from native completion and does not require an additional ordering operation or the serialisation of disjoint writes.

Several I/O completion port worker threads may process native completions concurrently. The RAS must still serialise its calls to OnWriteProgress() and ensure that the reported watermarks only increase. This serialisation applies to progress notification and does not prevent native writes from remaining concurrent.

Linux `io_uring` satisfies write-order requirements

The same contract can be met straightforwardly by a Linux implementation based on io_uring. The RAS describes each write with an IORING_OP_WRITE submission queue entry (SQE), including its file offset, buffer and size. When the kernel has finished processing the request, it places a corresponding completion queue entry (CQE) in the completion queue. The CQE reports the result that the equivalent write system call would have returned.

The buffer of an IORING_OP_WRITE request must remain valid while the request is in flight. The RAS therefore retains the buffer and operation record until it receives the CQE. It then records the native completion and advances numCompleted only across the contiguous prefix of completed WSNs. Requests may execute and complete out of order, so a CQE for a later WSN does not by itself allow the reported prefix to advance past an earlier pending request.

The caller waits for an OnWriteProgress() notification showing that the first write's WSN is less than numCompleted before issuing a second overlapping write. The SQE for the second write is consequently submitted only after the CQE for the first write has established that the first operation is complete. The two overlapping writes are not simultaneously in flight, so the required ordering follows from normal io_uring completion without linking the requests, serialising disjoint writes or introducing another ordering operation.

Submission and completion queues may be serviced concurrently, and an implementation may process CQEs on more than one thread. As with the Windows implementation, the RAS must combine those native results into monotonic prefix watermarks and invoke OnWriteProgress() sequentially. This affects progress reporting, not the concurrency of independent native writes.

RAS diagnostic check for pending overlapping writes

A RAS implementation can retain the file ranges of outstanding writes and check each new write for overlap in diagnostic builds. A range can be removed from this diagnostic set when its write is reported as part of the completed prefix. The diagnostic should identify both conflicting ranges and their WSNs.

The LSS normally leaves a long interval between writes to the same file range because it writes a log and only occasionally writes a new checkpoint. An overlap detected by the RAS is therefore likely to indicate incorrect checkpoint, segment-reuse or write-management behaviour. Checking this caller contract provides a useful way to help ensure that the LSS is correct.

Durability

Completion and durability are different properties. Completion means that a write has crossed the buffer-release and write-ordering boundary described above. It does not normally mean that the data would survive an operating-system crash or loss of power, because file data may remain in operating-system or device caches after the native write operation has completed.

numDurable reports the contiguous prefix of writes for which the RAS persistence guarantee has been established. A write with WSN wsn is durable when wsn < numDurable. Since durability implies completion, numDurable <= numCompleted always holds. The RAS must not advance numDurable merely because ordinary write completions have been received.

The RAS can make many writes durable with one persistence operation. When that operation succeeds, numDurable can advance directly to the end of the covered write prefix and the RAS reports the new value through OnWriteProgress(). This allows durability notifications to be batched rather than requiring a persistence operation for every write.

Asynchronous `Flush()`

Durability is requested explicitly with an asynchronous Flush() operation:


// Request durability of every write accepted before this call.
// Returns the exclusive end of the requested WSN prefix.
WSNCount Flush();

Flush() is thread-safe and is linearised with concurrent calls to Write(). It captures every write accepted before its linearisation point and returns the exclusive end of that prefix. If it returns target, the request has been satisfied when a later OnWriteProgress() notification reports target <= numDurable. A target of zero is valid and means that no writes preceded the flush.

The call returns without waiting for outstanding writes or for the storage device. Writes issued after the captured prefix may continue while the flush is pending. They are not required to become durable as part of that request, although a platform persistence operation may incidentally include some of them.

The name Flush() does not mean that the RAS has retained writes in an application buffer and should now submit them, nor does it mean that the RAS should try harder to send subsequent writes to the device. Writes are submitted promptly in the ordinary course of operation. Flush() specifically requests notification when the captured prefix has become durable.

Several pending flush requests may be coalesced. For example, requests with targets 20, 25 and 31 can be satisfied by one persistence operation covering the prefix [0,31). When it succeeds, the RAS can report numDurable == 31, thereby satisfying all three requests. The RAS must not report a prefix beyond the one it can prove durable.

Use by the LSS

The LSS writes complete Log Flush Units (LFUs) to the RAS. These writes are intended to be submitted to storage as quickly as possible rather than retained by the RAS while it waits for additional LFUs to accumulate. Ordinary LSS operation needs completion notifications so that write buffers and file ranges can be reused, but it does not generally need to know when each LFU becomes durable.

A checkpoint is one of the few operations that requires an explicit durability boundary. Before the next root-block division is rewritten, the last LFU on which that division depends must be durable. The checkpoint therefore requests a flush after writing that LFU and waits for the returned prefix to be reported as durable:


WSN lastLfuWsn = WriteLastLfu();
WSNCount flushTarget = ras.Flush();

// Continue asynchronously when OnWriteProgress() reports:
//     flushTarget <= numDurable

WriteRootBlockDivision();

The returned target will be greater than lastLfuWsn, because it is an exclusive prefix endpoint and includes at least that LFU. Waiting for flushTarget <= numDurable makes the dependency explicit without blocking a thread. Later log writes may be issued while the checkpoint flush is in progress.

Windows

On Windows, ordinary overlapped WriteFile() completion does not normally establish durability. The conventional operation for forcing buffered file data to storage is FlushFileBuffers(). In response to Flush(), the RAS first ensures that the writes in the target prefix have completed, invokes FlushFileBuffers() for the file, and advances numDurable only after the platform flush has returned successfully.

FlushFileBuffers() is a synchronous call rather than an overlapped I/O operation reported through an I/O completion port. A fully asynchronous RAS must therefore avoid calling it on a thread that must remain available to process other completions. It can run the flush on a shared blocking-I/O worker and deliver the result back to its normal completion mechanism. This does not consume a thread per write, but a worker thread is occupied while the persistence operation is in progress.

Windows also supports FILE_FLAG_WRITE_THROUGH, optionally combined with FILE_FLAG_NO_BUFFERING. With both flags, Windows requests that each write pass through the system and hardware caches to persistent media, subject to support from the storage hardware. Combining FILE_FLAG_NO_BUFFERING with overlapped I/O can provide high asynchronous throughput, but it imposes buffer, size and file-offset alignment requirements and gives up the benefits of the system file cache.

Write-through operation can allow numDurable to advance with the completed write prefix, whereas explicit flushing permits several writes to share one persistence barrier. The best choice depends on the storage device and workload. Calling FlushFileBuffers() after every small write is potentially expensive; batching writes before a flush normally amortises both the system-call and device-cache flush costs.

Linux

Linux io_uring provides IORING_OP_FSYNC, so a file synchronization request can itself be submitted asynchronously in response to Flush() and reported by a CQE. It can provide normal fsync() behaviour or, with IORING_FSYNC_DATASYNC, the data-oriented behaviour of fdatasync().

An fsync SQE is not automatically ordered after previously submitted write SQEs. If submitted as an independent request, it may execute before one of those writes has reached storage. The RAS must therefore establish the dependency explicitly. It can wait until all writes in the target prefix have completed before submitting IORING_OP_FSYNC, or use the appropriate io_uring linking or drain facility. It advances numDurable only after the synchronization CQE reports success.

The asynchronous interface means that no application thread needs to block while the device performs the synchronization. It does not remove the storage cost: dirty pages and device caches must still be flushed, and subsequent work that depends on durability must wait for that operation. As on Windows, issuing a synchronization request after every small write can significantly reduce throughput. Batching a prefix of writes behind one IORING_OP_FSYNC operation allows one device synchronization to advance numDurable across many WSNs.

Performance implications

Asynchrony prevents persistence latency from blocking an execution thread, but it cannot eliminate that latency or the underlying device work. Durability may require cached data to be written, a device cache to be flushed and storage ordering constraints to be honoured. These operations can be much more expensive than accepting data into a cache.

The principal performance choice is therefore the frequency at which Flush() is called. Flushing every write provides the smallest durability lag but may serialise the workload around device barriers. Flushing groups of writes permits higher throughput and allows numDurable to advance in larger increments, at the cost of a larger interval during which completed writes are not yet durable. The RAS interface should expose accurate progress without forcing a particular persistence frequency or batching policy.

72 Segment Cache

Purpose

The Version 2 SegmentCache will be similar to the current segment cache in its management of resident segment buffers, access counts and eviction. Its boundary will be narrower: it manages segments in memory, but it does not perform RAS I/O. Loading and writing are asynchronous workflows coordinated above the cache.

This separation is important in the event-driven LSS. A cache lookup is a synchronous memory operation. A RAS read is an asynchronous operation which may complete later and generate an event. Combining them in one call would conceal a state transition and give a function which sometimes returns immediately and sometimes starts an unrelated operation.

No upward reference to `LSS`

Neither SegmentCache nor Segment retains an LSS&. The LSS is the composition root and coordinator; its lower-level cache and segment objects must not use it as a service locator to reach the RAS, settings, error state or other LSS modules.

The cache can instead be constructed from the values it actually needs, such as the segment size and maximum number of resident segments. A segment needs its SegId, its buffer and the small amount of state required for cache residency. This makes both classes independently understandable and testable.


class SegmentCache
{
public:
    SegmentCache(std::size_t segmentSize, std::size_t maximumResidentSegments);
    // ...
};

class Segment
{
public:
    SegId GetSegId() const;
    std::span<octet_t> GetBuffer();
    std::span<const octet_t> GetBuffer() const;
    // ...
};

The signatures are illustrative. In particular, the buffer may use an internal view type rather than std::span. The architectural requirement is that the segment owns or refers to memory; it does not know how that memory is transferred to or from persistent storage.

`Segment` does not perform I/O

The following methods on the current Segment will not exist:


void ReadFromRAS();
void WriteToRAS();
void WriteSectionToRAS(int i1, int i2);

A segment is an in-memory object, not an active storage object. Removing these methods also removes its need to find the RAS through LSS&. Code coordinating a write obtains a view of the relevant bytes from the segment and submits that view to the RAS. Code coordinating a read arranges for the RAS to fill the final cache buffer directly.

Cache-only lookup

SegmentCache::GetSegment(SegId segid) only looks for the segment in the cache. It never reads from the RAS, starts a RAS request, waits for I/O or arranges for another module to load the segment. If the requested segment is not resident and available, the call reports a cache miss by returning an empty SegmentAccessor.


SegmentAccessor SegmentCache::GetSegment(SegId segid);

It may be preferable to name this operation FindSegment() or TryGetResidentSegment(), because those names make the possibility of a cache miss explicit. Regardless of its final name, lookup has cache-only semantics.

A successful lookup returns an accessor which pins the segment so it cannot be evicted while the caller uses it.

`SegmentAccessor`

SegmentAccessor represents a pin giving access to a resident segment. It is a move-only RAII type: constructing it acquires a pin, destroying it releases that pin, and moving it transfers responsibility for the pin to the destination. Copy construction and copy assignment are disabled so an accidental copy cannot silently create or duplicate a pin.


class SegmentAccessor
{
public:
    SegmentAccessor() = default;
    ~SegmentAccessor() { Reset(); }

    SegmentAccessor(const SegmentAccessor&) = delete;
    SegmentAccessor& operator=(const SegmentAccessor&) = delete;

    SegmentAccessor(SegmentAccessor&& other) noexcept;
    SegmentAccessor& operator=(SegmentAccessor&& other) noexcept;

    explicit operator bool() const { return segment_ != nullptr; }
    Segment& operator*() const;
    Segment* operator->() const;
    void Reset();

private:
    friend class SegmentCache;
    SegmentAccessor(SegmentCache& cache, Segment& segment);

    SegmentCache* cache_ = nullptr;
    Segment* segment_ = nullptr;
};

An empty accessor represents a cache miss. A non-empty accessor can be passed between functions by move without changing the underlying segment address or releasing its pin:


auto segment = cache.GetSegment(segid);
if (!segment)
    StartSegmentLoad(segid);
else
    ContinueRead(std::move(segment));

The moved-from accessor becomes empty. Exactly one accessor remains responsible for releasing that particular pin. If a second independent pin is needed, it must be acquired explicitly from the cache or through an explicitly named operation such as Clone(); ordinary copy syntax must not conceal the additional cache access count.

Use by asynchronous operations

An operation may take ownership of a SegmentAccessor and retain it while native I/O refers to the segment buffer. For example, a segment-write operation keeps the accessor until the RAS reports completion:


struct SegmentWriteOperation
{
    SegmentAccessor segment;
    RASWriteRequest request;
};

This makes the buffer-lifetime rule explicit and prevents eviction or recycling while the operation is outstanding. The accessor controls only the lifetime of resident memory; it does not initiate I/O and does not provide a route back to the complete LSS.

Lifetime and access rules

The SegmentCache must outlive every accessor which refers to it. If that ordering cannot be guaranteed structurally, accessors and the cache must share a separate control block whose lifetime extends until the final accessor is destroyed. Move operations should be noexcept so accessors can be stored and transferred reliably.

If segments can have concurrent readers but mutation requires exclusive access, the implementation should distinguish read-only and mutable access rather than allowing every accessor to return an unrestricted mutable reference. This can use separate accessor types or explicit operations on the cache. Pinning prevents eviction; it does not by itself provide synchronisation for concurrent access to the bytes.

Loading a segment

A cache miss is handled by an LSS-level read state machine or a dedicated segment-loading operation, not by GetSegment(). The normal sequence is:

Look for an available segment in the cache.
On a miss, reserve a cache entry and its final segment buffer.
Submit a RAS read which targets that buffer directly.
Return control while the RAS read is outstanding.
On completion, validate the segment and publish it as available in the cache.
Continue the logical operations waiting for that segment.

Reading directly into the reserved cache buffer avoids a staging copy. The segment must not be returned by ordinary cache lookup while its buffer is only partially filled or has not yet been validated.

Concurrent requests for the same segment

The implementation must prevent several cache misses for the same SegId from causing duplicate RAS reads. Reserving an entry for a load therefore records that the segment is being loaded. A later request can distinguish an available segment, an existing outstanding load and the need to start a new load.


enum class SegmentLookupResult
{
    Available,
    Loading,
    NeedToLoad,
    CacheFull
};

This state need not be exposed through the simple cache-only GetSegment() operation. It can be part of a separate atomic reservation API used by the LSS read coordinator. Logical reads which encounter Loading attach themselves to the existing load operation and are continued when its completion event is processed.

Writing segments

Writing is also coordinated outside Segment and SegmentCache. The writer determines the RAS offset and byte range, obtains a view of the segment buffer and submits the asynchronous write:


auto bytes = segment.GetBuffer();
ras.Write(offset, bytes, completion);

A section write uses a subview of the buffer rather than a method which lets the segment call the RAS. The segment buffer must remain alive and unchanged for as long as the native operation may reference it. The write operation therefore owns a cache pin or another lifetime handle until its completion is reported. Eviction cannot recycle that segment buffer while the write is outstanding.

I/O and integrity errors

There is no recovery from a segment read failure within the cache or loading state machine. Any RAS read or write error puts the open LSS into its zombie state, as does an integrity failure discovered while validating a segment. The LSS stops logical processing and does not retry the read, substitute another segment or resume waiting operations.

Native operations which were already submitted must still retain their operation records and buffers until the platform reports completion or cancellation. This lifetime requirement does not imply logical recovery: completions are drained only to finish the safe shutdown of outstanding native work after the LSS has become a zombie.

Responsibilities

The resulting division of responsibilities is:

Segment represents a segment identifier, its memory and the small amount of state needed for cache residency.
SegmentCache manages lookup, allocation, pinning, access counts, capacity and eviction of segment buffers.
The LSS read and write state machines decide when I/O is required, coalesce operations and react to RAS completion events.
The RAS transfers bytes asynchronously between persistent storage and buffers supplied by the LSS.

This boundary keeps synchronous memory management separate from asynchronous storage operations and removes hidden coupling from the lowest-level segment classes back to the complete LSS.

73 `Partition`

Purpose

An LSS can be divided into a number of Partition objects. An Partition is a coarse-grained physical and logging division of the store. It is heavier than a Space and normally contains many Spaces.

The purpose of partitioning is to increase parallelism when writing segments and to improve data locality. Independent partitions can build and submit segment writes concurrently. Data belonging to different uses of the LSS can be written to different segments, allowing related data to remain physically clustered.

Owned and shared components

Each Partition has its own:

log and current log position;
segment writer and write-progress state;
collection of Spaces and their four-level RPM roots; and
Segment Utilisation Table (SUT) for the segments assigned to the partition.

All partitions in an LSS share:

the Random Access Store (RAS); and
the Segment Cache.


LSS
    RAS
    SegmentCache

    Partition
        Log
        SegmentWriter
        SUT
        Space ...

    Partition
        Log
        SegmentWriter
        SUT
        Space ...

The RAS remains responsible for asynchronous access to the underlying storage, while the shared segment cache provides one bounded pool of resident segments across the LSS. Partitioning does not create a separate physical file or cache for every partition.

Relationship to Spaces

A Space is a lightweight logical namespace containing 32-bit Seids and a four-level RPM. It does not have its own log, segment writer, SUT, RAS or segment cache. Every Space belongs to one Partition, and packets addressed through that Space are written to the log of that partition.

An Partition is deliberately more coarse-grained. Creating a Space should be cheap enough to support many thousands of independent namespaces. Creating a partition establishes substantial logging, utilisation and write-coordination state, so the number of partitions should be much smaller.


class Partition
{
    // Heavyweight physical/logging domain
    std::mutex mutex;
    SegmentWriter writer;
    SUT sut;
    SpaceDirectory spaces;
};

Mutative transactions

Each Partition has a mutex which ensures that at most one thread at a time writes the log records for a mutative transaction into the partition's segments being prepared in memory. Transactions never span Partition objects.

The mutex orders transaction log records in the in-memory log. A transaction writes its log records followed by its snapshot record before releasing the mutex. A snapshot record does not need to identify its transaction because no other transaction can write records to that partition while the mutex is held.

As in Version 1, a lazy writer writes prepared segments to the RAS. It is allowed to be many segments behind the segments being prepared in memory. Completion of a mutative transaction does not mean its log records or snapshot record have been written to disk or made durable.

Transactions in different partitions can concurrently prepare their respective in-memory segments. Each partition has its own log, segment writer and lazy writer.

Mutex protection boundary

In Version 1 the SegmentWriter owns the transaction mutex, while the RPM7 root node and the SUT each have their own mutex. In Version 2 the transaction mutex belongs to Partition, where it protects the SegmentWriter, SUT and the mutable RPMs belonging to its Spaces. This places the mutex on the object which owns the complete partition-level mutation boundary.

A Version 2 mutative transaction therefore locks fewer mutexes than a Version 1 transaction, reducing mutex-acquisition overhead.

The partition mutex does not protect traversal through published immutable RPM roots. Readers retain an immutable root and traverse it without holding the partition mutex. Lazy loading into such roots uses the atomic child pointers described in the RPM chapter.

The partition mutex is not held while the lazy writer performs RAS I/O. Prepared segments can be written after the mutative transaction has released the mutex.

Parallel segment writing

Within each partition, the partition mutex allows one thread at a time to construct transaction log records in its in-memory segments. Different partitions can prepare log segments in memory at the same time. Their lazy writers can write prepared segments to the shared RAS concurrently.

Parallelism does not require one thread per partition. The Version 2 event-driven implementation can advance each partition's writer when work or RAS completions are available. A bounded shared I/O service can service writes for many partitions without giving every partition dedicated threads.

The RAS or an LSS-level segment allocator must assign non-overlapping physical segment locations to the partitions. A partition owns the contents and utilisation state of its assigned segments, while the shared RAS performs the physical writes.

Data locality

Without partitions, unrelated applications or workloads can append packets to the same sequence of log segments even when the workloads are accessed and cleaned independently. Assigning related Seid spaces to one partition keeps their packets in that partition's segments and prevents unrelated partitions from writing packets into those segments.

Several independent divisions of the store may each perform many transactions per second. With one log, successive transactions from those divisions append their packets to the same sequence of segments.

With separate partition logs, transactions in different partitions can execute concurrently and append packets to different segments. Successive transactions for one division therefore write to that partition's segments.

This can improve locality when reading, prefetching, caching and cleaning. A segment is more likely to contain packets used by the same related group of Spaces. Cleaning decisions based on the partition's SUT also operate on a physically coherent set of segments rather than a mixture of unrelated workloads.

Partitioning is not the only source of locality. Spaces localise RPM traversal and allocation metadata, while Partition localises physical log placement. The two mechanisms operate at different scales and complement one another.

Shared segment cache

The segment cache is shared so its memory limit can be applied across the complete LSS. Idle partitions do not retain private caches while an active partition is under memory pressure. Cache entries must identify segments unambiguously across partitions, either because SegId is unique within the whole LSS or because a cache key contains both a partition identifier and a partition-local segment identifier.

An Partition does not make the segment cache perform I/O. As described in the Segment Cache chapter, cache lookup, RAS submission and completion handling remain separate responsibilities.

SUT and cleaning

Each partition's SUT describes utilisation of the segments belonging to that partition. Packet replacement and deletion update the SUT associated with the packet's partition, and cleaning selects segments using that same utilisation information.

Cleaning state is logically partition-specific because logs and SUTs are partition-specific. This does not require a dedicated cleaner thread for every partition. An LSS-level cleaning service can schedule work across partitions while preserving the ownership of segment-utilisation state and log placement within each partition.

Granularity and cost

An Partition is not created merely to obtain a new 32-bit Seid namespace. That is the role of a Space. A partition is justified when a group of data benefits from an independent log-writing and physical-locality domain.

The implementation should expect many more Spaces than partitions. Partition-level buffers, queues, log tails, SUT state and checkpoint metadata can consume significant memory and leave unused space in partially filled segments. The number and assignment of partitions should therefore be chosen at a coarse workload boundary rather than automatically creating one partition for every Space.

Failure domain

Partitions share one open LSS and its RAS; they are not independent failure domains. An I/O or integrity error in any partition puts the LSS into its zombie state. Other partitions do not continue logical processing after such a failure.

CEDA Log Structured Store

Part I: Overview

1 Log Structured Store (LSS)

Summary of features

LSS is a persistent heap

Seids are virtual addresses

Three basic operations

Transactions

API

2 Serial element

3 Seid

Seid()

Seid(SeidLow low,SeidHigh high)

explicit Seid(uint64 v)

bool IsNull()

void SetNull()

SeidLow Low() const

void SetLow(SeidLow low)

SeidHigh High() const

void SetHigh(SeidHigh high)

uint64 Value() const

void SetValue(uint64 value)

Comparisons on Seids

Writing of a Seid to/from an Archive

Part II: Architecture

4 LSS - Under the hood

No need for Write Ahead Logging

Segments in the LSS File

Forward chaining of segments

The segment cleaner

5 Chunk size

6 Storage assumptions and failure model

Block size and alignment

Torn and incomplete writes

Root-block redundancy

Current implementation notes and user impact

7 Write ordering on storage devices

Four different properties

Why volatile write caches exist

ATA and SATA: FLUSH CACHE

SCSI and NVMe

Linux

Windows

Abrupt power loss

What the guarantee depends upon

8 Log structured store versus Write ahead logging

Write Ahead Logging

Write-ahead invariant

Performance consequences

Log structured store

Transaction path

Check-point publication

9 Check pointing

10 Cleaning

11 Lazy writer

12 Clustering

13 MSSN

Part III: Operations and administration

14 Reading serial elements

15 Writing serial elements

Example

16 Seid allocation

Serial element Ids

Allocation of affiliate Seids

Boot strapping a store

17 Durability

18 Backup and hot standby for the LSS

Check point identifiers

Hot standby configuration

Links

19 LssCompare.exe

20 LssApplyDeltas.exe

Part IV: API reference

21 CEDA LSS API

Public header

Simple example

Links

API

22 CreateOrOpenLSS

Example

`Seid()`

`Seid(SeidLow low,SeidHigh high)`

`explicit Seid(uint64 v)`

`bool IsNull()`

`void SetNull()`

`SeidLow Low() const`

`void SetLow(SeidLow low)`

`SeidHigh High() const`

`void SetHigh(SeidHigh high)`

`uint64 Value() const`

`void SetValue(uint64 value)`

`int flushTimeMilliSec`

`double cleanerUtilisationPercent`

`bool enableFileBuffering`

`bool enableWriteThrough`

`int maxNumSegmentsInCache`

`int numSegmentsPerCheckPoint`

`int segmentSize`

`bool forceIncrementMSSN`

`bool validateSUTDuringCheckPoint`

`void Close()`

`void SetMaxNumSegmentsInCache(int n)`

`MSSN GetMissingShutdownSeqNum() const`

`SeidHigh CreateSeidSpace()`

`Seid AllocateSeid(SeidHigh seidHigh)`

`bool ReserveSeid(Seid seid)`

`SeidLow PeekNextSeidLow(SeidHigh seidHigh)`

`bool AllocateAffiliateSeid(Seid& seid)`

`void GetSeidsInSeidSpace(xvector<SeidLow>& seidLows, SeidHigh seidHigh) const`

`bool SerialElementExists(Seid seid) const`

`ICloseableInputStream* ReadSerialElement(Seid seid) const`

`IContiguousSerialElement* ReadContiguousSerialElement(Seid seid) const`

`ILssTransaction* OpenTransaction()`

`void BlockUntilLowWaterMark() const`

`bool ReachedHighWaterMark() const`

`void GetStats(LssStats& stats) const`

`void DumpLSS(LssStats& stats, xostream& os, const LssDumpSettings& ds) const`

`bool RecurseSeidMap( xvector<Seid>& children, Seid seid = BEGIN_RECURSE_SEID_MAP, bool enableOverflowPackets = false) const`

`bool WriteInfoOnNodeForGivenSeid(Seid seid, xostream& os) const`

`void Close()`

`void FlushWhenClose()`

`ICloseableOutputStream* WriteSerialElement(Seid seid)`

`bool DeleteSerialElement(Seid seid)`

`void DeleteSeidSpace(SeidHigh seidHigh)`