21 CEDA LSS API

The LSS (Log Structured Store) is a persistent store for arbitrary sized binary objects, referred to as serial elements.

The LSS is supported on all flavors of 32 bit and 64 bit Windows from Windows 95 onwards. The store is written to the hard-disk (which could be FAT32 or NTFS) as a single file. This file grows as required to accommodate new data written to the store.

Public header

The API is defined in the header ILogStructuredStore.h, here it is reproduced without all the comments and other distractions and simplified slightly:

Simple example

The serial elements are written to the store within a transaction, and the LSS ensures atomicity of each transaction. i.e. all changes made to the store by a transaction are applied or else none are applied. For example, a transaction could fail to commit because of a power failure. The next time the store is opened, any uncommitted transactions are rolled back. This "recovery scan" is performed automatically whenever the store is opened. The time for a recovery scan is bounded, and on typical hardware will never take longer than a few seconds.

Serial elements are read or written as a byte stream, in a manner similar to the C functions fread() and fwrite(). The store can deal efficiently with very small and very large serial elements. Assuming compacting is good, the overhead is of the order of 20 bytes per object.

The LSS achieves excellent write performance, typically limited only by the maximum transfer rate of the hard-disk. Disk head seeks during writing of data are kept to a minimum by writing new data to the end of the log using large segments. By default segments are 512 kbyte.

When the LSS is opened, a background thread is automatically started that cleans segments with a poor utilisation (i.e. below a preset threshold). The data on a segment to be cleaned is written to the end of the log, allowing the segment to be returned to an internal free segment pool. Because of this, users of the LSS never need to concern themselves with "fragmentation" of the store.

However, a user of the LSS needs to be concerned with clustering related data together, in order to achieve maximum read performance. This is essentially achieved by writing related data close together in time (so the related serial elements tend to be written to the same segments). Note that rewriting individual serial elements over time has the effect of upsetting the clustering. Reclustering simply involves rewriting a collection of related serial elements to the end of the log. The background cleaner thread will automatically defragment the store.

It is important to note that the LSS is not concerned with concurrency control on access to the serial elements. It certainly doesn't provide strict two phase locking, or any other locking protocol to enforce serialisation of transactions. Instead, it assumes that a layer above the LSS is responsible for concurrency control.

Serial elements are identified by a 64 bit Seid (Serial element identifier). The LSS provides a mechanism for allocating new, unused Seids as required. The number of serial elements is actually limited by the maximum size of the store which is about 500 TB, rather than the size of the 64 bit Seid space.

Links

API

The following document the CEDA LSS API: