67 Introduction

This part serves as a design specification for a new version of the CEDA LSS. It will attempt to incorporate the proposals in Part VII. In particular, the aim is to support multi-version concurrency control (MVCC).

Version 2 is not required to be compatible with Version 1. Its APIs, in-memory structures and persistent file format may change where that produces a better design.

Although Version 2 is not being implemented in Rust, it is useful to confirm that the design has a straightforward mapping to a borrow checker. Borrow checking often forces ownership and lifetime relationships to be expressed clearly and therefore encourages good design.

The aim is for the Version 2 specification to be sufficiently precise and complete for AI code generators to implement it. If the design, interfaces, ownership rules and persistent formats are specified clearly enough, the resulting implementation will probably be both efficient and correct.

The specification must define the binary representation of the LSS file unambiguously.

Top-down organisation

This specification is organised top down. Early chapters establish the overall picture, principal abstractions and relationships; later chapters progressively drill down into their components, algorithms and representations. A reader should therefore encounter the context in which a detail exists before encountering the detail itself.

Trusted writer model

It is valid for the Version 2 implementation to assume that an LSS file was produced only by a correct LSS writer and has not been maliciously modified. The implementation does not need to defend against arbitrary hostile byte sequences or repeatedly validate logical invariants which its own writer guarantees. High performance takes precedence over additional validation which is redundant under this trusted writer model.

This assumption does not permit checksums to be skipped. All persisted regions protected by a 64-bit checksum must have that checksum verified whenever they are read from persistent storage and before their contents are trusted. Checksum verification is required during ordinary reads, lazy loading, cleaning, checkpoint processing and diagnostic traversal, not merely during a recovery scan. A cached immutable region need not be checksummed again on every access after it has been verified when loaded.

A valid 64-bit checksum makes undetected accidental corruption sufficiently unlikely for the LSS to rely on the data structures and invariants encoded in the verified region. This allows release builds to omit expensive checks whose purpose would only be to defend against a malicious writer or detect an implementation defect earlier. Checks required to establish record boundaries and avoid unsafe memory access are performed before unchecked deserialization begins.

Heavyweight LSS validation should be controlled by a dedicated preprocessor flag, independently of the ordinary debug or release build mode. For example:

#if CEDA_LSS_HEAVY_VALIDATION
    ValidateRpm(rpm);
    ValidateSut(sut);
    ValidatePacketChains(space);
#endif

Debug builds should normally enable this flag, but it must also be possible to enable it in an otherwise optimized release build. This permits realistic performance-oriented test runs, long soak tests and investigation of a deployment problem to retain extensive LSS checking without enabling all general-purpose debug behavior.

Heavy validation should verify RPM occupancy, node levels, packet-chain structure, SUT accounting, transaction sequencing, record identities, ownership and other invariants wherever practical. It is additional to always-required checksum, boundary and safety checks. Disabling the flag removes the expensive redundant checks justified by the trusted writer model; it must not remove validation needed to parse memory safely or verification of persisted 64-bit checksums.

It is also possible to select between safe and unsafe versions of the cxSerialise library. The safe version checks archive bounds and related serialization preconditions as fields are read or written. The unsafe version uses unchecked pointer advancement and is suitable after the enclosing record or packet boundary and its 64-bit checksum have already been validated. This choice is independent of CEDA_LSS_HEAVY_VALIDATION: an optimized build may use unsafe cxSerialise while enabling heavyweight logical checks, or use safe cxSerialise while omitting the expensive whole-structure checks.

If the implementation checks for a condition which is specified to be impossible and discovers that condition, it must not ignore it, repair it heuristically, skip the affected record or continue with a partially interpreted store. The LSS immediately enters its zombie state. No further logical processing or I/O is possible through that LSS, and the error is reported to its caller. The zombie state is permanent for that open LSS object; using the store again requires closing it and attempting a new open or recovery operation.