70 cxSerialise

cxSerialise will be a new header-only library factored out of cxUtils. It will provide the existing serialisation and deserialisation support, including archives such as InputArchive and the variable-length serialisation functions.

The LSS will use cxSerialise generally for its persistent file format so LSS files are platform-independent. An LSS file created on a little-endian machine can be read on a big-endian machine, and an LSS file created on a big-endian machine can be read on a little-endian machine.

Scope

The scope of cxSerialise is not yet fixed. It may be limited to serialisation to and from binary blobs held in memory. This narrow scope would cover bounded and unbounded memory archives, the binary wire format and variable-length serialisation, while leaving stream and paged-storage facilities in cxUtils.

Alternatively, cxSerialise may also include the stream interfaces currently provided by cxUtils, together with PagedBuffer and its archive/stream adaptors. This broader scope would allow serialisation over contiguous memory, streams and paged buffers to be provided by one library. The choice must preserve the requirement that cxSerialise has no dependency on cxUtils, Boost or any other external library; any included stream or paged-buffer code would therefore need to be factored into cxSerialise with those dependencies removed.

The broader scope might also encompass high-performance checksum support. Checksums are needed by the LSS to detect corruption in persisted data, and placing checksum support alongside the binary representation can allow data to be checksummed efficiently as it is serialised, copied or consumed. The same facility could also be useful for messaging over the wire, where messages may require integrity checks independently of the transport. The checksum API should work with contiguous binary blobs and, if streams and PagedBuffer are included, with those sources and destinations as well.

Dependencies

cxSerialise has no dependencies on any other CEDA library or on any third-party library, including Boost. In particular, it does not depend on cxUtils. The dependency is in the other direction: cxUtils may depend on and use cxSerialise. Code factored out of cxUtils must therefore be reorganised to remove its existing cxUtils and third-party dependencies.

Relevant cxUtils files

The core serialisation files to be factored out are:

  • Archive.h
  • ArchiveCollection.h
  • MemoryAccess.h (the relevant little-endian memory access support)
  • VariableLengthSerialise.h

The following files provide cxUtils-specific serialisation support or use the archive interfaces. They remain in cxUtils, which can use cxSerialise, but their includes and dependencies must be updated as part of the factoring:

  • ArchiveError.h
  • ArchiveHPTime.h
  • ArchiveMagic.h
  • BPlusTree.h
  • Crc32.h
  • File.h
  • Guid.h
  • HalfOpenInterval.h
  • Hex.h
  • IMsgOrientedInputStream.h
  • MD5.h
  • PagedBuffer.h
  • PagedBufferAsArchive.h
  • Range.h
  • RelocatableMap.h
  • SessionValueCache.h
  • xostream2.h

Selectable safe and fast archives

The library should provide two implementations of archive types such as InputArchive. An externally defined preprocessor flag selects between them when the library headers are included: 


// Defined consistently by the consuming project's build configuration.
#define CX_SERIALISE_SAFE_ARCHIVES 1
#include <Ceda/cxSerialise/Archive.h>
  • The fast implementation preserves the existing representation and behaviour. It performs no bounds checks and is suitable when the input has already been validated or maximum throughput is required.
  • The safe implementation is constructed from a bounded sequence of octets, preferably a std::span<const octet_t>, and checks every read, skip and variable-length decode before advancing. Truncated or malformed input produces a well-defined error rather than an out-of-bounds memory access.

Safe InputArchive representation

The safe InputArchive can contain a std::span<const octet_t> representing all unread input. As bytes are consumed, the archive advances the start of the span by replacing it with a trailing subspan. The span's data pointer is therefore the current read position and its size is the number of octets remaining: 


class SafeInputArchive
{
public:
    explicit SafeInputArchive(std::span<const octet_t> input) : remaining_(input) {}

    std::span<const octet_t> Read(std::size_t n)
    {
        if (n > remaining_.size())
            throw EndOfStreamException();

        auto result = remaining_.first(n);
        remaining_ = remaining_.subspan(n);
        return result;
    }

private:
    std::span<const octet_t> remaining_;
};

The archive should contain the span rather than publicly inherit from it or be an alias for it. This preserves the archive abstraction, prevents client code from bypassing its checked operations, and allows the representation to change without changing the public serialisation interface.

Using std::span does not by itself make deserialisation safe because unchecked operator[] can still access past the end. Each operation must check its complete range before calling first() or subspan(). This representation avoids maintaining a separate current pointer and end pointer and avoids unchecked pointer addition when advancing.

Variable-length integers need both input-bound checks and validation of their maximum encoded length and terminal byte. A compound decode should work on a local copy of the remaining span and only commit the advanced span to the archive after the entire value is valid. An error then leaves the archive position unchanged rather than partially consuming malformed input.

Source compatibility

Both implementations expose the same public type names and normal serialisation interface. Client code continues to use InputArchive, OutputArchive and the existing Serialise()/Deserialise() functions. Flipping the project-wide flag should therefore require virtually no source changes. Code which already knows the buffer size can use a span constructor in either mode: 


InputArchive ar(std::span(buffer));
Deserialise(ar, value);

For migration, the pointer-only constructor can remain available in fast mode. Safe mode cannot make that constructor safe because no bound is supplied; it should either be unavailable in safe mode or require a pointer and size. Making unsafe pointer-only construction a compile-time error in safe mode is preferable, because it identifies the relatively small number of call sites that must be changed to provide the buffer extent. After those boundary call sites use spans, application serialisation code can switch modes without changes.

Implementation selection

The two implementations can have internal names such as UnsafeInputArchive and SafeInputArchive, with InputArchive defined as an alias selected by the flag. Shared serialisation algorithms should be templates over the archive interface so they are not duplicated. The flag should have a documented default, while build systems set it explicitly for each consuming project.

The selection must be consistent across every translation unit and linked library in a program. Changing the flag can change archive layout, inline function definitions and instantiated template types; mixing modes can therefore cause one-definition-rule and ABI errors. The selected mode should be part of the exported build configuration, and a link-time configuration symbol or equivalent guard should be used where practical to make mismatches fail clearly. Data written in either mode must use exactly the same wire format, so the choice affects validation and performance but not persistent or exchanged data.

Platform-independent byte order

cxSerialise must be platform-independent with respect to little-endian and big-endian processor architectures. The existing cxUtils serialisation format already specifies little-endian byte order on the wire: Archive.h uses the SetUnalignedLE() and GetUnalignedLE() operations provided by MemoryAccess.h. cxSerialise must preserve this established little-endian wire format rather than introduce a new byte order.

The wire order does not depend on the native byte order of the machine producing it. A little-endian machine must be able to serialise data that is then deserialised correctly by a big-endian machine, and a big-endian machine must be able to serialise data that is then deserialised correctly by a little-endian machine.

Serialisation and deserialisation of every multi-octet primitive must perform the necessary native to wire and wire to native byte-order conversion. This applies to integral and floating-point values, fixed-size fields and any length or control fields used by the archive format. The safe and fast archive implementations must use exactly the same platform-independent wire representation. Cross-endian test vectors should verify byte-for-byte output and round-trip compatibility in both directions.

Floating-point serialisation assumes the IEEE 754 interchange formats: binary32 for 32-bit floating-point values and binary64 for 64-bit floating-point values. The IEEE 754 bit pattern is encoded using the byte order defined by the wire format; it must not be written using the machine's native byte order. Implementations should verify the required floating-point representation at compile time. A platform that does not provide compatible IEEE 754 binary32 and binary64 types is not directly supported unless it explicitly converts its native representation to and from these wire formats.

Scope and verification

The same pattern can be applied where bounds matter to output archives and other cursor-like archive types. Safe output archives must check remaining capacity before writing; sizing archives do not access a buffer but must still detect size arithmetic overflow. Tests should run the same serialisation suite in both configurations, verify byte-for-byte identical output, and exercise the safe mode with truncated input, malformed variable-length integers, zero-length buffers and sizes near arithmetic limits. Benchmarks should quantify the cost of safe mode rather than allowing the safe checks to affect the fast implementation.

Documentation and performance benchmarks

cxSerialise will have its own document describing the library, its wire format, public API, safe and unsafe configurations, error handling, portability guarantees and recommended usage. That document will include performance benchmarks which give users a good idea of the performance difference between the safe and unsafe versions.

The benchmarks should exercise representative primitive values, arrays, collections, variable-length integers and larger structured values. They should report serialisation and deserialisation throughput, latency where useful, and the relative overhead of bounds checking. Both versions must be measured using the same data, compiler, optimisation settings and hardware so the comparison is meaningful. The document should record this test environment and enough methodology for the results to be reproduced.