75 Log

Use of PartitionSeid

A 32-bit Seid is meaningful only within a selected Space. A log record can be processed without a previously selected Space handle, so records which identify serial elements or packets use an PartitionSeid:

struct PartitionSeid
{
    SpaceId spaceId;
    Seid seid;
};

The SpaceId selects a Space within an Partition and the 32-bit Seid selects a serial element or packet within that space. An PartitionSeid is therefore a 64-bit identity which is unambiguous within one partition. The partition is identified by the PartitionId in the containing LFU.

Log records corresponding to new versions of serial elements include an PartitionSeid in the packet header. Records for overflow packets and other packets having their own RPM entries likewise use the PartitionSeid of the packet they identify. Delete, relocation or other records which identify an existing serial element or packet also carry enough information to reconstruct its PartitionSeid.

LogRecordPosition

A LogRecordPosition is a 64-bit file offset measured from the start of the first segment, immediately after the root block.

The segment size must be a power of two. The SegId and offset within the segment can therefore be obtained from a LogRecordPosition using shifting and masking.

using LogRecordPosition = uint64;

SegId GetSegId(LogRecordPosition position, uint64 segmentSize)
{
    cxAssert(segmentSize != 0 && (segmentSize & (segmentSize - 1)) == 0);
    return static_cast<SegId>(position >> std::countr_zero(segmentSize));
}

uint64 GetOffsetWithinSegment(LogRecordPosition position, uint64 segmentSize)
{
    cxAssert(segmentSize != 0 && (segmentSize & (segmentSize - 1)) == 0);
    return position & (segmentSize - 1);
}

Log Flush Units

Segments do not have persistent headers or footers. They continue to contain Log Flush Units (LFUs), which are the independently identifiable and validated units appended to a partition log. Every LFU belongs to exactly one Partition and records its persistent PartitionId:

using PartitionId = uint32;

struct LogFlushUnitHeader
{
    Checksum64 checksum64;
    Guid checkpointId;
    PartitionId partitionId;
    FlushSeqNumber flushSeqNumber;
    int32 numBytesInPayload;
    SegId nextSegId;
};

LogFlushUnitHeader is 40 bytes.

The partition identity is stored with the LFU metadata and covered by the LFU's integrity checks.

All packets in an LFU belong to the partition identified by that LFU. When a packet contains an PartitionSeid, the LSS can verify that its SpaceId belongs to the stated partition. A mismatch is an integrity error and puts the LSS into its zombie state.

Partition-local log sequence

Each partition has its own log and writer, so LFU sequence numbers can be local to the partition. An LFU position in logical log order is qualified by both values:

    (PartitionId, LFU sequence number)

Each partition assigns its own LFU sequence numbers.

Partition identity within segments

A segment is assigned to one partition while that partition uses it, but there is no segment header which persistently records that assignment. The partition ID in each LFU identifies its partition log.

All valid LFUs in a segment must agree on the same PartitionId.

LFUs record the next SegId in the log. Each Partition therefore has an independent forward-chained sequence of segments from its last valid checkpoint. An independent recovery scan of that sequence can be performed when the partition is opened.

Self-identifying records

Using PartitionSeid makes the relevant log records self-identifying. Log scanning, cleaning, checkpoint processing, validation and diagnostic tools can select the correct Space and RPM without relying on external context.

This remains necessary when an Partition contains many Seid spaces. The partition identifies the physical log and SUT, while the SpaceId in the packet header identifies the four-level RPM which maps that packet's local Seid.

Using a 64-bit identity in self-describing log records does not remove the principal benefit of 32-bit Seids. High-volume references in B+Trees, RPMs and other structures whose address space is already known continue to store only the local 32-bit value.