74 Log

Each Partition has its own log. That log uses its own forward-chained sequence of segments, independently of the logs belonging to other partitions.

Use of `PartitionSeid`

A 32-bit Seid is meaningful only within a Space. There is no concept of a selected Space in the log, so records which identify serial elements or packets use a PartitionSeid:


struct PartitionSeid
{
    SpaceId spaceId;
    Seid seid;
};

The SpaceId selects a Space within a Partition and the 32-bit Seid selects a serial element or packet within that space. A PartitionSeid is therefore a 64-bit identity which is unambiguous within one partition.

Log records corresponding to new versions of serial elements include a PartitionSeid in the packet header. Records for overflow packets and other packets having their own RPM entries likewise use the PartitionSeid of the packet they identify. A DeleteSerialElement record identifies the head packet of the serial element being deleted. Other records which identify an existing serial element or packet also carry enough information to reconstruct its PartitionSeid.

When a packet contains a PartitionSeid, the LSS can verify that its SpaceId belongs to the Partition which owns the log. A mismatch is an integrity error and puts the LSS into its zombie state.

`LogRecordPosition`

A LogRecordPosition is a 64-bit byte offset measured from the beginning of Segment 0. Segment 0 begins immediately after the root block, so converting a LogRecordPosition to a physical RAS position adds the size of the root block.

Position zero is not a possible position for a real log record. The beginning of a segment containing log records is occupied by an LFU header, and the records in that LFU follow the header. Zero is therefore available as a sentinel without conflicting with a real record.

The value zero represents an absent or null log-record position. In-memory structures can store a LogRecordPosition directly and test it for zero instead of using std::optional<LogRecordPosition>. This convention applies to RPM entries, persistent RPM-root positions and other optional references to log records.

The segment size must be a power of two. The SegId and offset within the segment can therefore be obtained from a LogRecordPosition using shifting and masking.


using LogRecordPosition = uint64;

SegId GetSegId(LogRecordPosition position, uint64 segmentSize)
{
    cxAssert(segmentSize != 0 && (segmentSize & (segmentSize - 1)) == 0);
    return static_cast<SegId>(position >> std::countr_zero(segmentSize));
}

uint64 GetOffsetWithinSegment(LogRecordPosition position, uint64 segmentSize)
{
    cxAssert(segmentSize != 0 && (segmentSize & (segmentSize - 1)) == 0);
    return position & (segmentSize - 1);
}

Log Flush Units

Segments do not have persistent headers or footers. They contain Log Flush Units (LFUs), which are the independently identifiable and validated units appended to the log. Every LFU redundantly records the PartitionId of the Partition which owns the log:


using PartitionId = uint32;

struct LogFlushUnitHeader
{
    Checksum64 checksum64;
    Guid checkpointId;
    PartitionId partitionId;
    FlushSeqNumber flushSeqNumber;
    int32 numBytesInPayload;
    SegId nextSegId;
};

LogFlushUnitHeader is 40 bytes.

The partition identity is stored with the LFU metadata and covered by the LFU's integrity checks.

Partition-local log sequence

Each partition has its own log and writer, so LFU sequence numbers can be local to the partition. An LFU position in logical log order is qualified by both values:

    (PartitionId, LFU sequence number)

Each partition assigns its own LFU sequence numbers.

Partition identity within segments

A segment is assigned to one partition while that partition uses it, but there is no segment header which persistently records that assignment. The partition ID in each LFU identifies its partition log.

All valid LFUs in a segment must agree on the same PartitionId.

LFUs record the next SegId in the log. Each Partition therefore has an independent forward-chained sequence of segments from its last valid checkpoint. An independent recovery scan of that sequence can be performed when the partition is opened.

Self-identifying records

Using PartitionSeid makes the relevant log records self-identifying. Log scanning, cleaning, checkpoint processing, validation and diagnostic tools can select the correct Space and RPM without relying on external context.

This remains necessary when a Partition contains many Seid spaces. The partition identifies the physical log and SUT, while the SpaceId in the packet header identifies the variable-height RPM which maps that packet's local Seid.

Using a 64-bit identity in self-describing log records does not remove the principal benefit of 32-bit Seids. High-volume references in B+Trees, RPMs and other structures whose address space is already known continue to store only the local 32-bit value.

Previous73 Partition Next74.1 Time