39.2 LogRecord

Log records are written to flush units within segments. Obviously it is not permissible for a log record to overflow a FlushUnit or Segment.

The physical position of a log record is specified with a LogRecordPosition. This identifies the segment using a 32 bit SegId and a 32 bit integer offset within the segment.

struct LogRecordPosition
{
    SegId segid_;
    int32 offset_;
};

Given the physical position, it is possible to efficiently seek directly to that location and read the log record.

Kinds of log records

There is an enum corresponding to the 5 kinds of log records:

enum ELogRecordType
{
    LR_PACKET,
    LR_DELETE_PACKET,
    LR_SNAPSHOT,
    LR_TIMESTAMPED_SNAPSHOT,
    LR_NEXT_SEID_HIGH
};

A log record begins with a single octet which gives its type:

using LogRecordType = octet_t;

Since there are only 5 kinds of log records, we can use the low 4 bits of LogRecordType for recording the ELogRecordType and the high 4 bits provide 4 additional flags, which are used by LR_PACKET log records.

const int NEXT_SEID_PRESENT_BITPOS = 7;
const int HEAD_PACKET_BITPOS       = 6;
const int RPM_PACKET_BITPOS        = 5;
const int CLEANING_PACKET_BITPOS   = 4;

const LogRecordType NEXT_SEID_PRESENT_MASK = (LogRecordType) (1 << NEXT_SEID_PRESENT_BITPOS);
const LogRecordType HEAD_PACKET_MASK       = (LogRecordType) (1 << HEAD_PACKET_BITPOS);
const LogRecordType RPM_PACKET_MASK        = (LogRecordType) (1 << RPM_PACKET_BITPOS);
const LogRecordType CLEANING_PACKET_MASK   = (LogRecordType) (1 << CLEANING_PACKET_BITPOS);

Serialised format of log records

The serialised format of a log record is described by the following (binary) grammar

    <octet> = 8 bit unsigned integer

    <uint32> = <octet> <octet> <octet> <octet>
               (little endian binary representation of a 32 bit unsigned integer)

    <uint64> = <octet> <octet> <octet> <octet> <octet> <octet> <octet> <octet>
               (little endian binary representation of a 64 bit unsigned integer)

    <LogRecordType> = <octet>

    <SeidHigh> = <uint32>

    <seid> = <uint64>

    <next-seid> = <seid>

    <data-size> = <uint32>

    <data> = <octet>*

    <TxnSeqNumber> = <uint64>

    <HPTime> = <uint64>

    <PacketLogRecord> =
        <LogRecordType>
        <seid>
        <data-size>
        <data>
        [ <next-seid> ]

    <DeletePacketLogRecord> =
        <LogRecordType>
        <seid>

    <SnapShotLogRecord> =
        <LogRecordType>

    <TimeStampedSnapShotLogRecord> =
        <LogRecordType>
        <TxnSeqNumber>
        <HPTime>

    <NextSeidHighLogRecord> =
        <LogRecordType>
        <SeidHigh>

    <LogRecord> =
        <PacketLogRecord> |
        <DeletePacketLogRecord> |
        <SnapShotLogRecord> |
        <TimeStampedSnapShotLogRecord> |
        <NextSeidHighLogRecord>

Data alignment

Log records can appear at almost any offset position within a segment because log records are written consecutively within a flush unit and log records can be almost any size when serialised.

Therefore is not possible to use data structures for reading and writing the parameters of log records on architectures such as ARMv7 which don't support unaligned memory access for 32 bit and 64 bit values.

Code

enum ELogRecordType
{
    LR_PACKET,
    LR_DELETE_PACKET,
    LR_SNAPSHOT,
    LR_TIMESTAMPED_SNAPSHOT,
    LR_NEXT_SEID_HIGH
};

const int NEXT_SEID_PRESENT_BITPOS = 7;
const int HEAD_PACKET_BITPOS       = 6;
const int RPM_PACKET_BITPOS        = 5;
const int CLEANING_PACKET_BITPOS   = 4;

using LogRecordType = octet_t;

const LogRecordType NEXT_SEID_PRESENT_MASK = (LogRecordType) (1 << NEXT_SEID_PRESENT_BITPOS);
const LogRecordType HEAD_PACKET_MASK       = (LogRecordType) (1 << HEAD_PACKET_BITPOS);
const LogRecordType RPM_PACKET_MASK        = (LogRecordType) (1 << RPM_PACKET_BITPOS);
const LogRecordType CLEANING_PACKET_MASK   = (LogRecordType) (1 << CLEANING_PACKET_BITPOS);

// The size of a packet header is 13 octets:
//
//      LogRecordType   type        1 octet
//      Seid            seid        8 octets
//      int32           bufferSize  4 octets
const size_t PACKET_HEADER_SIZE = sizeof(LogRecordType) + sizeof(Seid) + sizeof(int32);

// Sequence number assigned to snapshot records in the log
using TxnSeqNumber = int64;
const size_t TIMESTAMPED_SNAPSHOT_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(TxnSeqNumber) + sizeof(HPTime);

const size_t NEXT_SEID_HIGH_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(SeidHigh);

const size_t DELETE_PACKET_LOG_RECORD_SIZE = sizeof(LogRecordType) + sizeof(Seid);

struct PacketInfo
{
    LogRecordType type_ = 0;           // Low two bits equals LR_PACKET
    Seid seid_;
    int32 bufferSize_ = 0;
    LogRecordPosition position_;       // Current position of the packet
    octet_t* buffer_ = nullptr;        // Points at the data of the packet (i.e. offset by PACKET_HEADER_SIZE)
    Seid nextSeid_;
    SegmentUnreserver segmentUnreserver_;      // Unreserves segid when destructs
    SegmentAccessor segmentAccessor_;          // Releases the segment when destructs
};

struct ILogRecordVisitor
{
    virtual void VisitPacketRecord(const PacketInfo& pi) = 0;
    virtual void VisitSnapshotRecord(LogRecordPosition nextPos, TxnSeqNumber txsn, HPTime timeStamp) = 0;
    virtual void VisitDeletePacketRecord(Seid seid) = 0;
    virtual void VisitNextSeidHighLogRecord(SeidHigh seidHigh) = 0;
};

void VisitLogRecordsInBuffer(octet_t* buffer, int segid, int s1, int s2, bool onlyVisitPackets, ILogRecordVisitor& visitor);

// Visit all log records within the given segment in the range [s1,s2) where s1 and s2 are integer
// offset positions into the buffer
// 'onlyVisitPackets' indicates whether the client is only interested in packet log records
void VisitLogRecordsInBuffer(Segment& segment, int s1, int s2, bool onlyVisitPackets, ILogRecordVisitor& visitor);

void VisitAllLogRecordsInSegment(Segment& segment, bool onlyVisitPackets, ILogRecordVisitor& visitor);

///////////////////////////////////////////////////////////////////////////////////////////////////

/*
RetrievePacketBufferAtGivenPosition() provides the basis for reading any given packet in the LSS.
It is suitabe for both data and RPM packets.

Typically the given LogRecordPosition will have been recorded by the RPM.

This function is fully threadsafe (any number of threads can access existing packet buffers).

The implementation calls GetSegment() on the segment cache which may block on I/O or on eviction
of a segment from the cache.  The corresponding ReleaseSegment() in performed when the PacketInfo
provided by the caller destructs (see PacketInfo::segmentAccessor_).

The segment is not reserved (in the SUT) by this function.
*/
void RetrievePacketBufferAtGivenPosition(LSS& lss, LogRecordPosition p, PacketInfo* pi);

bool RemoveChain(LSS& lss, Seid seid);