46 CheckPoint

A check point is performed at regular intervals to reduce the time required for crash recovery. The last valid check point divides the log at a position called the last valid check point position. This position is represented by a LogRecordPosition, which identifies a segment and an offset within that segment. It divides the log into two sections:

• The records before the last valid check point are referred to as the check pointed records.
• The records on or after the last valid check point are referred to as the recovery records. These are the records that are scanned to perform a recovery

The recovery segments are the segments that contain recovery records.

Check point identity and log position

Each log flush unit (LFU) contains a check point identity (CPID) and a 32-bit Flush Sequence Number (FSN). The CPID identifies the last completed check point from which recovery can begin. The FSN increases through the LFUs belonging to that recovery sequence.

The diagram shows check point cpid2 being made. Its check-point data, including dirty RPM nodes, is appended in the two highlighted LFUs with FSNs 22 and 23. Those LFUs still contain cpid1, because cpid1 remains the last completed check point while the new check point is being constructed.

After the final check-point LFU has been closed, SegmentWriter::UpdateFlushPos() returns a LogRecordPosition at the end of that LFU. The LSS assigns this value to positionOfLastCheckPoint_. It is exactly the position represented by the dotted line in the diagram: the boundary after the LFUs written by the check point and before the first LFU belonging to the new recovery sequence.

The LSS then calls SegmentBeingWrittenInMemory::SetCheckPointId(). This changes the already open but still empty LFU immediately to the right of the dotted line to cpid2 and resets its FSN to 1. Subsequent LFUs retain cpid2 and increment the FSN.

If the LFU immediately before the dotted line exactly fills a segment, the stored LogRecordPosition is represented as the end of that segment rather than offset zero of the next segment. It still denotes the same logical boundary in the LFU sequence.

Publication in the root block

The next root-block division is prepared with a coherent snapshot of the SUT root, RPM root, positionOfLastCheckPoint_, the new CPID, and the other check-point state. LSS::SerialiseRootBlockDivisionPayload() serialises positionOfLastCheckPoint_ directly into the division as a LogRecordPosition.

The checkpoint LFUs are made flushable and the log is flushed before RootBlock::WriteNextDivisionToDisk() writes the prepared division. Once that division is valid on disk, it publishes cpid2 as the newest check point and records the dotted-line position as the place from which its recovery scan begins.

The relevant source path is:

1. SegmentWriter::UpdateFlushPos() closes the last check-point LFU and returns the boundary position.
2. LSS::CheckPointWithGivenCpid() stores the position and switches the empty next LFU to the new CPID and FSN 1.
3. RootBlock::PrepareNextDivisionInMemory() serialises the check-point state into the next division.
4. RootBlock::WriteNextDivisionToDisk() writes that division after the log flush.

A checkpoint writes a coherent, up to date version of the RPM and SUT to the log and root block. Locking of the log ensures that no other thread is making changes to the RPM and SUT as they are check pointed.

Check pointing involves the following steps

1. Begin an LRS-op
2. Write any dirty RPM-nodes to the log
3. Write any dirty SUT-nodes to the log
4. Set lastCheckPoint to point at the end of the log
5. End the LRS-op
6. Flush the log
7. Write the SUT, root RPM-node and the position of the check point record to the root block.

After the check point is completed, the cleaner may be allowed to process more segments because the location of the last check point has advanced.

It is vital that the log be flushed to perform a check point. It is not allowable for the root block to reference log records that haven't been flushed yet.

A check point is not performed if the last LRS-op was a check point.

Check pointing is performed by a thread that sits in an infinite loop, performing check point operations then sleeping for a configurable time (say 3 minutes). The check point reduces the amount of recovery work after a crash.

We should flush the log more regularly than we check point to reduce the potential for significant data loss. Flushing every 20 seconds may be suitable.

The following code might be suitable for the check pointing thread

while(1)
{
    for (int i=0 ; i < numFlushesPerCheckPoint ; ++i)
    {
        FlushLog();
        Sleep(tmFlushInterval);
    }
    CheckPoint();
    Sleep(tmFlushInterval);
}

By sleeping for long intervals, check pointing represents a relatively small I/O and CPU load on the system.

TODO: It would be better to perform a check point after a configurable number of bytes has been written to the log since the last check point - because this relates to the time taken for recovery. During quiet periods check points will be far apart.

Thread safety

The LSS uses a mutex to make sure that only one thread does a check point at a given time.

// Used to write a single LRS for the purposes of a check point
class LRSWriterForCheckPoint
{
public:
    LRSWriterForCheckPoint(LSS& lss);
    ~LRSWriterForCheckPoint();

    // Allow for writing an RPM packet.
    // Used by the RPM to write its dirty nodes during a check point.
    /*gives*/ IOutputStream* WriteRPMPacket(Seid seid, RPMNode* node);

private:
    LSS& lss_;
};