49 DeltaFile

DeltaFileWriter is used to write LSS delta files. These provide a basis for backup of an LSS.

Delta files

     CPSN   0                    1                  2                       3
     CPID   {0000...}            {FC37...}          {4FB9...}               {EE28...}

 LSS   :    +--------------------+------------------+-----------------------+----------------

 Deltas:    <-------------------> <----------------> <---------------------> <---------------
                 delta 0               delta 1              delta 2               delta 3

            |                    |                  |                       |
            |                    |                  |                       |
          check                check              check                   check
          point                point              point                   point
           #0                   #1                 #2                      #3

Note the following

Delta files always begin at 0, and are sequenced by the CPSN (Check Point Sequence Number)
It is assumed that the initial, empty store is check pointed with CPSN=0 and CPID={00000000-0000-0000-0000-000000000000}
When all delta files are present (i.e. starting from 0), it is possible to create the initial level 0 by simply creating an empty store. I.e. there is no need to make a copy of the original empty store. This is because all empty stores are equivalent.
Delta files respect check point boundaries. Note in turn that check point boundaries respect both flush unit and transaction boundaries.
Each check point involves writing dirty RPM nodes to the end of the log and the delta file. So a delta file tends to finish with RPM packets. These are ignored when a delta file is applied to a level 0.
When LSS::SynchronousCheckPoint() is called, it (normally) tests whether the root RPM node is dirty. If the RPM is clean then there have been no changes to the store since the last check point so no check point is performed.
Data packet log records written by the cleaner don't represent a logical change to the store, and therefore should not be applied to the level 0. This is not merely a question of efficiency! The cleaner works at the level of packets, not whole serial elements, and there is no assumption of consistency between how the main LSS and a level 0 break up serial elements into packets.
It is possible that between two check points nothing but cleaning has occurred. So there are no "logical" changes to the store. Therefore when applying the delta to the level 0, it may be found that nothing needs to be done. After applying a delta to the level 0 and a checkpoint is performed, the RPM may be clean, yet the check point needs to be done anyway, just to make sure the level 0 root block records the new CPSN and CPID.
For correctness, the existing delta file (as well as the log) must be flushed before updating the root block. Otherwise, if a power failure occurs a delta file could be incomplete, making it impossible to bring the level 0 up to date.
To help ensure this we open the delta files with write-through cache.
The LSS may be on a different disk from the delta files. This makes it difficult to know for certain whether one is ahead of the other.
On recovery we always use the main LSS store for the recovery scan (not the delta file), and rewrite the last delta file from scratch (overwriting any previous data).
We are careful to initialise the segment writer so it continues using the last CPID and respects the FSN's. In any case, after recovery we immediately check point, (unless the RPM is clean).
During recovery only valid flush units may be written to the delta file. Flush units are always written in their entirety, even if recovery only actually made partial use of the flush unit. This may happen because recovery depends on finding snapshot log records.
Therefore to respect transaction atomicity, when a delta file is applied to a level 0, it is necessary to only apply changes when snapshot records are encountered. The delta file format definition must allow for some final log records that aren't committed by a snapshot record.

Writing delta file issues

It is not reasonable to break up delta files in an arbitrary way. For example, delta files should respect transaction boundaries or else applying delta files will be complicated by the need for transaction atomicity. Furthermore validation of adjacent delta files is made easier if they respect check point boundaries, for then we can think of each delta file as mapping the LSS from an input CPID to an output CPID. A Check Point Id (CPID) is a guid generated at each check point.

During a check point the LSS maximises concurrency by flushing the log (which can block on I/O) without blocking other threads from performing additional transactions. I.e. it releases the check point LRS mutex (used to write the dirty RPM nodes) *before* flushing the log. This can be a significant advantage, because check points are typically performed in the background by the lazy check pointer thread, so it makes sense for this thread to block while flushing the log without holding up threads that want to continue writing data to the segment cache.

Unfortunately this complicates the writing of the delta files. Conceptually during each check point the existing delta file should have its footer written and be closed, and the next delta file should be opened. However, once the check point releases the LRS mutex, another thread may write new data to the log. Furthermore there is nothing to stop the lazy writer waking up and needing to write the new data to a fresh delta file. This may happen concurrently with the check pointer which is synchronously flushing the log and old delta file.

Only one thread is able to write to the log and the delta files at a time. (Usually this is the lazy writer thread. Less often it is the thread that is flushing the log - either a client of the LSS, or else the thread doing a check point). It therefore makes sense for the thread writing to the delta files to itself work out when the old delta file needs to be closed and the next delta file opened (rather than the thread doing a check point). With this approach the delta file writer has no need to be thread-safe, because delta-files are only opened, written and closed by one thread at a time.

As the delta file writer writes sections of segments to disk, they can be scanned as a sequence of flush units. The flush unit headers tell the delta file writer precisely where the check point boundaries are. Note that check point boundaries always respect both flush unit boundaries and transaction boundaries.

A synchronous flush call is made to flush the log as part of a check point. This blocks out the lazy writer - because the mutex LazyWriter::writeSegmentsToDiskMutex_ ensures that only one thread is able to write segments to disk at a time. This calls


DeltaFileWriter::CloseDeltaFileIfOpen(int cpsn, const Guid& cpid)

The argument cpsn identifies the delta file to be closed. There is only a need to close the delta file if it hasn't been closed already.

Power failure while applying a delta

When a delta is applied to a level 0 we validate the CPID, CPSN and TXSN. Then we scan the delta file and apply each transaction in turn. During this process check points on the level 0 are not performed, because the flag LSS::forBackUp_ is set to true.

If there is a power failure then only some of the transactions may have been applied to the level 0. However, because no check points have been performed, in fact it is more a case that *none* of the delta file has been applied to the level 0. In fact the next time we try to apply the delta file, all transactions will need to be applied from scratch.

The TXSN in the root block makes it possible (in a future version of the LSS) to partially apply a delta to the LSS, and to later correctly apply the remainder of the delta file.

Code


class SFile
{
    cxNotCloneable(SFile)
public:
    SFile();
    ~SFile();

    void Open(ConstStringZ filename, bool read);
    bool IsOpen() const;
    void WriteBuffer(const void* buffer, int numBytes);
    void SetSize(int size);
    void Close();
private:
    FILE* fp_;
    xstring filename_;
    int numBytesWritten_;
};

class DeltaFileWriter
{
public:
    DeltaFileWriter(LSS& lss, ConstStringZ deltasDirPath);

    bool EnableWriteDeltaFiles() const { return enableWriteDeltaFiles_; }

    void SetCpsn(int cpsn, const Guid& cpid) { cpsn_ = cpsn; cpid_ = cpid; }

    // If writing to delta files is enabled then
    //      1. Opens the delta file (if not already opened)
    //      2. writes the given buffer to the delta file
    // Does nothing if writing to delta files is not enabled.
    // May throw FileException
    void Write(const void* buffer, int numBytes);

    // Called during a check point of the LSS.  If the delta file with given cpsn is still open
    // then write its footer.
    // May throw a FileException
    void CloseDeltaFileIfOpen(int cpsn, const Guid& cpid);

    void AbortDeltaFile();

    void TruncateDeltaFile(int fsn, int length);

private:
    bool DeltaFileIsOpen() const { return file_.IsOpen(); }

    // Throws FileException if there was an error writing the file
    void WriteFooterAndClose(const Guid& checkPointId);

    // Throws FileException if file couldn't be opened
    void OpenDeltaFile();

    // Throws FileException if there was an error writing the file
    void WriteBuffer(const octet_t* p1, const octet_t* p2);

    // Throws FileException if there was an error closing the file
    void CloseDeltaFile();

private:
    LSS& lss_;
    bool enableWriteDeltaFiles_;
    xstring deltasDirPath_;

    int fsn_;
    Guid cpid_;
    int cpsn_;

    SFile file_;
};

Previous48 ReadSerialElement Next50 CopyStore