61 Compacting the store

Status: unimplemented proposal. This chapter describes a possible iterative in-place compaction operation, not functionality provided by the current LSS implementation.

The user may want to compact the store to make it as small as possible. This can be useful before transferring, archiving, or deploying the store. In-place compaction avoids requiring enough additional storage for a complete second copy.

The LazyCleaner relocates live packets from segments with low utilisation to the end of the log, allowing the cleaned segments to become free. This reclaims space for reuse within the LSS, but does not by itself reduce the size of the file. A file can be truncated only when its final segments are free. Cleaning a segment in the middle of the file merely creates reusable space in the middle; the file becomes shorter only if cleaning and subsequent allocation happen to leave unused segments at the end. The proposed compaction operation is intended to arrange the live packets and free segments deliberately so the unused tail can be removed.

The LssCopy utility program can already be used to compact a store manually. It copies every live serial element from the source LSS into a newly created LSS, leaving obsolete packets and unused space behind. The resulting file is compact, but this is an out-of-place operation: it needs a second file and the user must replace the original store with the copy when appropriate. The proposal below concerns a distinct in-place compaction operation.

Iterative method

The proposed operation would repeat these steps:

1. Disable the background cleaner and checkpointer.
2. Clean every segment with utilisation below 100%.
3. Stop if no segments were cleaned.
4. Perform a checkpoint and return to step 2.

Segments that are cleaned become available so compaction can be done efficiently in-place.

In the first round of cleaning, all the obsolete packets for RPM-sections and serial elements will be removed. However, the subsequent check point may create new obsolete packets (some of the RPM-sections). These could be anywhere so a second round of cleaning may be required.

In the second round, only segments containing obsolete RPM-sections will be cleaned. This may still cause many objects to be moved to the end of the log. Then another checkpoint is performed. The previous checkpoint may contain RPM-sections that have now become obsolete. The process nevertheless converges because each pass groups the live RPM-sections into fewer, more densely utilised segments.

Schematic example

Each seven-character group below represents one segment. Within a segment:

• - represents an obsolete or unused packet position.
• D represents a live serial-element data packet.
• P represents a live RPM-section packet.
• C represents a checkpoint packet.

The table is schematic: after the initial row it shows the compacted live region, rather than preserving the original segment numbers or displaying obsolete segments that can be reused.

Size reduction of the store

If many objects are deleted from the store, it may be useful for the file size to drop back to a reasonable level. The proposal would repeatedly remove the final segment from the file, cleaning it first when necessary.