71.3 RAS Error Handling

All RAS errors are fatal to the open LSS

All RAS errors are handled in the same fundamental way. An error is propagated to code using the LSS, and the open LSS object enters a zombie state. A zombie LSS no longer performs logical processing or initiates I/O. Continuing to use a store after an I/O error could compound corruption or make recovery more difficult, so an LSS cannot return from the zombie state to normal operation.

The error notification does not need to identify the WSN of a failed write. The LSS does not attempt to skip that write or continue processing later writes, so the WSN would not affect recovery. It is useful to identify the kind of operation that failed:

enum class RASOperation
{
    Read,
    Write,
    Flush
};

void OnRASError(RASOperation operation, const RASError& error);

The error should retain the native platform error code and message. For a read or write, the file offset and size are also useful diagnostic information even though the LSS will not use them to continue operation.

Integrity errors detected above the RAS

Some failures are detected by the LSS rather than the RAS. For example, the RAS may successfully read the requested bytes and the LSS may then find an invalid checksum, root-block division, LFU or log record. These failures indicate possible store corruption and cause the same transition to the zombie state as a native RAS error.

The operation category and detailed cause remain valuable for diagnosis and data-recovery tools, but they do not determine whether ordinary LSS processing may continue. Read, write, flush and integrity failures are all fatal to the open LSS instance.

Zombie-state behaviour

After entering the zombie state, the LSS rejects new transactions, reads, writes, checkpoints and cleaning work. Existing operations that have not completed logically fail with the stored fatal error. A failed flush must never be followed by a root-block write that depended on that flush, and a later progress notification must not revive work that was abandoned because of the error.

Entering the zombie state does not imply that native operations already submitted to the operating system have stopped accessing their buffers. The implementation may request cancellation where that is practical, but it must retain every operation record and buffer until the platform reports that the operation has completed or been cancelled. Native completions may therefore continue to be drained after the logical LSS has stopped.

Multiple errors

Several outstanding operations may fail at approximately the same time. The transition to the zombie state is idempotent. The first fatal error is retained as the primary error and is the error propagated to users of the LSS, because later errors may be consequences of the first. Subsequent errors can be recorded as additional diagnostic information but do not replace the primary cause.

Error propagation and lifetime

An error encountered by a foreground operation is returned to the caller of that operation. A background write, flush, checkpoint or cleaning failure also requires a store-level error notification because no client operation may be waiting for it at the time. Once the LSS is a zombie, later public API calls fail with the stored fatal error rather than reporting an unrelated generic state error.

The fatal-error notification and state transition must be serialised with other LSS notifications and state changes. Destruction is safe only after all native operations have stopped referencing the RAS, the LSS and caller-owned buffers, and no further callback can occur.

Repair and data recovery

A zombie LSS is not repaired in place. Recovery requires closing the failed instance and using a separate diagnostic or repair tool. Such a tool can inspect root-block divisions, scan and validate LFUs, identify intact packet chains and copy recoverable data into a new store. Unlike the normal LSS, the tool operates on the assumption that persistent structures may be inconsistent or corrupt.