7 Write ordering on storage devices
Persistent data normally passes through several independently buffered layers: an application, operating-system page cache, filesystem, block layer and driver, controller or bridge, and finally a device with its own firmware, volatile RAM, and non-volatile media. RAID controllers, virtual machines, network storage, and USB-to-SATA bridges can add more layers. The order in which an application submits writes is therefore not, by itself, the order in which those writes become persistent.
This chapter describes the contract that a crash-consistent storage protocol needs from that stack. It is primarily about local block devices, but the same reasoning applies to other persistent storage: a higher layer can provide no stronger guarantee than the weakest layer through which its durability request passes.
Four different properties
Discussions of a "completed write" often conflate four independent properties:
- Visibility: a subsequent read observes the new data. It may merely be reading a cache.
- Durability: the data will survive the relevant failure, particularly abrupt loss of power.
- Ordering: write A is guaranteed to be durable before write B can become durable.
- Atomicity: after a failure a write is entirely old or entirely new, rather than torn.
A successful buffered write normally establishes visibility, not durability. A cache flush can establish durability for preceding writes, but does not make a large write atomic. Checksums and redundant records can detect or tolerate torn writes, but do not force earlier data to storage. A correct protocol states which property it needs at each step rather than relying on the word "write" to imply all four.
Why volatile write caches exist
HDDs and SSDs commonly acknowledge writes after accepting them into volatile device memory. This reduces latency, permits request merging and reordering, and lets the device schedule media updates efficiently. SSD firmware also performs address translation, garbage collection, wear levelling, and metadata updates whose physical order need not resemble the host's logical write order.
Disabling a device write cache is therefore a costly and usually unnecessary way to obtain a durability guarantee. The normal arrangement on commodity hardware is to leave the cache enabled and issue an explicit cache flush at the points required by the storage protocol. The 2008 Microsoft Research report Enforcing Database Recoverability on Disks that Lack Write-Through describes this approach for IDE disks. It reports that most single-disk IDE drives supported the ATA flush command even though most did not support per-write Force Unit Access, and explains both the performance cost and the unreliability of attempting to disable caching as the general solution.
ATA and SATA: FLUSH CACHE
For an ATA or SATA device with a volatile write cache, the FLUSH CACHE
and FLUSH CACHE EXT commands tell the device to complete cached writes
to non-volatile media before reporting successful completion. An operating system normally emits
these commands on behalf of a filesystem durability operation; applications using files do not send
ATA commands directly.
A completed flush provides a boundary for writes accepted before it. A protocol can therefore write a set of dependent records, wait for a flush, and only then issue the record that publishes or makes those records reachable. If the publication record must itself be durable before success is reported, it needs either a subsequent flush or a write with suitable FUA semantics.
Force Unit Access (FUA) is a property of an individual write: completion must not be reported until that write is in non-volatile storage. FUA is not the same as a flush of all prior cached writes. A pre-flush and an FUA publication write can be combined to express "everything before this write is durable first, and this write is durable before completion." Older ATA devices and parts of the SATA software stack have not always provided usable native FUA, so operating systems can implement the same result with cache flushes.
SCSI and NVMe
SCSI provides analogous mechanisms through SYNCHRONIZE CACHE and the
FUA bit on suitable write commands. NVMe provides a Flush command and an FUA bit on NVM write
commands. When a volatile write cache is present, a successful NVMe Flush makes the relevant earlier
writes persistent; an FUA write is not to complete until its data is persistent. These are interface
contracts rather than claims about the device's internal physical write sequence. The current
contracts are specified by the
NVM Express specifications.
NVMe also reports atomic-write units, including a distinct atomic-write unit for power-fail conditions. This distinction is important: a device may promise that a small write is atomic during normal operation while making a weaker promise if power is removed during the write. Atomic-write limits do not replace Flush or FUA, and Flush or FUA do not enlarge an atomic-write limit.
Linux
For ordinary file I/O, fsync() requests that modified file data and
the metadata needed to retrieve it be transferred to the storage device, and waits for the device to
report completion. fdatasync() may omit metadata that is not needed for
subsequent data retrieval. Creating or renaming a file can additionally require an
fsync() of its directory. These details are described by the
Linux fsync(2) manual page.
Direct I/O is not a substitute for a durability operation: bypassing the page cache does not, by
itself, force a volatile device cache to non-volatile media.
Within the Linux block layer, filesystems express the device-side requirements using a forced flush
and FUA. REQ_PREFLUSH guarantees that previously completed writes are
non-volatile before the associated write begins. REQ_FUA guarantees
that completion of the marked write is not reported until that write is non-volatile. The block layer
can translate FUA into a post-write flush when a device has a write cache but no native FUA support.
See the Linux kernel documentation on
explicit volatile write-back cache control.
Windows
On Windows, FlushFileBuffers()
flushes buffered information for a file to the device. A file opened with
FILE_FLAG_WRITE_THROUGH requests write-through behavior;
FILE_FLAG_NO_BUFFERING separately controls system caching and imposes
alignment requirements. The flags are independent. Microsoft describes their interaction and the
fact that support ultimately depends on the hardware in
File Buffering.
An explicit flush operation is normally preferable to treating every write as write-through. It preserves batching: many related writes can be submitted efficiently and then covered by one deliberate durability boundary. Write-through may still be useful for an individual write or for environments whose lower-level interface maps it efficiently to FUA.
Abrupt power loss
When power disappears, acknowledged data that exists only in volatile memory is lost. The failure can also interrupt a media update, producing a torn write, or interrupt SSD firmware while it is updating translation metadata. Drives with genuine power-loss protection use stored energy to drain volatile state and finish essential metadata updates. An external uninterruptible power supply is valuable, but is not equivalent: it does not cover a device, cable, controller, kernel, or firmware failure that prevents an orderly flush.
Device behavior under power fault has not always matched the simple model "only unflushed writes are lost." Zheng et al., Reliability Analysis of SSDs Under Power Fault (ACM TOCS, 2016), tested 17 commodity SSDs from six vendors over more than 3000 power-fault injections. Fourteen exhibited at least one surprising behavior, including bit corruption, shorn writes, unserializable writes, metadata corruption, or complete device failure. This result does not mean that every tested device ignored a successfully completed flush, and should not be read as a claim about that narrower case. It does show why software should use integrity checks and why a deployment requiring strong guarantees should validate its actual devices, firmware, controllers, and power-loss-protection claims.
The filesystem is another part of the contract. Pillai et al., All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications (OSDI 2014), found that persistence properties varied across six Linux filesystems and identified 60 crash vulnerabilities in eleven applications. The lesson is not that flushes are futile, but that a protocol must use the documented filesystem operations and must not infer persistence order from incidental behavior observed on one filesystem.
What the guarantee depends upon
On a typical PC with a directly attached SATA HDD or SSD, write caching can normally remain enabled. The application asks the operating system for durability, the filesystem and block layer translate that request into flush or FUA operations, and a conforming device honours them. IDE-era devices also commonly implemented ATA FLUSH CACHE. Thus it is inaccurate to say that most off-the-shelf disks must have their caches disabled to support ordered persistence.
The guarantee is conditional on the entire path. A RAID controller with unprotected write-back RAM, a USB bridge that drops cache-management commands, a hypervisor that acknowledges them too early, or firmware that falsely reports completion breaks the contract. Multiple independent devices also create multiple cache domains: flushing one does not flush another. Storage vendors and system operators therefore commonly qualify support by controller configuration, battery or capacitor health, firmware version, and power-loss-protection capability.