117 Schema evolution

In systems where value and entity types serve as the logical basis for a database, there is a practical requirement to accommodate type definition evolution.

Schema evolution is defined relative to component releases. Every type is implemented by exactly one component, and the release number of that component identifies the serialised representation used for values and entities of that type. Components that do not have a historical deserialisation contract do not need release metadata.

Type definition evolution is modelled by allowing attributes of a value or entity type to be added or dropped over time. The lifetime of each attribute is tied to release numbers of the component that implements the type.

Type-local evolution

The current attributes of a type are declared directly in the type body. Attributes that existed in older releases but are not current are declared in a dropped block. The schema block records when attributes are added or dropped.

Example: Point over time

Releases 0 and 1

No serialised representation of Point values existed in releases 0 and 1.

Release 2

Release 2 introduced Point with Cartesian coordinates.


value Point
{
    Float64 x;
    Float64 y;
}

Release 3

Release 3 added z. New attributes are added after existing attributes, so the serialised attribute order became (x,y,z). The schema block records that z was added in this release.


value Point
{
    Float64 x;
    Float64 y;
    Float64 z;

    schema
    {
        2:  +x +y
        3:  +z
    }
}

Release 4

Release 4 did not change the serialised representation of Point values. The release 3 representation therefore carried forward: serialised Point values still had (x,y,z).

Release 5

Release 5 dropped z. The current representation was back to (x,y), but z remained in the dropped block so serialised data from release 3 and release 4 could still be understood.


value Point
{
    Float64 x;
    Float64 y;

    dropped
    {
        Float64 z;
    }

    schema
    {
        2:  +x +y
        3:  +z
        5:  -z
    }
}

Release 6

Release 6 changed the current representation to polar coordinates. The old Cartesian fields moved to the dropped block, and the evolve functions described how to construct the new current fields from older serialised data.


value Point
{
    Float64 r;
    Float64 theta;

    dropped
    {
        Float64 x;
        Float64 y;
        Float64 z;
    }

    schema
    {
        2:  +x +y              // Release #2    : (x,y)
        3:  +z                 // Release #3,4  : (x,y,z)
        5:  -z                 // Release #5    : (x,y)
        6:  -x -y +r +theta    // Release #6-   : (r,theta)
    }

    evolve
    {
        r(x,y)     { r = sqrt(x*x + y*y); }
        theta(x,y) { theta = atan2(y,x); }
    }
}

Schema consistency

The release numbers in a schema block are release numbers of the component that implements the type. They must be strictly increasing. Each release number used as a schema change point must be declared in that component's manifest. A schema block does not need an entry for every declared release; a representation continues unchanged until the next schema change point. Before the first schema change point for a type, no serialised representation of values of that type exists.

Each attribute has a single lifetime. The schema adds an attribute once; the attribute is then either current or dropped by a later schema change. The schema cannot add the same attribute again after it has been dropped, even with the same type. Reusing the name would make the historical meaning of that attribute discontinuous; use a new attribute name when a later release introduces a distinct serialised field.

The schema block defines the serialised attribute order for each historical representation. A release entry is applied from left to right. Added attributes are appended to the serialised representation in the order in which they appear in the schema block. For example, +x +y means x appears before y in that release's serialised representation, and a later +z appends z after the existing fields.

Attribute order is part of the serialised representation. A schema evolution step adds attributes at the end or drops existing attributes; it does not reorder existing attributes. Source attribute names are not stored in the serialised representation. The serialised representation is determined by attribute positions. Therefore changing an attribute name in the source does not by itself define a schema evolution step, provided the attribute remains at the same serialised position with the same type.

Type names as dynamic type identifiers

Type names have a different status from attribute names, including enum attribute names, when serialised data uses dynamic type identifiers. For example, if a serialised value is statically known only as Shape but the actual value is a Circle, the serialised representation must identify the concrete type whose representation follows. The stable identifier available in Predica is the fully qualified type name, such as acme.Circle.

Therefore type names that can appear as dynamic type identifiers cannot generally be changed without creating a schema evolution issue. A type rename changes the identifier stored in old serialised data unless schema evolution for that type records the old name and maps it to the current type.

A schema block is consistent when all of the following conditions hold:

Each -name schema change refers to an attribute that has appeared in an earlier +name schema change.
Each -name schema change refers to an attribute that has not appeared in an earlier -name schema change.
A single release entry does not both add and drop the same attribute.
The attributes declared in the dropped block are exactly the attributes dropped by the complete schema block.
The current attributes declared directly in the type body are exactly the attributes that have been added using +name and not dropped.

The evolve block defines functions for current attributes that are derived from historical attributes during deserialisation. It is used when a new current attribute must be initialised from attributes present in older serialised representations.

Enum evolution

Enum schema evolution follows the same principle as value and entity evolution: serialisation is defined by integer representation rather than source names. An explicit representation-value expression is evaluated at compile time in the enum's generic environment. An attribute without an explicit expression has the value of the preceding attribute plus one, or zero if there is no preceding attribute. Attribute names are not stored in the serialised representation. Therefore changing an attribute name in the source does not by itself define a schema evolution step, provided its representation value remains unchanged for every valid instantiation.

To preserve serialisation compatibility during schema evolution, the author must add new enum attributes at the bottom of the enum declaration. This leaves existing implicit representation values unchanged. Every existing attribute must continue to have the same representation value for every valid instantiation, and each new attribute must have a value that does not collide with an existing historical value for any valid instantiation.

Existing enum attributes are dropped with an explicit remapping to a current attribute so old serialised values are deserialised. The dropped attribute is kept in the enum declaration with a dropped prefix, followed by as and the current replacement attribute. Dropped attributes are not values of the current enum type. They remain only as schema evolution metadata for interpreting old serialised values. A dropped entry remains in the sequence used to determine subsequent implicit values. It is an error for old serialised enum values to have no current interpretation.


enum OrderStatus
{
    pending = 0,
    accepted = 1,
    dropped cancelled = 2 as rejected,
    rejected = 3
}

Variant evolution

Variant types are closed: the attributes are listed in the variant declaration. Their serialised representation follows the same representation-value stability rule as enums, followed by the serialised representation of the selected variant attribute's value. Variant attributes do not have explicit discriminant values; the variant discriminant is the zero-based position of the attribute entry, counting both current and dropped entries. Attribute names are not stored in the serialised representation, so an attribute can be renamed without schema evolution when it keeps the same position and type.

New variant attributes are added at the bottom of the variant declaration. Existing attributes keep their position and type. Changing the type of an existing variant attribute is not an in-place change; it requires a new attribute or an explicit remapping from the old attribute.

A dropped variant attribute remains in the declaration as schema evolution metadata and remaps old serialised values to a current variant attribute. The remapping expression is evaluated with the old attribute value bound to the dropped attribute name and produces the value for the replacement attribute.


variant Shape
{
    Circle circle;
    Rectangle rectangle;
    dropped Square square as rectangle = square.boundingRectangle;
}

Dropped attributes

Dropped attributes remain part of schema evolution metadata, but they are not current attributes. Current schema expressions see only current attributes. Derived attributes, constraints, receiver expressions, and constructor initializers are encoded against the current type. If a current schema expression refers to a dropped attribute, the compiler reports that the attribute was dropped instead of treating it as a current attribute.

For example, after Point.x is dropped, a constructor such as Point { x = 1 } is a schema error.

Evolve functions

Evolve functions are used to generate deserialisation code. They are not ordinary current-schema expressions. Their parameters name historical input fields from older serialised representations.

In:


r(x,y) { r = sqrt(x*x + y*y); }

the names x and y are historical fields used while deserialising old data. They are not current attributes of this.

Component release dependencies

Component release metadata is declared in the component manifest. Release metadata is only needed for schema evolution; ordinary name lookup uses source-file using declarations.

A releases block is required for a component when serialised data contains values or entities implemented by that component and an older serialised representation must be deserialised by a newer release. This is the normal case for persistent database components. The release number is the key used to interpret the historical representation before evolve functions produce the current representation.

A releases block is therefore required when a component records type-local representation evolution with schema, dropped, or evolve, and when serialised representations of values or entities implemented by the component include serialised representation dependencies. Another component is a serialised representation dependency for a component release when a serialised value or entity of the component contains representation supplied by that other component, such as a has-a attribute whose type is implemented there or embedded inherited state whose representation is implemented there.

For each recorded component release, the release entry records the release number of every component that is a serialised representation dependency in that release. The dependency names are fully qualified component names. This lets old data be read deterministically after either the component or a represented dependency has evolved. Components used only for name lookup, constraints, formulas, contracts, or other non-serialised schema logic do not need release numbers in the entry.

Release metadata conforms to the schema evolution model only when the recorded release graph is coherent. Every dependency release number must name a release declared by that dependency component. For a dependency component that appears in more than one release entry, the recorded dependency release numbers must be non-decreasing as the depending component's release number increases. A later component release keeps using the same dependency release or moves to a newer one; it does not revert to an earlier dependency release.

A component with no historical deserialisation contract omits the releases block: its values and entities are not serialised for later reading across releases, and it has no historical serialised representations to describe. Omitting the block means the component declares no release sequence for schema evolution.


component SpaceShuttle;

releases
{
    0 :
    {
        name: "Release 1.0",
        date: (2012,April,18),
        acme.Geometry: 1,
        nasa.CommsModule: 0
    },
    1 :
    {
        name: "Release 1.1",
        date: (2015,July,11),
        acme.Geometry: 2,
        nasa.CommsModule: 3,
        nasa.Scrubber: 2
    },
    2 :
    {
        name: "Release 1.2",
        date: (2015,August,24),
        acme.Geometry: 2,
        nasa.Scrubber: 2
    },
    3 :
    {
        name: "Release 2.0",
        date: (2018,March,6),
        acme.Geometry: 4,
        nasa.Scrubber: 3,
        nasa.Telemetry: 0
    }
}

The manifest records release dependency versions, not the component's ordinary import graph. A source file uses using declarations to introduce component aliases or selected names.

Dependency graph

Definition (release dependency): component C1 is a release dependency of component C2 at release R when C2's release R entry records a release number for C1.

The release dependency graph for a recorded release must be acyclic. Equivalently, no component release directly or indirectly requires itself while interpreting a serialised representation.

The release dependency closure of a component release is the transitive closure of its release dependencies for that release.

Deployment upgrade independence

A component release fixes the releases of its serialised representation dependencies. This mapping is explicit and forms part of the component's release metadata. Release dates are metadata only and do not determine dependency versions.

This allows for:

Incremental deployment: a site running one release of a component does not need to upgrade a dependency unless that component release requires it.
Safe evolution: a dependency can evolve without forcing dependent components to evolve immediately.
Reproducibility: the system can be rebuilt at a historical release with no ambiguity.

Previous116 Name Resolution Next118 Standard Library