Schema evolution made easy

Applications age, and with age often comes the desire to change things. One of the more painful aspects of changing a deployed application is transforming existing data. This post shows some of the schema evolution techniques that we have used to update applications using the BKNR datastore.

With traditional database-based persistence solutions, converting existing databases is often done by first exporting the data using database tools or specialized exporter software and then importing it into the new database schema, possibly converting data while reading. Using explicit export and import steps can be tricky and time consuming. The export side often comes for free, but the importer will need to reinstantiate the correct objects in the new schema knowing both the export format and the new schema. Making such importers work reliably is an additional task that has no additional value, and after having converted the existing databases, the code written can typically be thrown away.

A common technique for making schema evolution less painful is to make only additive schema changes. Old information is preserved, and the new code does the conversion either on the fly or in a separate conversion run. This scheme leads to both cruft in the database and in the source code, and requires additional data cleanup steps to finalize the schema upgrade once the application has been migrated to the new code base. Substantial structural changes require more thought in this scheme, in particular if database side integrity constraints need to be considered.

All in all, schema evolution is a messy business.

Recently, we have been working on extending and updating two of the applications that we have developed using the BKNR datastore, and the schema evolution problem occured for us to be solved, too. When designing the store, we felt that schema evolution should be easy. After all, we have all our data in main memory, so we can always load the old data using the old application code, reload the application, and implement methods for the relevant CLOS functions that are called by the object system when classes are changed.

In practice, we found the need to load the old code, the data and then the new code in succession to be inconvenient. Keeping the old code around in the right places would be rather burdensome, and we also had some problems with (our usage of) CLOS in the face of large metadata changes.

As we keep snapshots of our data in a file, we decided that our schema evolution process would work by converting the data into the new schema while reading the snapshot file. Our snapshot file format contains serialized versions of all persistent CLOS objects that constitutes a data store. For each class used in the snapshot, a layout record is present in the snapshot file that contains the name of the class and the names of the slots of the class.

When reading a snapshot file, the object system compares the class layout in the snapshot against the class definitions of the running code. Inconsistencies detected are signalled, and restarts are used to select schema evolution actions.

Currently, we support the following schema evolution steps:

• Class removal: When a layout specifies a class that does not exist in the running application, instances of this class can be ignored.
• Class renames: When a class has been renamed, the new class name can be specified and the reading can be restarted.
• Slot removal: If a slot named in the snapshot file does not exist in the running code, the slot values of the instances can be ignored.

These three options are solely controlled through restarts established while reading the snapshot file. No code needs to be written to do these conversions.

Additionally, we have a slot conversion mechanism: If a slot has been renamed or its semantics have been changed, slot values of previous software versions can be converted. In order to do this, a method on the CONVERT-SLOT-VALUE-WHILE-RESTORING generic function must be defined. It receives the object being restored, the name of the slot as specified in the snapshot layout, and the value to set.

The default method of CONVERT-SLOT-VALUE-WHILE-RESTORING simply assigns the value provided to the slot with the given name. A specialized method could convert the value or assign it to a different slot of the same object. When reading a class layout from the snapshot file that contains an unknown slot name, a restart is offered to call CONVERT-SLOT-VALUE-WHILE-RESTORING for values of this slot.

The beauty of this form of schema evolution is that we can easily load snapshots from our production systems using new code, without any additional required conversion steps. We can regularily verify that our new code works with real data, and can incrementally refine our schema evolution process without having to resort to database tools or external formats that require special handling.

One could certainly improve on this, for example by making some of the evolution steps automatic so that one does not remember what restarts need to be chosen when loading older snapshots. For the moment, these tools serve us well, and it did not take a lot of time (and was fun) to implement them.