End-to-end schema migration tutorial

This tutorial walks through a complete, reviewed schema migration from start to finish: you will edit a model, inspect the generated migration script, review it, apply it with the CLI, and rerun the pipeline with every existing record preserved. It complements the conceptual schema migration section in the database chapter.

Added in version 2.4.

We assume a small project with a steering file analysis.toml whose database configuration points at a populated SQLite database and declares a migration directory (a project generated with mafw init already ships a db/migrations folder and a steering file wired to it):

[DBConfiguration]
URL = "sqlite:////data/project.db"
migration_directory = "migrations"

The migration_directory may be relative (resolved against the steering file’s directory) or absolute, and the URL may use ${VAR} environment-variable references.

and a registered model with a single name column:

import peewee
from mafw.db.db_model import MAFwBaseModel


class Sample(MAFwBaseModel):
    name = peewee.CharField()

    class Meta:
        table_name = 'sample'

The sample table already exists and holds experimental rows we must not lose.

Step 1 — Edit a model

Suppose we now want to annotate each sample with an optional note. We add a nullable column to the model (a supported, additive change):

class Sample(MAFwBaseModel):
    name = peewee.CharField()
    note = peewee.CharField(null=True)  # new, nullable

    class Meta:
        table_name = 'sample'

Editing the class alone does not change the live table. If we ran the analysis now against the old schema, the pipeline would be working against a stale table.

Step 2 — Inspect the generated migration script

Run a read-only diff first to confirm what the service detected:

mafw db diff analysis.toml

It reports one supported difference — add_column on sample.note. Now generate the migration script:

mafw db migrate generate analysis.toml

The command prints the absolute path of the generated script, for example /project/migrations/0001_<digest>.py. The database is not touched. If you prefer the processor-driven workflow, adding a SchemaMigrator after TableCreator in the pipeline generates exactly the same script and stops the pipeline before any data-processing step runs.

Step 3 — Review the migration script

Open the generated file. It declares a version, a content checksum, a summary of the change, and a migrate(migrator, database) function:

__migration_version__ = '0001'

# Human-readable summary of the detected change:
#   - add_column on sample.note

def migrate(migrator, database):
    """Apply this migration using the supplied per-backend migrator."""
    operations = []
    # Add the nullable column sample.note.
    operations.append(migrator.add_column('sample', 'note', nullable_field('VARCHAR')))
    if operations:
        migrator.migrate(*operations)

Read the summary and confirm the operation is what you intended. You may edit the file freely — the service applies your edited content verbatim and never regenerates or overwrites it. Commit the script to version control.

Step 4 — Apply it via mafw db migrate up

Check the state, then apply:

mafw db migrate status analysis.toml
mafw db migrate up analysis.toml

status lists 0001 as pending; up applies it inside a transaction (on SQLite and PostgreSQL), records it in the per-database history, and re-checks the schema against the models, reporting that it is now compatible. Your existing rows are preserved: the new note column is simply NULL for every pre-existing row.

Step 5 — Perform a successful rerun

Run the analysis again:

mafw run analysis.toml

This time there is no outstanding schema difference: if the pipeline includes a SchemaMigrator, it detects a clean state and lets the pipeline continue to the data-processing processors. The analysis completes successfully against the migrated schema, and every record created before the migration is still present with its original values.

That is the full reviewed cycle: edit → generate → review → apply → rerun, with your data preserved at every step.

Destructive changes follow the same cycle with one extra review step. If instead of adding a column you had removed one, the generated script would be flagged __migration_requires_review__ = True with the drop rendered as a commented, non-executing stub and a data-loss warning. You would complete the operation by hand, set the flag to False, and then mafw db migrate up would apply it — see Reviewing destructive migrations for the details.