.. include:: substitutions.rst

.. _migration_tutorial:

====================================
End-to-end schema migration tutorial
====================================

This tutorial walks through a complete, reviewed schema migration from start to
finish: you will edit a model, inspect the generated migration script, review it,
apply it with the CLI, and rerun the pipeline with every existing record
preserved. It complements the conceptual :ref:`schema migration section
<schema_migrations>` in the database chapter.

.. versionadded:: 2.4

We assume a small project with a steering file ``analysis.toml`` whose database
configuration points at a populated SQLite database and declares a migration
directory (a project generated with ``mafw init`` already ships a
``db/migrations`` folder and a steering file wired to it):

.. code-block:: toml

   [DBConfiguration]
   URL = "sqlite:////data/project.db"
   migration_directory = "migrations"

The ``migration_directory`` may be relative (resolved against the steering file's
directory) or absolute, and the ``URL`` may use ``${VAR}`` environment-variable
references.

and a registered model with a single ``name`` column:

.. code-block:: python

   import peewee
   from mafw.db.db_model import MAFwBaseModel


   class Sample(MAFwBaseModel):
       name = peewee.CharField()

       class Meta:
           table_name = 'sample'

The ``sample`` table already exists and holds experimental rows we must not lose.

Step 1 — Edit a model
---------------------

Suppose we now want to annotate each sample with an optional note. We add a
**nullable** column to the model (a supported, additive change):

.. code-block:: python

   class Sample(MAFwBaseModel):
       name = peewee.CharField()
       note = peewee.CharField(null=True)  # new, nullable

       class Meta:
           table_name = 'sample'

Editing the class alone does **not** change the live table. If we ran the analysis
now against the old schema, the pipeline would be working against a stale table.

Step 2 — Inspect the generated migration script
-----------------------------------------------

Run a read-only diff first to confirm what the service detected::

    mafw db diff analysis.toml

It reports one supported difference — ``add_column on sample.note``. Now generate
the migration script::

    mafw db migrate generate analysis.toml

The command prints the absolute path of the generated script, for example
``/project/migrations/0001_<digest>.py``. The database is **not** touched. If you
prefer the processor-driven workflow, adding a
:class:`~mafw.processor_library.db_init.SchemaMigrator` after
:class:`~mafw.processor_library.db_init.TableCreator` in the pipeline generates exactly the same
script and stops the pipeline before any data-processing step runs.

Step 3 — Review the migration script
------------------------------------

Open the generated file. It declares a version, a content checksum, a summary of
the change, and a ``migrate(migrator, database)`` function:

.. code-block:: python

   __migration_version__ = '0001'

   # Human-readable summary of the detected change:
   #   - add_column on sample.note

   def migrate(migrator, database):
       """Apply this migration using the supplied per-backend migrator."""
       operations = []
       # Add the nullable column sample.note.
       operations.append(migrator.add_column('sample', 'note', nullable_field('VARCHAR')))
       if operations:
           migrator.migrate(*operations)

Read the summary and confirm the operation is what you intended. You may edit the
file freely — the service applies your edited content verbatim and never
regenerates or overwrites it. Commit the script to version control.

Step 4 — Apply it via ``mafw db migrate up``
--------------------------------------------

Check the state, then apply::

    mafw db migrate status analysis.toml
    mafw db migrate up analysis.toml

``status`` lists ``0001`` as pending; ``up`` applies it inside a transaction (on
SQLite and PostgreSQL), records it in the per-database history, and re-checks the
schema against the models, reporting that it is now compatible. Your existing rows
are preserved: the new ``note`` column is simply ``NULL`` for every pre-existing
row.

Step 5 — Perform a successful rerun
-----------------------------------

Run the analysis again::

    mafw run analysis.toml

This time there is no outstanding schema difference: if the pipeline includes a
:class:`~mafw.processor_library.db_init.SchemaMigrator`, it detects a clean state and lets the
pipeline continue to the data-processing processors. The analysis completes
successfully against the migrated schema, and every record created before the
migration is still present with its original values.

That is the full reviewed cycle: **edit → generate → review → apply → rerun**, with
your data preserved at every step.

Destructive changes follow the same cycle with one extra review step. If instead
of adding a column you had *removed* one, the generated script would be flagged
``__migration_requires_review__ = True`` with the drop rendered as a commented,
non-executing stub and a data-loss warning. You would complete the operation by
hand, set the flag to ``False``, and then ``mafw db migrate up`` would apply it —
see :ref:`Reviewing destructive migrations <schema_migrations>` for the details.