Portable analysis tutorial

This tutorial starts from the same kind of small MAFw project as the project tutorial, but configures it so that the same steering file and database can be used from two computers. The files are still on a filesystem; the database stores portable references to them.

You need a shared data folder that both computers can access and one shared database service. If you only run the analysis on one computer, use the simpler physical-path workflow from the project tutorial instead.

Create a project with storage enabled

uvx mafw init detector-study --with-storage
cd detector-study

The generated steering file includes an example logical volume. The generated environment file points that volume to run/storage so the example is available locally. For a group analysis, replace this example folder with the shared data root on each computer as shown below. Keep large scientific data outside the Git repository.

Because the database records each volume’s persistent identity, the storage-enabled scaffold uses a file-backed local SQLite database at run/analysis.db. The ordinary scaffold uses an in-memory database to stay completely temporary. For a multi-computer study, replace the SQLite URL with the shared server database URL as in the configuration below.

The CLI can also be explicit about the default:

uvx mafw init detector-study --without-storage

This creates a project without a storage section and leaves a comment linking to the storage guide.

Configure two computers

Assume the study’s shared raw files and output folder are mounted at different locations on two laboratory computers:

Computer	Raw data folder	Results folder
PC1	`/mnt/lab-a/measurements`	`/mnt/lab-a/results`
PC2	`/shared/measurements`	`/shared/results`

Store these non-secret settings in conf/detector_study_PC1.env and conf/detector_study_PC2.env respectively:

ANALYSIS_RAW_ROOT=/mnt/lab-a/measurements
ANALYSIS_RESULTS_ROOT=/mnt/lab-a/results
ANALYSIS_DB_URL=postgresql://analysis.example.org:5432/detector_study

ANALYSIS_RAW_ROOT=/shared/measurements
ANALYSIS_RESULTS_ROOT=/shared/results
ANALYSIS_DB_URL=postgresql://analysis.example.org:5432/detector_study

In the steering file, refer to one stable name:

env_files = ['detector_study.env', 'secrets.env']

[StorageConfiguration.raw]
root = "${ANALYSIS_RAW_ROOT}"
type = "filesystem"
access = "read-only"

[StorageConfiguration.results]
root = "${ANALYSIS_RESULTS_ROOT}"
type = "filesystem"
access = "full"

[DBConfiguration]
URL = "${ANALYSIS_DB_URL}"

[DBConfiguration.authentication]
method = "env"
username = "ANALYSIS_DB_USER"
password = "ANALYSIS_DB_PASSWORD"

On PC1, select its file with a symbolic link:

ln -sfn detector_study_PC1.env conf/detector_study.env

On PC2, point the same link to that computer’s settings:

ln -sfn detector_study_PC2.env conf/detector_study.env

The shared steering file and the two named environment files can be committed to version control. Keep conf/secrets.env out of version control; it is intended for database passwords and other secrets. If symbolic links are not available on your operating system, use a regular local file with the selected name instead.

Important

Both roots must lead to the same shared directories on PC1 and PC2, not to separate copies of the data. Keep using the same shared database service as well. MAFw checks a UUID marker in each volume against the database’s volume record. Do not copy or recreate either identity independently.

Use a portable reference in a processor

An input parameter can accept either a physical path or a logical reference. For a shared analysis, configure it as a logical reference:

from pathlib import Path

from mafw.decorators import single_loop
from mafw.processor import ActiveParameter, InputFilePath, Processor
from mafw.storage import LogicalFileReference


@single_loop
class DetectorAnalysis(Processor):
    measurement_file: ActiveParameter[InputFilePath | LogicalFileReference] = ActiveParameter(
        'measurement_file',
        default=Path('../data/example.csv'),
        help_doc='Measurement file to analyse.',
    )

    def process(self) -> None:
        value = self.measurement_file
        # Logical references need the initialized StorageService attached
        # by the application (or supplied by the processor's caller).
        # MAFw reports a clear error if that service is missing.
        with self.open_file(value, 'r') as stream:
            measurement = stream.readline().strip()
        self.result = float(measurement)

Use self.open_file for normal file reading and writing. If a scientific library needs a filesystem path so it can open the file itself, use self.resolve_path(value) and pass the resulting path to that library.

In the steering GUI, the annotation offers a selector for physical path or logical reference. To select raw:run-42/detector-1.csv, the corresponding TOML value is represented explicitly:

[DetectorAnalysis.measurement_file]
__param_type__ = "LogicalFileReference"
value = "raw:run-42/detector-1.csv"

A physical choice remains a regular TOML string such as measurement_file = "../data/detector-1.csv". Lists may mix both types; the Steering GUI keeps the selection for each list element.

Keep portable references in the database

When a processor records file locations in your own model, use StorageField or StorageListField if the model should accept both physical and logical values:

from peewee import FloatField

from mafw.db import StorageField
from mafw.db.db_model import MAFwBaseModel
from mafw.storage import LogicalFileReference


class DetectorMeasurement(MAFwBaseModel):
    source_file = StorageField()
    value = FloatField()


DetectorMeasurement.create(
    source_file=LogicalFileReference.from_string('raw:run-42/detector-1.csv'),
    value=3.2,
)

A processor that queries DetectorMeasurement receives either a physical path or a LogicalFileReference from source_file. Use self.open_file(...) for normal I/O, or self.resolve_path(...) when an external library requires a filesystem path. StorageField keeps the model’s database column usable if you later switch between physical and logical file references.

Run the same analysis from both computers

Keep the steering file and database configuration shared. Before running on each computer, make sure its selected environment file points at the expected mounted folders and the shared database URL. Provide the database username and password in secrets.env, which is excluded from Git. Validate the steering file and run the analysis:

mafw steering validate -f conf/detector_study_steering.toml
mafw run conf/detector_study_steering.toml

The database continues to contain raw:run-42/detector-1.csv after the move. Only the environment values differ: PC1 resolves the raw volume under /mnt/lab-a/measurements and PC2 resolves it under /shared/measurements. The database row and steering-file reference are unchanged.

For a longer explanation of access policies, volume registration, checksums, file cleanup, and the current filesystem-only backend, see How to store and find your files.