.. include:: substitutions.rst .. _storage_tutorial: Portable analysis tutorial ========================== This tutorial starts from the same kind of small MAFw project as the :doc:`project tutorial `, but configures it so that the same steering file and database can be used from two computers. The files are still on a filesystem; the database stores portable references to them. You need a shared data folder that both computers can access and one shared database service. If you only run the analysis on one computer, use the simpler physical-path workflow from the project tutorial instead. Create a project with storage enabled ------------------------------------- .. code-block:: console uvx mafw init detector-study --with-storage cd detector-study The generated steering file includes an example logical volume. The generated environment file points that volume to ``run/storage`` so the example is available locally. For a group analysis, replace this example folder with the shared data root on each computer as shown below. Keep large scientific data outside the Git repository. Because the database records each volume's persistent identity, the storage-enabled scaffold uses a file-backed local SQLite database at ``run/analysis.db``. The ordinary scaffold uses an in-memory database to stay completely temporary. For a multi-computer study, replace the SQLite URL with the shared server database URL as in the configuration below. The CLI can also be explicit about the default: .. code-block:: console uvx mafw init detector-study --without-storage This creates a project without a storage section and leaves a comment linking to the storage guide. Configure two computers ----------------------- Assume the study's shared raw files and output folder are mounted at different locations on two laboratory computers: .. list-table:: :header-rows: 1 * - Computer - Raw data folder - Results folder * - PC1 - ``/mnt/lab-a/measurements`` - ``/mnt/lab-a/results`` * - PC2 - ``/shared/measurements`` - ``/shared/results`` Store these non-secret settings in ``conf/detector_study_PC1.env`` and ``conf/detector_study_PC2.env`` respectively: .. code-block:: ini ANALYSIS_RAW_ROOT=/mnt/lab-a/measurements ANALYSIS_RESULTS_ROOT=/mnt/lab-a/results ANALYSIS_DB_URL=postgresql://analysis.example.org:5432/detector_study .. code-block:: ini ANALYSIS_RAW_ROOT=/shared/measurements ANALYSIS_RESULTS_ROOT=/shared/results ANALYSIS_DB_URL=postgresql://analysis.example.org:5432/detector_study In the steering file, refer to one stable name: .. code-block:: toml env_files = ['detector_study.env', 'secrets.env'] [StorageConfiguration.raw] root = "${ANALYSIS_RAW_ROOT}" type = "filesystem" access = "read-only" [StorageConfiguration.results] root = "${ANALYSIS_RESULTS_ROOT}" type = "filesystem" access = "full" [DBConfiguration] URL = "${ANALYSIS_DB_URL}" [DBConfiguration.authentication] method = "env" username = "ANALYSIS_DB_USER" password = "ANALYSIS_DB_PASSWORD" On PC1, select its file with a symbolic link: .. code-block:: console ln -sfn detector_study_PC1.env conf/detector_study.env On PC2, point the same link to that computer's settings: .. code-block:: console ln -sfn detector_study_PC2.env conf/detector_study.env The shared steering file and the two named environment files can be committed to version control. Keep ``conf/secrets.env`` out of version control; it is intended for database passwords and other secrets. If symbolic links are not available on your operating system, use a regular local file with the selected name instead. .. important:: Both roots must lead to the same shared directories on PC1 and PC2, not to separate copies of the data. Keep using the same shared database service as well. MAFw checks a UUID marker in each volume against the database's volume record. Do not copy or recreate either identity independently. Use a portable reference in a processor --------------------------------------- An input parameter can accept either a physical path or a logical reference. For a shared analysis, configure it as a logical reference: .. code-block:: python from pathlib import Path from mafw.decorators import single_loop from mafw.processor import ActiveParameter, InputFilePath, Processor from mafw.storage import LogicalFileReference @single_loop class DetectorAnalysis(Processor): measurement_file: ActiveParameter[InputFilePath | LogicalFileReference] = ActiveParameter( 'measurement_file', default=Path('../data/example.csv'), help_doc='Measurement file to analyse.', ) def process(self) -> None: value = self.measurement_file # Logical references need the initialized StorageService attached # by the application (or supplied by the processor's caller). # MAFw reports a clear error if that service is missing. with self.open_file(value, 'r') as stream: measurement = stream.readline().strip() self.result = float(measurement) Use ``self.open_file`` for normal file reading and writing. If a scientific library needs a filesystem path so it can open the file itself, use ``self.resolve_path(value)`` and pass the resulting path to that library. In the steering GUI, the annotation offers a selector for physical path or logical reference. To select ``raw:run-42/detector-1.csv``, the corresponding TOML value is represented explicitly: .. code-block:: toml [DetectorAnalysis.measurement_file] __param_type__ = "LogicalFileReference" value = "raw:run-42/detector-1.csv" A physical choice remains a regular TOML string such as ``measurement_file = "../data/detector-1.csv"``. Lists may mix both types; the Steering GUI keeps the selection for each list element. Keep portable references in the database ----------------------------------------- When a processor records file locations in your own model, use :class:`~mafw.db.StorageField` or :class:`~mafw.db.StorageListField` if the model should accept both physical and logical values: .. code-block:: python from peewee import FloatField from mafw.db import StorageField from mafw.db.db_model import MAFwBaseModel from mafw.storage import LogicalFileReference class DetectorMeasurement(MAFwBaseModel): source_file = StorageField() value = FloatField() DetectorMeasurement.create( source_file=LogicalFileReference.from_string('raw:run-42/detector-1.csv'), value=3.2, ) A processor that queries ``DetectorMeasurement`` receives either a physical path or a :class:`~mafw.storage.LogicalFileReference` from ``source_file``. Use ``self.open_file(...)`` for normal I/O, or ``self.resolve_path(...)`` when an external library requires a filesystem path. ``StorageField`` keeps the model's database column usable if you later switch between physical and logical file references. Run the same analysis from both computers ----------------------------------------- Keep the steering file and database configuration shared. Before running on each computer, make sure its selected environment file points at the expected mounted folders and the shared database URL. Provide the database username and password in ``secrets.env``, which is excluded from Git. Validate the steering file and run the analysis: .. code-block:: console mafw steering validate -f conf/detector_study_steering.toml mafw run conf/detector_study_steering.toml The database continues to contain ``raw:run-42/detector-1.csv`` after the move. Only the environment values differ: PC1 resolves the ``raw`` volume under ``/mnt/lab-a/measurements`` and PC2 resolves it under ``/shared/measurements``. The database row and steering-file reference are unchanged. For a longer explanation of access policies, volume registration, checksums, file cleanup, and the current filesystem-only backend, see :doc:`How to store and find your files `.