How to store and find your files

MAFw keeps the contents of your scientific files on the filesystem. The database stores the information needed to find those files and relate them to the results of your analysis. This keeps database tables small while allowing processors to reuse earlier measurements and outputs.

There are two good ways to describe a file. Use a physical path for a simple analysis that stays on one computer. Use a logical reference when the same analysis and database must work on several computers. You can use either approach in your own processors; MAFw also allows the two approaches to coexist.

Choose the workflow that fits your study

	Local workflow	Portable workflow
File name in the database	Physical path, such as ../data/sample.csv	Logical reference, such as raw:sample.csv
Data location	A folder on one computer	A shared filesystem available to each participating computer
Database	Usually a local SQLite file	One shared database service when computers share the same analysis
Setup	Minimal	Configure storage volumes and local environment files
Moving the analysis	Paths may need updating or data may need to be copied to the same path	Keep the database and steering file; configure each computer’s local mount point

If you are learning MAFw or working on one computer, start with physical paths. If a group needs to move the same analysis between computers without rewriting file references, use logical storage. A logical reference does not copy or synchronize data: the computers still need access to the same shared files and, when sharing one analysis database, to the same database service.

Local workflow: physical paths

Physical paths are ordinary pathlib.Path values. They are easy to understand and are a good choice for a first project, a laptop analysis, or a workflow whose data will remain in one directory structure.

An example database model stores the file path and its checksum:

from mafw.db.db_model import MAFwBaseModel
from mafw.db.fields import FileChecksumField, FileNameField


class Measurement(MAFwBaseModel):
    filename = FileNameField(checksum_field='checksum')
    checksum = FileChecksumField()

When MAFw runs this analysis, the database row contains the physical path. If the project is at /work/study and the steering file is /work/study/conf/run.toml, a processor parameter set to ../data/sample.csv resolves to /work/study/data/sample.csv.

This is convenient, but the database remembers that computer’s path. If the same data live at /srv/lab/study-data on another computer, the old path does not automatically point there. You can update the database paths, copy the data to the same directory layout, or use the portable workflow below.

Portable workflow: logical references

A logical reference has a volume name and a relative file name. For example, raw:run-42/sample.csv means “the file run-42/sample.csv inside the volume named raw.” It does not include the computer-specific location of that volume.

Your database model can store physical paths and logical references using the same recommended field:

from pathlib import Path

from mafw.db import StorageField
from mafw.db.db_model import MAFwBaseModel


class Measurement(MAFwBaseModel):
    source_file = StorageField()

Measurement.create(source_file=Path('/data/measurements/sample.csv'))

For an explicit logical reference object, use:

from mafw.storage import LogicalFileReference

source = LogicalFileReference.from_string('raw:run-42/sample.csv')
Measurement.create(source_file=source)

MAFw’s standard PlotterOutput and OrphanFile tables also use the self-describing StorageListField, so their database schema can represent either kind of file reference.

Technical detail: one database field for both kinds of path

StorageField stores physical paths and logical references in one text column. Ordinary physical paths keep their existing text form; the field adds a marker when needed to distinguish a logical reference from a physical path. You do not need to read or edit this stored marker. This lets a model keep the same database column if you later change its application workflow from physical paths to logical references. See custom fields for the field choices and checksum behavior.

A Python string assigned to StorageField is a physical path. Assign a LogicalFileReference object to store a logical reference.

Configure logical volumes

Add a StorageConfiguration section to the steering file. Each named entry is a logical volume. Its root is the local filesystem folder that contains the volume’s files; use an environment variable so that the steering file can be shared while each computer supplies its own root.

env_files = ['my_analysis.env', 'secrets.env']

[DBConfiguration]
URL = "postgresql://analysis-server.example.org:5432/study"

[StorageConfiguration.raw]
root = "${ANALYSIS_RAW_ROOT}"
type = "filesystem"
access = "read-only"

[StorageConfiguration.results]
root = "${ANALYSIS_RESULTS_ROOT}"
type = "filesystem"
access = "full"

Here raw and results are the volume names used in logical references. The raw data volume is read-only because this analysis only reads those files; the results volume allows writing and MAFw-managed deletion. Use the least permissive access that suits the work. See Access policies and safe cleanup for the exact behavior.

The storage service uses the database to register and validate the identity of each volume. A database section and a working database connection are therefore required whenever storage is enabled. MAFw creates the StorageVolume standard table as part of its normal database setup.

Keep one environment file for each computer

The steering file can name a stable environment file, while each computer selects its own version through a local symbolic link. Keep the environment files in the project because they contain non-secret path settings; do not put passwords or access tokens in them.

For example, commit these files:

conf/my_analysis_PC1.env:

ANALYSIS_RAW_ROOT=/mnt/lab-a/measurements
ANALYSIS_RESULTS_ROOT=/mnt/lab-a/analysis-results

conf/my_analysis_PC2.env:

ANALYSIS_RAW_ROOT=/shared/measurements
ANALYSIS_RESULTS_ROOT=/shared/analysis-results

The steering file refers to my_analysis.env. On computer 1, select its settings with:

ln -sfn my_analysis_PC1.env conf/my_analysis.env

On computer 2, use:

ln -sfn my_analysis_PC2.env conf/my_analysis.env

The link is relative to conf/ and is not itself part of the shared steering file. On systems where symbolic links are unavailable or restricted, put the selected values in a regular conf/my_analysis.env file instead. The env_files setting is resolved relative to the steering file. In case you have forgotten, here is how to use environmental variables in steering files and how to use dotenv files.

The root paths must already exist and identify the intended data folders. On all computers, the logical volume must lead to the same shared data and carry the same MAFw UUID marker. Do not copy a database to create a second identity for the same shared volume; coordinate with your database administrator before replacing either the database or the marker.

In the Steering GUI, select the Storage section in the left-hand tree to enable storage and add or edit volumes. The GUI writes the same StorageConfiguration section; it does not replace the need to provide the local environment values. See the Steering GUI guide.

Access policies and safe cleanup

Each volume has one access policy:

• read-only allows reads through the storage service. Writes and MAFw-managed deletion are refused.

• protected allows reads and writes through the storage service, but MAFw-managed deletion is refused.

• full allows reads, writes, and MAFw-managed deletion.

These policies protect operations performed through MAFw’s storage service; they are not operating-system permissions or a sandbox. A person or program with direct filesystem access can still change the files. Ask your system administrator to set appropriate filesystem permissions for shared data.

MAFw’s orphan cleanup is designed to remove files after their database rows are deleted and the file-removal trigger has queued the references. Cleanup runs during a later processor execution. Physical paths are removed directly; logical references are deleted through the storage service. If a logical reference belongs to a protected or read-only volume, deletion is refused and the queue entry remains for a later retry. Do not enable automatic file-removal triggers on tables where deleting a database row must not lead to deletion of its files.

For processor code, open_file() enforces the configured policy when it opens a logical reference. If a processor resolves a reference to a pathlib.Path and then opens that path directly, it bypasses this policy check. Use direct paths only when that is intentional and appropriate for your analysis.

Use file references in a processor

Use the same processor pattern for a physical path or a logical reference. MAFw’s resolve_path() resolves a known file value to the location used on the current computer. It expands environment variables and resolves relative physical paths from the steering file; logical references are resolved through the configured storage service. The configured value remains unchanged in the steering file.

For ordinary file reading and writing, use open_file(). It accepts either kind of value: physical paths are resolved from the steering file, while logical references are opened through the storage service so their volume policy is enforced.

from pathlib import Path

from mafw.decorators import single_loop
from mafw.processor import ActiveParameter, InputFilePath, Processor
from mafw.storage import LogicalFileReference
from mafw.tools.path_resolution import PathValidationPolicy


@single_loop
class ReadMeasurements(Processor):
    input_file: ActiveParameter[InputFilePath | LogicalFileReference] = ActiveParameter(
        'input_file',
        default=Path('../data/sample.csv'),
        help_doc='Physical path or logical reference to the input measurement.',
    )

    def process(self) -> None:
        value = self.input_file
        # A logical reference needs the initialized StorageService supplied
        # by the application (or by the code that constructs this processor).
        # MAFw reports a clear error if that service is missing.
        with self.open_file(value, validation=PathValidationPolicy.EXISTING_FILE) as stream:
            first_line = stream.readline()
        self.result = first_line

When an external library needs a filesystem path instead of a file object, call self.resolve_path(value) and pass the returned pathlib.Path to that library. Path-based access is useful for libraries that open files themselves, but it does not enforce the storage volume’s access policy.

The annotation lets the Steering GUI offer physical and logical choices. A physical value is an ordinary TOML string. A logical value is saved with an explicit type marker, so a value such as raw:sample.csv is not confused with a physical path. Lists can mix both alternatives and retain the selected type of each element. open_file handles either value at runtime; a logical value requires an initialized storage service from the application or caller.

[ReadMeasurements]
input_file = "../data/sample.csv"

For a logical choice, the Steering GUI writes an explicitly typed table:

[ReadMeasurements.input_file]
__param_type__ = "LogicalFileReference"
value = "raw:run-42/sample.csv"

For a processor that reads file references from a database, use StorageField in the model and query the rows in get_items. The field returns either a pathlib.Path or a LogicalFileReference, so a processor can apply the same resolution pattern to values from parameters or database records.

from mafw.db import StorageField
from mafw.db.db_model import MAFwBaseModel
from mafw.processor import Processor


class MeasurementFile(MAFwBaseModel):
    source_file = StorageField()


class ProcessStoredMeasurements(Processor):
    def get_items(self):
        return MeasurementFile.select()

    def process(self):
        path = self.resolve_path(self.item.source_file)
        # Continue with the same analysis for either stored reference type.
        self.result = path.stat().st_size

If your processor uses resolve_path and then performs direct path operations on a logical value, remember that it is no longer using open_file’s access-policy check. Choose the method that matches the safety policy of your workflow.

Filesystem paths in steering files

Physical filesystem paths and logical references follow different rules:

• A physical relative path is based on the folder containing the steering file. For /work/study/conf/run.toml, ../data/sample.csv means /work/study/data/sample.csv even if you run MAFw from another terminal folder.

• Environment variables are expanded at runtime. Use ${DATA_ROOT}/sample.csv for a required variable or ${CACHE_DIR:-/tmp/mafw-cache} for a fallback value.

• A logical reference such as raw:run-42/sample.csv is relative to its configured volume. The volume root is resolved using the steering file’s location and environment mapping.

• MAFw keeps the configured text as written; resolving it does not rewrite the steering file or turn the configured parameter into an absolute path.

The Steering GUI’s path role controls which browse dialog is offered. It does not itself resolve the path or confirm that an input exists. A processor uses self.open_file(...) for ordinary file access, or self.resolve_path(...) when a library needs a filesystem path. Save As to another folder can change the effective location of a relative physical path; logical references continue to use their volume name and relative path, but still need a valid local volume configuration.

The path resolver accepts filesystem paths, not database URLs or glob patterns. Resolve a literal folder first, then use Path.glob to search within it. For paths in a secondary configuration file, resolve that file first and then use its location as the context for its own relative paths.

Technical detail: explicit path contexts

A helper outside a Processor can use resolve_path() with an explicit PathResolutionContext. A processor already receives the context of its steering file and should normally use self.resolve_path. The standalone helper does not guess a base folder.

Logical reference rules

The volume name is case-sensitive. It must be a non-empty ASCII name using letters, numbers, underscores, or hyphens, for example raw or analysis-1. A colon or semicolon is not allowed in a volume name.

The serialized form is <volume>:<relative/path>. Parsing splits at the first colon, which separates the volume name from the path. The path uses portable POSIX forward slashes, even when the analysis runs on Windows. It must be relative, cannot contain a Windows drive or UNC prefix, and cannot contain a colon, semicolon, or any .. parent-directory component.

Accepted	Rejected
raw:run-42/sample.csv	raw:/sample.csv (absolute path)
raw:folder/./sample.csv normalizes to raw:folder/sample.csv	raw:../sample.csv (parent traversal)
raw:folder//sample.csv normalizes repeated separators	raw:folder\\sample.csv (backslash separator)
raw:sample.csv	raw:folder;sample.csv (semicolon is reserved for list storage)

Constructing a LogicalFileReference checks this syntax; it does not check whether the named volume is configured or whether the file exists. Those checks happen when the storage service is initialized or the reference is resolved.

Technical detail: field values and checksums

LogicalFileReferenceField stores one logical reference as text and preserves None as SQL NULL. LogicalFileReferenceListField stores references separated by semicolons, preserves order and duplicates, and maps an empty or null list to []. Empty members are invalid. The checksum_field option records an association; assigning a logical reference does not calculate a checksum. Use a checksum calculated from a resolved physical file when your analysis requires one.

Checksums and file cleanup

Checksums let MAFw or your processor detect whether a physical file has changed. A FileNameField can calculate and save a checksum through its checksum_field association. A logical reference names a file but does not itself open or read that file; calculate its checksum only after resolving the reference to the current physical file.

When a row containing registered file fields is deleted, MAFw can queue those references in the standard OrphanFile table. A later processor execution attempts cleanup. Failed logical deletions remain queued, for example when the volume is not configured or its policy does not allow deletion. This is a cleanup aid, not a substitute for backups or a confirmation step: enable file removal triggers only when the table’s files are safe to remove with their rows.

The physical-only FileNameField and FileNameListField remain supported for existing analyses. You can adopt logical references incrementally. For a model that should support both from the start, use StorageField or StorageListField.

Current capabilities and limitations

The filesystem is the storage backend currently implemented by MAFw. The logical-reference and storage-service APIs provide a consistent way to name and access files, but an object-store backend such as S3 is not currently available. Keep shared storage mounted and accessible on every participating computer, and ask your local IT team about backup, permissions, and availability.

SQLite is a good local database for one computer. For several computers sharing one evolving analysis, use one supported server database service and coordinate its backups and access with your IT team. A logical reference does not make a local SQLite database shared, and it does not synchronize files.

For a complete worked example, continue to the portable analysis tutorial. The original single-computer project tutorial remains a good first example with physical paths.

What’s next

With file locations configured, continue to database filters to learn how to select only the records that need to be analysed. The Steering GUI guide explains how to edit storage volumes and other analysis settings graphically.