mafw.tools.toml_tools

The module provides tools to read / write / modify specific TOML files.

Functions

dump_processor_parameters_to_toml(...)	Dumps a toml file with processor parameters.
generate_steering_file(output_file, processors)	Generates a steering file.
load_steering_file(steering_file[, validate])	Load a steering file for the execution framework.
path_encoder(obj)	Encoder for PathItem.
processor_validator(processors)	Validates that all items in the list are valid processor instances or classes.

Classes

PathItem(t, value, original, trivia)

TOML item representing a Path

class mafw.tools.toml_tools.PathItem(t, value, original, trivia)

Bases: String

TOML item representing a Path

unwrap() → Path

Returns as pure python object (ppo)

mafw.tools.toml_tools._add_db_configuration(database_conf: dict[str, Any] | None = None, db_engine: str = 'sqlite', doc: TOMLDocument | None = None) → TOMLDocument

Add the DB configuration to the TOML document

The expected structure of the database_conf dictionary is one of these two:

option1 = {
    'DBConfiguration': {
        'URL': 'sqlite:///:memory:',
        'pragmas': {
            'journal_mode': 'wal',
            'cache_size': -64000,
            'foreign_keys': 1,
            'synchronous': 0,
        },
    }
}

option2 = {
    'URL': 'sqlite:///:memory:',
    'pragmas': {
        'journal_mode': 'wal',
        'cache_size': -64000,
        'foreign_keys': 1,
        'synchronous': 0,
    },
}

We will always convert the option1 in option2.

Parameters:

• database_conf (dict) – A dictionary with the database configuration. See comments above. If None, then the default is used.

• db_engine (str, Optional) – The database engine. It is used only in case the provided database configuration is invalid to retrieve the default configuration. Defaults to sqlite.

• doc (TOMLDocument, Optional) – The TOML document to add the DB configuration. If None, one will be created.

Returns:

The modified document.

Return type:

TOMLDocument

Raises:

UnknownDBEngine – if the database_conf is invalid and the db_engine is not yet implemented.

mafw.tools.toml_tools.dump_processor_parameters_to_toml(processors: list[ProcessorClassProtocol] | ProcessorClassProtocol, output_file: Path | str) → None

Dumps a toml file with processor parameters.

This helper function can be used when the parameters of one or many processors have to be dumped to a TOML file. For each Processor in the processors a table in the TOML file will be added with their parameters is the shape of parameter name = value.

It must be noted that processors can be:

• a list of processor classes (list[type[Processor]])

• a list of processor instances (list[Processor]])

• one single processor class (type[Processor])

• one single processor instance (Processor)

What value of the parameters will be dumped?

Good question, have a look at this explanation.

param processors:

One or more processors for which the parameters should be dumped.

type processors:

list[type[Processor | Processor]] | type[Processor] | Processor

param output_file:

The name of the output file for the dump.

type output_file:

Path | str

raise KeyAlreadyPresent:

if an attempt to add twice, the same processor is made.

raise TypeError:

if the list contains items different from Processor classes and instances.

mafw.tools.toml_tools.generate_steering_file(output_file: Path | str, processors: list[ProcessorClassProtocol] | ProcessorClassProtocol, database_conf: dict[str, Any] | None = None, db_engine: str = 'sqlite') → None

Generates a steering file.

Parameters:

• output_file (Path | str) – The output filename where the steering file will be save.

• processors (list[type[Processor] | Processor], type[Processor], Processor) – The processors list for which the steering file will be generated.

• database_conf (dict, Optional) – The database configuration dictionary

• db_engine – A string representing the DB engine to be used. Possible values are: sqlite, postgresql and mysql.

Type:

str

mafw.tools.toml_tools.load_steering_file(steering_file: Path | str, validate: bool = True) → dict[str, Any]

Load a steering file for the execution framework.

Changed in version v2.0.0: Introduce support for replica names along with base names in file validation

Parameters:

• steering_file (Path, str) – The path to the steering file.

• validate (bool, Optional) – A flag to validate the content. Defaults to True.

Returns:

The configuration dictionary.

Return type:

dict

Raises:

FileNotFound – if steering_file does not exist.

mafw.tools.toml_tools.path_encoder(obj: Any) → Item

Encoder for PathItem.

mafw.tools.toml_tools.processor_validator(processors: list[ProcessorClassProtocol]) → bool

Validates that all items in the list are valid processor instances or classes.

Parameters:

processors (list[type[Processor] | Processor]) – The list of items to be validated.

Returns:

True if all items are valid.

Return type:

bool