mafw.examples.importer_example

The module provides one concrete implementation of an Importer, as it was used in the autorad paper2.

Classes

ImporterExample(*args, **kwargs)

An exemplary implementation of an importer processor.

InputElement(*args, **kwargs)

A model to store the input elements

Exceptions

InputElementDoesNotExist

Exception raised if the InputElement does not exist
exception mafw.examples.importer_example.InputElementDoesNotExist[source]

Bases: DoesNotExist

Exception raised if the InputElement does not exist

class mafw.examples.importer_example.ImporterExample(*args: Any, **kwargs: Any)[source]

Bases: Importer

An exemplary implementation of an importer processor.

This importer subclass is looking for tif files in the input_folder and using the information stored in the filename, all required database fields will be obtained.

For this importer, we will use a filename parser including two compulsory elements and one optional ones. Those are:

  • Sample name, in the form of sample_xyz, where xyz are three numerical digits (compulsory).

  • Exposure time, in the form of expYh, where Y is the exposure time in hours. It can be an integer number or a floating number using . as decimal separator (compulsory).

  • Resolution, in the form of rXYu, where XY is the readout granularity in micrometer. It is optional and its default is 25 µm if not provided.

This processor is subclassing:

  1. The start(): in order to be sure that the database table is existing.

  2. The get_items(): where the list of input files is retrieved. There we also verify that the table is still updated using the verify_checksum() and in case the user wants to process only new files, we will have to filter out from the list all items already stored in the database.

  3. The process(): where we create a dictionary with the values to be stored in the database. This approach allows a much more efficient database transaction.

  4. The finish(): where we actually do the database insert in a single go.

  5. The format_progress_message(): to keep the user informed about the progress (optional).

Processor parameters

  • input_folder: The input folder from where the images have to be imported. (default: ‘/builds/kada/mafw’)

  • parser_configuration: The path to the TOML file with the filename parser configuration (default: ‘parser_configuration.toml’)

  • recursive: Extend the search to sub-folder (default: True)

Constructor parameters

Parameters:

  • name (str, Optional) – The name of the processor. If None is provided, the class name is used instead. Defaults to None.

  • description (str, Optional) – A short description of the processor task. Defaults to the processor name.

  • config (dict, Optional) – A configuration dictionary for this processor. Defaults to None.

  • looper (LoopType, Optional) – Enumerator to define the looping type. Defaults to LoopType.ForLoop

  • user_interface (UserInterfaceBase, Optional) – A user interface instance to be used by the processor to interact with the user.

  • timer (Timer, Optional) – A timer object to measure process duration.

  • timer_params (dict, Optional) – Parameters for the timer object.

  • database (Database, Optional) – A database instance. Defaults to None.

  • database_conf (dict, Optional) – Configuration for the database. Default to None.

  • remove_orphan_files (bool, Optional) – Boolean flag to remove files on disc without a reference to the database. See Standard tables and _remove_orphan_files(). Defaults to True

  • replica_id (str, Optional) – The replica identifier for the current processor.

  • create_standard_tables (bool, Optional) – Boolean flag to create std tables on disk. Defaults to True

  • kwargs – Keyword arguments that can be used to set processor parameters.

finish() → None[source]

The finish method overload.

Here is where we do the database insert with a on_conflict_replace to cope with the unique constraint.

format_progress_message() → None[source]

Customizes the progress message with information about the current item.

The user can overload this method in order to modify the message being displayed during the process loop with information about the current item.

The user can access the current value, its position in the looping cycle and the total number of items using Processor.item, Processor.i_item and Processor.n_item.

get_items() → Collection[Any][source]

Retrieves the list of element to be imported.

The base folder is provided in the configuration file, along with the recursive flags and all the filter options.

Returns:

The list of items full file names to be processed.

Return type:

list[Path]

process()[source]

The process method overload.

This is where the whole list of files is scanned.

The current item is a filename, so we can feed it directly to the FilenameParser interpret command, to have it parsed. To maximise the efficiency of the database transaction, instead of inserting each file singularly, we are collecting them all in a list and then insert all of them in the finish() method.

In case the parsing is failing, then the element is skipped and an error message is printed.

start() → None[source]

The start method.

The filename parser is ready to use because it has been already configured in the super method. We need to be sure that the input table exists, otherwise we create it from scratch.

class mafw.examples.importer_example.InputElement(*args, **kwargs)[source]

Bases: MAFwBaseModel

A model to store the input elements

DoesNotExist

alias of InputElementDoesNotExist