MAFw: Modular Analysis Framework

A software library for scientists written by scientists!

Description

Modular Analysis Framework (MAFw) is a python tool to run analytical steps in a consistent manner and to generate suitable outputs in the form of graphs and database tables.

The idea behind MAFw is to offer data scientists a framework where they will be able to implement complex analytical tasks in a well defined environment in a way that they can focus only on the data analysis without bothering with all other ancillary things, like interfaces to database, job submission and so on. In order for this scheme to work, the developer / scientist needs to respect some boundary conditions when drafting the code, but the overall advantage will enormously exceed the small imposed freedom limitations.

The core of MAFw is the Processor, the class that is responsible to perform the analytical task. The Processor I/O is based on a strong collaboration between a relational database structure and files on disc. The use of the database is per se not compulsory, but it will greatly simplify the analysis workflow.

In general the processor is:

• gathering the relevant input from one or more DB tables (location of input files, processing parameters…),

• performing its analytical job,

• and updating a DB output table with the main outcome including the location where output files are saved on disc.

By inheriting from the base Processor class, user developed processors will come with some superpowers, like the ability to exchange data with the database back-end, displaying progress to the user, generating output graphs and so on. The scientist tasks will be limited to the implementation of the analysis code.

Once the data scientists have created their processor libraries, they will be able to chain processors one after the other in a very simple way inside a so-called steering file and MAFw will take care to run them.

Are you ready to embark? Let us start with our tutorial and you will master MAFw in a couple of hours!

This documentation is also available as a PDF file.

Contents:

• First steps
  • Installation
    • Requirements
  • Contributing
  • Testing
  • Citing MAFw
• Introduction
  • Statement of need
  • MAFw conceptual design
  • The way ahead
    • A G- / T- UI for MAFw
    • Parallel processing
    • Introduce interactivity
• Processor: The core of MAFw
  • Execution workflow
    • The single loop execution
    • The for loop execution
    • The while loop execution
    • How to switch from one loop type to another
  • Subclassing
    • List of methods to be overloaded
    • Customize and document you processor
  • Processor parameters
    • Public ActiveParameter and private PassiveParameter
    • Parameter configuration
    • The limitation of ActiveParameter
    • Saving parameter configuration
    • Parameter typing
  • What’s next
• Processor Examples
  • Simple and looping processors
  • Modify the for loop cycle using the LoopingStatus
  • For and while loop execution workflow
  • Importing elements to the database
    • Retrieving information from filenames
    • The basic importer
    • The ImportExample processor
• ProcessorList: Combine your processors in one go
  • The ProcessorExitStatus
  • Resources acquisition and distribution
  • What’s next
• Plugins: import your processors
  • The quick and dirty: code inside MAFw
  • The clean and effective: turn your library into a plugin
  • What’s next
• Run your processors from the command line
  • List the processors
  • Prepare a generic steering file
  • Run the steering file
  • Debugging your processors
  • The exit code
  • Commands abbreviation
  • What’s next
• Database: your new buddy!
  • Database: one name, many implementations
  • Peewee: a simple and small ORM
    • Database drivers
  • One class for each table
  • Ready, go!
    • Configuring other types of databases
  • Triggers: when the database works on its own
    • The signal approach
    • The trigger approach
    • Triggers on different databases
  • Standard tables
  • Custom fields
    • Removing widow rows
    • Pruning orphan files
    • Keeping the entries updated
  • Filters: let us do only what is needed to be done!
    • How to configure a filter
    • Explicit operation configuration
    • How to use a filter
  • Multi-primary key columns
    • Advanced new_only filtering with multiple column primary keys
  • Importing an existing DB
  • What’s next
• Display your results
  • Add diagnostic information to your processors
  • Generate high level graphs
    • Getting the data
    • Plotting the data
    • Mixin approach
• A step by step tutorial for a real experimental case study
  • The experimental scenario
    • Task 0. Generating the data
    • Task 1. Building your data bank
    • Task 2. Do the analysis
    • Task 3. Prepare a relation plot
  • The ‘code’
    • The database definition
    • The processor library
    • The plugin module
  • Run the code!
• API
  • mafw
    • mafw_hookimpl
    • mafw.active
    • mafw.db
    • mafw.decorators
    • mafw.enumerators
    • mafw.examples
    • mafw.hookspecs
    • mafw.mafw_errors
    • mafw.plugin_manager
    • mafw.plugins
    • mafw.processor
    • mafw.processor_library
    • mafw.runner
    • mafw.scripts
    • mafw.timer
    • mafw.tools
    • mafw.ui

Indices and tables

• Index

• Module Index

• Search Page