mafw.db.db_filter

Database filter module for MAFW.

This module provides classes and utilities for creating and managing database filters using Peewee ORM. It supports various filtering operations including simple conditions, logical combinations, and conditional filters where one field’s criteria depend on another.

The module implements a flexible filter system that can handle:

Simple field comparisons (equality, inequality, greater/less than, etc.)
Complex logical operations (AND, OR, NOT)
Conditional filters with dependent criteria
Nested logical expressions
Support for various data types and operations

Key components include:

FilterNode: Abstract base class for filter nodes
ConditionNode: Represents individual field conditions
LogicalNode: Combines filter nodes with logical operators
ConditionalNode: Wraps conditional filter conditions
ModelFilter: Main class for building and applying filters to models
ProcessorFilter: Container for multiple model filters in a processor

The module uses a hierarchical approach to build filter expressions that can be converted into Peewee expressions for database queries. It supports both simple and complex filtering scenarios through a combination of direct field conditions and logical expressions.

Changed in version v2.0.0: Major overhaul introducing conditional filters and logical expression support.

Example usage:

from mafw.db.db_filter import ModelFilter

# Create a simple filter
flt = ModelFilter(
    'Processor.__filter__.Model',
    field1='value1',
    field2={'op': 'IN', 'value': [1, 2, 3]},
)

# Bind to a model and generate query
flt.bind(MyModel)
query = MyModel.select().where(flt.filter())

See also

peewee - The underlying ORM library used for database operations

LogicalOp - Logical operation enumerations used in filters

Module Attributes

`Token`	Type definition for a logical expression token
`NameNode`	An atom is a tuple of the literal string 'NAME' and the value
`NotNode`	A NOT node is a tuple of 'NOT' and a recursive node
`BinaryNode`	AND/OR nodes are tuples of the operator and two recursive nodes
`ExprNode`	The main recursive type combining all options
`TOKEN_SPECIFICATION`	Token specifications
`MASTER_RE`	Compiled regular expression to interpret the logical expression grammar

Functions

tokenize(text)

Tokenize a logical expression string into a list of tokens.

Classes

`ConditionNode`(field, operation, value[, name])	Represents a single condition node in a filter expression.
`ConditionalFilterCondition`(condition_field, ...)	Represents a conditional filter where one field's criteria depends on another.
`ConditionalNode`(conditional[, name])	Wraps `ConditionalFilterCondition` behaviour as a `FilterNode`.
`ExprParser`(text)	Recursive descent parser producing a simple Abstract Syntax Tree (AST).
`FilterNode`()	Abstract base for nodes.
`LogicalNode`(op, *children)	Logical combination of child nodes.
`ModelFilter`(name_, **kwargs)	Class to filter rows from a model.
`ProcessorFilter`([data])	A special dictionary to store all `Filters` in a processors.

Exceptions

ParseError

Exception raised when parsing a logical expression fails.

exception mafw.db.db_filter.ParseError[source]

Bases: ValueError

Exception raised when parsing a logical expression fails.

This exception is raised when the parser encounters invalid syntax in a logical expression string.

class mafw.db.db_filter.ConditionNode(field: str | None, operation: LogicalOp | str, value: Any, name: str | None = None)[source]

Bases: FilterNode

Represents a single condition node in a filter expression.

This class encapsulates a single filtering condition that can be applied to a model field. It supports various logical operations through the LogicalOp enumerator or string representations of operations.

Added in version v2.0.0.

Initialize a condition node.

Parameters:

field (str | None) – The name of the field to apply the condition to.
operation (LogicalOp | str) – The logical operation to perform.
value (Any) – The value to compare against.
name (str | None, Optional) – Optional name for this condition node.

to_expression(model: type[Model]) → Expression[source]

Convert this condition node to a Peewee expression.

This method translates the condition represented by this node into a Peewee expression that can be used in database queries.

Parameters:

model (type[Model]) – The model class containing the field to filter.

Returns:

A Peewee expression representing this condition.

Return type:

peewee.Expression

Raises:

RuntimeError – If the node has no field to evaluate.
ValueError – If an unsupported operation is specified.
TypeError – If operation requirements are not met (e.g., IN operation requires list/tuple).

Bases: object

Represents a conditional filter where one field’s criteria depends on another.

This allows expressing logic like: “IF field_a IN [x, y] THEN field_b IN [1, 2] ELSE no constraint on field_b”

Example usage:

# Filter: sample_id in [1,2] if composite_image_id in [100,101]
condition = ConditionalFilterCondition(
    condition_field='composite_image_id',
    condition_op='IN',
    condition_value=[100, 101],
    then_field='sample_id',
    then_op='IN',
    then_value=[1, 2],
)

# This generates:
# WHERE (composite_image_id IN (100, 101) AND sample_id IN (1, 2))
#    OR (composite_image_id NOT IN (100, 101))

Initialise a conditional filter condition.

Parameters:

condition_field (str) – The field to check for the condition
condition_op (str | LogicalOp) – The operation for the condition (e.g., ‘IN’, ‘==’)
condition_value (Any) – The value(s) for the condition
then_field (str) – The field to filter when condition is true
then_op (str | LogicalOp) – The operation to apply when condition is true
then_value (Any) – The value(s) for the then clause
else_field (str | None) – Optional field to filter when condition is false
else_op (str | LogicalOp | None) – Optional operation when condition is false
else_value (Any | None) – Optional value(s) for the else clause
name (str | None, Optional) – The name of this condition. Avoid name clashing with model fields. Defaults to None

to_expression(model: type[Model]) → Expression[source]

Convert this conditional filter to a Peewee expression.

The resulting expression is: (condition AND then_constraint) OR (NOT condition AND else_constraint)

Which logically means:

When condition is true, apply then_constraint
When condition is false, apply else_constraint (or no constraint)

Parameters:: model (type[Model]) – The model class containing the fields
Returns:: A Peewee expression
Return type:: peewee.Expression

class mafw.db.db_filter.ConditionalNode(conditional: ConditionalFilterCondition, name: str | None = None)[source]

Bases: FilterNode

Wraps ConditionalFilterCondition behaviour as a FilterNode.

This class serves as an adapter to integrate conditional filter conditions into the filter node hierarchy, allowing them to be treated uniformly with other filter nodes during expression evaluation.

Added in version v2.0.0.

Initialize a conditional node.

Parameters:

conditional (ConditionalFilterCondition) – The conditional filter condition to wrap
name (str | None, Optional) – Optional name for this conditional node

to_expression(model: type[Model]) → Expression[source]

Convert this conditional node to a Peewee expression.

This method delegates the conversion to the wrapped conditional filter condition’s to_expression() method.

Parameters:: model (type[Model]) – The model class to generate the expression for
Returns:: A Peewee expression representing this conditional node
Return type:: peewee.Expression

class mafw.db.db_filter.ExprParser(text: str)[source]

Bases: object

Recursive descent parser producing a simple Abstract Syntax Tree (AST).

The parser handles logical expressions with the following grammar:

expr    := or_expr
or_expr := and_expr ("OR" and_expr)*
and_expr:= not_expr ("AND" not_expr)*
not_expr:= "NOT" not_expr | atom
atom    := NAME | "(" expr ")"

AST nodes are tuples representing different constructs:

(“NAME”, “token”): A named element (field name or filter name)
(“NOT”, node): A negation operation
(“AND”, left, right): An AND operation between two nodes
(“OR”, left, right): An OR operation between two nodes

Added in version v2.0.0.

Initialize the expression parser with a logical expression string.

Parameters:: text (str) – The logical expression to parse

accept(*kinds: str) → tuple[str, str] | None[source]

Accept and consume the next token if it matches one of the given types.

Parameters:: kinds (str) – Token types to accept
Returns:: The consumed token if matched, otherwise None
Return type:: Token | None

expect(kind: str) → tuple[str, str][source]

Expect and consume a specific token type.

Parameters:: kind (str) – The expected token type
Returns:: The consumed token
Return type:: Token
Raises:: ParseError – If the expected token is not found

parse() → tuple[Literal['NAME'], str] | tuple[Literal['NOT'], ExprNode] | tuple[Literal['AND', 'OR'], ExprNode, ExprNode][source]

Parse the entire logical expression and return the resulting AST.

Returns:: The abstract syntax tree representation of the expression
Return type:: ExprNode
Raises:: ParseError – If the expression is malformed

parse_and() → tuple[Literal['NAME'], str] | tuple[Literal['NOT'], ExprNode] | tuple[Literal['AND', 'OR'], ExprNode, ExprNode][source]

Parse an AND expression.

Returns:: The parsed AND expression AST node
Return type:: ExprNode

parse_atom() → tuple[Literal['NAME'], str] | tuple[Literal['NOT'], ExprNode] | tuple[Literal['AND', 'OR'], ExprNode, ExprNode][source]

Parse an atomic expression (NAME or parenthesised expression).

Returns:: The parsed atomic expression AST node
Return type:: ExprNode
Raises:: ParseError – If an unexpected token is encountered

parse_not() → tuple[Literal['NAME'], str] | tuple[Literal['NOT'], ExprNode] | tuple[Literal['AND', 'OR'], ExprNode, ExprNode][source]

Parse a NOT expression.

Returns:: The parsed NOT expression AST node
Return type:: ExprNode

parse_or() → tuple[Literal['NAME'], str] | tuple[Literal['NOT'], ExprNode] | tuple[Literal['AND', 'OR'], ExprNode, ExprNode][source]

Parse an OR expression.

Returns:: The parsed OR expression AST node
Return type:: ExprNode

peek() → tuple[str, str] | None[source]

Peek at the next token without consuming it.

Returns:: The next token if available, otherwise None
Return type:: Token | None

class mafw.db.db_filter.FilterNode[source]

Bases: object

Abstract base for nodes.

class mafw.db.db_filter.LogicalNode(op: str, *children: FilterNode)[source]

Bases: FilterNode

Logical combination of child nodes.

This class represents logical operations (AND, OR, NOT) applied to filter nodes. It enables building complex filter expressions by combining simpler filter nodes with logical operators.

Added in version v2.0.0.

Initialize a logical node.

Parameters:

op (str) – The logical operation (‘AND’, ‘OR’, ‘NOT’)
children (FilterNode) – Child filter nodes to combine with the logical operation

to_expression(model: type[Model]) → Expression | bool[source]

Convert this logical node to a Peewee expression.

This method evaluates the logical operation on the child nodes and returns the corresponding Peewee expression.

Parameters:: model (type[Model]) – The model class to generate the expression for
Returns:: A Peewee expression representing this logical node
Return type:: peewee.Expression | bool
Raises:: ValueError – If an unknown logical operation is specified

class mafw.db.db_filter.ModelFilter(name_: str, **kwargs: Any)[source]

Bases: object

Class to filter rows from a model.

The filter object can be used to generate a where clause to be applied to Model.select().

The construction of a ModelFilter is normally done via a configuration file using the from_conf() class method. The name of the filter is playing a key role in this. If it follows a dot structure like:

ProcessorName.__filter__.ModelName

then the corresponding table from the TOML configuration object will be used.

For each processor, there might be many Filters, up to one for each Model used to get the input list. If a processor is joining together three Models when performing the input select, there will be up to three Filters collaborating on making the selection.

The filter configuration can contain the following key, value pair:

key / string pairs, where the key is the name of a field in the corresponding Model

key / numeric pairs

key / arrays

key / dict pairs with ‘op’ and ‘value’ keys for explicit operation specification

All fields from the configuration file will be added to the instance namespace, thus accessible with the dot notation. Moreover, the field names and their filter value will be added to a private dictionary to simplify the generation of the filter SQL code.

The user can use the filter object to store selection criteria. He can construct queries using the filter contents in the same way as he could use processor parameters.

If he wants to automatically generate valid filtering expression, he can use the filter() method. In order for this to work, the ModelFilter object be bound to a Model. Without this binding the ModelFilter will not be able to automatically generate expressions.

For each field in the filter, one condition will be generated according to the following scheme:

Filter field type	Logical operation	Example
Numeric, boolean	==	Field == 3.14
String	GLOB	Field GLOB ‘*ree’
List	IN	Field IN [1, 2, 3]
Dict (explicit)	op from dict	Field BIT_AND 5

All conditions will be joined with a AND logic by default, but this can be changed.

The ModelFilter also supports logical expressions to combine multiple filter conditions using AND, OR, and NOT operators. These expressions can reference named filter conditions within the same filter or even combine conditions from different filters when used with ProcessorFilter.

Conditional filters allow expressing logic like: “IF field_a IN [x, y] THEN field_b IN [1, 2] ELSE no constraint on field_b”

Consider the following example:

class MeasModel(MAFwBaseModel):
    meas_id = AutoField(primary_key=True)
    sample_name = TextField()
    successful = BooleanField()
    flags = IntegerField()
    composite_image_id = IntegerField()
    sample_id = IntegerField()


# Traditional simplified usage
flt = ModelFilter(
    'MyProcessor.__filter__.MyModel',
    sample_name='sample_00*',
    meas_id=[1, 2, 3],
    successful=True,
)

# New explicit operation usage
flt = ModelFilter(
    'MyProcessor.__filter__.MyModel',
    sample_name={'op': 'LIKE', 'value': 'sample_00%'},
    flags={'op': 'BIT_AND', 'value': 5},
    meas_id={'op': 'IN', 'value': [1, 2, 3]},
)

# Logical expression usage
flt = ModelFilter(
    'MyProcessor.__filter__.MyModel',
    sample_name={'op': 'LIKE', 'value': 'sample_00%'},
    flags={'op': 'BIT_AND', 'value': 5},
    meas_id={'op': 'IN', 'value': [1, 2, 3]},
    __logic__='sample_name AND (flags OR meas_id)',
)

# Conditional filter usage
flt = ModelFilter(
    'MyProcessor.__filter__.MyModel',
    sample_name='sample_00*',
    composite_image_id=[100, 101],
    sample_id=[1, 2],
    __conditional__=[
        {
            'condition_field': 'composite_image_id',
            'condition_op': 'IN',
            'condition_value': [100, 101],
            'then_field': 'sample_id',
            'then_op': 'IN',
            'then_value': [1, 2],
        }
    ],
)

flt.bind(MeasModel)
filtered_query = MeasModel.select().where(flt.filter())

The explicit operation format allows for bitwise operations and other advanced filtering.

TOML Configuration Examples:

[MyProcessor.__filter__.MyModel]
sample_name = "sample_00*"  # Traditional GLOB
successful = true           # Traditional equality

# Explicit operations
flags = { op = "BIT_AND", value = 5 }
score = { op = ">=", value = 75.0 }
category = { op = "IN", value = ["A", "B", "C"] }
date_range = { op = "BETWEEN", value = ["2024-01-01", "2024-12-31"] }

# Logical expression for combining conditions
__logic__ = "sample_name AND (successful OR flags)"

# Conditional filters
[[MyProcessor.__filter__.MyModel.__conditional__]]
condition_field = "composite_image_id"
condition_op = "IN"
condition_value = [100, 101]
then_field = "sample_id"
then_op = "IN"
then_value = [1, 2]

# Nested conditions with logical expressions
[MyProcessor.__filter__.MyModel.nested_conditions]
__logic__ = "a OR b"
a = { op = "LIKE", value = "test%" }
b = { op = "IN", value = [1, 2, 3] }