Development tools

MAFw ships with a unified development tool called devtools that supports maintenance tasks such as building versioned documentation trees, preparing tagged releases, and verifying dependency compatibility.

These tools are not part of the scientific user-facing workflow: a typical MAFw user does not need them, and can safely ignore this section. They exist for the benefit of the development team and for community members who want to contribute to MAFw in a consistent, reproducible way.

Requirements

The devtools CLI requires additional Python packages that are installed when MAFw is set up with the optional [devtools] or [dev] feature:

pip install mafw[devtools]

Dependency

Minimum supported version

Description

packaging

>=26.2, <27

to parse and compare dependency version specifiers

requests

>=2.34.0, <3

to interact with the GitLab API for registry operations

psutil

>=7.2.2, <8

to manage background processes (documentation server)

ruamel.yaml

>=0.18.14, <0.19

to read and write YAML files preserving comments and formatting (pre-commit config)

How to invoke devtools

The unified tool is available as a console entry point from an activated development environment:

devtools --help

Or via Hatch (recommended for CI/CD and for reproducible local runs):

hatch run dev.py3.14:devtools --help

Note

The Hatch environment name includes the Python version (for example dev.py3.14). On CI, the version is typically driven by the PYTHON_VERSION variable in .gitlab-ci.yml.

Command structure

devtools
├── completion       Manage shell completion
│   ├── install
│   ├── uninstall
│   └── show
├── documentation    Build and manage versioned documentation
│   ├── build          Build multiversion documentation tree
│   ├── current        Build only the current working tree
│   ├── clean          Remove output directories
│   ├── prune          Remove old versions to respect a size limit
│   ├── redirects      Generate GitLab Pages _redirects file
│   ├── landing        Generate root landing page
│   ├── requirements   Generate RST requirement files from pyproject.toml
│   ├── registry       Interact with the GitLab Generic Package Registry
│   │   ├── upload
│   │   ├── download
│   │   └── delete
│   └── server         Local HTTP server helper
│       ├── start
│       ├── status
│       ├── stop
│       └── restart
├── release          Release lifecycle management
│   └── create        Prepare a new tagged release
├── dependencies     Dependency verification and maintenance
│   ├── freeze         Freeze dependency upper bounds in pyproject.toml
│   ├── unfreeze       Remove frozen dependency upper bounds
│   ├── latest         Rolling compatibility for the latest dependencies
│   │   ├── check        Verify compatibility with newest dependencies
│   │   └── compare      Compare dependencies against reference pylock files
│   ├── oldest         Oldest dependency stack verification
│   │   └── check        Verify compatibility with oldest supported dependencies
│   ├── audit          Audit dependency vulnerabilities using pip-audit
│   └── registry       GitLab registry for dependency reference files
│       ├── upload
│       ├── download
│       ├── delete
│       └── prune
└── toolchain         Centralized development tool version management
│   ├── list             Show all managed tools
│   ├── version          Show current versions without online checks
│   ├── check            Compare current vs latest versions
│   ├── update           Bring tools to their latest versions
│   ├── verify           Check configuration consistency
│   └── bootstrap        Install missing host tools
└── python            Manage the Python version policy in [tool.mafw]
    ├── list             Show configured versions and their roles
    ├── add              Add a supported (or --experimental) version
    ├── remove           Remove a supported or experimental version
    ├── default          Change the default development interpreter
    ├── free-threaded    Select the free-threaded test interpreter
    ├── check            Check canonical vs derived consistency
    └── sync             Re-derive configuration from [tool.mafw]

Shell completion

devtools exposes shell TAB completion through the completion command group. The feature is available for bash, zsh, fish and PowerShell.

Changed in version 2.4: Added PowerShell completion support.

Install it in the active virtual environment with:

devtools completion install

If you only want to activate directly (but not permanently) the completion functionality, use:

eval "$(devtools completion show)"

For PowerShell, use:

Invoke-Expression (devtools completion show --shell powershell)

Documentation commands

Purpose and idea

The documentation group builds the MAFw documentation using Sphinx and produces a directory tree that contains:

  • One folder per stable tag (for example v2.1.0).

  • A stable alias pointing to the latest stable tag.

  • Optionally a dev alias for the current branch if it is ahead of stable.

  • A latest folder containing a build of the current working tree.

This makes it possible to publish multiple documentation versions (HTML and optionally PDF) in a single GitLab Pages site, while keeping the workflow reproducible and CI-friendly.

Typical usage examples

Build only the current documentation (current)

This is the fastest way to validate documentation changes on your current branch.

devtools documentation current -y

Rebuild from scratch:

devtools documentation current -y --from-scratch

Build a multiversion documentation tree (build)

devtools documentation build --min-vers v1.0.0 --build-pdf

How documentation is built on CI

In the GitLab pipeline, the multiversion documentation is generated by the doc_build_all job:

hatch run dev.py${PYTHON_VERSION}:devtools documentation build \
  --min-vers $DOC_MIN_VERS \
  --build-pdf \
  --max-size $DOC_MAX_SIZE \
  --zip-filepath /tmp/zips \
  --with-zip-file \
  --with-upload-zip \
  --with-cached-packages

Start a local documentation server

After building the documentation locally, serve the generated tree over HTTP:

devtools documentation server start -d docs/build -p 8000

Even though one can browse directly the generated files in the browser (file:///), serving the pages via a real server assures the proper functioning of all JavaScript and version switching mechanism.

Release commands

Purpose and idea

The release group orchestrates the MAFw release lifecycle: version bumps, changelog updates, and tagging. It enforces consistency and automates repetitive tasks.

Release reproducibility and rolling compatibility

MAFw uses open lower bounds during development (for example rich>=13.9.4) to allow working against newer upstream releases. For tagged releases, however, reproducibility matters.

The release create command applies a freeze → release → unfreeze pattern:

  1. Freeze: rewrite dependency specifiers with explicit upper bounds based on the highest resolved version.

  2. Release: the release commit includes the frozen pyproject.toml.

  3. Unfreeze: remove the computed upper bounds so development continues unrestricted.

Documentation target version

The documentation target version is the major.minor value used by the release workflow to annotate public API docstrings with Sphinx directives such as versionadded and versionchanged.

Prepare a new release (release create)

The release create command automates the entire release pipeline. It requires a positional argument SEGMENTS which is passed to Hatch (e.g., minor,rc, rc, or release).

devtools release create minor,rc --dry-run

Logical workflow summary

  1. Safety checks: Verify the current branch is main and the working tree is clean.

  2. Freeze: Rewrite pyproject.toml to add explicit upper bounds to dependencies.

  3. Bump: Update the project version using hatch version.

  4. Doc target: Update the documentation target version used by docstring version directives.

  5. Metadata: Update NOTICE.txt and CHANGELOG.md.

  6. Requirements: Update the requirements RST files and README.rst with frozen dependencies.

  7. Notes: Optionally generate a release_note_vX.Y.Z.md file.

  8. Tag: Commit the changes and create a local git tag.

  9. Unfreeze: Remove the computed upper bounds from pyproject.toml.

  10. Requirements: Restore the requirements RST files and README.rst to open upper bounds.

  11. Commit: Create a second commit on main with the unfrozen dependencies.

  12. Push: Optionally push both commits and the tag to the remote repository.

Dependencies commands

Purpose and idea

The dependencies group ensures MAFw is compatible with both the latest and oldest supported dependency stacks. It also provides tools for freezing upper bounds, auditing vulnerabilities, and managing reference files in the GitLab Generic Package Registry.

Verify dependency stacks

Verify against newest dependencies:

devtools dependencies latest check

Verify against oldest supported dependencies (using lowest-direct resolution):

devtools dependencies oldest check

Both commands support --preserve-envs to keep hatch environments for local debugging.

Deprecated since version 2.3: The --remove-envs/--no-remove-envs option on dependencies oldest check is deprecated. Use --preserve-envs instead.

Debugging CI resource exhaustion

The nightly scheduled pipelines run dependencies latest check and dependencies oldest check across every supported Python version in a single job. Because each version creates, uses and removes fresh hatch-test and types environments, a job that stalls or is killed by the Kubernetes orchestrator often points to a resource limit on the runner (disk space, inodes, memory or CPU quota) rather than to a genuine dependency incompatibility.

Two controls — both togglable through environment variables so they can be driven from the GitLab scheduled-pipeline variables without editing .gitlab-ci.yml — help diagnose such situations.

Added in version 2.4: The group-level --show-resources option and the MAFW_DEPS_COMPARE_WITH_REF environment variable override.

Group-level --show-resources option

The dependencies group accepts --show-resources/--no-show-resources (default: --no-show-resources), also controlled by the MAFW_DEPS_SHOW_RESOURCES environment variable. When enabled, a best-effort resource snapshot is printed after the unit-test phase, after the static-typing phase, and at the end of each Python version iteration for both the latest and oldest check commands. The snapshot reports:

  • disk usage (df -h) and inode usage (df -i);

  • cgroup v2 memory statistics (memory.current, memory.max, memory.events);

  • cgroup v2 CPU statistics (cpu.stat).

The probes never raise: a missing cgroup pseudo-file (for example on a non-Kubernetes host) is reported inline and does not abort the job.

# Enable explicitly on the command line
devtools dependencies --show-resources latest check

# Or enable through the environment (e.g. a GitLab scheduled-pipeline variable)
MAFW_DEPS_SHOW_RESOURCES=1 devtools dependencies latest check

Forcing every Python version to be tested

By default dependencies latest check skips the test and typing phases for a Python version when its freshly compiled lockfile is identical to the stored reference (--compare-with-ref). This is efficient for routine runs, but it hides the resource-exhaustion scenario on reruns, because already-verified versions are skipped. Set MAFW_DEPS_COMPARE_WITH_REF to a false value (for example 0) to force every version through the full test and typing phases regardless of the reference:

MAFW_DEPS_COMPARE_WITH_REF=0 devtools dependencies latest check

Command-line flags still take precedence over the environment variable, so an explicit --compare-with-ref on the command line overrides MAFW_DEPS_COMPARE_WITH_REF.

Note

When comparison is disabled and --gitlab-ref is not used, the local and registry reference lockfiles are not advanced at the end of a version. This is intentional for a pure debugging run. Keep --gitlab-ref enabled (as the pipeline does) if the references should still be updated.

Compare dependency versions (dependencies latest compare)

The compare subcommand detects ecosystem drift by comparing the latest resolved dependency versions against reference pylock files.

devtools dependencies latest compare
devtools dependencies latest compare -o drift-report.md -f markdown
devtools dependencies latest compare -o drift-report.json -f json
devtools dependencies latest compare --gitlab-ref

Freeze and unfreeze

devtools dependencies freeze
devtools dependencies unfreeze

Manage dependency reference files (dependencies registry)

To speed up CI/CD pipelines, devtools can cache dependency reference files in the GitLab Generic Package Registry.

devtools dependencies registry upload --all
devtools dependencies registry download --all
devtools dependencies registry prune

Added in version 2.3: The dependencies registry prune command.

Vulnerabilities check (dependencies audit)

A scheduled pipeline executed every night verifies that there are no known vulnerabilities for all MAFw dependencies. To run the audit manually:

devtools dependencies audit

Toolchain commands

Purpose and idea

The toolchain group centralizes management of all development tool versions used by MAFw. It covers four main activities:

  • Checking whether tools are at their latest versions.

  • Reporting current tool versions without checking for newer releases.

  • Updating tools following their declared update policy (pipx upgrade for host tools, lower-bound bump for project tools).

  • Verifying configuration consistency across pyproject.toml and .pre-commit-config.yaml.

  • Bootstrapping missing tools for a fresh development environment.

Tools are categorized as either host (installed system-wide via pipx) or project (managed through pyproject.toml). By default, only project tools are processed; host tools are included explicitly via --include-host.

Command tree

devtools toolchain
├── list       – Show all managed tools
├── version    – Show current versions without online checks
├── check      – Compare current vs latest versions
├── update     – Bring tools to their latest versions
├── verify     – Check configuration consistency
└── bootstrap  – Install missing host tools

Group-level options

The toolchain group accepts two mutually exclusive flags that control whether host tools (installed via pipx) participate in subcommand processing:

--include-host

Include host tools (hatch, uv) in the operation. When specified, both host and project tools are processed.

--exclude-host

Explicitly exclude host tools from the operation. This is the default behavior when neither option is provided.

If both --include-host and --exclude-host are specified, the CLI rejects the invocation with an error message.

Subcommand-level filtering options

Each subcommand (list, version, check, update, verify, bootstrap) accepts two repeatable options for filtering individual tools by name:

-i / --include <name>

Process only the named tools. Can be repeated to include multiple tools. Unrecognized names produce a warning but do not abort.

-e / --exclude <name>

Process all applicable tools except the named ones. Can be repeated to exclude multiple tools.

These two options are mutually exclusive: specifying both -i and -e in the same invocation results in an error.

Example — update only ruff and mypy:

devtools toolchain update -i ruff -i mypy

Example — check all project tools except sphinx:

devtools toolchain check -e sphinx

List (toolchain list)

Displays a table of all applicable managed tools showing their name and category.

devtools toolchain list

To include host tools in the listing:

devtools toolchain --include-host list

Check (toolchain check)

Compares the currently configured version of each tool against its latest available version and reports whether tools are in sync. Exits with a non-zero exit code if any tool is out of date or if a version detection error occurred.

devtools toolchain check

Include host tools in the check:

devtools toolchain --include-host check

Version (toolchain version)

Reports the current version for each applicable tool without querying PyPI or checking whether a newer release exists. This makes it suitable for CI jobs that need to record the versions used by the job. Missing versions are shown as N/A; detection errors are shown in the table and cause the command to exit with a non-zero status.

devtools toolchain --include-host version

As with other toolchain commands, host tools can be omitted by leaving out --include-host, and the -i/--include or -e/--exclude options can limit the reported tools.

Update (toolchain update)

Brings tools to their latest versions following each tool’s update policy. After a successful update, a post-update validation hook runs (for example, the full test suite for pytest, or ruff check for ruff). If validation fails, the update is reverted automatically.

devtools toolchain update

Update only a specific tool:

devtools toolchain update -i ruff

Include host tools in the update:

devtools toolchain --include-host update

Verify (toolchain verify)

Checks that all tool configurations are internally consistent. For example, it detects when the ruff version in pyproject.toml differs from the rev field in .pre-commit-config.yaml, or when the pipx-installed uv version does not satisfy the declared constraint in the hatch-uv environment. Exits with code 1 if any inconsistency is found.

devtools toolchain verify

Bootstrap (toolchain bootstrap)

Installs missing tools for a fresh development environment. For host tools, this runs pipx install; project tools that are already declared in pyproject.toml are typically available after a pip install -e .[dev] and are skipped. Exits with a non-zero code if any installation fails.

devtools toolchain --include-host bootstrap

Managed tools

The following table lists all tools managed by the toolchain command group:

Table 1 Managed tools

Tool

Category

Update policy

hatch

host

pipx upgrade

uv

host

pipx upgrade + pyproject.toml hatch-uv env sync

pytest

project

Lower bound bump in test optional-dependencies

mypy

project

Lower bound bump in types env extra-dependencies

sphinx

project

Lower bound bump in doc optional-dependencies

ruff

project

Lower bound bump in dev optional-dependencies + pre-commit rev sync

pre-commit

project

Lower bound bump in dev optional-dependencies

git-cliff

project

Lower bound bump in dev optional-dependencies

pip-audit

project

Lower bound bump in dev optional-dependencies

Python version commands

Purpose and idea

MAFw supports several CPython versions, together with a dedicated free-threaded interpreter for concurrency testing. Historically, the Python version configuration was scattered across many sections of pyproject.toml — package classifiers, the Hatch test/dev/typing matrices, the nogil/gil environments, and the pre-push hooks — so adding, removing, or changing a version meant editing several places by hand and risking inconsistencies.

Added in version 2.4: The python command group centralizes Python version management.

The python group makes the [tool.mafw] section of pyproject.toml the single source of truth and keeps every derived location synchronized with it. The command group manages configuration only: it never installs or removes interpreters, never installs or resolves dependencies, never runs tests, and never edits the GitLab CI configuration.

Canonical configuration

Four keys under [tool.mafw] describe the policy:

[tool.mafw]
supported-python = ["3.11", "3.12", "3.13", "3.14"]
experimental-python = ["3.15", "3.16"]
default-python = "3.14"
free-threaded-python = "3.14t"
supported-python

Stable CPython minor versions officially supported by MAFw. These are advertised in the package classifiers and exercised by the regular test, development, and typing matrices.

experimental-python

Versions under evaluation — prereleases or stable releases whose dependency ecosystem is not yet fully compatible. They are not advertised as supported and are kept out of the mandatory matrices; they live only in a dedicated experimental environment.

default-python

The supported interpreter used for primary development operations such as linting, documentation builds, and static typing. It must be one of the supported versions; an experimental version can never be the default.

free-threaded-python

The independently selected free-threaded interpreter (for example 3.14t) used for concurrency and performance testing. Its base minor version (3.14) must be supported. The default and free-threaded interpreters are chosen independently.

Note

Adding a version records an intent to support or evaluate it. It does not verify that MAFw’s dependencies are compatible with that version — dependency compatibility remains the responsibility of the dependencies command group and the regular CI matrix.

Command tree

devtools python
├── list            – Show configured versions and their roles
├── add             – Add a supported (or --experimental) version
├── remove          – Remove a supported or experimental version
├── default         – Change the default development interpreter
├── free-threaded   – Select the free-threaded test interpreter
├── check           – Check canonical vs derived consistency
└── sync            – Re-derive configuration from [tool.mafw]

All modifying commands (add, remove, default, free-threaded, sync) accept --dry-run, which prints the unified diff of the intended changes without writing any file.

List (python list)

Displays the configured versions and their roles in a table. The command is read-only and does not require any interpreter other than the one running it.

devtools python list

Add (python add)

Adds a stable version to the supported list, synchronizing the classifiers and the mandatory Hatch matrices. If the version is currently experimental, it is promoted (removed from the experimental list). With --experimental the version is registered for evaluation instead, without touching the classifiers or the mandatory matrices.

# Register an upcoming release for experimental evaluation
devtools python add 3.16 --experimental

# Promote a release to officially supported (preview first)
devtools python add 3.15 --dry-run
devtools python add 3.15

Failure conditions. Adding a prerelease (e.g. 3.16a1) as supported is rejected; a supported version cannot also be experimental and vice versa; re-adding an existing version is rejected as a duplicate. In each case the command exits with a non-zero status and leaves pyproject.toml unchanged.

Remove (python remove)

Removes a version from whichever list contains it and resynchronizes the derived configuration.

devtools python remove 3.11

Failure conditions. Removing the current default-python, or the base version of the current free-threaded-python, is rejected until that role is reassigned. Removing a version that is not configured reports that it is not present. The file is left unchanged on any rejection.

Default (python default)

Changes the default development interpreter. Version-specific development commands and the default-dependent pre-push hooks are updated to the new version, while deliberately pinned interpreter selections (for example a pre-push test pinned to 3.12) and the free-threaded interpreter are left untouched.

devtools python default 3.15

Failure conditions. The version must be officially supported; an experimental-only or unknown version is rejected. Setting the current default again is reported as a no-op.

Free-threaded (python free-threaded)

Selects the interpreter used for free-threaded testing. The dedicated free-threaded test matrix, the nogil environment, and the aligned gil performance environment (set to the same minor version) are updated consistently, as are the free-threaded pre-push commands. The default interpreter is left unchanged.

devtools python free-threaded 3.15t

Failure conditions. The value must be an X.Yt selector whose base minor version is officially supported; otherwise the command is rejected and the file is left unchanged.

Check (python check)

Performs a read-only consistency check between the canonical configuration and every derived location, reporting each mismatch with the affected TOML key. It exits with status 0 when the configuration is consistent and a non-zero status otherwise. It installs nothing, resolves no dependencies, runs no tests, and never modifies the file.

devtools python check

Sync (python sync)

Re-derives every managed location from the canonical [tool.mafw] configuration, repairing drift in a single step. It validates the canonical configuration first (an invalid source is never silently repaired), preserves unrelated settings and formatting, and is idempotent.

# Review the corrections first, then apply them
devtools python sync --dry-run
devtools python sync

Worked example: promoting a release

Let’s start from the stable situation in which MAFw supports a range of Python versions (say from 3.11 to 3.14) and the first release of the next python version is been released to that MAFw maintainers and developers can start experiment with it. They can start a new branch where the single source of truth in the pyproject.toml is like this:

[tool.mafw]
supported-python = ["3.11", "3.12", "3.13", "3.14"]
experimental-python = ["3.15"]
default-python = "3.14"
free-threaded-python = "3.14t"

The next step is to add experimental support for 3.15

devtools python add --experimental 3.15

The consequences of this command are mainly two. First of all the new tool.mafw section will read now as:

[tool.mafw]
supported-python = ["3.11", "3.12", "3.13", "3.14"]
experimental-python = ["3.15"]
default-python = "3.14"
free-threaded-python = "3.14t"

And there will be a new experimental environment matrix added to the toml file

[[tool.hatch.envs.experimental.matrix]]
python = ["3.15"]

It is worth to remind that this operation did not install any Python 3.15 distribution nor created any environment with MAFw installed. It has just adapted the pyprojecto.toml file.

The next step for the maintainer / developer is to add one features at the time in the experimental environment to test for example that all dependencies are being installed correctly and that the whole test suite is successfully executed.

When all those steps are accomplished, then 3.15 can be promoted from experimental to officially supported.

Promote it and verify:

devtools python add 3.15
devtools python check

Resulting [tool.mafw]:

[tool.mafw]
supported-python = ["3.11", "3.12", "3.13", "3.14", "3.15"]
experimental-python = []
default-python = "3.14"
free-threaded-python = "3.14t"

The classifiers and the standard, development, and typing matrices now include 3.15, while the default interpreter and the free-threaded interpreter are deliberately left unchanged. It is important to repeat once again that adding a python version to the supported list does not mean that it has been tested. The CLI tool is just adapting the pyproject.toml description, tests and verifications are under the responsibility of the maintainer / developer working on this.

Making 3.15 the default is a separate, explicit step:

devtools python default 3.15