.. _doc_dev_tools: Development tools ================= MAFw ships with a small set of **development tools** that support maintenance tasks such as building versioned documentation trees and preparing tagged releases. These tools are **not** part of the scientific user-facing workflow: a typical MAFw user does not need them, and can safely ignore this section. They exist for the benefit of the development team and for community members who want to contribute to MAFw in a consistent, reproducible way. Because they automate development activities, they may depend on additional Python packages or external executables that are installed only when MAFw is installed with the optional ``[dev]`` feature (or when using the Hatch development environments). How to invoke the tools ----------------------- You can invoke each tool in two equivalent ways: 1. **Console entry point** (from an activated development environment). 2. **Hatch script** (recommended for CI/CD and for reproducible local runs). The table below provides a quick lookup. .. list-table:: Development tools lookup table :header-rows: 1 :widths: 25 25 50 * - Source module - Console entry point - Hatch script (development env) * - ``src/mafw/scripts/doc_versioning.py`` - ``multiversion-doc`` - ``hatch run dev.py3.14:multidoc`` * - ``src/mafw/scripts/release_mgt.py`` (release) - ``release-mgt release`` - ``hatch run dev.py3.14:release`` * - ``src/mafw/scripts/release_mgt.py`` (deps) - ``release-mgt deps`` - ``hatch run dev.py3.14:deps`` .. note:: The Hatch environment name includes the Python version (for example ``dev.py3.14``). On CI, the version is typically driven by the ``PYTHON_VERSION`` variable in ``.gitlab-ci.yml``. doc_versioning.py (``multiversion-doc``) ---------------------------------------- Purpose and idea ^^^^^^^^^^^^^^^^ ``multiversion-doc`` builds the MAFw documentation using Sphinx and produces a directory tree that contains: * One folder per stable tag (for example ``v2.1.0``). * A ``stable`` alias pointing to the latest stable tag. * Optionally a ``dev`` alias for the current branch if it is ahead of ``stable``. * A ``latest`` folder containing a build of the current working tree. This makes it possible to publish *multiple* documentation versions (HTML and optionally PDF) in a single GitLab Pages site, while keeping the workflow reproducible and CI-friendly. Getting help ^^^^^^^^^^^^ To display the full help, use: .. code-block:: console multiversion-doc --help Command structure ^^^^^^^^^^^^^^^^^ The tool is organised into subcommands: .. code-block:: console multiversion-doc ├── current Build only the current working tree documentation (latest/) ├── build Build a multiversion documentation tree (tags + stable/dev) ├── clean Remove output directories (all/ or latest/) ├── prune Remove old versions to respect a size limit ├── redirects Generate GitLab Pages `_redirects` file ├── landing Generate a root `index.html` landing page for Pages ├── registry Interact with the GitLab Generic Package Registry │ ├── upload │ ├── download │ └── delete ├── completion Manage shell completion │ ├── install │ ├── uninstall │ └── show └── server Local HTTP server helper ├── start ├── status ├── stop └── restart Shell completion ^^^^^^^^^^^^^^^^ ``multiversion-doc`` also exposes shell TAB completion through the ``completion`` command group. The feature is available for ``bash``, ``zsh`` and ``fish`` only. Install it in the active virtual environment with: .. code-block:: console multiversion-doc completion install Issuing this command will install the completion in the local virtual environment and will make it available the next time you activate the environment. If you only want to activate directly (but not in a permanent way) the completion functionality, use: .. code-block:: console eval "$(multiversion-doc completion show)" Typical usage examples ^^^^^^^^^^^^^^^^^^^^^^ Build only the current documentation (``current``) """""""""""""""""""""""""""""""""""""""""""""""""" This is the fastest way to validate documentation changes on your current branch. .. code-block:: console multiversion-doc current --help Typical build (HTML only): .. code-block:: console multiversion-doc current -y Rebuild from scratch (recommended when you changed Sphinx configuration, extensions, or generated files): .. code-block:: console multiversion-doc current -y --from-scratch Logical workflow summary 1. Optionally delete the ``docs/build`` output and Sphinx generated trees (``--from-scratch``). 2. Run Sphinx for the current working tree into ``docs/build/doc/latest``. 3. Write a ``sphinx-build.log`` into the output directory for later inspection. Build a multiversion documentation tree (``build``) """"""""""""""""""""""""""""""""""""""""""""""""""" This builds one documentation folder per stable tag and assembles the published tree under ``docs/build/doc``. .. code-block:: console multiversion-doc build --help Typical build (HTML + PDF, keep only versions >= v1.0.0): .. code-block:: console multiversion-doc build --min-vers v1.0.0 --build-pdf Logical workflow summary 1. List stable tags and select the version range (``--min-vers`` and other selectors). 2. For each selected tag, check out the tag (via git worktrees) and run Sphinx. 3. Create ``stable`` (and optionally ``dev``) aliases. 4. Optionally prune old versions if the output exceeds the configured size limit. How documentation is built on CI (``doc_build_all``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In the GitLab pipeline, the multiversion documentation is generated by the ``doc_build_all`` job in ``.gitlab-ci.yml`` using a Hatch script: .. code-block:: console hatch run dev.py${PYTHON_VERSION}:multidoc build \ --min-vers $DOC_MIN_VERS \ --build-pdf \ --max-size $DOC_MAX_SIZE \ --zip-filepath /tmp/zips \ --with-zip-file \ --with-upload-zip \ --with-cached-packages In this workflow: * ``--with-zip-file`` creates a per-tag zip archive of the generated documentation. * ``--with-upload-zip`` uploads those zip files to the GitLab Generic Package Registry (package name ``mafw-docs``). * ``--with-cached-packages`` allows reusing already published zip files instead of rebuilding older tags, making the job faster and more reproducible. Start a local documentation server (``server start``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ After building the documentation locally, you can serve the generated tree over HTTP: .. code-block:: console multiversion-doc server start -d docs/build -p 8000 Then open your browser at ``http://127.0.0.1:8000`` and navigate to the ``doc/`` folder. release_mgt.py (``release-mgt``) --------------------------------- Purpose and idea ^^^^^^^^^^^^^^^^ ``release-mgt`` is a comprehensive maintenance tool that manages the MAFw release lifecycle and ensures dependency compatibility. It is designed to automate repetitive tasks, enforce consistency, and verify the framework against both the latest and the oldest supported dependency stacks. The tool is organised into two main command groups: * **Releases** (``release``): orchestrating version bumps, changelog updates, and tagging. * **Dependencies** (``deps``): freezing upper bounds for reproducibility, verifying compatibility with newest and oldest dependency stacks, and managing reference files in the GitLab Generic Package Registry. Getting help ^^^^^^^^^^^^ To display the top-level help, use: .. code-block:: console release-mgt --help To get help on specific command groups: .. code-block:: console release-mgt release --help release-mgt deps --help Command structure ^^^^^^^^^^^^^^^^^ The tool is organised into subcommands: .. code-block:: console release-mgt ├── release │ └── create Prepare a new tagged release ├── deps │ ├── freeze Freeze dependency upper bounds in pyproject.toml │ ├── unfreeze Remove frozen dependency upper bounds │ ├── latest │ │ └── check Verify compatibility with the newest dependencies │ ├── oldest │ │ └── check Verify compatibility with the oldest supported dependencies │ ├── audit Audit dependency vulnerabilities using pip-audit │ └── registry Interact with the GitLab Generic Package Registry │ ├── upload │ ├── download │ └── delete └── completion Manage shell completion ├── install ├── uninstall └── show Shell completion ^^^^^^^^^^^^^^^^ ``release-mgt`` also exposes shell TAB completion through the ``completion`` command group. The feature is available for ``bash``, ``zsh`` and ``fish`` only. Install it in the active virtual environment with: .. code-block:: console release-mgt completion install Issuing this command will install the completion in the local virtual environment and will make it available the next time you activate the environment. If you only want to activate directly (but not in a permanent way) the completion functionality, use: .. code-block:: console eval "$(release-mgt completion show)" Typical usage examples ^^^^^^^^^^^^^^^^^^^^^^ Prepare a new release (``release create``) """""""""""""""""""""""""""""""""""""""""" The ``release create`` command automates the entire release pipeline. It requires a positional argument ``SEGMENTS`` which is passed to Hatch (e.g., ``minor,rc``, ``rc``, or ``release``). .. code-block:: console release-mgt release create minor,rc --dry-run Logical workflow summary 1. **Safety checks**: Verify the current branch is ``main`` and the working tree is clean. 2. **Freeze**: Rewrite ``pyproject.toml`` to add explicit upper bounds to dependencies (see the "Release reproducibility" section below). 3. **Bump**: Update the project version using ``hatch version``. 4. **Doc target**: Update the documentation target version used by docstring version directives. 5. **Metadata**: Update ``NOTICE.txt`` and ``CHANGELOG.md``. 6. **Requirements**: Update the requirements RST files and ``README.rst`` with the frozen dependencies. 7. **Notes**: Optionally generate a ``release_note_vX.Y.Z.md`` file. 8. **Tag**: Commit the changes (including requirements and ``README.rst``) and create a local git tag. 9. **Unfreeze**: Remove the computed upper bounds from ``pyproject.toml``. 10. **Requirements**: Restore the requirements RST files and ``README.rst`` to open upper bounds. 11. **Commit**: Create a second commit on ``main`` with the unfrozen dependencies and updated requirements. 12. **Push**: Optionally push both commits and the tag to the remote repository. Documentation target version ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The documentation target version is the ``major.minor`` value used by the release workflow to annotate public API docstrings with Sphinx directives such as ``versionadded`` and ``versionchanged``. It is intentionally coarser than the full package version: the release process updates it when the docstring history should advance, and contributors should keep it aligned with the intended documentation milestone rather than the exact patch release number. When preparing a release, the ``release create`` command updates this value automatically in CI or can prompt for a manual confirmation locally. In both cases, the target should never move backward relative to the release being prepared. Verify dependency stacks (``deps latest/oldest check``) """"""""""""""""""""""""""""""""""""""""""""""""""""""" MAFw maintains strict compatibility with a range of dependency versions. These commands use ``uv`` to create isolated environments and run tests. Verify against newest dependencies: .. code-block:: console release-mgt deps latest check Verify against oldest supported dependencies (using ``lowest-direct`` resolution): .. code-block:: console release-mgt deps oldest check Logical workflow summary (``latest check``) 1. Generate a new dependency lock file (``pylock.py3.XX.toml``) using ``uv pip compile``. 2. Compare the new lock file with a reference file (locally or from the GitLab registry). 3. If dependencies have changed, run the test suite and static type checking. 4. Optionally update the reference file in the GitLab registry. Release reproducibility and rolling compatibility ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ MAFw uses open lower bounds during development (for example ``rich>=13.9.4``) to allow working against newer upstream releases. For tagged releases, however, reproducibility matters: users installing the same MAFw version today or months from now should receive a dependency set that is compatible with what was validated at release time. To achieve this, ``release-mgt`` applies a **freeze → release → unfreeze** pattern: 1. **Freeze**: before committing the release artifacts, the script rewrites dependency specifiers in ``pyproject.toml`` by first compiling the project dependencies with ``uv pip compile`` for the supported CPython versions and then adding an explicit upper bound based on the highest resolved version observed for each dependency: - For major-versioned dependencies (``X.*`` with ``X > 0``) the upper bound is ``<(X + 1)``. - For ``0.*`` dependencies the upper bound is ``<0.(minor + 1)``. Dependencies that are already pinned or already have an upper bound are left untouched. 2. **Release**: the release commit (and corresponding tag) includes the frozen ``pyproject.toml``, ensuring that the tagged revision remains reproducible and compatible over time. 3. **Unfreeze**: immediately after creating the tag, the script removes the computed upper bounds and commits the change on ``main`` so development continues against unconstrained newer dependency versions. Manage dependency reference files (``deps registry``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To speed up CI/CD pipelines, ``release-mgt`` can cache dependency reference files in the GitLab Generic Package Registry. This avoids running full test suites if the resolved dependencies haven't changed since the last verification. .. code-block:: console release-mgt deps registry upload --all release-mgt deps registry download --all Vulnerabilities check (``deps audit``) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The fact that you are able to install and execute MAFw with the oldest as well as the newest set of dependencies does not necessarily means that everything is safe. The beauty of open source code is that source codes are constantly audit for new vulnerabilities and those are publicly advertised so that everyone can take action against them. Despite the fact MAFw is based on tools and libraries for which there are rarely severe vulnerabilities that might pose at risk your system and your data, we constantly performed vulnerability scan to assure maximum safety. A scheduled pipeline executed everynight is verifying that there are no known vulnerabilities for all MAFw dependencies for both the oldest and newest ecosystem; if you want to run the audit yourself, just use the following command: .. code-block:: console release-mgt deps audit