Development introduction#
If you are interested in learning about how the NexusLIMS back-end works or adding new features, these instructions should get you up and running with a development environment that will allow you to modify how the code operates.
Currently, running the NexusLIMS record building back-end code is supported and tested on Linux and MacOS. It does not currently work on Windows due to some specific implementation choices (although contributions would be accepted for this if it’s an important platform for your use case).
Installation#
NexusLIMS uses the uv framework
to manage dependencies and create reproducible deployments. This means that
installing the nexusLIMS package will require
installing
uv. uv will automatically manage Python versions for you, so you don’t need
to install Python separately. NexusLIMS requires Python 3.11 or 3.12.
Installing uv is usually as simple as:
curl -LsSf https://astral.sh/uv/install.sh | sh
Danger
Never run code downloaded straight from the internet just because someone tells you to! Make sure to download and inspect the script to ensure it doesn’t do anything nasty before you run it.
Once uv is installed, clone the NexusLIMS repository using git,
and then change to the root folder of the repository. Running the following
sync command will make uv install the dependencies specified
in the uv.lock file into a local python virtual environment (so they
don’t interfere with other Python installations on your system). It will also
install the nexusLIMS Python package in “editable” mode so you can make
changes locally for development purposes and still run things.
$ uv sync
Note
uv automatically creates virtual environments in a local .venv folder
by default, keeping all project files in one directory. This eliminates the need
for additional configuration that was required with other package managers.
To see information about your current environment, you can run
$ uv python list to see available Python versions or $ uv pip list
to see installed packages. This approach keeps the project
files all in one directory, rather than having the Python virtualenv in your
user folder somewhere else on the system. uv manages this automatically
without requiring additional configuration.
Setting up the environment#
Note
For Development: If you’re just developing NexusLIMS and running the test suite, you do not need a full production environment. The integration tests include dockerized versions of all external dependencies (NEMO, CDCS, etc.), so you can develop and test without real credentials or infrastructure.
Production Environment Setup#
To interact with the remote systems from which NexusLIMS harvests information in a production deployment,
it is necessary to provide credentials for authentication and the paths in which
to search for new data files and where to write dataset previews, as well as
the path to the NexusLIMS database.
These values should be set by copying the .env.example file from the git
repository into a file named .env in the base directory (in the same folder
as the README.md and pyproject.toml files).
nexusLIMS makes use of the
dotenv library to dynamically
load variables from this file when running any nexusLIMS code.
As an example, the .env file content should look something like the
following (substituting real credentials, of course). See Configuration
for a complete reference of all environment variables.
NX_CDCS_TOKEN='your-api-token-here'
NX_INSTRUMENT_DATA_PATH='/path/to/mmfnexus/mount'
NX_DATA_PATH='/path/to/nexusLIMS/mount/mmfnexus'
NX_DB_PATH='/path/to/nexusLIMS/nexuslims_db.sqlite'
NX_NEMO_ADDRESS_1='https://path.to.nemo.com/api/'
NX_NEMO_TOKEN_1='authentication_token'
NX_NEMO_STRFTIME_FMT_1="%Y-%m-%dT%H:%M:%S%z"
NX_NEMO_STRPTIME_FMT_1="%Y-%m-%dT%H:%M:%S%z"
NX_NEMO_TZ_1="America/New_York"
Development Environment#
For development and testing, the test suite handles environment setup automatically:
Unit tests mock external dependencies and don’t require any configuration
Integration tests use Docker Compose to spin up temporary instances of NEMO, CDCS, and other services
All test fixtures create isolated temporary directories and databases
Simply run:
./scripts/run_tests.sh
No .env file or production credentials required!
Getting into the environment#
Once the package is installed using uv, the code can be used
like any other Python library within the resulting virtual environment.
uv allows you to run a single command inside that environment by
using the uv run command from the repository:
$ uv run python
To use other commands in the NexusLIMS environment, you can also “activate”
the environment using the $ source .venv/bin/activate command from within the cloned
repository. This will activate the virtual environment that ensures all commands will have
access to the installed packages and environment variables set appropriately.
Pre-commit Hooks#
To ensure code quality and consistency, NexusLIMS uses pre-commit hooks that automatically run linting checks before each commit. This prevents commits with linting errors from being pushed to the repository.
Installing Pre-commit Hooks#
Pre-commit is included as a dev dependency, so it’s already installed when you run uv sync. To set up the git hooks:
$ uv run pre-commit install
From now on, linting checks will run automatically whenever you attempt to commit code. If linting errors are found, the commit will be blocked until you fix them.
Running Pre-commit Manually#
To run linting checks on all files without committing:
$ uv run pre-commit run --all-files
To run checks on specific files:
$ uv run pre-commit run --files <file1> <file2>
Bypassing Pre-commit (Not Recommended)#
If you need to bypass the pre-commit hooks in exceptional cases:
$ git commit --no-verify
Warning
Only use --no-verify in exceptional cases. Pre-commit hooks exist to maintain code quality standards across the project.
Testing and Documentation#
Unit Testing#
To run the complete unit test suite, use the provided shell script:
$ ./scripts/run_tests.sh
This will run all unit tests with coverage reporting. To run specific tests:
$ uv run pytest tests/unit/test_extractors.py
To run a specific test:
$ uv run pytest tests/unit/test_extractors.py::TestClassName::test_method_name
To generate matplotlib baseline figures for image comparison tests, run:
$ ./scripts/generate_mpl_baseline.sh
Integration Testing#
Integration tests validate end-to-end workflows using real Docker services (NEMO, CDCS, PostgreSQL, etc.) instead of mocks. These tests ensure the complete system works together correctly.
# run unit and integration tests together using the included script (requires Docker)
$ ./scripts/run_tests.sh --integration
# Run integration tests (requires Docker)
$ uv run pytest tests/integration/ -v
# Run with coverage
$ uv run pytest tests/integration/ -v --cov=nexusLIMS
For detailed information about integration testing, setup, architecture, and best practices, see the Integration Testing Guide.
Note: Integration tests require Docker and Docker Compose to be installed and running. Unit tests do not require Docker and can be run independently.
Documentation#
To build the documentation for the project, run:
$ ./scripts/build_docs.sh
The documentation will then be present in the ./_build/ directory.
Building new records#
The most basic feature of the NexusLIMS back-end is to check the
database for any logs (typically inserted by the
NEMO harvester) with a status of
'TO_BE_BUILT'. This can be accomplished simply by running the
record_builder module directly via:
$ uv run nexuslims build-records
This command will find any records that need to be built, build their .xml files, and then upload them to the front-end CDCS instance. Consult the record building documentation for more details.
Using other features of the library#
Once you are in a python interpreter (such as python, ipython,
jupyter, etc.) from the uv environment, you can access the
code of this library through the nexusLIMS package if you want to do other
tasks, such as extracting metadata or building previews images, etc.
For example, to extract the metadata from a .tif file saved on the
FEI Quanta, run the following code using the
get_quanta_metadata() function:
from nexusLIMS.extractors.plugins.quanta_tif import get_quanta_metadata
meta = get_quanta_metadata("path_to_file.tif")
The meta variable will then contain a dictionary with the extracted
metadata from the file.
Contributing#
AI Policy#
AI-assisted development is allowed and used extensively in NexusLIMS itself. We find it a useful tool when applied with care and judgment. That said, a few expectations apply to AI-assisted contributions:
Disclose AI use in your pull request description. A brief note is fine — just be honest about which parts were AI-assisted.
You are responsible for the code you submit. AI output must be reviewed, understood, and tested by you before submitting. The same code review standards apply regardless of how the code was written.
Automated contributions are not accepted. Pull requests generated and submitted by an AI agent without meaningful human involvement will be closed. A human must be the author and must stand behind the contribution.
How to contribute#
To contribute, please fork the repository, develop your addition on a
feature branch
within your forked repo, and submit a pull request to the main
branch to have it included in the project. Contributing to the package
requires that every line of code is covered by a test case. This project uses
testing through the pytest library,
and features that do not pass the test cases or decrease coverage will not be
accepted until suitable tests are included (see the tests directory
for examples) and that the coverage of any new features is 100%.
For example, the following output is shown when running the full test suite, which will highlight any lines that are not covered:
$ cd <path_to_repo>
$ uv run ./scripts/run_tests.sh --integration
Running unit and integration tests...
====================================== test session starts ===========================================
platform darwin -- Python 3.11.14, pytest-9.0.1, pluggy-1.6.0
Matplotlib: 3.10.7
Freetype: 2.6.1
rootdir: /Users/josh/git_repos/datasophos/NexusLIMS
configfile: pyproject.toml
plugins: docker-3.2.5, timeout-2.4.0, mpl-0.17.0, cov-7.0.0
collected 993 items
tests/integration/test_cdcs_integration.py ..................................... [ 3%]
tests/integration/test_cli.py .......... [ 4%]
tests/integration/test_end_to_end_workflow.py .... [ 5%]
tests/integration/test_fileserver.py .... [ 5%]
tests/integration/test_fixtures_smoke.py ............ [ 6%]
tests/integration/test_nemo_integration.py ....................................... [ 10%]
tests/integration/test_nemo_multi_instance.py ................ [ 12%]
tests/integration/test_partial_failure_recovery.py ..... [ 12%]
tests/unit/cli/test_process_records.py ............................ [ 15%]
tests/unit/test___init__.py ........ [ 16%]
tests/unit/test___main__.py . [ 16%]
tests/unit/test_cdcs.py ............. [ 17%]
tests/unit/test_config.py ................... [ 19%]
tests/unit/test_extractors/test_basic_metadata.py .. [ 19%]
tests/unit/test_extractors/test_digital_micrograph.py ................................ [ 23%]
tests/unit/test_extractors/test_edax.py ... [ 23%]
tests/unit/test_extractors/test_extractor_module.py .......................................... [ 27%]
tests/unit/test_extractors/test_fei_emi.py ......................... [ 30%]
tests/unit/test_extractors/test_instrument_profile_modules.py ................... [ 32%]
tests/unit/test_extractors/test_orion_HIM.py ................................ [ 35%]
tests/unit/test_extractors/test_plugins.py .................................... [ 38%]
tests/unit/test_extractors/test_profiles.py ................................. [ 42%]
tests/unit/test_extractors/test_quanta_tif.py ................ [ 43%]
tests/unit/test_extractors/test_registry.py ................................................... [ 52%]
tests/unit/test_extractors/test_schemas.py ................................... [ 56%]
tests/unit/test_extractors/test_tescan_tif.py ........................................ [ 60%]
tests/unit/test_extractors/test_thumbnail_generator.py .................................... [ 63%]
tests/unit/test_extractors/test_utils.py ... [ 64%]
tests/unit/test_extractors/test_xml_serialization.py ................... [ 66%]
tests/unit/test_harvesters/test_init.py .. [ 66%]
tests/unit/test_harvesters/test_nemo_api.py ......................................... [ 70%]
tests/unit/test_harvesters/test_nemo_connector.py ......................... [ 72%]
tests/unit/test_harvesters/test_nemo_utils.py ......... [ 73%]
tests/unit/test_harvesters/test_reservation_event.py ...... [ 74%]
tests/unit/test_instruments.py ................. [ 76%]
tests/unit/test_network_resilience.py ............... [ 77%]
tests/unit/test_record_builder/test_activity.py ............. [ 78%]
tests/unit/test_record_builder/test_record_builder.py ......................... [ 81%]
tests/unit/test_schemas/test_em_glossary.py ............................................ [ 85%]
tests/unit/test_schemas/test_metadata.py ............................................... [ 90%]
tests/unit/test_schemas/test_pint_types.py ................. [ 92%]
tests/unit/test_schemas/test_units.py .................................. [ 95%]
tests/unit/test_sessions.py ........ [ 96%]
tests/unit/test_utils.py ................................. [ 99%]
tests/unit/test_version.py . [100%]
====================================== tests coverage ================================================
____________________ coverage: platform darwin, python 3.11.14-final-0 _______________________________
Name Stmts Miss Cover Missing
------------------------------------------------------------------------------------------------------
nexusLIMS/__init__.py 18 0 100%
nexusLIMS/__main__.py 3 0 100%
nexusLIMS/builder/__init__.py 0 0 100%
nexusLIMS/builder/record_builder.py 199 0 100%
nexusLIMS/cdcs.py 117 0 100%
nexusLIMS/cli/__init__.py 0 0 100%
nexusLIMS/cli/process_records.py 144 0 100%
nexusLIMS/config.py 153 0 100%
nexusLIMS/db/__init__.py 7 0 100%
nexusLIMS/db/session_handler.py 98 0 100%
nexusLIMS/extractors/__init__.py 168 0 100%
nexusLIMS/extractors/base.py 38 0 100%
nexusLIMS/extractors/plugins/__init__.py 1 0 100%
nexusLIMS/extractors/plugins/basic_metadata.py 27 0 100%
nexusLIMS/extractors/plugins/digital_micrograph.py 350 0 100%
nexusLIMS/extractors/plugins/edax.py 106 0 100%
nexusLIMS/extractors/plugins/fei_emi.py 271 0 100%
nexusLIMS/extractors/plugins/orion_HIM_tif.py 199 0 100%
nexusLIMS/extractors/plugins/preview_generators/__init__.py 4 0 100%
nexusLIMS/extractors/plugins/preview_generators/hyperspy_preview.py 358 0 100%
nexusLIMS/extractors/plugins/preview_generators/image_preview.py 62 0 100%
nexusLIMS/extractors/plugins/preview_generators/text_preview.py 86 0 100%
nexusLIMS/extractors/plugins/profiles/__init__.py 47 0 100%
nexusLIMS/extractors/plugins/profiles/fei_titan_stem_643.py 25 0 100%
nexusLIMS/extractors/plugins/profiles/fei_titan_tem_642.py 53 0 100%
nexusLIMS/extractors/plugins/profiles/jeol_jem_642.py 22 0 100%
nexusLIMS/extractors/plugins/quanta_tif.py 353 0 100%
nexusLIMS/extractors/plugins/tescan_tif.py 248 0 100%
nexusLIMS/extractors/profiles.py 29 0 100%
nexusLIMS/extractors/registry.py 226 0 100%
nexusLIMS/extractors/utils.py 195 0 100%
nexusLIMS/extractors/xml_serialization.py 48 0 100%
nexusLIMS/harvesters/__init__.py 5 0 100%
nexusLIMS/harvesters/nemo/__init__.py 36 0 100%
nexusLIMS/harvesters/nemo/connector.py 256 0 100%
nexusLIMS/harvesters/nemo/exceptions.py 2 0 100%
nexusLIMS/harvesters/nemo/utils.py 68 0 100%
nexusLIMS/harvesters/reservation_event.py 130 0 100%
nexusLIMS/instruments.py 91 0 100%
nexusLIMS/schemas/__init__.py 5 0 100%
nexusLIMS/schemas/activity.py 216 0 100%
nexusLIMS/schemas/em_glossary.py 111 0 100%
nexusLIMS/schemas/metadata.py 101 0 100%
nexusLIMS/schemas/pint_types.py 41 0 100%
nexusLIMS/schemas/units.py 102 0 100%
nexusLIMS/utils.py 222 0 100%
nexusLIMS/version.py 2 0 100%
------------------------------------------------------------------------------------------------------
TOTAL 5043 0 100%
Coverage HTML written to dir tests/coverage
Coverage XML written to file coverage.xml
================================ 993 passed in 146.72s (0:02:26) =====================================
Release Process#
This section describes how to create a new release of NexusLIMS.
Quick Start#
# Standard release
./scripts/release.sh "2.0.0"
# Preview what will happen (dry-run)
./scripts/release.sh "2.0.1" --dry-run
# Auto-approve prompts
./scripts/release.sh "2.0.2" --yes
Prerequisites#
Clean working directory: Commit or stash any uncommitted changes
Towncrier fragments: Add changelog fragments to
docs/changes/for all changes since last releaseBranch: Typically on
mainbranch (or your release branch)Tests passing: Ensure all tests pass with
./scripts/run_tests.shLinting passing: Ensure linting passes with
./scripts/run_lint.sh
Creating Towncrier Fragments#
Before releasing, create changelog fragments for all notable changes:
# Fragment naming: <number>.<type>.md or +<number>.<type>.md
# Use <number> to link to GitHub issue #<number>
# Use +<number> for fragments without issue links
# Types: feature, bugfix, enhancement, doc, misc, removal
# Example fragments:
echo "Added new NEMO harvester for multi-instance support" > docs/changes/42.feature.md
echo "Fixed metadata extraction for FEI TIFF files" > docs/changes/43.bugfix.md
echo "Improved performance of file clustering algorithm" > docs/changes/+44.enhancement.md
Fragment types correspond to sections in the changelog:
feature: New features
bugfix: Bug fixes
enhancement: Enhancements
doc: Documentation improvements
misc: Miscellaneous/Development changes
removal: Deprecations and/or Removals
Release Workflow#
1. Prepare the Release#
# Ensure you're on the correct branch
git checkout main
git pull origin main
# Run tests
./scripts/run_tests.sh
# Run linting
./scripts/run_lint.sh
# Preview the changelog
./scripts/release.sh "2.0.0" --draft
2. Create the Release#
# Interactive release (recommended for first time)
./scripts/release.sh "2.0.0"
# Or with auto-confirmation
./scripts/release.sh "2.0.0" --yes
The script will:
✓ Update version in
pyproject.toml✓ Generate changelog from towncrier fragments (adds to
docs/changelog.md)✓ Delete consumed changelog fragments
✓ Commit changes
✓ Create git tag for the version
✓ Push to remote (with confirmation)
3. Monitor Automated Process#
Once the tag is pushed, GitHub Actions automatically:
Builds distribution packages (wheel and sdist)
Publishes to PyPI
Creates GitHub Release with auto-generated notes
Deploys versioned documentation to GitHub Pages
Monitor progress at: datasophos/NexusLIMS
Script Options#
./scripts/release.sh [VERSION] [OPTIONS]
Options:
-h, --help Show help message
-d, --dry-run Run without making changes (preview only)
-y, --yes Skip confirmation prompts
--no-push Don't push to remote (tag locally only)
--draft Generate draft changelog without committing
Version Numbering#
Follow Semantic Versioning:
Major (X.0.0): Breaking changes, incompatible API changes
Minor (x.Y.0): New features, backwards-compatible
Patch (x.y.Z): Bug fixes, backwards-compatible
Pre-release versions:
2.0.0-alpha1: Alpha releases2.0.0-beta1: Beta releases2.0.0-rc1: Release candidates
Troubleshooting#
“Working directory is not clean”#
Commit or stash your changes:
git status
git add .
git commit -m "Your changes"
“No towncrier fragments found”#
Add at least one changelog fragment:
echo "Your change description" > docs/changes/+1.misc.md
Need to fix a release tag#
If you need to redo a release:
# Delete local tag
git tag -d v2.0.0
# Delete remote tag (CAUTION!)
git push origin :refs/tags/v2.0.0
# Re-run release script
./scripts/release.sh 2.0.0
Release workflow failed on GitHub#
Check the Actions tab for error details. Common issues:
PyPI credentials not configured (requires trusted publishing setup)
Package build errors (test locally with
uv build)Documentation build errors (test with
./scripts/build_docs.sh)
Manual Release (Without Script)#
If you need to release manually:
# 1. Update version
sed -i 's/version = ".*"/version = "2.0.0"/' pyproject.toml
# 2. Generate changelog
uv run towncrier build --version=2.0.0 --yes
# 3. Commit and tag
git add pyproject.toml docs/
git commit -m "Release v2.0.0"
git tag -a v2.0.0 -m "Release 2.0.0"
# 4. Push
git push origin main
git push origin v2.0.0
Post-Release Checklist#
Verify package on PyPI: https://pypi.org/project/nexusLIMS/
Check GitHub Release: datasophos/NexusLIMS
Verify documentation: https://datasophos.github.io/NexusLIMS/stable/
Test installation:
pip install nexusLIMS==2.0.0Announce release (if applicable)
Update dependent projects/deployments