Python cleanup - #234
Conversation
Move Python dependencies and Ruff policy into pyproject.toml, require Python 3.12 or newer, and pin Ruff 0.16.1 with a committed uv lockfile. Remove Python shebangs while preserving non-Python entrypoints, and update legacy generator documentation to use uv-managed Python commands. Ignore uv virtual environments and remove the legacy requirements.txt and .ruff.toml configuration.
Run Ruff and Maven through the locked uv environment, pin uv 0.12.1, and require Python 3.12 or newer in CI. Remove pip-based dependency installation and update the CI matrix to supported Python versions.
Modernize model-tooling logging, exception handling, datetime usage, and file management. Remove the legacy unknown mx import and obsolete generator code while adding shebang rule exclusions for Ruff.
Pin setup-uv by commit, rely on pyproject.toml for the uv version, and remove redundant development dependency flags from GitHub Actions.
Replace the legacy generateDS-based implementation and xsd-fu executable with an installable xsd-fu package under xsd-fu/src. Add typed schema parsing, normalized model construction, Java configuration, template rendering, generation settings, converter validation, structured errors, and module and console entry points. Package the Java templates in the wheel using uv_build and expose model, metadata, and combined generation targets through the new CLI. Add dry-run support, generated-file and dependency listing, package overrides, custom template directories, deterministic output ordering, conflicting-output detection, and atomic file replacement. Document installation, generation, converter validation, development commands, and the compatibility contract. Update ome-xml Maven code generation to invoke the locked xsd-fu command once during generate-sources for both model and metadata APIs. Remove the legacy Python generator, modeltools modules, utility code, and executable while relocating the templates into the packaged source tree. Add unit tests for the CLI, Java configuration, schema normalization, model introspection, rendering helpers, and converter validation. Add integration parity tests and SHA-256 fixtures for the 2012-06, 2013-06, 2015-01, and 2016-06 schemas to preserve generated Java output compatibility. Move the project to the xsd-fu distribution name, require Python 3.13, pin uv, Ruff, pytest, pytest-cov, and ty, add xmlschema and the uv_build backend, refresh uv.lock, and expand .gitignore for Python caches, coverage, virtual environments, and build artifacts. Expand Python CI from Ruff-only checks into separate quality and test jobs covering lockfile verification, formatting, linting, type checking, package building, and branch-covered tests. Run CI on pull requests, pushes to master, and manual dispatches, with concurrency cancellation, timeouts, uv caching, and Python 3.13. Update Maven CI to use Python 3.13 across the Java and operating-system matrix, install the locked development environment, pin uv and setup-java versions, cache from uv.lock, and add job timeouts. Pin checkout to actions/checkout v7.0.0 and CodeQL init/analyze to v4.37.6 by commit SHA, disable persisted checkout credentials, and retain pinned setup-java v5.7.0 and setup-uv actions.
|
To clarify, this makes |
Add deterministic transform catalog generation with CI verification, replacing the legacy helper script. Resolve schema imports from checked-in files, track dependencies, validate Java package and output paths, and use atomic writes. Validate generated metadata APIs during Maven builds and document the uv-backed build and development workflow.
|
575088d hardens |
Enforce the Maven version required by the build plugins and update the documented minimum version. Simplify Python CI setup and use non-development dependencies in Maven CI.
…ehavior - Use --no-sync in python calls as locked env is already ensures adn doesnt need to rechecked over and over again - Fix maven.yml comment about caching
|
We are unable to consider this contribution at this time which fully rewrites the |
|
@sbesson could you clarify this for me a bit how should I continue? Should i open a post on Image.sc to discuss this? |
The figures reported by GitHub (+5,434, -11,726) can certainly be broken down into more granular categories needing different degrees of review. This does not significantly modify the scale of this contribution.
We would recommend using a GitHub issue describing the bug or feature request rather than an image.sc post. At this stage, we are looking for a clear and concise explanation of the problem rather than a detailed technical implementation. As mentioned above, the current focus of the maintainers of the ome-xml component is Bio-Formats 9. Updating the code-generation tooling has not been identified as a task as part of this roadmap or raised as a blocker. We are reviewing issues on a monthly basis across Bio-Formats components. |
2 participants
Hej,
While investigating the CI failure of #233, and tracking them down to the larger filter list enabled by
ruff0.16.0(see https://astral.sh/blog/ruff-v0.16.0), I found broader problems with the tooling, including that the repository contained a mixture of Python 2- and Python 3-era code, vendoredgenerateDSsources, legacy entry points, duplicated configuration, no version pinning, etc.Rather than adding increasingly broad Ruff exceptions around code that was difficult to maintain, I found that a clean rewrite around
xmlschema, replaces the custom schema-processing implementation, providing a clearer and more maintainable foundation for validating and traversing OME XML Schema documents.Changes
Replace the legacy
generateDS-based implementation andxsd-fu/xsd-fuscript with an installablexsd-fupackage underxsd-fu/src.The new implementation adds:
xmlschemaxsd-fuandpython -m xsd_fuThe command-line interface now supports:
MetadataStore,MetadataRetrieve, andMetadataConverterAPIsGeneration is performed in memory before writing any files. Output paths are checked for conflicts, generated files are ordered deterministically, and writes use temporary files followed by atomic replacement.
Move all Java templates into the package so installed wheels can generate sources without depending on the repository layout.
Remove the legacy Python package tree, vendored
generateDSimplementation, obsolete utility code, and old executable. The generated output remains compatible with the existing OME Java APIs, as verified against the frozen output contract.Update
ome-xml/pom.xmlto invoke the lockedxsd-fu generatecommand once duringgenerate-sources, producing both model and metadata APIs in a single validated pass.Python tooling
Modernize the project configuration by:
xsd-fuxmlschemauv_buildfor packagingxsd-fuconsole scriptxsd-fu/srcuv.lock.gitignorefor Python caches, coverage output, virtual environments, and build artifactsAdd documentation covering installation, generation, converter validation, custom generation options, the compatibility contract, and development commands.
The single output change is, which should not be a problem donwstream, is the switch from camelCase fields such as
maxOccursto the new model’s snake_case fields such asmax_occurs, in line with PEP8, in the template model attributes.CI and GitHub Actions
Expand Python CI from a Ruff-only job into separate quality and test jobs. The quality job now verifies the lockfile, installs the locked development environment, checks formatting, runs Ruff linting, runs
ty, and builds source and wheel distributions. The test job runs the Python 3.13 test suite with branch coverage.Run Python CI for pull requests, pushes to
master, and manual workflow dispatches. Add concurrency cancellation, job timeouts, uv caching based onuv.lock, and consistent Python and uv versions.Update Maven CI to use Python 3.13 across the Java 11, 17, 21, and 25 matrix on Ubuntu, Windows, and macOS. Install the locked development environment with development dependencies, configure uv caching, and add build and release timeouts. Now with proper versioning etc. the python matrix is no longer needed etc.
Update and harden the GitHub Actions dependencies:
actions/checkoutfrom v6 to v7.0.0 and pin it by commit SHAactions/setup-javato v5.7.0 by commit SHAThis is only a small surface level cleanup, for more maintainability a dependabot should probably be added, also keeping the Node depreciation on Actions in mind.
Tests
Add focused unit tests for:
Add integration tests with SHA-256 manifests for every generated file from the 2012-06, 2013-06, 2015-01, and 2016-06 OME schemas. These tests ensure that the rewrite preserves the existing generated Java output. The source of truth is 1957bd7, with some reservations, outlined in the README. Maintaining the same failure surface etc. with the
MetadataConverter.