Add semantic schemas and access-rights support

.github/workflows/ci.yml

@@ -13,7 +13,7 @@ jobs:
         strategy:
             fail-fast: false
             matrix:
-                python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+                python-version: ['3.10', '3.11', '3.12', '3.13', '3.14']
 
         steps:
 

.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 repos:
 
     - repo: https://github.com/pre-commit/pre-commit-hooks
-      rev: v4.4.0
+      rev: v6.0.0
       hooks:
           - id: check-json
           - id: check-yaml
@@ -14,53 +14,50 @@ repos:
           - id: trailing-whitespace
 
     - repo: https://github.com/psf/black
-      rev: 23.1.0
+      rev: 26.5.1
       hooks:
           - id: black
             args: [--line-length, "79"]
 
     - repo: https://github.com/pycqa/isort
-      rev: 5.12.0
+      rev: 8.0.1
       hooks:
           - id: isort
-            args: [--profile, black, --filter-files]
+            args: [--profile, black, --filter-files, --line-length, "79"]
 
     - repo: https://github.com/asottile/pyupgrade
-      rev: v3.3.1
+      rev: v3.21.2
       hooks:
           - id: pyupgrade
             args: [--py38-plus]
 
     - repo: https://github.com/PyCQA/flake8
-      rev: 6.0.0
+      rev: 7.3.0
       hooks:
           - id: flake8
             args: [--count, --show-source, --statistics, '--ignore', 'E501,E203,W503,E201,E202,E221,E222,E231,E241,E271,E272,E702,E713']
             # additional_dependencies: [flake8-bugbear==21.3.1, pep8-naming]
             log_file: flake8.log
 
     - repo: https://github.com/asottile/setup-cfg-fmt
-      rev: v2.2.0
+      rev: v3.2.0
       hooks:
           - id: setup-cfg-fmt
 
     - repo: https://github.com/PyCQA/bandit
-      rev: 1.7.5
+      rev: 1.9.4
       hooks:
       - id: bandit
         args: ["-r"]
         files: ^(dsms)/.*
-        additional_dependencies:
-          - "pbr==2.0.0"
-          - setuptools
 
     - repo: local
       hooks:
       - id: pylint
         name: pylint
         entry: pylint
-        args: ["--rcfile=.pylintrc", "--extension-pkg-whitelist='pydantic'"]
-        language: python
+        args: ["--rcfile=.pylintrc", "--extension-pkg-whitelist=pydantic"]
+        language: system
         types: [python]
         require_serial: true
         files: ^(dsms)/.*

.pylintrc

@@ -72,6 +72,7 @@ disable= no-name-in-module,
     cyclic-import,
     import-outside-toplevel,
     too-many-arguments,
+    too-many-positional-arguments,
     too-many-nested-blocks,
     dangerous-default-value,
     too-many-public-methods,
@@ -80,6 +81,7 @@ disable= no-name-in-module,
     too-many-branches,
     too-many-lines,
     too-many-statements,
+    global-statement,
 
 
 
@@ -245,7 +247,7 @@ ignored-classes=optparse.Values,thread._local,_thread._local
 # (useful for modules/projects where namespaces are manipulated during runtime
 # and thus existing member attributes cannot be deduced by static analysis). It
 # supports qualified module names, as well as Unix pattern matching.
-ignored-modules=
+ignored-modules=dotenv,pydantic,pydantic_settings,yaml,oyaml,requests,click,pandas,rdflib,PIL,segno
 
 # Show a hint with possible names when a member name was not found. The aspect
 # of finding the hint is based on edit distance.

CHANGELOG.md

@@ -0,0 +1,114 @@
+# Changelog
+
+All notable changes to this project are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
+Versioning follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [5.0.0] — unreleased
+
+### Added
+
+**User model**
+- `User` gains optional `firstName`, `lastName`, and `email` fields populated from the Keycloak user profile.
+
+**Access control (RBAC)**
+- `KItemAccessProperties` model with `user_access` and `group_access` lists for per-KItem role assignments.
+- `Role` enum (`MEMBER=1`, `CONTRIBUTOR=2`, `OWNER=3`) and `OperationType` enum (`create`, `read`, `update`, `delete`, `manage`). Roles serialize as lowercase strings on the wire (`"member"`, `"contributor"`, `"owner"`).
+- `RoleMapping` enum with `get_operations`, `min_access_level`, and `max_access_level` helpers.
+- `UserAccessProperty` and `GroupAccessProperty` sub-models.
+- `DSMS.user_groups` and `DSMS.users` cached properties with `refresh_user_groups()` / `refresh_users()` invalidation.
+- `DSMS.get_user(user_id)` convenience method.
+- `Group`, `User`, `GroupList`, `UserList` models (`dsms.knowledge.groups`).
+- `INTERNAL_GROUP` / `PUBLIC_GROUP` constants (IDs `dsms:internal` / `dsms:public`), configurable via environment variables.
+- `refresh_public_groups(config)` to avoid import-time staleness when custom group IDs are used.
+
+**KType v2 semantic-spec subsystem**
+- `KTypeSpec` model capturing the full `ktype_spec` database record: ontology classes, relations, semantic schema references, inheritance, tags, versioning, stash state.
+- Sub-models: `OntologyClassSpec`, `SemanticSchemaRef`, `RelationSpec`.
+- `KTypeV2(KType)` response model with optional `spec: KTypeSpec` field.
+- Request models: `CreateKTypeRequest`, `ImportFromUrlRequest`, `KTypeSpecPayload`.
+- Remote-repository models: `RemoteKTypeSummary`, `RemoteKTypeVersion`, `RemoteSchemaInfo`, `RemoteSchemaVersionInfo`, `SpecDiffField`, `RemoteDiffOut`.
+- Full v2 CRUD surface on `DSMS`: `get_v2_ktypes`, `get_v2_ktype`, `create_v2_ktype`, `import_v2_ktype`, `update_v2_ktype`, `delete_v2_ktype`, `restore_v2_ktype_stash`, `refresh_v2_ktype`, `export_v2_ktype`, `list_remote_v2_ktypes`, `list_remote_schemas`, `list_remote_ktype_versions`, `get_v2_ktype_remote_diff`.
+- `DSMS.get_ktypes_by_parent(parent_id)` for the `?parent=` filter on the v1 list endpoint.
+
+**Schema data on KItems**
+- `KItemSchemaData` model (`schema_id`, `content`) and `KItemSchemaDataList` helper with `.by_schema_id` lookup.
+- `KItem.schema_data: Optional[List[KItemSchemaData]]` field.
+- Commit flow syncs `schema_data` changes via `PUT`/`DELETE` on `/api/knowledge/{kitem_id}/schema-data/{schema_id}`.
+
+**Context SPARQL**
+- `SparqlInterface.query_context(context_id, query)` — `POST /api/knowledge/sparql/context`.
+- `SparqlInterface.graph_context(context_id, query)` — `POST /api/knowledge/graph/context`.
+
+**Search and list additions**
+- `DSMS.search()` gains `contexts: List[str]` (filter by context KItem IDs) and `attachment_extensions: List[str]` (filter by file extension).
+- `DSMS.get_kitems()` gains `name: str` for substring filtering.
+
+**KItemCompactedModel additions**
+- `has_contexts: bool` — whether the KItem belongs to at least one context.
+- `attachment_extensions: Optional[List[str]]` — unique file extensions in the KItem's attachments.
+- `avatar_exists` moved from `KItem` to the shared `KItemCompactedModel` base.
+
+### Changed
+
+- `KItem.contexts` field now properly tracks changes in `_get_kitems_diffs()`.
+- `KItem.access_properties` changes are tracked and committed.
+- `get_user_by_id` now accepts `dsms` as its first argument (consistent with all util functions) and returns a typed `User` object.
+- `Role.min_access_level` / `max_access_level` now return `Role` objects and raise `ValueError` for operations not granted by any role (e.g. `CREATE`).
+
+### Deprecated
+
+- `KItem.authors` — the server no longer populates this field. Use `access_properties` instead.
+- `KItem.rdf_exists` — the server no longer populates this field.
+- `KItem.user_groups` — legacy field still supported but superseded by `access_properties`.
+
+### Maintenance
+
+- Upgraded pre-commit hooks: `pre-commit-hooks` v4→v6, `black` 23→26, `isort` 5→8, `setup-cfg-fmt` v2→v3, `bandit` 1.9.3→1.9.4.
+- Aligned `isort` line length to match `black`'s 79-character limit.
+- Relaxed dependency pins: `rdflib>=6,<8`, `pandas>=2,<4`, `segno>=1.6,<2`, `pydantic-settings>=2,<3`, `oyaml>=1`.
+- Dropped Python 3.8 and 3.9 support.
+
+---
+
+## [4.0.0]
+
+### Added
+
+- Pydantic v2 migration (`BaseModel`, `field_validator`, `field_serializer`, `model_dump`).
+- Service-account authentication via `client_id` / `client_secret` (Keycloak).
+- `KItemAccessProperties` groundwork, `UserGroup` model.
+- `AppConfig` and `DSMS.apps` for managing application configurations.
+- `ProcessSchema` and `WebformSchema` models on `KType`.
+
+### Changed
+
+- Minimum Python version raised to 3.10.
+- `KType.id` accepts both `UUID` and `str`.
+- `Configuration` uses `pydantic-settings` for environment-variable loading.
+
+---
+
+## [3.x]
+
+- Triplestore / SPARQL interface (`SparqlInterface`, subgraph CRUD).
+- `Attachment`, `Avatar`, `ExternalLink`, `LinkedKItems` property models.
+- DataFrame integration via `dataframe` field on `KItem`.
+- `DSMS.search()` initial implementation.
+
+---
+
+## [2.x]
+
+- Initial Pydantic v1 models for `KItem` and `KType`.
+- Basic CRUD operations via `DSMS.add()`, `DSMS.delete()`, `DSMS.commit()`.
+- Annotation support.
+
+---
+
+## [< 2.0.0]
+
+- Initial release.

CONTRIBUTING.md

@@ -0,0 +1,151 @@
+# Contributing to DSMS Python SDK
+
+Thank you for considering a contribution! This document explains how to set up a development environment, run the checks we require, and submit changes.
+
+---
+
+## Table of Contents
+
+1. [Development setup](#development-setup)
+2. [Code style and linting](#code-style-and-linting)
+3. [Pre-commit hooks](#pre-commit-hooks)
+4. [Testing](#testing)
+5. [Branching and commits](#branching-and-commits)
+6. [Opening a pull request](#opening-a-pull-request)
+7. [Versioning](#versioning)
+8. [Reporting issues](#reporting-issues)
+
+---
+
+## Development setup
+
+```bash
+git clone git@github.com:MI-FraunhoferIWM/dsms-python-sdk.git
+cd dsms-python-sdk
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+```
+
+The `[dev]` extra installs all development dependencies including linters, test runners, and the pre-commit framework.
+
+---
+
+## Code style and linting
+
+We enforce a consistent style automatically via pre-commit hooks (see below). The key rules are:
+
+| Tool      | Configuration                                 |
+|:---------:|:---------------------------------------------:|
+| `black`   | Line length 79, enforced on all `.py` files   |
+| `isort`   | Profile `black`, line length 79               |
+| `flake8`  | Default rules, line length inferred from black |
+| `pylint`  | `fail-under=10.0`; see `.pylintrc` for disabled checks |
+| `bandit`  | Security linting                              |
+| `pyupgrade` | Enforces modern Python syntax              |
+
+Do **not** bypass hooks with `--no-verify`. If a hook fails, fix the underlying issue.
+
+---
+
+## Pre-commit hooks
+
+Install the hooks once after cloning:
+
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+Run manually against all changed files:
+
+```bash
+pre-commit run --files <file1> <file2> ...
+```
+
+Run against all files in the repo:
+
+```bash
+pre-commit run --all-files
+```
+
+To update hook versions to the latest stable releases:
+
+```bash
+pre-commit autoupdate
+```
+
+---
+
+## Testing
+
+Run the unit test suite with:
+
+```bash
+pytest
+```
+
+Tests live under `tests/`. We do not mock the database in integration tests — if you add a test that touches the backend, it must run against a real DSMS instance configured via environment variables (see `Configuration` in `dsms/core/configuration.py`).
+
+**Tutorial notebooks** can be tested against a live instance with:
+
+```bash
+./scripts/run_notebooks.sh
+```
+
+To re-execute notebooks and save outputs in-place (for documentation commits):
+
+```bash
+./scripts/run_notebooks.sh --refresh
+```
+
+See `scripts/run_notebooks.sh --help` (or read the script header) for full usage. Requires `pip install -e ".[docs,tests]"` and a reachable DSMS instance.
+
+---
+
+## Branching and commits
+
+- Base feature branches off `main`.
+- Use descriptive branch names, e.g. `feature/ktype-v2-subsystem` or `fix/search-context-filter`.
+- Keep commits focused. One logical change per commit.
+- Write commit messages in the imperative mood: *"Add schema_data field to KItem"*, not *"Added"* or *"Adding"*.
+- Do not amend published commits.
+
+---
+
+## Opening a pull request
+
+1. Push your branch and open a PR against `main`.
+2. Fill in the PR template — at minimum, describe **what** changed and **why**.
+3. Ensure all CI checks pass before requesting a review.
+4. Add an entry to `CHANGELOG.md` under the relevant unreleased section.
+5. Update any affected documentation in `docs/`.
+
+PRs that introduce new public API surface should update:
+- `dsms/knowledge/__init__.py` — re-export new models.
+- `docs/dsms_sdk/dsms_kitem_schema.md` or `docs/dsms_sdk/dsms_sdk.md` — document new fields/methods.
+
+---
+
+## Versioning
+
+This project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html):
+
+- **MAJOR** — breaking API changes.
+- **MINOR** — backward-compatible new functionality.
+- **PATCH** — backward-compatible bug fixes.
+
+The version is set in `setup.cfg` (`version = vMAJOR.MINOR.PATCH`). Update it and `CHANGELOG.md` together as part of a release PR.
+
+The SDK version must stay compatible with the target DSMS backend version. See the compatibility table in `README.md`.
+
+---
+
+## Reporting issues
+
+Please open an issue at <https://github.com/MI-FraunhoferIWM/dsms-python-sdk/issues> and include:
+
+- SDK version (`pip show dsms-sdk`).
+- Python version.
+- A minimal reproducible example.
+- The full traceback if applicable.