Standardize dataset metadata in DATASETS.toml

.github/PULL_REQUEST_TEMPLATE.md

@@ -4,44 +4,22 @@
 
 <!-- What are you adding or changing, and why? -->
 
-## Dataset license checklist
-
-**All new datasets must pass this checklist before merge.** See the
-[Adding new datasets](../blob/master/README.md#adding-new-datasets-licensing-requirements)
-section of the README for full details.
-
-- [ ] I have identified the **original source** of the dataset (URL,
-      paper, repository, or upload page) and recorded it in the license
-      file.
-- [ ] I have identified the **exact license** of the dataset by reading
-      it at the source. I have not guessed.
-- [ ] The license **permits commercial use**. Non-commercial licenses
-      (CC BY-NC, CC BY-NC-SA, CC BY-NC-ND, "research only", etc.) are
-      not accepted without explicit maintainer approval and prominent
-      labeling.
-- [ ] The license **permits redistribution** from a BSD-licensed
-      project (pyvista).
-- [ ] I have added a **license file** alongside the dataset:
-  - For a dataset in its own directory: a `LICENSE` file inside that
-    directory.
-  - For a loose file in `Data/`: a `<filename>.license` companion file.
-- [ ] The license file includes:
-  - [ ] Source URL
-  - [ ] Verbatim license statement or link
-  - [ ] Required attribution text (if any)
-  - [ ] An `SPDX-License-Identifier:` line
-- [ ] If the license is **ShareAlike** (CC BY-SA, ODbL, etc.), I have
-      prominently flagged the ShareAlike obligation so downstream users
-      are aware that derivatives must be re-shared under the same
-      license.
-
-## Declared license
-
-- **SPDX identifier:** <!-- e.g. CC0-1.0, CC-BY-4.0, MIT, LicenseRef-... -->
-- **Source URL:** <!-- where the dataset originated -->
-- **Commercial use:** <!-- permitted / not permitted -->
-- **Attribution required:** <!-- yes/no — if yes, what text? -->
-
-## Additional notes
-
-<!-- Anything else reviewers should know. -->
+## Dataset checklist
+
+Only needed if this PR adds or changes files under `Data/`. See
+[CONTRIBUTING.md](https://github.com/pyvista/data/blob/master/CONTRIBUTING.md)
+for the field reference.
+
+- [ ] I added a `[[dataset]]` block to `DATASETS.toml` for every file I added,
+      in alphabetical order by `name`.
+- [ ] I did **not** add a `LICENSE`, `README`, `CITATION` or `.license` file
+      under `Data/`.
+- [ ] I read the licence **at the source** and recorded it in
+      `SPDX-License-Identifier`. I did not guess.
+- [ ] The licence permits redistribution from this repository (Apache-2.0)
+      and from PyVista (MIT), and permits commercial use. (Non-commercial data needs maintainer approval
+      and `commercial_use = false` on its `[license.*]` table.)
+- [ ] If the licence requires credit, I filled in `attribution`.
+- [ ] If I changed the data after downloading it, I set `modified = true` and
+      described the change in `modification`.
+- [ ] `python tools/validate_datasets.py` passes locally.

.github/workflows/validate-datasets.yml

@@ -0,0 +1,78 @@
+name: Validate dataset metadata
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  validate:
+    name: DATASETS.toml
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          filter: blob:none
+          persist-credentials: false
+          sparse-checkout-cone-mode: false
+          sparse-checkout: |
+            /*
+            !/Data/
+
+      - uses: actions/setup-python@v7
+        with:
+          python-version: '3.12'
+
+      - name: Validate DATASETS.toml
+        id: validate
+        shell: bash
+        run: |
+          set -o pipefail
+          python tools/validate_datasets.py 2>&1 | tee validation.log
+
+      - name: Test the validator
+        if: always() && steps.validate.conclusion != 'skipped'
+        shell: bash
+        run: |
+          python -m pip install --quiet pytest
+          python -m pytest tools -q
+
+      - name: Explain the failure
+        if: failure() && steps.validate.conclusion == 'failure'
+        shell: bash
+        run: |
+          {
+            echo '## Dataset metadata check failed'
+            echo
+            echo 'Every file under `Data/` must be described by a `[[dataset]]` block in'
+            echo '`DATASETS.toml`. Per-dataset licence, copying, copyright, notice, readme'
+            echo 'and citation files (any letter case, singular or plural, bare or with any'
+            echo 'extension: `LICENSE`, `LICENSE.MIT`, `COPYING`, `NOTICE`, `README.md`,'
+            echo '`CITATION.cff` and the like) and `.license` sidecars are not accepted'
+            echo 'under `Data/`: that information belongs in `DATASETS.toml`,'
+            echo 'which is what PyVista reads to publish provenance in the Dataset Gallery.'
+            echo
+            echo 'Run the same check locally with:'
+            echo
+            echo '```bash'
+            echo 'python tools/validate_datasets.py'
+            echo '```'
+            echo
+            echo 'Full output:'
+            echo
+            echo '```'
+            cat validation.log
+            echo '```'
+            echo
+            echo 'See [CONTRIBUTING.md](https://github.com/pyvista/data/blob/master/CONTRIBUTING.md)'
+            echo 'for the field reference and worked examples.'
+          } >> "$GITHUB_STEP_SUMMARY"

.gitignore

@@ -1,2 +1,4 @@
 *.unzip
 .DS_Store
+tools/__pycache__/
+.pytest_cache/