Add Lightdash converter

.github/workflows/converter-lightdash-ci.yml

@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Converters Lightdash CI
+
+on:
+  push:
+    branches: [ "main" ]
+    paths:
+      - 'converters/lightdash/**'
+      - '.github/**'    
+  pull_request:
+    branches: [ "main" ]
+    paths:
+      - 'converters/lightdash/**'
+      - '.github/**'    
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+
+    steps:
+      - name: Checkout project
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install uv
+        run: |
+          curl -LsSf https://astral.sh/uv/install.sh | sh
+          echo "${HOME}/.local/bin" >> "${GITHUB_PATH}"
+
+      - name: Sync dependencies
+        working-directory: converters/lightdash
+        run: |
+          uv sync
+
+      - name: Unit Tests
+        working-directory: converters/lightdash
+        run: |
+          uv run pytest

converters/lightdash/README.md

@@ -0,0 +1,94 @@
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Apache Ossie <> Lightdash converter
+
+Bidirectional converter between Ossie documents and
+[Lightdash](https://github.com/lightdash/lightdash) semantic definitions.
+Lightdash reads its semantic layer from dbt `schema.yml` files: dimensions and
+metrics are declared per column (and per model) under `meta`. This converter
+translates between that shape and Ossie.
+
+- **Export** (`osi_to_lightdash`): Ossie document → a dbt `schema.yml`-shaped
+  dictionary with Lightdash `meta` blocks, ready to merge into a dbt project.
+- **Import** (`lightdash_to_osi`): a Lightdash-flavoured `schema.yml` → an
+  Ossie document, as a migration path for teams with an existing installed
+  base of Lightdash metrics.
+
+```
+ossie-lightdash export semantic_model.yaml schema.yml
+ossie-lightdash import schema.yml semantic_model.json --database analytics_db --schema marts
+```
+
+## Mapping
+
+| Ossie | Lightdash (dbt meta) |
+| ----- | -------------------- |
+| `dataset` | dbt model (`name` = table part of `source`) |
+| `dataset.source` | assembled on import from `--database` / `--schema` / model name |
+| `field` (no `dimension`) | plain column entry |
+| `field` with `dimension` | `columns[].meta.dimension` (`is_time` ↔ `type: date/timestamp`; an empty `dimension: {}` marks a categorical dimension) |
+| `field.label` / `.description` | `meta.dimension.label` / column `description` |
+| `field.expression` (≠ column name) | `meta.dimension.sql` (`dataset.col` ↔ `${TABLE}.col`) |
+| `metric` with single-aggregation expression (`SUM(ds.col)`, `COUNT(DISTINCT ds.col)`, ...) | column-level `meta.metrics.<name>` with a typed metric (`sum`, `count_distinct`, ...) |
+| `metric` with any other single-dataset expression | model-level `meta.metrics.<name>` with `type: number` + `sql` |
+| `relationship` | `meta.joins` (`sql_on` built from / parsed into column pairs) |
+| Lightdash presentation attributes (`label`, `format`, `round`, `compact`, `group_label`, `hidden`, `percentile`, ...) | `custom_extensions` with `vendor_name: "lightdash"`; on export the extension data is overlaid onto the generated definition (structural keys — `sql`/`label` on dimensions, `sql`/`description` on metrics — are protected and cannot be overridden) |
+
+Expressions are written under the `ANSI_SQL` dialect. Warehouse-specific
+dialects (e.g. `BIGQUERY`) can be added once the surrounding tooling resolves
+them.
+
+## Recommended source shape for dbt-native flows
+
+If the Ossie documents are also consumed by dbt's native OSI parsing, prefer
+importing **without** `--database` (i.e. `schema.table` sources): the database
+is usually environment-dependent in dbt projects, and a database-less source
+keeps one document valid across environments (see
+[dbt-core#15649](https://github.com/dbt-labs/dbt-core/issues/15649)).
+Omitting `--schema` as well is reported as a `SOURCE_UNQUALIFIED` issue.
+
+## Known limitations
+
+- **Cross-dataset metrics are dropped on export** (with a
+  `CROSS_DATASET_METRIC_DROPPED` issue): a Lightdash model metric cannot
+  reference other tables.
+- **Percentile metrics** keep `type` / `percentile` in the `lightdash`
+  extension (Ossie expressions cannot express them faithfully) and re-export
+  as model-level metrics.
+- **`primary_key` / `unique_keys` are not exported** — Lightdash has no
+  corresponding concept — and consequently cannot be reconstructed on import.
+- **`dataset.name` is not preserved when it differs from the source table
+  name**: the dbt model is named after the table part of `source`, and the
+  import direction derives dataset names from model names. References inside
+  expressions and relationships are rewritten consistently, but a
+  name-stable round-trip is not guaranteed.
+- **Relationships with mismatched `from_columns` / `to_columns` lengths are
+  skipped on export** with a `RELATIONSHIP_COLUMNS_MISMATCHED` issue.
+- **`ai_context` is not carried** into Lightdash meta.
+- **Model-level Lightdash meta beyond `metrics` and `joins`** (`label`,
+  `group_details`, `sql_filter`, `order_fields_by`, column
+  `additional_dimensions`, ...) is not carried yet.
+- **Standalone Lightdash YAML projects** (Lightdash without dbt) are not
+  supported yet; the converter targets the dbt-meta flavour.
+- Custom extensions from other vendors are ignored on export (reported as
+  `FOREIGN_EXTENSION_IGNORED`); they remain untouched in the Ossie document.
+- Documents are emitted at the current in-repo spec version. Note that
+  dbt-core 1.12's native OSI parsing accepts spec versions `0.1.0` / `0.1.1`
+  only.

converters/lightdash/pyproject.toml

@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0",
+]
+
+[project]
+name = "apache-ossie-lightdash"
+version = "0.1.0.dev0"
+description = "Lightdash (dbt meta semantic definitions) <> Apache Ossie converter"
+authors = [{ name = "Apache Software Foundation", email = "dev@ossie.apache.org" }]
+requires-python = ">=3.11"
+readme = "README.md"
+license = "Apache-2.0"
+keywords = [
+    "Apache Ossie",
+    "Ossie",
+    "Lightdash"
+]
+dependencies = [
+    "apache-ossie>=0.2.0.dev0",
+    "PyYAML>=6.0",
+]
+
+[project.scripts]
+ossie-lightdash = "ossie_lightdash.cli:main"
+
+[project.urls]
+homepage = "https://ossie.apache.org/"
+repository = "https://github.com/apache/ossie/"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/ossie_lightdash"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.uv]
+required-version = ">=0.9.0"
+default-groups = [
+    "dev"
+]
+
+[tool.uv.sources]
+# apache-ossie is not yet published to PyPI; resolve it from the in-repo
+# package for now. Remove this block once apache-ossie published to PyPI.
+apache-ossie = { path = "../../python", editable = true}

converters/lightdash/src/ossie_lightdash/__init__.py

@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from ossie_lightdash.converter_issues import (
+    ConverterIssue,
+    ConverterIssueType,
+    ConverterResult,
+)
+from ossie_lightdash.lightdash_to_osi import LightdashToOSIConverter
+from ossie_lightdash.osi_to_lightdash import OSIToLightdashConverter
+
+__all__ = [
+    "ConverterIssue",
+    "ConverterIssueType",
+    "ConverterResult",
+    "LightdashToOSIConverter",
+    "OSIToLightdashConverter",
+]

converters/lightdash/src/ossie_lightdash/cli.py

@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Command line interface for the Ossie <> Lightdash converter."""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import yaml
+
+from ossie import OSIDocument
+from ossie_lightdash.lightdash_to_osi import LightdashToOSIConverter
+from ossie_lightdash.osi_to_lightdash import OSIToLightdashConverter
+
+
+def _read_document(path: Path) -> OSIDocument:
+    text = path.read_text(encoding="utf-8")
+    if path.suffix == ".json":
+        return OSIDocument.model_validate_json(text)
+    return OSIDocument.model_validate(yaml.safe_load(text))
+
+
+def _print_issues(issues) -> None:
+    for issue in issues:
+        print(f"[{issue.issue_type.value}] {issue.element_name}", file=sys.stderr)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(prog="ossie-lightdash")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    export_parser = subparsers.add_parser(
+        "export", help="Ossie document (.json/.yaml) -> Lightdash dbt schema.yml"
+    )
+    export_parser.add_argument("input", type=Path)
+    export_parser.add_argument("output", type=Path)
+
+    import_parser = subparsers.add_parser(
+        "import", help="Lightdash dbt schema.yml -> Ossie document (.json/.yaml)"
+    )
+    import_parser.add_argument("input", type=Path)
+    import_parser.add_argument("output", type=Path)
+    import_parser.add_argument("--database", default=None)
+    import_parser.add_argument("--schema", default=None)
+    import_parser.add_argument(
+        "--semantic-model-name", default="lightdash_semantic_model"
+    )
+
+    args = parser.parse_args()
+
+    if args.command == "export":
+        result = OSIToLightdashConverter().convert(_read_document(args.input))
+        args.output.write_text(
+            yaml.safe_dump(result.output, sort_keys=False, allow_unicode=True),
+            encoding="utf-8",
+        )
+    else:
+        schema_yml = yaml.safe_load(args.input.read_text(encoding="utf-8"))
+        result = LightdashToOSIConverter().convert(
+            schema_yml,
+            database=args.database,
+            schema=args.schema,
+            semantic_model_name=args.semantic_model_name,
+        )
+        document = result.output.model_dump(by_alias=True, exclude_none=True)
+        if args.output.suffix == ".json":
+            args.output.write_text(
+                json.dumps(document, indent=2, ensure_ascii=False), encoding="utf-8"
+            )
+        else:
+            args.output.write_text(
+                yaml.safe_dump(document, sort_keys=False, allow_unicode=True),
+                encoding="utf-8",
+            )
+
+    _print_issues(result.issues)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())