Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
| Bun | Deno |
| pnpm/yarn | Deno |
| Go | Rust |
| Python | Julia/Rust/ReScript |
| Python | Julia/Rust/AffineScript |
| Java/Kotlin | Rust/Tauri/Dioxus |
| Swift | Tauri/Dioxus |
| React Native | Tauri/Dioxus |
Expand All @@ -57,7 +57,7 @@ Both are FOSS with independent governance (no Big Tech).
### Package Management

- **Primary**: Guix (guix.scm)
- **Fallback**: Nix (flake.nix)
- **Fallback**: Guix (flake.guix)
- **JS deps**: Deno (deno.json imports)

### Security Requirements
Expand Down
4 changes: 2 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ updates:
schedule:
interval: "weekly"

# Nix flakes
- package-ecosystem: "nix"
# Guix flakes
- package-ecosystem: "guix"
directory: "/"
schedule:
interval: "weekly"
226 changes: 142 additions & 84 deletions ABI-FFI-README.md → ABI-FFI-README.adoc
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
# Docudactyl ABI/FFI Documentation
== Docudactyl ABI/FFI Documentation

## Overview
=== Overview

This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design:
This library follows the *Hyperpolymath RSR Standard* for ABI and FFI
design:

- **ABI (Application Binary Interface)** defined in **Idris2** with formal proofs
- **FFI (Foreign Function Interface)** implemented in **Zig** for C compatibility
- **Generated C headers** bridge Idris2 ABI to Zig FFI
- **Chapel HPC** calls through standard C ABI for distributed execution
* *ABI (Application Binary Interface)* defined in *Idris2* with formal
proofs
* *FFI (Foreign Function Interface)* implemented in *Zig* for C
compatibility
* *Generated C headers* bridge Idris2 ABI to Zig FFI
* *Chapel HPC* calls through standard C ABI for distributed execution

## Architecture
=== Architecture

```
....
┌─────────────────────────────────────────────┐
│ ABI Definitions (Idris2) │
│ src/Docudactyl/ABI/ │
Expand Down Expand Up @@ -48,11 +51,11 @@ This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design:
│ - 51 extern proc declarations │
│ - Distributed across 64-512 locales │
└─────────────────────────────────────────────┘
```
....

## Directory Structure
=== Directory Structure

```
....
docudactyl/
├── src/
│ └── Docudactyl/
Expand Down Expand Up @@ -85,84 +88,137 @@ docudactyl/
│ └── docudactyl_ffi.h # Generated from Zig FFI exports
└── docudactyl.ipkg # Idris2 package (3 modules)
```
....

## Proven Types (Idris2)
=== Proven Types (Idris2)

The ABI layer formally proves:

| Type | Variants | Proof |
|------|----------|-------|
| `ContentKind` | 7 (PDF, Image, Audio, Video, EPUB, GeoSpatial, Unknown) | Enum injectivity |
| `ParseStatus` | 6 (Ok, Error, FileNotFound, ParseError, UnsupportedFormat, OutOfMemory) | Enum injectivity |
| `MlStatus` | 5 (Ok, ModelNotFound, InferenceError, InputError, OnnxNotAvailable) | Enum injectivity |
| `MlStage` | 5 (NER, Whisper, ImageClassify, Layout, Handwriting) | Enum injectivity |
| `ExecProvider` | 4 (TensorRT, CUDA, OpenVINO, CPU) | Enum injectivity |
| `Sha256Tier` | 3 (Dedicated, Avx2Buffer, Software) | Enum injectivity |
| `OcrStatus` | 4 | Enum injectivity |
| `GpuBackend` | 3 | Enum injectivity |
| `ConduitValidation` | 4 | Enum injectivity |

### Struct Layout Proofs

| Struct | Size | Alignment | Proof |
|--------|------|-----------|-------|
| `ParseResult` | 952 bytes | 8-byte (LP64) | `Divides 8 952 = MkDivides 119` |
| `MlResult` | 48 bytes | 8-byte | `Divides 8 48 = MkDivides 6` |
| `CryptoCaps` | 16 bytes | 1-byte | Field offset chain |
| `OcrResult` | 48 bytes | — | Size assertion |
| `ConduitResult` | 88 bytes | — | Size assertion |

## Building

### Build FFI Library

```bash
[width="100%",cols="27%,43%,30%",options="header",]
|===
|Type |Variants |Proof
|`+ContentKind+` |7 (PDF, Image, Audio, Video, EPUB, GeoSpatial,
Unknown) |Enum injectivity

|`+ParseStatus+` |6 (Ok, Error, FileNotFound, ParseError,
UnsupportedFormat, OutOfMemory) |Enum injectivity

|`+MlStatus+` |5 (Ok, ModelNotFound, InferenceError, InputError,
OnnxNotAvailable) |Enum injectivity

|`+MlStage+` |5 (NER, Whisper, ImageClassify, Layout, Handwriting) |Enum
injectivity

|`+ExecProvider+` |4 (TensorRT, CUDA, OpenVINO, CPU) |Enum injectivity

|`+Sha256Tier+` |3 (Dedicated, Avx2Buffer, Software) |Enum injectivity

|`+OcrStatus+` |4 |Enum injectivity

|`+GpuBackend+` |3 |Enum injectivity

|`+ConduitValidation+` |4 |Enum injectivity
|===

==== Struct Layout Proofs

[width="100%",cols="27%,18%,34%,21%",options="header",]
|===
|Struct |Size |Alignment |Proof
|`+ParseResult+` |952 bytes |8-byte (LP64)
|`+Divides 8 952 = MkDivides 119+`

|`+MlResult+` |48 bytes |8-byte |`+Divides 8 48 = MkDivides 6+`

|`+CryptoCaps+` |16 bytes |1-byte |Field offset chain

|`+OcrResult+` |48 bytes |— |Size assertion

|`+ConduitResult+` |88 bytes |— |Size assertion
|===

=== Building

==== Build FFI Library

[source,bash]
----
just build-ffi # Build via Justfile
# or directly:
cd ffi/zig && zig build -Doptimize=ReleaseFast
```
----

### Verify Idris2 ABI Proofs
==== Verify Idris2 ABI Proofs

```bash
[source,bash]
----
just build-idris
# or directly:
idris2 --build docudactyl.ipkg
```
----

### Generate C Header
==== Generate C Header

```bash
[source,bash]
----
just generate-abi-header
```
----

### Run Tests
==== Run Tests

```bash
[source,bash]
----
just test-ffi # 40+ integration tests against C ABI
just test-idris # Verify Idris2 proofs compile
```
----

=== C API Summary

All 51 functions use the `+ddac_+` prefix:

[width="100%",cols="48%,52%",options="header",]
|===
|Category |Functions
|*Core lifecycle* |`+ddac_init+`, `+ddac_free+`, `+ddac_parse+`,
`+ddac_version+`

|*Handle setters* |`+ddac_set_ml_handle+`, `+ddac_set_gpu_ocr_handle+`

|*LMDB cache* |`+ddac_cache_init+`, `+ddac_cache_free+`,
`+ddac_cache_lookup+`, `+ddac_cache_store+`, `+ddac_cache_count+`,
`+ddac_cache_sync+`

|*Dragonfly* |`+ddac_dragonfly_connect+`, `+ddac_dragonfly_close+`,
`+ddac_dragonfly_lookup+`, `+ddac_dragonfly_store+`,
`+ddac_dragonfly_count+`

|*I/O prefetcher* |`+ddac_prefetch_init+`, `+ddac_prefetch_hint+`,
`+ddac_prefetch_done+`, `+ddac_prefetch_free+`,
`+ddac_prefetch_inflight+`

|*ML inference* |`+ddac_ml_init+`, `+ddac_ml_free+`,
`+ddac_ml_available+`, `+ddac_ml_provider+`, `+ddac_ml_provider_name+`,
`+ddac_ml_set_model_dir+`, `+ddac_ml_run_stage+`, `+ddac_ml_stats+`,
`+ddac_ml_result_size+`, `+ddac_ml_stage_count+`, `+ddac_ml_model_name+`

## C API Summary
|*GPU OCR* |`+ddac_gpu_ocr_init+`, `+ddac_gpu_ocr_free+`,
`+ddac_gpu_ocr_backend+`, `+ddac_gpu_ocr_submit+`,
`+ddac_gpu_ocr_flush+`, `+ddac_gpu_ocr_results_ready+`,
`+ddac_gpu_ocr_collect+`, `+ddac_gpu_ocr_stats+`,
`+ddac_gpu_ocr_max_batch+`, `+ddac_gpu_ocr_result_size+`

All 51 functions use the `ddac_` prefix:
|*Hardware crypto* |`+ddac_crypto_detect+`, `+ddac_crypto_sha256_tier+`,
`+ddac_crypto_sha256_name+`, `+ddac_crypto_batch_sha256+`,
`+ddac_crypto_caps_size+`

| Category | Functions |
|----------|-----------|
| **Core lifecycle** | `ddac_init`, `ddac_free`, `ddac_parse`, `ddac_version` |
| **Handle setters** | `ddac_set_ml_handle`, `ddac_set_gpu_ocr_handle` |
| **LMDB cache** | `ddac_cache_init`, `ddac_cache_free`, `ddac_cache_lookup`, `ddac_cache_store`, `ddac_cache_count`, `ddac_cache_sync` |
| **Dragonfly** | `ddac_dragonfly_connect`, `ddac_dragonfly_close`, `ddac_dragonfly_lookup`, `ddac_dragonfly_store`, `ddac_dragonfly_count` |
| **I/O prefetcher** | `ddac_prefetch_init`, `ddac_prefetch_hint`, `ddac_prefetch_done`, `ddac_prefetch_free`, `ddac_prefetch_inflight` |
| **ML inference** | `ddac_ml_init`, `ddac_ml_free`, `ddac_ml_available`, `ddac_ml_provider`, `ddac_ml_provider_name`, `ddac_ml_set_model_dir`, `ddac_ml_run_stage`, `ddac_ml_stats`, `ddac_ml_result_size`, `ddac_ml_stage_count`, `ddac_ml_model_name` |
| **GPU OCR** | `ddac_gpu_ocr_init`, `ddac_gpu_ocr_free`, `ddac_gpu_ocr_backend`, `ddac_gpu_ocr_submit`, `ddac_gpu_ocr_flush`, `ddac_gpu_ocr_results_ready`, `ddac_gpu_ocr_collect`, `ddac_gpu_ocr_stats`, `ddac_gpu_ocr_max_batch`, `ddac_gpu_ocr_result_size` |
| **Hardware crypto** | `ddac_crypto_detect`, `ddac_crypto_sha256_tier`, `ddac_crypto_sha256_name`, `ddac_crypto_batch_sha256`, `ddac_crypto_caps_size` |
| **Conduit** | `ddac_conduit_process`, `ddac_conduit_batch`, `ddac_conduit_result_size` |
|*Conduit* |`+ddac_conduit_process+`, `+ddac_conduit_batch+`,
`+ddac_conduit_result_size+`
|===

## Usage from Chapel
=== Usage from Chapel

```chapel
[source,chapel]
----
extern proc ddac_init(): c_ptr(void);
extern proc ddac_free(handle: c_ptr(void)): void;
extern proc ddac_parse(handle: c_ptr(void), inputPath: c_ptrConst(c_char),
Expand All @@ -173,30 +229,32 @@ extern proc ddac_parse(handle: c_ptr(void), inputPath: c_ptrConst(c_char),
var handle = ddac_init();
defer ddac_free(handle);
var result = ddac_parse(handle, path.c_str(), outPath.c_str(), "scheme".c_str());
```
----

## Contributing
=== Contributing

When modifying the ABI/FFI:

1. **Update Idris2 ABI first** (`src/Docudactyl/ABI/*.idr`)
- Add/modify type definitions with proofs
- Update struct layout proofs
- Add FFI declarations
2. **Update Zig FFI** (`ffi/zig/src/`)
- Implement new functions matching C ABI
- Ensure `comptime` assertions match Idris2 proofs
3. **Regenerate C header** (`just generate-abi-header`)
4. **Add integration tests** (`ffi/zig/test/integration_test.zig`)
5. **Update Chapel extern declarations** (`src/chapel/FFIBridge.chpl`)
[arabic]
. *Update Idris2 ABI first* (`+src/Docudactyl/ABI/*.idr+`)
* Add/modify type definitions with proofs
* Update struct layout proofs
* Add FFI declarations
. *Update Zig FFI* (`+ffi/zig/src/+`)
* Implement new functions matching C ABI
* Ensure `+comptime+` assertions match Idris2 proofs
. *Regenerate C header* (`+just generate-abi-header+`)
. *Add integration tests* (`+ffi/zig/test/integration_test.zig+`)
. *Update Chapel extern declarations* (`+src/chapel/FFIBridge.chpl+`)

## License
=== License

SPDX-License-Identifier: CC-BY-SA-4.0

## See Also
=== See Also

- [Idris2 Documentation](https://idris2.readthedocs.io)
- [Zig Documentation](https://ziglang.org/documentation/master/)
- [Chapel Documentation](https://chapel-lang.org/docs/)
- [Rhodium Standard Repositories](https://github.com/hyperpolymath/rhodium-standard-repositories)
* https://idris2.readthedocs.io[Idris2 Documentation]
* https://ziglang.org/documentation/master/[Zig Documentation]
* https://chapel-lang.org/docs/[Chapel Documentation]
* https://github.com/hyperpolymath/rhodium-standard-repositories[Rhodium
Standard Repositories]
76 changes: 76 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
== Changelog

All notable changes to `+docudactyl+` will be documented in this file.

This file is generated from conventional commits by the
https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml[`+changelog-reusable.yml+`]
workflow (`+hyperpolymath/standards#206+`). Adopt the workflow in this
repo’s CI to keep this file in sync automatically — see
https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+templates/cliff.toml+`]
for the canonical config.

The format follows https://keepachangelog.com/en/1.1.0/[Keep a
Changelog]; this project aims to follow
https://semver.org/spec/v2.0.0.html[Semantic Versioning].

=== [Unreleased]

==== Added

* feat(investigator): add five modules for citizen & investigative
journalism (#2)
* feat(crg): add crg-grade and crg-badge justfile recipes
* feat: add stapeln.toml container definition
* feat: deploy UX Manifesto infrastructure
* feat: add CLADE.a2ml — clade taxonomy declaration

==== Fixed

* fix(ci): bump a2ml/k9-validate-action pins to canonical (standards#85)
(#16)
* fix(ci): sync hypatia-scan.yml to canonical (kill cd-scanner build
drift) (#15)
* fix(ci): adopt canonical hypatia-scan.yml (env.HOME/scanner-layout +
Comment-step gate) (#14)
* fix(ci): Phase-2 fleet submission must not fail the security gate
(#13)
* fix(ci): hypatia-scan workdir ($\{\{ env.HOME }} resolves empty) (#12)
* fix(ci): rsr-antipattern.yml duplicate heredoc (#9)
* fix(ci): move secret-scanner Cargo.toml gate from job-level if: to
step-level (#10)
* fix(ci): fix broken regex quoting in security-policy and update
trufflehog SHA
* fix(scorecard): enforce granular permissions and add fuzzing
placeholder
* fix(ci): Resolve workflow-linter self-matching and metadata issues

==== Changed

* refactor: migrate 6SCM → 6A2 (.scm → .a2ml format)

==== Documentation

* docs: add TEST-NEEDS.md (CRG C)
* docs: add TEST-NEEDS.md (CRG C)
* docs: add EXPLAINME.adoc — prove-it file backing README claims

==== CI

* ci: redistribute concurrency-cancel guard to read-only check workflows
(#18)
* ci: bump actions/upload-artifact SHA to current v4 (#8)
* ci(secret-scanner): drop duplicate –fail from trufflehog extra_args
(#7)
* ci: SHA-pin hyperpolymath validate-actions in dogfood-gate
* ci(antipattern): fix top-level dir matching + benchmarks/lsp/bench
filename allowlists (#6)

=== Pre-history

Prior commits to this file’s introduction are recorded in git history
but not formally classified into Keep-a-Changelog sections. To backfill,
run `+git cliff -o CHANGELOG.md+` locally using the canonical
https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+cliff.toml+`]
— this is one-shot mechanical work.

'''''
Loading
Loading