Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 50 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,14 @@ jobs:
- name: Install
run: cd py && pip install -e ".[dev]"

- name: Typecheck
- name: Typecheck (advisory)
# Surfaced but non-blocking: glyph/patch.py has pre-existing nullability
# errors in the apply logic (tracked for a follow-up cleanup). This shows
# them as a visible CI annotation instead of swallowing them with `|| true`.
continue-on-error: true
run: |
pip install mypy
cd py && mypy glyph/__init__.py glyph/types.py glyph/patch.py --ignore-missing-imports --no-error-summary || true
cd py && mypy glyph/__init__.py glyph/types.py glyph/patch.py --ignore-missing-imports --no-error-summary

- name: Test with coverage
run: cd py && pytest tests/ -v --tb=short --co -q 2>/dev/null; pytest tests/ -v --cov=glyph --cov-report=term-missing --cov-fail-under=80
Expand Down Expand Up @@ -138,7 +142,8 @@ jobs:
run: cd js && npm ci

- name: Typecheck
run: cd js && npx tsc --noEmit 2>/dev/null || true
# Gating: tsc --noEmit is clean and must stay clean.
run: cd js && npx tsc --noEmit

- name: Test with coverage
run: cd js && npx jest --coverage
Expand Down Expand Up @@ -347,12 +352,45 @@ jobs:
fi
echo "All checks passed. Safe to publish."

# ─── Release Meta: per-ecosystem tag↔version gate ──────────────────
# A single v* tag must NOT blindly fan out to both registries. This job
# compares the tag against each package's own version and emits a publish
# flag per ecosystem. PyPI/npm publish only when the tag matches THAT
# package's version, so a tag can ship one ecosystem without the other.
release-meta:
name: Release Meta
needs: [publish-gate]
runs-on: ubuntu-latest
# Custom `if:` overrides the implicit success() that `needs:` would apply, so
# it MUST re-assert the gate explicitly — otherwise a tag push whose suite
# FAILED would still run this job and trigger the publish jobs (fail-open).
if: startsWith(github.ref, 'refs/tags/v') && needs.publish-gate.result == 'success'
outputs:
pypi: ${{ steps.check.outputs.pypi }}
npm: ${{ steps.check.outputs.npm }}
steps:
- uses: actions/checkout@v4
- id: check
run: |
TAG="${GITHUB_REF#refs/tags/v}"
# Anchor the key and accept either quote style (PEP 621 allows both).
PY=$(grep -m1 -E '^version[[:space:]]*=' py/pyproject.toml | sed -E "s/.*=[[:space:]]*[\"']([^\"']+)[\"'].*/\1/")
JS=$(grep -m1 '"version"' js/package.json | sed -E 's/.*"version"[[:space:]]*:[[:space:]]*"([^"]+)".*/\1/')
echo "Tag=$TAG py=$PY js=$JS"
if [ "$TAG" = "$PY" ]; then echo "pypi=true" >> "$GITHUB_OUTPUT"; else echo "pypi=false" >> "$GITHUB_OUTPUT"; fi
if [ "$TAG" = "$JS" ]; then echo "npm=true" >> "$GITHUB_OUTPUT"; else echo "npm=false" >> "$GITHUB_OUTPUT"; fi
if [ "$TAG" != "$PY" ] && [ "$TAG" != "$JS" ]; then
echo "::error::Tag v$TAG matches neither py ($PY) nor js ($JS) version — refusing to publish"
exit 1
fi
echo "Will publish: PyPI=$([ "$TAG" = "$PY" ] && echo yes || echo no), npm=$([ "$TAG" = "$JS" ] && echo yes || echo no)"

# ─── Publish: PyPI ─────────────────────────────────────────────────
publish-pypi:
name: Publish to PyPI
needs: [publish-gate]
needs: [release-meta]
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
if: needs.release-meta.outputs.pypi == 'true'
permissions:
id-token: write
steps:
Expand All @@ -372,19 +410,20 @@ jobs:
# ─── Publish: npm ──────────────────────────────────────────────────
publish-npm:
name: Publish to npm
needs: [publish-gate]
needs: [release-meta]
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
if: needs.release-meta.outputs.npm == 'true'
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}
registry-url: 'https://registry.npmjs.org'
- name: Build and publish
run: |
cd js && npm ci && npm run build 2>/dev/null || true
npm publish --access public
# Fail closed: a failed build must abort the publish (no `|| true`).
- name: Build
run: cd js && npm ci && npm run build
- name: Publish
run: cd js && npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}

4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ coverage/

# Generated
explainer.html
gauntlet/scenarios/report.json

# AI tool context dumps (not source)
glyph_*_context.txt

# Hypothesis test database
.hypothesis/
2 changes: 1 addition & 1 deletion PARITY_ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,4 +80,4 @@ critiques are already handled or moot. The real remaining core is a focused **P0
- **Does Loose preserve types or intentionally collapse to JSON-like?** (Determines whether time/ID/bytes round-trip through Loose or only through Typed.) — *biggest fork; decide first, it shapes P0.*
- **Keep or drop Decimal128 and EmitV2** as advertised features, or move both behind experimental?
- **`@lyph` vs `@glyph`** header spelling — parser accepts both (`parse_header.go:38`); pick one canonical for the emitter.
- **Float rule** (already-open) — shortest-roundtrip (Go) vs threshold (Py/JS); unify or keep documented-divergent.
- ~~**Float rule** — shortest-roundtrip (Go) vs threshold (Py/JS)~~ **RESOLVED:** unified to shortest-round-trip + safe-integer typing; byte-identical across Go/Py/JS (`tests/all_impl_parity_test.py`, `py/tests/test_golden_corpus.py`). Authoritative rule: `docs/CANONICAL_FORMS.md` §3.
20 changes: 13 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,12 +84,14 @@ Poor fits:
| Language | Package | Docs |
|----------|---------|------|
| Python | `pip install glyph-py` | [Python README](./py/README.md) |
| Go | `go get github.com/Neumenon/glyph` | [Go README](./go/README.md) |
| Go | in-repo / source preview — build under `go/` (`go get` not yet a stable path) | [Go README](./go/README.md) |
| JavaScript / TypeScript | `npm install cowrie-glyph` | [JS README](./js/README.md) |
| Rust | parked in `attic/rust/glyph-codec/` — emit-only, not published | [Rust README](./attic/rust/glyph-codec/README.md) |
| C | parked in `attic/c/glyph-codec/` — emit-only, build from source | [C README](./attic/c/glyph-codec/README.md) |

> **Note:** Rust and C ports are parked in `attic/`. They emit canonical GLYPH-Loose but are not conformance ports (no text parser, no patch/GS1/pack). They are not published; `cargo add glyph-rs` is not a valid install path.
>
> **Go status:** the Go codec is a full conformance implementation, but it is currently an **in-repo / source preview**. The module lives under `go/`, and external `go get github.com/Neumenon/glyph` / `go mod tidy` do not yet resolve cleanly (module is in a subdirectory and an optional dev-only bridge pulls an unpublished dependency). Use it from a checkout of this repo — `cd go && go build ./...` — until the external module packaging is stabilized. See the [Go README](./go/README.md) for details.

## Examples

Expand Down Expand Up @@ -144,12 +146,15 @@ Repeated keys are emitted once. The savings show up exactly where agent traces h
### 4. Patch with base fingerprint

```glyph
@patch base=sha256:f35719430d98a2fe @ops=[
{op=replace path=[memory 3 status] value=done}
]
@patch @target=m:session @base=9202d6f0ad620860
= steps[2].status done
~ turn +1
@end
```

The patch records a `base` fingerprint. In the GS1 stream layer, the cursor enforces it — rejecting any patch whose `base` does not match the current state's fingerprint, so a stale patch fails explicitly instead of silently corrupting state. (Standalone `apply_patch` records the base but does not itself verify it today; outside the stream layer the receiver must check the fingerprint before applying.)
A patch is a header line (`@patch` with optional `@target=` and `@base=`), one operation per line, and an `@end` footer. The operation verbs are `=` set, `+` append, `-` delete, and `~` numeric delta.

`@base=` records a 16-hex digest of the base state's canonical form (the first 16 hex of `sha256(canonical_bytes)`), identical across Go, Python, and JS. In the GS1 stream layer (Go and JS) the cursor enforces it — rejecting any patch whose `base` does not match the current state, so a stale patch fails explicitly instead of silently corrupting state. Standalone `apply_patch` does not auto-verify; outside the stream layer, call `verify_patch_base(base, patch)` (Go `VerifyPatchBase`) before applying.

### 5. Stream frame (GS1) — Go and JS only

Expand Down Expand Up @@ -189,8 +194,9 @@ These hold across the conformance-tested implementation surface:
parse(emit(x)) = x
emit(parse(s)) = canonical(s)
fingerprint(x) = SHA256(canonical_no_tabular_bytes(x)) # Go/Python/JS value identity
patch.base records the fingerprint of the base state; GS1 cursor layer enforces
base matching on the stream; standalone ApplyPatch does NOT verify
patch.base = first 16 hex of SHA256(canonical_loose_bytes(base)); GS1 cursor
enforces base matching on the stream; standalone ApplyPatch does NOT
verify (call verify_patch_base / VerifyPatchBase first)
JSON ↔ GLYPH preserves JSON-domain meaning
conformance impls (Go/Python/JS) agree byte-for-byte on canonical form for the shared corpus
```
Expand Down
8 changes: 5 additions & 3 deletions docs/API_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,10 @@ The purpose is to give the current package names, import surfaces, and the core
| Language | Package | Primary Doc |
|----------|---------|-------------|
| Python | `glyph-py` | [../py/README.md](../py/README.md) |
| Go | `github.com/Neumenon/glyph` | [../go/README.md](../go/README.md) |
| Go | in-repo / source preview (module under `go/`; `go get` not yet a stable path) | [../go/README.md](../go/README.md) |
| JavaScript / TypeScript | `cowrie-glyph` | [../js/README.md](../js/README.md) |
| Rust | parked in `attic/` — emit-only, not published | [../attic/rust/glyph-codec/README.md](../attic/rust/glyph-codec/README.md) |
| C | parked in `attic/` — emit-only, build from source | [../attic/c/glyph-codec/README.md](../attic/c/glyph-codec/README.md) |
| C | parked in `attic/c/` — emit-only, build from source | [../attic/c/glyph-codec/README.md](../attic/c/glyph-codec/README.md) |

## Shared Concepts

Expand Down Expand Up @@ -70,7 +70,9 @@ fingerprint = glyph.fingerprint_loose(glyph.from_json(data))
Use the `glyph` module after installing `glyph-py`. The Python README is the current source of truth for the shipped Python surface.

### Go
The module is `github.com/Neumenon/glyph`. Import the codec package as:
**In-repo / source preview.** The Go codec is a full conformance implementation, but it is not yet a polished external module: the module lives under `go/`, and `go get github.com/Neumenon/glyph` / `go mod tidy` do not yet resolve cleanly (subdirectory layout plus an optional dev-only bridge that pulls an unpublished dependency). Use it from a checkout of this repo (`cd go && go build ./...`) until module packaging is stabilized.

Within the module, the import path is:

```go
import "github.com/Neumenon/glyph/glyph"
Expand Down
43 changes: 23 additions & 20 deletions docs/CANONICAL_FORMS.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,19 +83,29 @@ Int(-100) → -100

### 3.1 Canonical Rule (D4)

The canonical float form MUST satisfy BOTH of the following conditions simultaneously:
The canonical float form MUST satisfy ALL of the following conditions simultaneously:

1. **Shortest round-trip digits.** Use the minimum number of significant digits such that
`parse(emit(f)) == f` holds exactly (IEEE 754 double round-trip).
2. **Always include a decimal point.** A float MUST be distinguishable from an int at the
lexical level. If shortest-round-trip produces no decimal point or exponent character
(i.e. the value is a whole number), append `.0`.

This rule is **identical across Go, Python, and JS** and across all emit modes. It supersedes
the threshold-based rule (exponent when `exp < -4` or `exp >= 15`) documented in
`LOOSE_MODE_SPEC.md:29,34-38` and `SPECIFICATIONS.md:54,59-63`, which was disclosed as an
open divergence. The threshold rule is hereby retired. Implementations MUST migrate to the
shortest-round-trip-with-decimal-point rule stated here.
3. **Exponential notation boundary.** Render in exponential form (`e±NN`, lowercase `e`,
exponent zero-padded to ≥ 2 digits) when the decimal exponent is `<= -5` or `>= 6`;
otherwise use plain decimal. Examples: `999999.9` (exp 5) → `999999.9`; `1234567.5`
(exp 6) → `1.2345675e+06`; `0.0001` (exp -4) → `0.0001`; `0.00001` (exp -5) → `1e-05`.

This rule is **unified and byte-identical across Go, Python, and JS** (verified by the
cross-implementation corpus — see `tests/all_impl_parity_test.py`). It replaced an earlier
threshold-based digit rule (exponent when `exp < -4` or `exp >= 15`) that was once an open
divergence; that rule is **retired** and all three implementations now emit identically.

> **Loose vs typed layer.** Conditions 1–3 describe the canonical form of a value whose type
> is *float* (typed mode, or an explicitly-constructed float). In **GLYPH-Loose**, JSON
> numbers first go through safe-integer typing: an integer-valued number within
> `|n| <= 2^53-1` becomes an **integer** literal (so `0`, `42`, `1000` — never `0.0`/`42.0`),
> and only non-integer or out-of-window numbers are floats and formatted by this rule. So a
> loose canonical never shows `3.0`; it shows `3`.

**Special values:**

Expand Down Expand Up @@ -123,17 +133,10 @@ lower-case `e` MUST be used) already contains a non-integer indicator and satisf
without an additional `.0`. The `.0` suffix is only required when neither a `.` nor `e` is
present in the shortest-round-trip string.

**Current divergence to fix (ground truth):**

- `emitFloat` in `emit.go:148-153` uses `'f'/-1` format which never uses exponent notation,
then appends `.0` for whole numbers. This satisfies condition 2 but can produce unnecessarily
long strings for large values (e.g. `1e21` would become `1000000000000000000000.0`). This
MUST be corrected to use the shortest-round-trip format.
- `canonFloat` in `canon.go:50-55` uses integer format for whole numbers below `1e6`, breaking
condition 2 for those values (e.g. `Float(1.0)` → `"1"` instead of `"1.0"`). This MUST be
corrected.
- `writeCanonLoose` in `loose.go:475` uses bare `'g'` format with no decimal-point guard,
also breaking condition 2 for whole-number floats. This MUST be corrected.
**Resolved (W2 — verified byte-identical across Go, Python, and JS):** `emitFloat`,
`canonFloat`, and `writeCanonLoose` now all emit the shortest-round-trip form with a
guaranteed decimal point or exponent (`Float(1.0) → "1.0"`, `Float(1e21) → "1e+21"`,
`Float(-0.0) → "0.0"`). The historical per-path divergences once tracked here are closed.

---

Expand Down Expand Up @@ -543,8 +546,8 @@ Notes on the table:

| Topic | This document | LOOSE_MODE_SPEC.md | SPECIFICATIONS.md |
|-------|--------------|--------------------|--------------------|
| Float format | Section 3 (D4 — supersedes) | §Float Formatting (threshold rule — retired) | §Float Formatting (threshold rule — retired) |
| Float zero / negative zero (G6) | Section 3.1: `Float(0.0)→"0.0"`, `Float(-0.0)→"0.0"` (D4 — **supersedes**) | "Zero: always 0" — **RETIRED** | Not specified |
| Float format | Section 3 (D4 — **authoritative**) | §Number Formatting (defers to §3; unified) | §Number Formatting (defers to §3; unified) |
| Float zero / negative zero (G6) | Section 3.1: float type `Float(0.0)→"0.0"`, `Float(-0.0)→"0.0"`; loose collapses to int `0` (see §3.1 layer note) | §Number Formatting: loose zero → `0` (safe-int collapse) | §Number Formatting: loose zero → `0` |
| NaN/Inf | Section 4 (D3) | "NaN/Infinity: Rejected with error" | "NaN/Infinity: Rejected with error" |
| Bare-string rule | Section 5 (D8 — conservative) | §String Bare-Safe Rule (allows Unicode) | §String Bare-Safe Rule (allows Unicode) |
| Bytes form | Section 6 (D6) | Not addressed | `b64"..."` mentioned in type table |
Expand Down
15 changes: 6 additions & 9 deletions docs/GLYPH_T_SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,10 +185,9 @@ summary for reference:
then appends `.0` if no decimal point is present. This correctly ensures
`Float(1.0) → "1.0"` and `Float(0.1) → "0.1"`.

**Known bug in `canonFloat` (canon.go:39-65):** integral floats below 1e6
are emitted without a decimal point (e.g. `Float(1.0) → "1"`). This breaks
the D4 rule and causes cross-language fingerprint divergence. Fix in W2:
`canonFloat` must append `.0` for integral values, matching `emitFloat`.
**Resolved (W2):** `canonFloat` now appends `.0` for integral floats, so
`Float(1.0) → "1.0"` — byte-identical across Go, Python, and JS (verified by the
cross-implementation corpus).

#### Bytes canonical form (D6)

Expand Down Expand Up @@ -599,9 +598,9 @@ Both share the same canonical scalar forms (after bugs are fixed). The
`FingerprintLoose` (SHA-256 over `CanonicalizeLoose`) is the stable
cross-language hash; it MUST be byte-identical across Go, Python, and JS.

Float unification (D4) is required for cross-language fingerprint parity.
See `LOOSE_MODE_SPEC.md §Float Formatting` and `SPECIFICATIONS.md:205` for
the currently documented divergence.
Float unification (D4) is required for cross-language fingerprint parity, and is
**resolved** — the float rule is byte-identical across Go, Python, and JS. See
`LOOSE_MODE_SPEC.md §Number Formatting` and `CANONICAL_FORMS.md §3` (authoritative).

### 6.3 Cross-mode value identity

Expand All @@ -628,7 +627,6 @@ may change or be removed without notice (see `doc.go:69-73`):
| Issue | Severity | Work item |
|-------|----------|-----------|
| `emit.go:108` wrong time format (offset-preserving) | High | W2 |
| `canonFloat` drops decimal point for integral floats | High | W2 |
| `emit.go:111-116` unquoted unsafe refs | High | W2/W3 |
| `emit_packed.go:269`, `emit_tabular.go:202` raw-bytes bug | High | W3 |
| `parseLooseValue` no `b64"..."` branch | High | W3 |
Expand All @@ -637,6 +635,5 @@ may change or be removed without notice (see `doc.go:69-73`):
| `parsePathToSegs` silently ignores `Atoi` error | Medium | W6 |
| `parsePathToSegs` unescaped map-key body | Medium | W6 |
| `parseRefIDFromTarget` first-`:` split (no escaping) | Medium | W6 |
| `canonFloat` in Python/JS still uses threshold rule (D4) | High | W8 |
| Time sub-second trimming in all emit paths | High | W2/W3 |
| NaN/Inf guard missing in `canonFloat`/`writeCanonLoose` | Medium | W2/W3 |
Loading
Loading