Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ jobs:
with:
python-version: '3.13'
enable-cache: true
- name: Spelling
run: make spelling
- name: Workflow contract tests
run: make test-workflow-contracts
- name: Test
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,9 @@ target/
**/*.rs.bk
.memdb/
__pycache__/
.coverage
.pytest_cache/
.uv-cache/
.uv-tools/
.typos-oxendict-base.json
.typos-oxendict-base.toml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ def reciprocal(n: float) -> float:
return result
```

`else` emphasises the happy path and avoids odd control‑flow within `try`
`else` emphasizes the happy path and avoids odd control‑flow within `try`
blocks.

## 4) Message construction for raises (EM101/EM102) and logging practice (LOG004/LOG007/LOG009/LOG014/LOG015, TRY401)
Expand All @@ -110,7 +110,7 @@ raise RuntimeError(msg)
EM101/EM102 prefer a single message object; this reduces duplication and
clarifies intent.

### Logging: parameterised messages, module loggers, correct APIs
### Logging: parameterized messages, module loggers, correct APIs

```python
import logging
Expand Down
2 changes: 1 addition & 1 deletion .rules/python-pyproject.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ package = true
Reddit[^5])
- `description`, `readme`, `requires-python`: provide clarity about the
project and help tools like PyPI. (Python Packaging[^4], Reddit[^5])
- `license`, `authors`, `keywords`, `classifiers`: standardised metadata,
- `license`, `authors`, `keywords`, `classifiers`: standardized metadata,
which improves discoverability. (Python Packaging[^4], Reddit[^5])
- `dependencies`: runtime requirements, expressed in PEP 508 syntax.
(Astral Docs[^1], RidgeRun.ai[^2])
Expand Down
2 changes: 1 addition & 1 deletion .rules/python-typing.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

Use `Enum` for fixed sets of related constants. Use `enum.auto()` to avoid
repeating values manually. Use `IntEnum` or `StrEnum` when interoperability
with integers or strings is required (e.g. for database or JSON serialisation).
with integers or strings is required (e.g. for database or JSON serialization).

```python
import enum
Expand Down
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
examples demonstrating the usage and outcome of the function. Test
documentation should omit examples where the example serves only to reiterate
the test logic.
- **Keep file size managable.** No single code file may be longer than 400
- **Keep file size manageable.** No single code file may be longer than 400
lines. Long switch statements or dispatch tables should be broken up by
feature and constituents colocated with targets. Large blocks of test data
should be moved to external data files.
Expand Down Expand Up @@ -59,7 +59,7 @@
- Where a bug is being fixed, a unittest has been provided demonstrating the
behaviour being corrected both to validate the fix and to guard against
regression.
- Passes all relevant unit and behavioral tests according to the guidelines
- Passes all relevant unit and behavioural tests according to the guidelines
above. (Use `make test` to verify).
- Passes lint checks. (Use `make lint` to verify).
- Adheres to formatting standards tested using a formatting validator. (Use
Expand Down Expand Up @@ -106,7 +106,7 @@
- **Separate Atomic Refactors:** If refactoring is deemed necessary:
- Perform the refactoring as a **separate, atomic commit** *after* the
functional change commit.
- Ensure the refactoring adheres to the testing guidelines (behavioral tests
- Ensure the refactoring adheres to the testing guidelines (behavioural tests
pass before and after, unit tests added for new units).
- Ensure the refactoring commit itself passes all quality gates.

Expand Down Expand Up @@ -173,7 +173,7 @@ project:
- Lint rule suppressions must be tightly scoped and include a clear reason.
- Prefer `expect` over `allow`.
- Use `rstest` fixtures for shared setup.
- Replace duplicated tests with `#[rstest(...)]` parameterised cases.
- Replace duplicated tests with `#[rstest(...)]` parameterized cases.
- Prefer `mockall` for mocks/stubs.
- Prefer `.expect()` over `.unwrap()`.
- Use `concat!()` to combine long string literals rather than escaping newlines
Expand All @@ -199,7 +199,7 @@ project:
`into_inner()` and `to_path_buf()`; avoid attempting
`impl From<Wrapper> for PathBuf` because of the orphan rule. Prefer explicit
tuple structs whenever bespoke validation or tailored trait surfaces are
required, customising `Deref`, `AsRef`, and `TryFrom` per type. Use
required, customizing `Deref`, `AsRef`, and `TryFrom` per type. Use
`the-newtype` when defining traits and needing blanket implementations that
apply across wrappers satisfying `Newtype + AsRef/AsMut<Inner>`, or when
establishing a coherent internal convention that keeps trait forwarding
Expand All @@ -212,7 +212,7 @@ project:
- Write unit and behavioural tests for new functionality. Run both before and
after making any change.
- Use `rstest` fixtures for shared setup.
- Replace duplicated tests with `#[rstest(...)]` parameterised cases.
- Replace duplicated tests with `#[rstest(...)]` parameterized cases.
- Prefer `mockall` for mocks/stubs.
- Mock non-deterministic dependencies (e.g., environment variables and the
system clock) using dependency injection with the `mockable` crate (traits
Expand Down
35 changes: 31 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
.PHONY: help all clean test test-workflow-contracts build release lint typecheck fmt check-fmt markdownlint nixie bless
.PHONY: help all clean test test-workflow-contracts build release lint typecheck \
fmt check-fmt markdownlint nixie spelling spelling-helper-test bless

APP ?= lag-complexity
APP ?= lagc
CARGO ?= cargo
BUILD_JOBS ?=
CLIPPY_FLAGS ?= --all-targets --all-features -- -D warnings
MDLINT ?= markdownlint-cli2
NIXIE ?= nixie
WHITAKER ?= whitaker
UV ?= uv
UV_ENV = UV_CACHE_DIR=.uv-cache UV_TOOL_DIR=.uv-tools
RUFF_VERSION ?= 0.15.12
TYPOS_VERSION ?= 1.48.0

build: target/debug/$(APP) ## Build debug binary
release: target/release/$(APP) ## Build release binary

all: release ## Default target builds release binary
all: release spelling ## Build the release binary and enforce spelling

clean: ## Remove build artifacts
$(CARGO) clean
Expand Down Expand Up @@ -39,9 +44,31 @@ fmt: ## Format Rust and Markdown sources
check-fmt: ## Verify formatting
$(CARGO) fmt --all -- --check

markdownlint: ## Lint Markdown files
markdownlint: spelling ## Lint Markdown files and enforce spelling
find . -type f -name '*.md' -not -path './target/*' -print0 | xargs -0 $(MDLINT)

spelling: spelling-helper-test ## Enforce en-GB-oxendict spelling in Markdown prose
@$(UV_ENV) $(UV) run scripts/generate_typos_config.py
@git ls-files -z '*.md' | \
xargs -0 -r env $(UV_ENV) $(UV) tool run typos@$(TYPOS_VERSION) \
--config typos.toml --force-exclude
Comment on lines +51 to +54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail on generated spelling config drift

When a PR changes typos.local.toml or the shared dictionary advances, this target rewrites the tracked typos.toml and then immediately runs typos against that uncommitted output. Because the recipe never checks git diff --exit-code typos.toml after regeneration, CI can pass while the committed generated config remains stale, leaving fresh or offline checkouts to use a different spelling policy than the one CI validated.

Useful? React with 👍 / 👎.

Comment on lines +51 to +54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail on generated spelling config drift

When a PR changes typos.local.toml or the shared dictionary advances, this target rewrites the tracked typos.toml and then immediately runs typos against that uncommitted output. Because the recipe never checks git diff --exit-code typos.toml after regeneration, CI can pass while the committed generated config remains stale, leaving fresh or offline checkouts to use a different spelling policy than the one CI validated.

Useful? React with 👍 / 👎.


spelling-helper-test: ## Validate the shared spelling-policy integration
@$(UV_ENV) $(UV) tool run ruff@$(RUFF_VERSION) format --isolated \
--target-version py313 --check scripts/generate_typos_config.py \
scripts/typos_rollout.py scripts/typos_rollout_cache.py \
scripts/tests/test_typos_rollout.py
@$(UV_ENV) $(UV) tool run ruff@$(RUFF_VERSION) check --isolated \
--target-version py313 scripts/generate_typos_config.py \
scripts/typos_rollout.py scripts/typos_rollout_cache.py \
scripts/tests/test_typos_rollout.py
@PYTHONPATH=scripts $(UV_ENV) $(UV) run --no-project --python 3.13 \
--with pytest==9.0.2 --with pytest-cov==7.0.0 \
python -m pytest scripts/tests/test_typos_rollout.py \
-c /dev/null --rootdir=. -p no:cacheprovider \
--cov=generate_typos_config --cov=typos_rollout \
--cov=typos_rollout_cache --cov-fail-under=90

nixie: ## Validate Mermaid diagrams
# CI currently requires --no-sandbox; remove once nixie supports
# environment variable control for this option
Expand Down
8 changes: 4 additions & 4 deletions docs/complexity-antipatterns-and-refactoring-strategies.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## I. Introduction: The Challenge of Software Complexity

Software development is an inherently complex endeavor. As systems evolve and
Software development is an inherently complex endeavour. As systems evolve and
features are added, the intricacy of the codebase tends to increase, often
leading to challenges in maintenance, scalability, and developer productivity.
"Any fool can write code that a computer can understand. Good programmers write
Expand Down Expand Up @@ -95,7 +95,7 @@ Cognitive Complexity is incremented based on three main rules 8:
3. **Shorthand Discount:** Structures that allow multiple statements to be read
as a single unit (e.g., a well-named method call) do not incur the same
penalties as the raw statements they encapsulate. Method calls are generally
"free" in terms of cognitive complexity, as a well-chosen name summarises
"free" in terms of cognitive complexity, as a well-chosen name summarizes
the underlying logic, allowing readers to grasp the high-level view before
diving into details. However, recursive calls do increment the score.8

Expand All @@ -110,7 +110,7 @@ Thresholds and Implications:

Code with high Cognitive Complexity is harder to read, understand, test, and
modify.8 SonarQube, for example, raises issues when a function's Cognitive
Complexity exceeds a certain threshold, signaling that the code should likely
Complexity exceeds a certain threshold, signalling that the code should likely
be refactored into smaller, more manageable pieces.8 The primary impact of high
Cognitive Complexity is a slowdown in development and an increase in
maintenance costs.8
Expand Down Expand Up @@ -338,7 +338,7 @@ perform auto-refactoring for certain languages.6

### C. Red Flags Portending the Bumpy Road

Recognising early warning signs can prevent minor complexity issues from
Recognizing early warning signs can prevent minor complexity issues from
escalating into full-blown Bumpy Roads.

1. **Increasing Cognitive Complexity Scores:** A rising Cognitive Complexity
Expand Down
8 changes: 8 additions & 0 deletions docs/developers-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ This guide records development practices that are specific to maintaining this
repository. Follow the project-wide guidance in `AGENTS.md` first, then use
this guide for workflow automation details.

## Spelling policy

Run `make spelling` to enforce en-GB-oxendict prose spelling. The generated
`typos.toml` starts from the shared estate dictionary, refreshes its untracked
local cache only when the authority is newer, and then applies the narrow
repository policy in `typos.local.toml`. Edit the local policy and regenerate
the configuration rather than changing generated entries by hand.

## Continuous Integration workflow

The Continuous Integration (CI) workflow lives in `.github/workflows/ci.yml`.
Expand Down
2 changes: 1 addition & 1 deletion docs/lag-complexity-function-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -381,7 +381,7 @@ require a wider retrieval strategy.
and results in a dedicated `VarianceError::Empty`. This guards against silent
NaNs and ensures caller intent is explicit.
- **Non-finite Inputs:** `NaN` or infinite values yield `NaN`. Callers should
sanitise embeddings if such values are not expected.
sanitize embeddings if such values are not expected.

#### Embedding providers

Expand Down
6 changes: 3 additions & 3 deletions docs/model-training-pipeline-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Key improvements in this design include:
(training, evaluation, etc.) is executed on dedicated ephemeral cloud
instances. Prefect tasks use Terraform (OpenTofu distribution) to provision
the required **spot or on-demand VM instances** (with GPU for training, CPU
for export/evaluation) and tear them down afterward. This ensures
for export/evaluation) and tear them down afterwards. This ensures
**cloud-agnostic deployment** – by swapping Terraform modules, the pipeline
can target AWS, GCP, or other providers uniformly. It also maximizes cost
efficiency by using spot instances and releasing resources immediately when
Expand Down Expand Up @@ -102,7 +102,7 @@ adhere to several core principles in designing the workflow:
versioned. We use **containerization** (Docker images) and
infrastructure-as-code to ensure that a given pipeline run can be exactly
replicated in the future. This is critical for auditing and for guaranteeing
consistent model behavior between training and production. The pipeline will
consistent model behaviour between training and production. The pipeline will
incorporate configuration files (or Prefect configuration blocks) for all
adjustable parameters (dataset version, model architecture, learning rates,
instance types, etc.), which are stored in version control. By fixing these
Expand Down Expand Up @@ -220,7 +220,7 @@ automate their lifecycle:
2.3). The Terraform provisioning task can request a spot VM (e.g., AWS EC2
Spot or GCP Preemptible VM) for training; if the request is not fulfilled or
the instance is reclaimed mid-training, the Prefect flow will handle
launching a replacement and resuming training. The **cost modeling** takes
launching a replacement and resuming training. The **cost modelling** takes
into account the expected interruptions and restart overhead, ensuring that
even with occasional restarts, using spot instances yields net savings. For
cases where spot is not available or for final runs, the pipeline can easily
Expand Down
20 changes: 10 additions & 10 deletions docs/model-training-pipeline-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,10 @@ progress tracking.
- [ ] 1.1.3.4. Wire alerting (Slack, PagerDuty, email) to Prefect block
failures and add runbook links for operator response.

### 1.2. Standardise compute procurement and cost controls
### 1.2. Standardize compute procurement and cost controls

- [ ] Automate OpenTofu modules for g4dn.xlarge, g6.xlarge, and c6i.xlarge
instances with provider parameterisation and teardown under 10 minutes.
instances with provider parameterization and teardown under 10 minutes.
- [ ] Validate spot-to-on-demand fallbacks by injecting capacity failures and
confirming retries succeed within three orchestration attempts.
- [ ] Publish hourly cost benchmarks for each instance class with variance
Expand All @@ -75,7 +75,7 @@ progress tracking.

- [ ] Build ingestion scripts that hydrate `/datasets/raw/` from approved
sources and emit checksums recorded in metadata manifests.
- [ ] Create preprocessing jobs that materialise `/datasets/processed/`
- [ ] Create preprocessing jobs that materialize `/datasets/processed/`
artefacts with schema validation and rejection workflows for anomalies.
- [ ] Introduce calibration dataset generation producing 100–500 curated
samples stored under `/datasets/processed/{dataset}/calibration/`.
Expand All @@ -84,7 +84,7 @@ progress tracking.

- [ ] Develop the `OrdinalRegressionTrainer` with cutpoint ordering guarantees
and unit tests covering boundary cases (monotonicity, identical labels).
- [ ] Parameterise model head replacement logic to support both BERT-like and
- [ ] Parameterize model head replacement logic to support both BERT-like and
ViT architectures, validated on representative text and image datasets.
- [ ] Achieve ≥90% statement coverage on the training module using `pytest`
plus fixtures from `rstest`.
Expand All @@ -100,7 +100,7 @@ progress tracking.
- [ ] Capture structured metrics (loss, cutpoints, throughput) and emit them to
Prefect Orion and the observability stack with comparison dashboards.

## 3. Operationalise export and optimisation
## 3. Operationalize export and optimization

### 3.1. Build ONNX export automation

Expand All @@ -111,7 +111,7 @@ progress tracking.
- [ ] Execute export and verification Prefect tasks on CPU OpenTofu modules,
proving GPU instances are torn down before CPU provisioning.
- [ ] Store FP32 exports under `/models/onnx/{experiment}/fp32/` with metadata
capturing opset, optimiser flags, and git references.
capturing opset, optimizer flags, and git references.

### 3.2. Enforce parity verification gates

Expand All @@ -122,13 +122,13 @@ progress tracking.
- [ ] Schedule nightly parity spot-checks on retained models to detect drift in
dependencies (CUDA, ONNX Runtime) before release branches freeze.

### 3.3. Deliver quantisation and benchmarking workflow
### 3.3. Deliver quantization and benchmarking workflow

- [ ] Codify static INT8 quantisation scripts with calibration dataset reuse
- [ ] Codify static INT8 quantization scripts with calibration dataset reuse
and artefact write-back to `/models/onnx/{experiment}/int8/`.
- [ ] Measure INT8 throughput versus FP32 on target CPU instances, documenting
improvements and acceptable accuracy deltas (<1% MAE loss).
- [ ] Integrate quantised-model verification into CI with `atol=1e-2`
- [ ] Integrate quantized-model verification into CI with `atol=1e-2`
tolerances, including latency smoke tests under realistic batch sizes.

## 4. Integrate with the Rust inference surface
Expand All @@ -151,7 +151,7 @@ progress tracking.
- [ ] Benchmark end-to-end inference latencies (p50, p95) under representative
traffic, ensuring INT8 pipelines meet sub-50 ms targets on t4g.large.

### 4.3. Operationalise release governance
### 4.3. Operationalize release governance

- [ ] Define promotion criteria linking training experiment success, parity
checks, and Rust e2e tests before artefacts become GA releases.
Expand Down
10 changes: 5 additions & 5 deletions docs/opentofu-hcl-syntax-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ infrastructure. OpenTofu, as an Infrastructure as Code (IaC) tool, reads this
declarative configuration and takes on the responsibility of figuring out how
to achieve that state.

This approach is centered on the concept of state reconciliation. OpenTofu
This approach is centred on the concept of state reconciliation. OpenTofu
maintains a state file (by default, `terraform.tfstate`) that records the
real-world resources it manages. When a configuration is applied, OpenTofu
compares the desired state defined in the HCL files with the current state
Expand Down Expand Up @@ -620,7 +620,7 @@ provider's API.2 A data source is declared using a

`data` block.

#### Syntax and Behavior
#### Syntax and Behaviour

The syntax is `data "<PROVIDER>_<TYPE>" "<NAME>" {... }`.25

Expand Down Expand Up @@ -760,7 +760,7 @@ of the resource or module.5

- **Syntax**: `count = <WHOLE_NUMBER>`

- **Behavior**: It is best suited for creating multiple copies of a resource
- **Behaviour**: It is best suited for creating multiple copies of a resource
that are identical or vary only in ways that can be derived from a simple
numeric index.

Expand Down Expand Up @@ -822,7 +822,7 @@ in the collection.29

- **Syntax**: `for_each = <MAP_OR_SET_OF_STRINGS>`

- **Behavior**: It creates a more stable association between the configuration
- **Behaviour**: It creates a more stable association between the configuration
and the real-world resource. Each instance is tracked by the map key or set
value, not by a transient numeric index.

Expand Down Expand Up @@ -1000,7 +1000,7 @@ of operations for creating, updating, and destroying resources.
dependencies is implicitly. When one resource's argument references an
attribute of another resource (e.g., `subnet_id = aws_vpc.main.id`), OpenTofu
automatically infers that the VPC must be created before the subnet. It
analyzes all such references to build the dependency graph.19
analyses all such references to build the dependency graph.19

- **Explicit Dependencies with** `depends_on`: In some rare cases, a dependency
exists that cannot be inferred from expression references. This typically
Expand Down
Loading
Loading