Skip to content
Merged
134 changes: 99 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Experiment

# Current Status

Prooflight is currently in **Milestone 1: Foundation Layer**.
Prooflight is currently in **Milestone 2: Execution Core**.

Implemented:

Expand All @@ -87,6 +87,13 @@ Implemented:
- static type checking
- linting and formatting
- CI-ready development workflow
- runtime abstraction
- execution context
- execution orchestration
- immutable execution events
- execution recorder
- lifecycle event tracking
- structured execution results

The current implementation establishes the core abstraction that future evaluation components will build upon:

Expand Down Expand Up @@ -185,14 +192,22 @@ Reproducibility is treated as a first-class engineering requirement.

## 4. Observability

Future versions will record:
Prooflight treats execution history as a first-class artifact.

Current capabilities include:

- execution lifecycle events
- ordered event recording
- execution failure tracking
- structured execution outcomes

- execution traces
- agent decisions
- tool usage
- failures
Future versions will extend this into:

- agent trajectories
- tool interactions
- resource consumption
- evaluation outcomes
- evaluation metrics
- replay systems

The goal is that every evaluation result can be reconstructed.

Expand All @@ -208,13 +223,43 @@ prooflight/
├── src/
│ └── prooflight/
│ ├── __init__.py
│ └── domain/
│ │
│ ├── domain/
│ │ ├── __init__.py
│ │ └── experiment.py
│ │
│ ├── events/
│ │ ├── __init__.py
│ │ └── event.py
│ │
│ ├── recorder/
│ │ ├── __init__.py
│ │ └── recorder.py
│ │
│ ├── execution/
│ │ ├── __init__.py
│ │ ├── context.py
│ │ ├── executor.py
│ │ └── result.py
│ │
│ └── runtime/
│ ├── __init__.py
│ └── experiment.py
│ └── runtime.py
│
├── tests/
│ └── domain/
│ └── test_experiment.py
│ ├── domain/
│ │ └── test_experiment.py
│ │
│ ├── events/
│ │ └── test_event.py
│ │
│ ├── recorder/
│ │ └── test_recorder.py
│ │
│ └── execution/
│ ├── test_context.py
│ ├── test_executor.py
│ └── test_result.py
│
├── .github/
│ └── workflows/
Expand Down Expand Up @@ -291,11 +336,13 @@ Current tests verify:
- seed validation
- path normalization
- experiment immutability
- event validation
- recorder behaviour
- execution lifecycle
- execution result handling

Future tests will cover:

- runtime execution
- event generation
- benchmark execution
- telemetry
- replay
Expand All @@ -317,42 +364,59 @@ Completed:

---

## Milestone 2: Execution Runtime
## Milestone 2: Execution Core ✅

Planned:
Completed:

- runtime abstraction
- model adapters
- asynchronous execution
- agent interfaces
- environment interfaces
- event model
- execution context
- executor orchestration
- immutable event model
- execution recorder
- lifecycle event tracking
- execution result model

Architecture:
### Execution Architecture

```
Milestone 2 establishes the execution foundation:

```text
Experiment
|
▼
ExecutionContext
|
▼
Executor
|
+----------------+
| |
▼ ▼
Runtime Recorder
|
▼
Events

|
▼
Executor returns:

Runtime Executor
ExecutionResult
```

|
▼
The separation allows Prooflight to distinguish between:

Agent
**Execution history**

|
▼
"What happened?"

Environment
Captured through events.

|
▼
and:

Events
```
**Execution outcome**

"What was the final result?"

Captured through `ExecutionResult`.

---

Expand Down
7 changes: 7 additions & 0 deletions src/prooflight/events/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""
Execution event models.
"""

from .event import Event

__all__ = ["Event"]
56 changes: 56 additions & 0 deletions src/prooflight/events/event.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""
Immutable execution event.

Events are the fundamental record of everything that happens during an
experiment. They are intentionally generic so that new event types can be
introduced without changing the core event model.
"""

from __future__ import annotations

from datetime import UTC, datetime
from typing import Any
from uuid import UUID, uuid4

from pydantic import BaseModel, ConfigDict, Field, field_validator


class Event(BaseModel):
"""Immutable execution event."""

model_config = ConfigDict(
frozen=True,
extra="forbid",
)

id: UUID = Field(
default_factory=uuid4,
description="Unique identifier for the event.",
)

timestamp: datetime = Field(
default_factory=lambda: datetime.now(UTC),
description="Time at which the event occurred.",
)

name: str = Field(
min_length=1,
description="Machine-readable event name.",
)

payload: dict[str, Any] = Field(
default_factory=dict,
description="Structured event data.",
)

@field_validator("name")
@classmethod
def validate_name(cls, value: str) -> str:
"""Reject empty event names."""

value = value.strip()

if not value:
raise ValueError("Event name cannot be empty.")

return value
16 changes: 16 additions & 0 deletions src/prooflight/execution/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
"""
Execution lifecycle components.

This package contains objects responsible for managing the lifecycle
of an experiment execution.
"""

from .context import ExecutionContext
from .executor import Executor
from .result import ExecutionResult

__all__ = [
"ExecutionContext",
"Executor",
"ExecutionResult",
]
56 changes: 56 additions & 0 deletions src/prooflight/execution/context.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""
Execution context.

An ExecutionContext represents the state shared across a single experiment
execution.

It acts as a dependency boundary. Instead of passing many independent
objects between execution components, the context provides a single stable
interface.

The context currently contains:
- Experiment definition
- Event Recorder

Future components such as:
- Runtime
- Artifact Store
- Telemetry
- Evaluation state

will be added here when they become necessary.
"""

from __future__ import annotations

from pydantic import BaseModel, ConfigDict

from prooflight.domain import Experiment
from prooflight.recorder import Recorder


class ExecutionContext(BaseModel):
"""
Immutable container for one experiment execution.

A context belongs to exactly one execution lifecycle.

Once created, the execution dependencies should not be replaced.
This guarantees that every component participating in an execution
observes the same execution state.
"""

model_config = ConfigDict(
frozen=True,
arbitrary_types_allowed=True,
)

experiment: Experiment
"""
Experiment specification being executed.
"""

recorder: Recorder
"""
Recorder responsible for collecting execution events.
"""
Loading
Loading