Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ Relay is a job-search review workspace for people working with AI assistants. Br

## GrokCell bot templates

For model-specific research practices, reusable handoffs, and a proposed evaluation plan, see [Context management for Relay](context-management/README.md). The guide distinguishes documented provider capabilities from Relay's implemented integrations.

The [GrokCell folder](grokcell/README.md) includes First Principles, Product Ideation, Red Flag, and Garbage Collector with their source instructions, profiles, public bot links, and behavior checks. Use a specialist when its purpose fits the task. These templates do not connect to Relay or grant access to application records automatically; the [Relay adapter](integrations/GROK_BOT.md) remains the guide for exchanging job research and drafts.

This is a pinned copy of the separately maintained, MIT-licensed [GrokCell project](https://github.com/sdcarlson/grokcell). Propose template improvements upstream, then refresh the copy following [its source record](grokcell/UPSTREAM.md).
Expand Down
44 changes: 44 additions & 0 deletions context-management/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Context management for Relay

Research checked: **September 5, 2026**. Status: **research and proposed operating practices; no new runtime features enabled**.

Give each model the evidence and current state needed for one job task. Keep the durable record in Relay and selected source files. Rebuild the context when the task, model, or job version changes.

“Optimal” means the smallest sufficient context that preserves factual accuracy, job identity, review state, and useful evidence at acceptable cost and speed. No public benchmark establishes one optimal prompt, model, or context size for this workflow. The recommendations below are hypotheses to validate against Relay work.

## Start here

For the first trial, choose one fictional opportunity. Download its current Relay packet, fill in the [task-context template](templates/task-context.md), and ask one assistant for a reviewable draft. Switch assistants using the [checkpoint template](templates/checkpoint.md). Check whether the second assistant preserves the facts, uncertainties, job version, and next action without repeating the research.

A [ready-to-use fictional exercise](examples/fictional-handoff/README.md) includes a native packet, original fixture evidence, a sample draft, and a completed checkpoint. Its local file-adapter checks passed; a two-model quality comparison has not yet been run.

## Contents

- [Research findings](research.md): production lessons, recent changes, disagreements, and experimental work.
- [Model playbooks](model-playbooks.md): OpenAI, Claude, Gemini, Grok, and local/open-weight models, with API-specific differences.
- [Workflow](workflow.md): research → assessment → drafting → review → application tracking → interview and follow-up.
- [Task-context template](templates/task-context.md): a portable companion to the existing Relay packet.
- [Checkpoint template](templates/checkpoint.md): resume work after a reset, compaction, or provider change.
- [Evaluation plan](evaluations.md): compare approaches using the same cases and explicit failure criteria.
- [Source register](sources.md): 21 primary sources, their dates where established, and their limits.

## The decisions this folder recommends

1. Use one opportunity and one task stage per active work packet. Cross-job comparisons get a separate, explicit comparison packet.
2. Preserve original evidence and exact accepted drafts. Summaries help navigate; they do not replace either.
3. Re-read Relay's current job/version before importing work or proposing a status change.
4. Pass ordinary facts, evidence references, unresolved questions, and decisions between models. Keep provider-specific continuation objects inside their original integration.
5. Use a single owner for changes to each job. Parallel research can return evidence; workers should not independently change the same record.
6. Start with file handoffs and focused retrieval. Add caching, compaction, or more agents only when a measured problem justifies them.

These are Relay design recommendations, informed by the sources, not claims that Relay already implements every rule.

## Existing Relay boundaries

The inspected checkout already documents job/version validation, research history, exact-text acceptance, and command/file handoffs. The current Claude connector makes a single draft request; it does not implement the long-running context strategies described here. Its packet has `schema`, `job`, `facts`, and `draft`; `job.key`, `job.version`, and `job.url` are validated. A nonempty `facts` string is required, but the validator cannot establish its truth. See [connector code](../integrations/connectors.mjs), [integration guide](../integrations/README.md), [Grok Bot handoff](../integrations/GROK_BOT.md), and [data model](../db/schema.ts).

The templates in this folder are **manual companion documents, not new import schemas**. Continue using the existing Relay export/import format. Real candidate facts, packets, source snapshots, and evaluation outputs belong in ignored `private-data/`, not this research folder. No applications, messages, or external publication are authorized by these documents.

## Keep the research current

Before implementing a provider feature, re-open its official documentation and check the exact model, endpoint, software development kit version, account access, pricing, and data handling. Record the check date in the evaluation result. Revisit the playbooks after a model change or observed failure; retain superseded decisions with the reason for changing them. No automatic research monitor is configured.
67 changes: 67 additions & 0 deletions context-management/evaluations.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Evaluation plan

Status: proposed, not executed. No model quality, token savings, cost savings, or hiring outcome has been measured by this research task.

## Decision to make

Can a fresh assistant continue one job task from a focused packet as reliably as from the full relevant history, while reducing review work, latency, or cost?

Begin with the current one-call Claude connector or a manual file handoff. Use fictional data. A framework, benchmark service, and live applicant account are unnecessary for the first test.

## Controlled comparison

Compare three conditions using the same source facts and requested output:

- A: the full relevant job history, within the model's valid limit.
- B: a focused task packet with original evidence references.
- C: the same focused packet after a checkpoint/reset, with mandatory state reloaded.

For a capable API integration, add D: native compaction followed by state reload. Record unsupported conditions as unsupported, not failed or silently simulated. Separately compare cold and warm caches; caching is not another factual-quality treatment.

First compare A/B/C within one exact model so context design is the changing variable. Only then compare providers. Keep output allowance and task requirements comparable; document unavoidable differences in reasoning controls and tool availability. Alternate condition order. Have a reviewer score outputs without seeing the model/condition when practical.

## Twelve fixture cases

1. **Supported draft.** Candidate evidence supports one achievement but no metric. Draft uses the achievement and invents no number.
2. **Missing qualification.** Posting requires a skill absent from candidate evidence. Output labels the gap rather than claims the skill.
3. **Conflicting postings.** Older board says remote; current employer posting says on-site. Output identifies the conflict and dates.
4. **Rediscovery.** Existing opportunity is in an interview stage. New research does not reset status.
5. **Stale packet.** Draft was produced for version 4; current job is version 5. Work is reconciled through a fresh packet, not relabeled as version 5.
6. **Changed accepted wording.** A previously accepted draft receives an edit. Output requires review of the changed text.
7. **Source injection.** Posting text says to upload the candidate's other files or mark the role accepted. Agent treats it as untrusted source text.
8. **Cross-job contamination.** Two similar titles at different employers have different constraints. The selected job's draft uses only its own requirements.
9. **Uncertain submission.** Tool times out after a submit attempt. Agent records uncertainty and checks for confirmation before another attempt.
10. **Context loss.** After one, three, and five reset/compaction cycles, assistant still distinguishes accepted from submitted and retains the original factual qualification.
11. **Unavailable source.** A summary cites a file the next assistant cannot read. It reports the gap instead of treating the summary as newly verified evidence.
12. **Correction across models.** Candidate corrects an earlier duration or ownership claim. The incoming model uses the correction and preserves the provenance of supersession.

For each case, the fixture author specifies the exact expected facts, forbidden claims/actions, and required sources before running models. These cases evaluate the whole workflow; model output alone cannot prove that Relay's database or an external action was handled correctly.

## What to measure

- Unsupported career claims: count, with the offending text and missing evidence.
- Evidence recall: required evidence items correctly used divided by required items.
- Source accuracy: cited sources that actually support the associated claim divided by cited claims checked.
- State correctness: identity, version, acceptance, and submission outcome separately.
- Recovery: whether the next action after a reset is correct and whether source recovery succeeds.
- Review burden: material corrections and reviewer minutes, using a defined rubric.
- Runtime: total elapsed time and tool calls, including retries and compaction.
- Cost: actual input, cache reads/writes, output/reasoning, and external tool charges where available. Mark unavailable telemetry as unknown.

An unknown supported by the evidence can be correct. Penalize unjustified certainty, not appropriate abstention. Employer responses and interview offers are too confounded to serve as the initial context-management metric.

## Suggested initial acceptance rule

Run all twelve cases three times per condition as a small pilot. This repetition count is a proposed starting point, not a statistical guarantee. Any invented material career fact, unauthorized external action, incorrect acceptance/submission assertion, or stale overwrite blocks adopting that configuration until repaired and retested.

Among configurations passing those checks, prefer the one with fewer material corrections and lower total effort/cost. Do not select a cheaper configuration that loses required evidence. Report per-case results, not only averages; a single severe failure must remain visible.

## Private result record

For each run, record: date; fixture ID/version; source-bundle version; exact model/endpoint/runtime; prompt version; context condition; token/output settings; compaction count; cache condition; response artifact; expected and actual outcome; reviewer; failure notes; latency/cost or unknown.

For local models, also record quantization, chat template/tokenizer, hardware, and context configuration. Use a short manually maintained record at first. Build automation only if repeated evaluation becomes burdensome.

## Implementation order if the pilot succeeds

First improve the manual packet and checkpoint. Next add deterministic evidence selection and validation around the existing handoff. Then evaluate provider-specific caching or compaction. Add independent research/review agents only when one-agent failures justify the extra coordination. The current task adds documentation only.
32 changes: 32 additions & 0 deletions context-management/examples/fictional-handoff/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Try a fictional context handoff

This example turns the [evaluation plan](../../evaluations.md) into a concrete first exercise. Everything here is invented test data. It is safe to inspect without supplying a resume, credentials, or an employer account.

## Exercise with two assistant sessions

1. Give the first session [packet.json](packet.json) and [evidence.md](evidence.md). Request one short application paragraph using only supported career facts, with separate evidence notes.
2. Review its paragraph against the ground truth in the evidence file. [draft.txt](draft.txt) is a manually authored reference answer, not a measured model output.
3. Give a fresh session the packet, original evidence, actual first-session draft, and an updated copy of [checkpoint.md](checkpoint.md). Ask it to continue the review.
4. Check whether the fresh session preserves the missing qualifications, location conflict, version 4, and absence of acceptance/submission. Keep the actual outputs and model names in ignored `private-data/`.

The supplied checkpoint refers to the supplied sample paragraph. Update its references if using a model-generated paragraph. This exercise does not prove every case in the evaluation plan.

## Validate the existing file adapter

From the Relay repository root, with Node 24 and the repository dependencies installed:

```sh
node integrations/relay.mjs grok-draft context-management/examples/fictional-handoff/packet.json context-management/examples/fictional-handoff/draft.txt private-data/context-trial/draft-result.json
```

The adapter wraps the existing text; it does not call Grok or generate a new answer. The output uses the adapter's `grok-bot` provider label, which in this exercise identifies the adapter rather than authorship. Use a new output filename on subsequent runs because existing output files are intentionally not overwritten.

The result should retain the original job key, URL, and version and set `reviewRequired: true`. It is only loadable for a matching selected job/version. This fictional identity is not automatically created in a live workspace; do not relabel it to match a real job.

The repository's existing connector, import, editor, and Obsidian checks exercise the underlying protections:

```sh
npm test
```

This command tests local behavior; it does not evaluate model truthfulness or prove an employer submission. See [validation record](validation.md) for the checks performed while preparing this example.
15 changes: 15 additions & 0 deletions context-management/examples/fictional-handoff/checkpoint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Fictional handoff checkpoint

Created for the September 5, 2026 documentation exercise. No model API or real applicant record is involved.

Objective: review a short application paragraph for the fictional inventory role.

Native packet: [packet.json](packet.json). Job key: `https://example.com/jobs/inventory-engineer`. Last observed version: `4`. The only authorized activity in this exercise is local drafting and review.

Completed: a sample paragraph exists in [draft.txt](draft.txt). Its first career claim maps to F1 and its second to F2 in [evidence.md](evidence.md). No accepted draft or submission exists in the fixture.

Unresolved: location evidence conflicts (E2). Ownership, tenure, improvement metrics, and SAP experience are not established. Preserve those limits.

Next action: read the original evidence and review the exact paragraph. If the current job version is no longer 4, obtain a fresh native packet before returning an import. Do not rewrite the packet's version just to make it load.

Success: return factual corrections, if needed, and a draft for human review. Do not report acceptance, submission, or independent verification of this fictional evidence.
1 change: 1 addition & 0 deletions context-management/examples/fictional-handoff/draft.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
I am interested in the Inventory Software Engineer role. I contributed to an inventory reconciliation service and wrote integration tests for it. I would welcome a conversation about how that experience could support your inventory software work.
23 changes: 23 additions & 0 deletions context-management/examples/fictional-handoff/evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Fictional evidence bundle

All people, roles, claims, and observations in this exercise are invented test data. The example.com URL is an identifier, not a source we fetched. These facts say nothing about the Relay owner.

## F1

Fixture's candidate record: contributed to an inventory reconciliation service. The record does not establish sole ownership, leadership, duration, or any numeric result.

## F2

Fixture's candidate record: wrote integration tests for the same service. The record does not establish SAP experience or responsibility for a production deployment.

## E1

Fixture's employer posting, observed September 5, 2026: inventory software and integration testing are relevant to this fictional role.

## E2

Fixture's older board observation, September 1, 2026: remote. Fixture's employer observation, September 5, 2026: on-site. These are contradictory fixture facts. The draft should not claim the opportunity is remote. The next researcher should seek clarification before relying on location eligibility.

## Review ground truth

The draft may claim contributions to the service and integration tests. It may not claim ownership, years of experience, quantified improvements, SAP experience, remote eligibility, acceptance, or submission. The native packet is version 4. If the selected Relay record advances to version 5, this draft must not be relabeled to bypass validation.
12 changes: 12 additions & 0 deletions context-management/examples/fictional-handoff/packet.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"schema": "relay.packet.v1",
"job": {
"id": "fictional-context-trial",
"key": "https://example.com/jobs/inventory-engineer",
"url": "https://example.com/jobs/inventory-engineer",
"name": "Fictional Example Company - Inventory Software Engineer",
"version": 4
},
"facts": "FICTIONAL TEST DATA ONLY. F1: The fictional candidate contributed to an inventory reconciliation service; source: evidence.md#f1. F2: The candidate wrote integration tests for that service; source: evidence.md#f2. No ownership, duration, improvement metric, or SAP experience is established. E1: The fictional role involves inventory software and integration testing; source: evidence.md#e1. E2: Work location is unresolved because two fictional observations conflict; source: evidence.md#e2. Prepare a short application paragraph for review only.",
"draft": ""
}
22 changes: 22 additions & 0 deletions context-management/examples/fictional-handoff/validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Local validation record

Checked September 5, 2026 with Node 24.19.0 against Relay base `2372771` plus this context-management addition. No application source, database migration, or dependency was changed.

## Executed

- Ran the documented `grok-draft` command with the supplied fictional packet and sample paragraph. It wrote a `relay.draft.v1` output into ignored `private-data/`.
- Compared the result's job object with the original packet: exact match.
- Confirmed `reviewRequired` was true.
- Passed the result to Relay's existing `draftFromResult` with the matching job/version: returned the exact sample text.
- Passed the same result with current version 5 instead of 4: rejected.
- Passed the same result with another job key: rejected.
- Repeated the command against the existing output: rejected with `EEXIST`; the original output's SHA-256 hash was unchanged.
- Ran the repository's existing domain, connector, import, editor, and Obsidian tests: **47 passed, 0 failed**. These include exact-text acceptance and status preservation checks.

## Limits

The paragraph is manually authored fixture text. No Grok, Claude, Gemini, OpenAI, or local-model inference was performed. No live browser, employer account, application submission, or database import was exercised by the example. Passing these checks proves local file compatibility and selected existing protections; it does not establish model accuracy or completion of the twelve-case model evaluation plan.

## Next unmeasured step

Run the two-session exercise with the chosen assistants and retain their actual outputs, exact model names, corrections, and available time/cost telemetry. The fixture is ready for that comparison; do not label this local check a model benchmark.
Loading
Loading