Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .github/workflows/theseus-engine.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,21 @@ on:
- cron: "0 0 1 * *"
workflow_dispatch:
inputs:
reprocess:
description: 'Reprocess state. Type "all" to wipe and rebuild completely, or "last" to re-calculate the most recent period.'
required: false
run_mode:
description: 'Run Mode (Default is Resume Run)'
required: true
type: choice
options:
- ""
- "last"
- "all"
default: ""
- "Resume Run (Pick up where left off)"
- "Complete Extraction (Wipe and rebuild all)"
default: "Resume Run (Pick up where left off)"
full_fossil_scan:
description: 'Run a full deep-scan for fossils (uncheck to only do a fast survivor update)'
required: false
type: boolean
default: false


concurrency:
group: theseus-data-engine
cancel-in-progress: false
Expand Down Expand Up @@ -72,11 +72,11 @@ jobs:
continue-on-error: true
env:
REPO_NAME: ${{ matrix.repo }}
REPROCESS_INPUT: ${{ inputs.reprocess }}
RUN_MODE: ${{ inputs.run_mode }}
run: |
ARGS=()
if [ -n "$REPROCESS_INPUT" ]; then
ARGS+=(--reprocess "$REPROCESS_INPUT")
if [[ "$RUN_MODE" == "Complete Extraction (Wipe and rebuild all)" ]]; then
ARGS+=(--reprocess "all")
fi
if [ "${{ inputs.full_fossil_scan }}" != "true" ]; then
ARGS+=(--update-survivor)
Expand Down
17 changes: 17 additions & 0 deletions data/raw/claude-code_history.jsonl

Large diffs are not rendered by default.

36 changes: 27 additions & 9 deletions data/raw/langchain_history.jsonl

Large diffs are not rendered by default.

111 changes: 111 additions & 0 deletions data/raw/numpy_history.jsonl

Large diffs are not rendered by default.

65 changes: 65 additions & 0 deletions data/raw/react_history.jsonl

Large diffs are not rendered by default.

Large diffs are not rendered by default.

34 changes: 34 additions & 0 deletions data/raw/zed_history.jsonl

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/state/claude-code_state.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"commit_hash":"bdb04fc52421537f63bbcb9685e1c0905689f8f7","file_compositions":{".devcontainer/devcontainer.json":{"2025":57},".claude/commands/commit-push-pr.md":{"2025":19},".claude-plugin/marketplace.json":{"2025":149,"2026":1},".claude/commands/triage-issue.md":{"2026":74},".devcontainer/init-firewall.sh":{"2025":137},".devcontainer/Dockerfile":{"2025":91},".claude/commands/dedupe.md":{"2025":16,"2026":11},".github/ISSUE_TEMPLATE/bug_report.yml":{"2025":188},".github/ISSUE_TEMPLATE/config.yml":{"2025":14},".gitattributes":{"2025":2},".github/ISSUE_TEMPLATE/feature_request.yml":{"2025":132},".github/ISSUE_TEMPLATE/documentation.yml":{"2025":117},".github/workflows/claude.yml":{"2025":35,"2026":8},".github/workflows/claude-issue-triage.yml":{"2025":19,"2026":28},".github/workflows/backfill-duplicate-comments.yml":{"2025":43,"2026":1},".github/workflows/auto-close-duplicates.yml":{"2025":30,"2026":1},".github/workflows/claude-dedupe-issues.yml":{"2025":72,"2026":22},".github/workflows/issue-lifecycle-comment.yml":{"2026":27},".github/workflows/issue-opened-dispatch.yml":{"2025":28},".github/workflows/non-write-users-check.yml":{"2026":47},".github/workflows/lock-closed-issues.yml":{"2025":92},".github/ISSUE_TEMPLATE/model_behavior.yml":{"2025":220},".github/workflows/log-issue-events.yml":{"2025":40},".github/workflows/remove-autoclose-label.yml":{"2025":42},".github/workflows/sweep.yml":{"2026":31},".vscode/extensions.json":{"2025":8},".gitignore":{"2025":1},"Script/run_devcontainer_claude_code.ps1":{"2025":152},"LICENSE.md":{"2025":1},"examples/mdm/README.md":{"2026":28},"README.md":{"2025":42,"2026":30},"examples/hooks/bash_command_validator_example.py":{"2025":83},"examples/mdm/macos/com.anthropic.claudecode.mobileconfig":{"2026":56},"SECURITY.md":{"2025":9,"2026":3},"examples/mdm/managed-settings.json":{"2026":5},"examples/mdm/macos/com.anthropic.claudecode.plist":{"2026":11},"examples/settings/settings-lax.json":{"2026":6},"examples/mdm/windows/ClaudeCode.admx":{"2026":28},"examples/mdm/windows/Set-ClaudeCodePolicy.ps1":{"2026":28},"examples/settings/settings-bash-sandbox.json":{"2026":18},"examples/mdm/windows/en-US/ClaudeCode.adml":{"2026":31},"examples/settings/README.md":{"2026":35},"examples/settings/settings-strict.json":{"2026":28},"plugins/README.md":{"2025":77},"plugins/agent-sdk-dev/.claude-plugin/plugin.json":{"2025":9},"plugins/agent-sdk-dev/README.md":{"2025":208},"plugins/agent-sdk-dev/agents/agent-sdk-verifier-py.md":{"2025":140},"plugins/claude-opus-4-5-migration/.claude-plugin/plugin.json":{"2025":9},"plugins/agent-sdk-dev/agents/agent-sdk-verifier-ts.md":{"2025":145},"plugins/claude-opus-4-5-migration/README.md":{"2025":21},"plugins/agent-sdk-dev/commands/new-sdk-app.md":{"2025":176},"feed.xml":{"2026":619},"plugins/code-review/commands/code-review.md":{"2025":81,"2026":28},"plugins/claude-opus-4-5-migration/skills/claude-opus-4-5-migration/references/effort.md":{"2025":70},"plugins/code-review/.claude-plugin/plugin.json":{"2025":9},"plugins/claude-opus-4-5-migration/skills/claude-opus-4-5-migration/SKILL.md":{"2025":105},"plugins/claude-opus-4-5-migration/skills/claude-opus-4-5-migration/references/prompt-snippets.md":{"2025":106},"plugins/code-review/README.md":{"2025":258},"plugins/commit-commands/.claude-plugin/plugin.json":{"2025":9},"plugins/explanatory-output-style/.claude-plugin/plugin.json":{"2025":9},"plugins/commit-commands/commands/commit-push-pr.md":{"2025":20},"plugins/commit-commands/commands/clean_gone.md":{"2025":52},"plugins/commit-commands/README.md":{"2025":225},"plugins/explanatory-output-style/hooks-handlers/session-start.sh":{"2025":15},"plugins/commit-commands/commands/commit.md":{"2025":17},"plugins/explanatory-output-style/README.md":{"2025":72},"plugins/explanatory-output-style/hooks/hooks.json":{"2025":15},"plugins/feature-dev/.claude-plugin/plugin.json":{"2025":9},"plugins/feature-dev/agents/code-reviewer.md":{"2025":46},"plugins/feature-dev/agents/code-explorer.md":{"2025":51},"plugins/feature-dev/agents/code-architect.md":{"2025":34},"plugins/frontend-design/README.md":{"2025":31},"plugins/feature-dev/README.md":{"2025":412},"plugins/feature-dev/commands/feature-dev.md":{"2025":125},"plugins/frontend-design/.claude-plugin/plugin.json":{"2025":9},"plugins/hookify/.claude-plugin/plugin.json":{"2025":9},"plugins/frontend-design/skills/frontend-design/SKILL.md":{"2025":42},"plugins/hookify/.gitignore":{"2025":30},"plugins/hookify/README.md":{"2025":340},"plugins/hookify/commands/list.md":{"2025":82},"CHANGELOG.md":{"2025":1177,"2026":2969},"plugins/hookify/commands/configure.md":{"2025":128},"plugins/hookify/commands/help.md":{"2025":175},"plugins/hookify/agents/conversation-analyzer.md":{"2025":176},"plugins/hookify/commands/hookify.md":{"2025":231},"plugins/hookify/examples/dangerous-rm.local.md":{"2025":14},"plugins/hookify/core/rule_engine.py":{"2025":313},"plugins/hookify/core/config_loader.py":{"2025":297},"plugins/hookify/examples/require-tests-stop.local.md":{"2025":22},"plugins/hookify/hooks/posttooluse.py":{"2025":66},"plugins/hookify/examples/console-log-warning.local.md":{"2025":14},"plugins/hookify/hooks/hooks.json":{"2025":49},"plugins/hookify/hooks/stop.py":{"2025":59},"plugins/hookify/examples/sensitive-files-warning.local.md":{"2025":18},"plugins/hookify/hooks/userpromptsubmit.py":{"2025":58},"plugins/learning-output-style/.claude-plugin/plugin.json":{"2025":9},"plugins/hookify/hooks/pretooluse.py":{"2025":74},"plugins/learning-output-style/hooks/hooks.json":{"2025":15},"plugins/learning-output-style/hooks-handlers/session-start.sh":{"2025":15},"plugins/learning-output-style/README.md":{"2025":93},"plugins/plugin-dev/agents/skill-reviewer.md":{"2025":184},"plugins/hookify/skills/writing-rules/SKILL.md":{"2025":374},"plugins/plugin-dev/agents/agent-creator.md":{"2025":176},"plugins/plugin-dev/agents/plugin-validator.md":{"2025":184},"plugins/plugin-dev/README.md":{"2025":402},"plugins/plugin-dev/skills/agent-development/SKILL.md":{"2025":415},"plugins/plugin-dev/skills/agent-development/examples/agent-creation-prompt.md":{"2025":238},"plugins/plugin-dev/commands/create-plugin.md":{"2025":415},"plugins/plugin-dev/skills/agent-development/references/agent-creation-system-prompt.md":{"2025":207},"plugins/plugin-dev/skills/agent-development/scripts/validate-agent.sh":{"2025":217},"plugins/plugin-dev/skills/command-development/README.md":{"2025":272},"plugins/plugin-dev/skills/agent-development/references/triggering-examples.md":{"2025":491},"plugins/plugin-dev/skills/agent-development/examples/complete-agent-examples.md":{"2025":427},"plugins/plugin-dev/skills/agent-development/references/system-prompt-design.md":{"2025":411},"plugins/plugin-dev/skills/command-development/examples/plugin-commands.md":{"2025":557},"plugins/plugin-dev/skills/command-development/examples/simple-commands.md":{"2025":504},"plugins/plugin-dev/skills/command-development/SKILL.md":{"2025":834},"plugins/plugin-dev/skills/command-development/references/advanced-workflows.md":{"2025":722},"plugins/plugin-dev/skills/command-development/references/frontmatter-reference.md":{"2025":463},"plugins/plugin-dev/skills/command-development/references/testing-strategies.md":{"2025":702},"plugins/plugin-dev/skills/command-development/references/interactive-commands.md":{"2025":920},"plugins/plugin-dev/skills/command-development/references/documentation-patterns.md":{"2025":739},"plugins/plugin-dev/skills/hook-development/SKILL.md":{"2025":712},"plugins/plugin-dev/skills/hook-development/examples/load-context.sh":{"2025":55},"plugins/plugin-dev/skills/hook-development/examples/validate-bash.sh":{"2025":43},"plugins/plugin-dev/skills/command-development/references/marketplace-considerations.md":{"2025":904},"plugins/plugin-dev/skills/command-development/references/plugin-features-reference.md":{"2025":609},"plugins/plugin-dev/skills/hook-development/examples/validate-write.sh":{"2025":38},"plugins/plugin-dev/skills/hook-development/scripts/README.md":{"2025":164},"plugins/plugin-dev/skills/hook-development/references/patterns.md":{"2025":346},"plugins/plugin-dev/skills/hook-development/references/advanced.md":{"2025":479},"plugins/plugin-dev/skills/hook-development/scripts/hook-linter.sh":{"2025":153},"plugins/plugin-dev/skills/hook-development/references/migration.md":{"2025":369},"plugins/plugin-dev/skills/mcp-integration/examples/stdio-server.json":{"2025":26},"plugins/plugin-dev/skills/mcp-integration/examples/sse-server.json":{"2025":19},"plugins/plugin-dev/skills/hook-development/scripts/test-hook.sh":{"2025":252},"plugins/plugin-dev/skills/mcp-integration/examples/http-server.json":{"2025":20},"plugins/plugin-dev/skills/hook-development/scripts/validate-hook-schema.sh":{"2025":159},"plugins/plugin-dev/skills/mcp-integration/references/server-types.md":{"2025":536},"plugins/plugin-dev/skills/plugin-settings/examples/read-settings-hook.sh":{"2025":65},"plugins/plugin-dev/skills/plugin-settings/examples/create-settings-command.md":{"2025":98},"plugins/plugin-dev/skills/plugin-settings/examples/example-settings.md":{"2025":159},"plugins/plugin-dev/skills/mcp-integration/references/tool-usage.md":{"2025":538},"plugins/plugin-dev/skills/mcp-integration/SKILL.md":{"2025":554},"plugins/plugin-dev/skills/mcp-integration/references/authentication.md":{"2025":549},"plugins/plugin-dev/skills/plugin-settings/scripts/parse-frontmatter.sh":{"2025":59},"plugins/plugin-dev/skills/plugin-structure/README.md":{"2025":109},"plugins/plugin-dev/skills/plugin-settings/scripts/validate-settings.sh":{"2025":101},"plugins/plugin-dev/skills/plugin-settings/SKILL.md":{"2025":544},"plugins/plugin-dev/skills/plugin-settings/references/real-world-examples.md":{"2025":395},"plugins/plugin-dev/skills/plugin-settings/references/parsing-techniques.md":{"2025":549},"plugins/plugin-dev/skills/plugin-structure/examples/minimal-plugin.md":{"2025":83},"plugins/plugin-dev/skills/plugin-structure/SKILL.md":{"2025":476},"plugins/plugin-dev/skills/plugin-structure/references/component-patterns.md":{"2025":567},"plugins/plugin-dev/skills/plugin-structure/examples/standard-plugin.md":{"2025":587},"plugins/pr-review-toolkit/.claude-plugin/plugin.json":{"2025":9},"plugins/plugin-dev/skills/plugin-structure/references/manifest-reference.md":{"2025":552},"plugins/pr-review-toolkit/agents/code-reviewer.md":{"2025":47},"plugins/plugin-dev/skills/plugin-structure/examples/advanced-plugin.md":{"2025":765},"plugins/plugin-dev/skills/skill-development/SKILL.md":{"2025":637},"plugins/plugin-dev/skills/skill-development/references/skill-creator-original.md":{"2025":209},"plugins/pr-review-toolkit/README.md":{"2025":313},"plugins/pr-review-toolkit/agents/code-simplifier.md":{"2025":83},"plugins/pr-review-toolkit/agents/comment-analyzer.md":{"2025":70},"plugins/pr-review-toolkit/commands/review-pr.md":{"2025":189},"plugins/pr-review-toolkit/agents/pr-test-analyzer.md":{"2025":69},"plugins/ralph-wiggum/.claude-plugin/plugin.json":{"2025":9},"plugins/pr-review-toolkit/agents/type-design-analyzer.md":{"2025":110},"plugins/pr-review-toolkit/agents/silent-failure-hunter.md":{"2025":130},"plugins/ralph-wiggum/README.md":{"2025":179},"plugins/ralph-wiggum/commands/cancel-ralph.md":{"2025":10,"2026":8},"plugins/ralph-wiggum/commands/ralph-loop.md":{"2025":17,"2026":1},"plugins/ralph-wiggum/hooks/hooks.json":{"2025":15},"plugins/ralph-wiggum/hooks/stop-hook.sh":{"2025":177},"plugins/ralph-wiggum/commands/help.md":{"2025":126},"plugins/security-guidance/.claude-plugin/plugin.json":{"2025":6,"2026":4},"plugins/security-guidance/README.md":{"2026":116},"plugins/ralph-wiggum/scripts/setup-ralph-loop.sh":{"2025":176,"2026":27},"plugins/security-guidance/hooks/_base.py":{"2026":156},"plugins/security-guidance/hooks/ensure_agent_sdk.py":{"2026":225},"plugins/security-guidance/hooks/diffstate.py":{"2026":438},"plugins/security-guidance/hooks/extensibility.py":{"2026":289},"plugins/security-guidance/hooks/review_api.py":{"2026":398},"plugins/security-guidance/hooks/hooks.json":{"2025":12,"2026":59},"plugins/security-guidance/hooks/session_state.py":{"2026":160},"plugins/security-guidance/hooks/sg-python.sh":{"2026":44},"plugins/security-guidance/hooks/patterns.py":{"2026":345},"plugins/security-guidance/hooks/gitutil.py":{"2026":722},"scripts/auto-close-duplicates.ts":{"2025":277},"scripts/edit-issue-labels.sh":{"2026":84},"scripts/gh.sh":{"2026":96},"scripts/comment-on-duplicates.sh":{"2026":95},"scripts/backfill-duplicate-comments.ts":{"2025":213},"scripts/sweep.ts":{"2026":168},"plugins/security-guidance/hooks/llm.py":{"2026":1696},"scripts/lifecycle-comment.ts":{"2026":53},"scripts/issue-lifecycle.ts":{"2026":38},"plugins/security-guidance/hooks/security_reminder_hook.py":{"2025":82,"2026":2110},"demo.gif":{"2025":53434}}}
2 changes: 1 addition & 1 deletion data/state/langchain_state.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/state/numpy_state.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/state/react_state.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/state/tensorflow_state.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions data/state/zed_state.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ dev = [
]

[tool.pylint]
init-hook = "import sys; sys.path.insert(0, 'scripts')"
init-hook = "import sys; sys.path.insert(0, '.')"

[tool.pylint.format]
max-line-length = 120
Expand Down
14 changes: 7 additions & 7 deletions scripts/analyse_repository.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,8 +237,8 @@ def _filter_snapshots(

:param all_periods: Full list of (period, commit) tuples.
:param processed_periods: Set of period strings already on disk.
:param reprocess: Optional ``YYYY-MM`` period to force re-processing.
:return: List of (period, commit) tuples that need processing.
:param reprocess: Optional target ('all', 'last', or 'YYYY-MM') to force re-processing.
:return: A list of (YYYY-MM, commit_hash) tuples to process.
"""
result: list[tuple[str, str]] = []
for period, commit in all_periods:
Expand Down Expand Up @@ -415,7 +415,7 @@ def process_repository(

:param repo_slug: GitHub ``owner/name`` slug.
:param data_dir: Path to the ``data/`` output directory.
:param reprocess: Optional ``YYYY-MM`` period to force re-processing.
:param reprocess: Optional target ('all', 'last', or 'YYYY-MM') to force re-processing.
"""
repo_name = repo_slug.split("/")[-1]
temp_repo_path = f"./temp_workdir_{repo_slug.replace('/', '__')}"
Expand Down Expand Up @@ -467,7 +467,7 @@ def main() -> None:
CLI flags
---------
--repo NAME Process only the given repository (by config name).
--reprocess YYYY-MM Re-process a specific snapshot period even if it already
--reprocess TARGET Re-process 'all', 'last', or a specific snapshot period even if it already
exists on disk.
"""
parser = argparse.ArgumentParser(
Expand All @@ -481,9 +481,9 @@ def main() -> None:
)
parser.add_argument(
"--reprocess",
metavar="YYYY-MM",
metavar="TARGET",
default=None,
help="Re-process a specific snapshot period (e.g. '2023-06').",
help="Re-process a target. Pass 'all' to wipe and rebuild, 'last' for the latest period, or a specific period like '2023-06'.",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
)
args = parser.parse_args()

Expand Down Expand Up @@ -521,7 +521,7 @@ def main() -> None:
logger.info("Processing %d repositories", len(selected_targets))

if args.reprocess:
logger.info("Re-processing period: %s", args.reprocess)
logger.info("Re-processing target: %s", args.reprocess)

max_top_level_workers = min(
len(selected_targets),
Expand Down
12 changes: 6 additions & 6 deletions scripts/run_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

Usage::

python -m scripts.run_pipeline [--repo NAME] [--reprocess YYYY-MM] [--update-survivor]
python -m scripts.run_pipeline [--repo NAME] [--reprocess TARGET] [--update-survivor]

Runs all three stages in sequence on one or more repositories:

Expand Down Expand Up @@ -37,7 +37,7 @@ def run_pipeline(
Run the full pipeline (analyse → fossils → cleanup) for all repositories.

:param repo: Optional repository name to process (None = all repos).
:param reprocess: Optional ``YYYY-MM`` period to force re-process.
:param reprocess: Optional target ('all', 'last', or 'YYYY-MM') to force re-process.
:param update_survivor: If ``True``, skip genesis scan and only refresh the
survivor (living) fossil. Designed for monthly cron ticks.
:return: ``True`` if any stage failed, ``False`` otherwise.
Expand Down Expand Up @@ -145,8 +145,8 @@ def main() -> None:

CLI flags
---------
--repo NAME Process only this repository (by config name).
--reprocess YYYY-MM Re-process a specific snapshot period.
--repo NAME Process only this repository.
--reprocess TARGET Re-process 'all', 'last', or a specific snapshot period.
--update-survivor Skip genesis scan; refresh only the survivor fossil
(designed for monthly cron ticks).
"""
Expand All @@ -169,9 +169,9 @@ def main() -> None:
)
parser.add_argument(
"--reprocess",
metavar="YYYY-MM",
metavar="TARGET",
default=None,
help="Re-process a specific snapshot period (e.g. '2023-06').",
help="Re-process a target. Pass 'all' to wipe and rebuild, 'last' for the latest period, or a specific period like '2023-06'.",
)
parser.add_argument(
"--update-survivor",
Expand Down
Loading