Skip to content

[Research]: Benchmark which :free OpenRouter models reliably support tools #68

Description

@shauryagangrade

Problem

README warns some free models return "No endpoints found that support tool
use", and free-tier rate limits vary wildly — but there's no data on which
models actually work with GCode's tool schema and stream reliably.

Proposed approach

  • Run a scripted benchmark: for each :free model on OpenRouter, send a
    fixed prompt that requires one tool call (e.g. read_file), record
    success/failure, tool-support, latency, and 429 rate-limit behavior.
  • Publish results as a table in the README/models docs and propose a curated
    default list plus a warning badge in /models for models known to lack tools.

Where to look

  • gcode/models.py (supports_tools derivation) — cross-check vs reality.
  • README.md:78-85 (Models section).

Acceptance criteria

  • A reproducible script + published results table.
  • README recommends specific models; /models can flag untested/failing ones.

Difficulty

Medium — great non-Python-heavy contribution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationhelp wantedExtra attention is neededquestionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions