Skip to content

Add WebHarbor task validation script#45

Open
Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:add-task-validator
Open

Add WebHarbor task validation script#45
Lxr-max wants to merge 1 commit into
aiming-lab:mainfrom
Lxr-max:add-task-validator

Conversation

@Lxr-max

@Lxr-max Lxr-max commented Jun 2, 2026

Copy link
Copy Markdown

Summary

This PR adds a small repository-level validator for WebHarbor task files:

  • scripts/validate_tasks.py
  • scripts/test_validate_tasks.py

It validates the current sites/*/tasks.jsonl schema used in this repo without introducing any new website contribution.

Why this is useful

WebHarbor reviews depend heavily on task quality and consistency. This script gives contributors and reviewers a quick way to catch:

  • malformed JSONL
  • missing required task fields
  • duplicate task IDs within a file or across sites
  • localhost/upstream URL mistakes
  • task IDs that do not match the current site naming convention
  • suspicious task wording, answer-leak heuristics, and bad markers

That makes task review more repeatable and lowers the chance of avoidable PR feedback.

CLI examples

python scripts/validate_tasks.py
python scripts/validate_tasks.py --site amazon
python scripts/validate_tasks.py --tasks sites/amazon/tasks.jsonl
python scripts/validate_tasks.py --strict
python scripts/validate_tasks.py --json

Validation categories

  • JSONL parsing and object shape
  • Required fields: id, web_name, web, upstream_url, ques
  • Duplicate IDs per file and across all sites
  • Localhost URL validation for web
  • Upstream URL validation for upstream_url
  • Port cross-checking against websyn_start.sh
  • Suspicious phrases and safety-oriented warnings
  • Simple answer-leak and bad-marker heuristics

Checks run

py -m py_compile scripts/validate_tasks.py
py scripts/test_validate_tasks.py
py scripts/validate_tasks.py
py scripts/validate_tasks.py --site amazon
py scripts/validate_tasks.py --tasks sites/amazon/tasks.jsonl
py scripts/validate_tasks.py --strict
py scripts/validate_tasks.py --json

Results:

  • py_compile: passed
  • self-tests: 6 tests passed
  • full-repo validation: 15 sites, 15 task files, 643 tasks, 0 errors, 0 warnings
  • --site amazon: passed
  • --tasks sites/amazon/tasks.jsonl: passed
  • --strict: passed against the current repo state
  • --json: returned valid machine-readable JSON summary

Scope notes

  • This is not a new website contribution.
  • No HF assets are involved.
  • .assets-revision was not modified.
  • No existing site implementation was changed.
  • No site task files were modified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant