ICLayout-Bench

An MIT-licensed framework for developing and evaluating integrated-circuit layout agents, with separately licensed public tasks, independent observation, a generic HTTP client, a shared evaluation engine and a local evaluation service. 简体中文

Run public tasks locally to debug your Agent before participating in an operator's unified evaluation service. Both paths use the same HTTP contract, isolated EDA execution and independent scoring implementation. Local results are development evidence; formal results require operator-controlled conditions and verification. Operators may add hidden tasks, admission, controlled reruns and reviewed releases.

Tasks use declared electrical and area quality targets with circuit-specific metric weights (layout). Same-condition source simulation remains required; its measurements supply the electrical targets when none are declared explicitly. Scores range from 0 to 100 after capping each metric at its target; feasible reference layouts may score well below 100. See scoring and task design.

Install and run

Use Python 3.12+ and install the ICLayout-Bench wheel. Engine source stays on GitHub; tasks and the official core selection live in an independent Dataset repository. No Dataset is bundled in the Python package.

Supply an HF Dataset repo ID and optional fixed --revision once published, or a local Dataset working copy for development. The Dataset source and revision are explicit caller inputs. Pull a prebuilt Docker Hub tools image or build and customize the Dockerfile; see image setup. With that image:

python -m benchmarking.engine.preview --dataset /path/to/dataset \
  run --case NAND2_X1 --image iclayout-eda-open:local --output build/reference-check
python -m benchmarking.run --config experiment.toml \
  --dataset /path/to/dataset --output results/new-experiment

Reference evaluation makes no model calls. The experiment runner starts the local service and the selected participant harness. It records the dataset commit, input digests and tool identities. HF manages downloaded task caches; resource installation and derived PDK caches are automatic. There is no prepared case export. Only declared inputs enter the isolated solver workspace.

Supported native harnesses save local workspace checkpoints by default. Resume an interrupted experiment with the same inputs and --output plus --resume; its original deadline still applies. See recovery.

For remote participation, use --endpoint and operator credentials instead of a local Dataset. Docker and PDK resources are not needed on the participant host. See running for experiment configuration and output contracts, and tools for reference checks and resource management.

Maintain configurations, provider settings and complete results in your own experiment project outside this checkout. The running guide explains how to prepare a fixed wheel environment, create configurations and retain results.

Licensing and public data

The engine is MIT licensed. Independently distributed circuits, reference layouts, PDKs and tools retain their own licenses. Consult the Dataset's Dataset card and collection notices; public availability does not grant additional redistribution or commercial-use rights.

Dataset process manifests select pinned external resources. The engine wheel provides the benchmarking namespace and its engine, service, participants and result subpackages. A local Dataset is an explicit data input, never an importable dependency on the design authoring workspace or on another Python source checkout.

Guides

Topic	Guide
Protocol, ownership and trust	Architecture
Local self-testing, remote participation and analysis	Running
Task contracts, scoring and qualification	Tasks
Tool images and resource preparation	Tools
Result storage, comparisons and presentation data	Result storage
Formal reruns and disclosure	Admission
Verification and contribution	Contributing

Validation scope

Public owns client/observation tests, engine regressions, container isolation checks and public reference/counterexample checks. Offline tests need neither Docker nor operator applications. Container/EDA tests require the declared tools. Operators verify admission, controlled reruns and disclosure using that same engine. Synthetic protocol runs are not model scores; reference evaluation is not an Agent measurement.