Run the benchmark

Requires uv and Docker. Harbor fetches the tasks from Harbor Hub.

Run all 70 tasks

git clone https://github.com/TIGER-AI-Lab/WebsiteBench.git
cd WebsiteBench
uv sync
uv run harbor run --dataset websitebench/WebsiteBench@v1 \
  --agent codex --model '<model>' \
  --allow-environment-host snapshot.debian.org \
  --allow-environment-host raw.githubusercontent.com \
  --allow-environment-host github.com \
  --allow-environment-host nodejs.org \
  --allow-environment-host registry.npmjs.org

Run one task

uv run harbor run --task websitebench/medium@v1 \
  --agent codex --model '<model>' \
  --allow-environment-host snapshot.debian.org \
  --allow-environment-host raw.githubusercontent.com \
  --allow-environment-host github.com \
  --allow-environment-host nodejs.org \
  --allow-environment-host registry.npmjs.org

The repository pins Harbor 0.25.0.
The same tasks are on Hugging Face. Pass --path to run a downloaded task.
Set --agent and --model to what you want to evaluate.
--allow-environment-host opens the agent installer’s download hosts during setup. For Claude Code, allow snapshot.debian.org and downloads.claude.ai.
The Docker host needs Linux 6.7 or newer and nftables fib support.
Each task allows the agent 16 hours and the verifier 8 hours, and requests 8 CPUs, 8 GiB of memory and 20 GiB of disk.

Setup details on GitHub Tasks on Hugging Face Tasks on Harbor Hub Browse the 70 tasks