If you’re evaluating AI coding agents — Claude Code, OpenHands, Codex CLI, whatever — you’re probably doing it wrong. Running them against your own infra is dangerous. Running them manually is slow. Running them unrepeatably is pointless.
Harbor fixes that. It’s a framework from the creators of Terminal-Bench that lets you define sandboxed agent tasks, run evaluations against any agent/model combo, and scale across cloud providers.
What It Does
Harbor wraps each agent task in an isolated environment (Docker locally, or Daytona/Modal/LangSmith/Novita Sandbox in the cloud). You specify:
- A dataset — built-in benchmarks (Terminal-Bench 2.0, SWE-Bench, Aider Polyglot) or your own
- An agent — Claude Code, OpenHands, Codex CLI, or custom
- A model — whatever backend you want to test
- Concurrency — how many environments to run in parallel
uv tool install harbor
# Run Terminal-Bench-2.0 with Claude Code, 4 parallel instances
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 4
# Scale to 100 parallel instances on Daytona
harbor run --dataset terminal-bench@2.0 \
--agent claude-code \
--model anthropic/claude-opus-4-1 \
--n-concurrent 100 \
--env daytona
Why This Matters
Most agent evaluations today fall into one of two buckets:
- Manual ad-hoc — someone runs a few prompts, watches the output, says “looks good.” No repeatability, no comparability.
- Fixed benchmarks — SWE-Bench reports pass@k on a static set of PRs. Useful, but they don’t tell you how an agent performs in your workflow against your tooling.
Harbor bridges the gap. You can run established benchmarks for comparability, but you can also write your own task definitions. The sandbox ensures the agent can’t break anything real, and the structured output (pass/fail, latency, cost) lets you compare runs across models and agent versions.
From the Terminal-Bench Team
The team behind this built Terminal-Bench, which was already the gold standard for evaluating agents in CLI environments. Harbor is that same philosophy extracted into a general framework — not just one benchmark, but a harness for running any benchmark, any agent, any model.
Practical Applications
For a DevOps workflow like ours:
- Test agents before deploying — run a candidate model/agent combo against your infra playbook tasks in a sandbox, see where it fails
- Regression testing — pin a specific agent+model config, re-run after updates
- Model comparison — same task set, different models, objective scoring
- RL data generation — Harbor can produce rollout data for reinforcement learning optimization of your own models
Getting Started
uv tool install harbor
harbor datasets list # see available benchmarks
harbor run --help # explore options
harbor run -d "terminal-bench@2.0" -m "anthropic/claude-opus-4-1" -a "claude-code"
The Harbor Cookbook has full end-to-end examples. The docs cover custom environments and provider config.
Verdict
Harbor is one of those tools that fills a real gap in the ecosystem. Agent evaluation has been either too manual (eyeball the output) or too rigid (fixed benchmarks only). Harbor lets you slot in your own tasks, run them at scale, and get structured results. If you’re deploying agents in any serious capacity, this is worth a look.