What it is
A UC Berkeley RDI “living” benchmark and dataset for evaluating computer-use AI agents on long-horizon, economically valuable professional workflows with verifiable outcomes.
Gabriel’s notes
Agents’ Last Exam (ALE) is a UC Berkeley RDI project that evaluates generalist, computer-using AI agents on long-horizon, economically valuable professional workflows with verifiable outcomes. The project describes itself as a rolling / living benchmark: the task pool grows over time (reported as 1,500+ tasks collected toward a 5,000-task target across 55 targeted sub-industries), while the public benchmark slice discussed in their technical writeups is smaller (around 150 tasks).
Quick take: ALE is the closest thing I’ve seen to a “please stop telling me your agent is job-ready” meter. It’s a broad, verifiable benchmark that’s less about vibes and more about whether an agent can ship real artifacts in messy, tool-heavy workflows.
What jumped out at me: ALE’s headline is basically “useful agents exist; job-ready agents don’t (yet)”. In their announcement results, Fable 5, GPT-5.5, and Composer 2.5 land in a similar overall performance band—but on the hardest tier, every frontier agent they tested scored 0% success. Meanwhile, cost per task diverges hard: about US$15.70 (Fable 5) vs. US$3.80 (GPT-5.5) vs. US$1.33 (Composer 2.5). On that same hardest tier, they report an average full pass rate of 2.6% across mainstream configurations. And yes: one of the most common faceplants is still agents declaring victory before verifying their own work.
I saved this under Research because it’s a rare benchmark that forces the conversation back onto real workflows, real artifacts, and real failure modes—where the expensive lies usually hide.
Good fit if you want to:
- Sanity-check “agentic” product claims against a benchmark focused on economically valuable professional work.
- Compare not just pass rate, but cost and wall-clock runtime tradeoffs across agent/harness setups.
- Study failure modes (e.g., premature “done” claims) and design better verification loops.
- Track progress over time as the benchmark evolves and task coverage expands.
Pricing snapshot (auto-enriched):
The benchmark site/paper are publicly accessible. Your actual cost is the cost of running an agent with a chosen model/harness. In ALE’s published comparison, estimated cost per task varies substantially across common stacks (e.g., ~US$1.33 to ~US$15.70 per task in their three-agent comparison).
Work-use / compliance snapshot (auto-enriched):
Dataset licensing is described as CC BY 4.0 (attribution required) and code as Apache-2.0. If you contribute workflows, the project’s competition/contributor terms include concrete requirements that submissions must not contain sensitive data like HIPAA-protected health information (PHI), personally identifiable financial information, or confidential enterprise data without authorization; contributors also grant broad rights for reuse and relicensing of submitted workflows. If you need a legal interpretation for your org, treat this as “talk to counsel,” not vibes.
Alternatives (auto-enriched):
- Terminal-Bench: A benchmark focused specifically on hard, realistic command-line tasks (great if your agent lives in the terminal; narrower scope than ALE’s multi-app professional workflows).
- WebArena: A realistic, reproducible web environment + benchmark for autonomous web-browsing agents (excellent for web navigation/action; not a broad cross-occupation workflow benchmark like ALE).
Before you adopt it:
- Decide what you’ll optimize for: raw pass rate, mean score, cost per solved task, or wall-clock time. ALE makes it hard to hide from tradeoffs.
- Budget time for verification plumbing (artifact checks, self-check prompts, test harnesses). “Looks done” is an anti-signal here.
- If you plan to contribute tasks, run a strict internal review for sensitive data and IP exposure first.
Sources:
- https://agents-last-exam.org/blogs/job-ready-agents
- https://agents-last-exam.org/terms
- https://creativecommons.org/licenses/by/4.0/
- https://arxiv.org/abs/2601.11868
- https://arxiv.org/abs/2307.13854