The SEO Agent Benchmark is a planned test of how well AI agents handle real SEO jobs, like diagnosing a traffic drop or writing a content brief. We haven't run it yet, so there are no scores or rankings here.
Agent demos are easy to make look impressive. A benchmark (a fixed set of tasks used to compare systems fairly) is how you find out what happens on the tasks nobody picked to show off.
Status
Planned. No data has been collected or published for this study yet. Anything you see quoted elsewhere as coming from this study isn't ours.
The question
Given the same inputs, how accurately, cheaply, and quickly do different SEO agents complete realistic tasks, and how often do they make things up?
Why it matters
Choosing or building an agent without a fair comparison means trusting demos. A shared set of tasks lets anyone test their own agent against the same bar.
How we plan to run it
- A fixed set of realistic tasks where the right answer is known in advance.
- The same data and instructions given to every agent tested.
- Scores for accuracy, invented claims, cost, and time taken.
- A comparison against a person doing the same tasks.
- Publishing the tasks, so others can re-run the test.
What the finished study will include
- The full methodology, including its limits.
- The sample size, and who or what was included.
- Findings, including the unexciting ones.
- Charts.
- Downloadable data, where we can share it.
- Citations for anything we build on.
- Publication and update dates.
Until then
You can run a small version of this on your own agent today:
- Pick ten past cases where you already know the answer, like pages whose drop you've already diagnosed.
- Run your agent on each and score the answers yourself.
- Count any claims it made that weren't supported by the data.
- Re-run the same ten after every change you make.
Where to go next
- Build SEO agents
- SEO agent with Python: includes a simple evaluation loop
- All research