OpenAI’s Evals repo sits at roughly 18.5k GitHub stars, and it has company. A 2026 comparison counted 11 new open-source eval frameworks that emerged in the past 12 months, stacked on top of established names like DeepEval and Ragas. Benchmarking LLM agents used to mean trusting a vendor’s leaderboard screenshot, and that era is over because the tooling flipped open source. This changes what you should demand from anyone selling you an agent.
The direct answer for anyone skimming: open-source eval frameworks are now the default way to benchmark LLM agents. A MorphLLM comparison of AI agent evaluation frameworks lists the seven tools teams actually choose among: DeepEval, Braintrust, Arize Phoenix, OpenAI Evals, RAGAS, LangSmith, and Galileo. The open-source entries match the paid ones on metrics, and the closed tools now compete on the ops layer around the metrics instead.
The Cambrian Explosion, Counted
The author of a 2026 open-source comparison wrote it plainly: “I spent the last month digging into 11 open-source eval frameworks that emerged in the past 12 months,” and called the market a “Cambrian explosion.” When a category produces eleven new entrants in a year, it stops being a research curiosity and becomes infrastructure.
The license breakdown tells the story. DeepEval is Apache 2.0, OpenAI Evals is MIT, and Ragas is Apache 2.0. Arize Phoenix sits under the Elastic License 2.0, which an MLflow comparison article calls partially open source, while LangSmith is closed source. That same MLflow article calls MLflow “the most widely adopted open source AI engineering platform with 30M+ monthly downloads,” which tells you where the center of gravity sits.
Maintenance cadence matters more than metric counts when you run a two-person shop. A GitHub release entry shows a new DeepEval release on September 6, 2026, and that signal matters because an eval framework that goes quiet strands your test suite with it. I would take an actively maintained framework with 50 metrics over a stalled one with 200, and release logs are how you check before you commit.
The Defaults, and Where They Disagree
DeepEval is the default for agent teams on Python. Apache 2.0, built by Confident AI, and it plugs into an interface you already have: Python, Pytest, and a CLI. It ships 50+ research-backed metrics covering RAG, agents, chatbots, single-turn, multi-turn, and safety use cases, including G-Eval, task completion, and faithfulness, plus span-level scoring for agent traces. The MLflow article puts the pitch in one line: “With 50+ research-backed metrics and a familiar testing interface, DeepEval makes it easy to add LLM evaluation to existing CI/CD workflows.” The limitation, per Confident AI’s own comparison: no UI, collaboration, or production monitoring out of the box.
OpenAI Evals is the pick when your data cannot leave the building. It pairs a framework with an open-source registry of benchmarks, and the repo treats private evaluation as a first-class use case: “You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.” Its Completion Function Protocol extends evals to tool-using agents, so you are not stuck benchmarking base models only.
Ragas began as the standard for RAG evaluation, with metrics like faithfulness, context precision, context recall, and answer relevancy. Whether it covers agents depends on which comparison you read, because the same Confident AI page describes Ragas as RAG-only with no agent, chatbot, or production support. That disagreement is the most useful data point in this section. Coverage claims in this category are marketing until you run them against your own tasks.
The newest entrants cluster around multi-agent systems. The 2026 comparison names MASEval (MIT, on PyPI, arXiv 2603.08835) as the strongest option for evaluating full multi-agent systems, with OpenAgentBench as runner-up. LangWatch does simulation-based agent testing including voice and multi-turn, though the comparison flags a younger community and lighter human-metric alignment than the established tools.
The Real Gap Is the Ops Layer, Not the Metrics
My read after watching this category fill up is simple: metrics are commoditized, and the expensive part is everything around them. DeepEval’s missing UI and production monitoring is exactly the wedge the paid platforms sell against. Nobody wins a deal anymore by claiming better faithfulness scoring, because the open frameworks match them there.
The interesting entrants understand this. The awesome-evals catalog includes a TypeScript-native entry with versioned EvalSets, offline JSON/Markdown/JUnit reports, baseline regression gates for CI, and optional production sampling scored asynchronously. Look at what that list actually is: build-pipeline tooling, not a research artifact. Regression gates mean an eval can fail a deploy the same way a unit test does, and JUnit output means it plugs into CI tooling you already run.
The implication for a small shop is blunt. An eval that is not wired into CI is a demo, not a safeguard. The win is not picking the perfect framework, it is making your agent’s behavior a test that runs on every commit.
One Agent Score Still Lies Until the Config Ships With It
Open tooling fixes convenience, not fairness, and the research side knows it. UniACE, presented in a paper titled “A Unified Framework for the Evaluation of LLM Agentic Capabilities” (arXiv 2605.27898), integrates diverse benchmarks into a standardized instruction-tool-environment format, executes agents through a fixed ReAct-style architecture in a controllable sandbox, and keeps an offline setting with curated snapshots to separate framework effects from environment effects. The authors’ own conclusion deserves quoting: “These findings motivate reporting agent benchmark outcomes as properties of an explicit evaluation configuration, enabling more interpretable and reproducible cross-benchmark comparisons.”
Read that as a buyer’s defense. Your agent’s benchmark score depends on the evaluation configuration that produced it, so a vendor quoting a number without the config is quoting noise. Ask what ran, in what sandbox, against which snapshot, and treat a refusal to answer as the answer. Research groups keep releasing code for exactly this reason, AgentQuest among them with its repository at nec-research/agentquest.
What a Lean Shop Should Do This Week
My agency ships agent automations for small businesses, and the rule I would hand anyone starting now is this: benchmark on your own traffic, not on a public leaderboard. Pick one framework, DeepEval if you are a Python shop and OpenAI Evals if private data is the constraint, then pull ten real requests from your logs into eval cases. Wire it into CI with a regression gate, keep the reports, and attach them to the next client invoice.
The open-source default made this cheap to start and expensive to skip. If you want a second pair of eyes on wiring evals into your pipeline, that is the work Mediascout does, so reach out and tell me what your agent is supposed to be doing.
