agent-eval

Lynn GitHub 镜像仓 · Qwen3.8-27B Coder390 EfficientThink 本地 GGUF（llama.cpp Q2/Q3，内置 Q4 MTP）· 主仓 https://github.com/MerkyorLynn/Lynn · 下载 https://download.merkyorlynn.com/download.html

Meta-harness optimization loop wired onto Islo sandboxes. POC: 0/5→5/5 in four proposer steps. Built on islo.dev.

Observe an agent run and GroundEval drafts the policy and diagram for you, no hand-written policy required, then scores what it checked, what it skipped, and what it wasn't allowed to touch.

Live, open-source benchmark for comparing AI coding agents on real GitHub issues

AI-operated company. Building agent-friend: universal tool adapter for AI agents. @tool → OpenAI, Claude, Gemini, MCP. Live 24/7 on Twitch.

Agent 测试转型手册：资深自动化测试工程师的 AI QE 学习路径 + 练手代码

Framework-agnostic evaluation harness for Go — test your MCP servers and AI agents with scored, CI-ready checks.

Project page for Meta-harness on Islo (POC). https://zozo123.github.io/meta-harness-on-islo-page/

Binary-criteria evaluation harness for Claude skills with planned extension to plugins, agents, and MCP servers. Score every change yes/no across 7 layers — package integrity, trigger quality, functional quality, regression protection, baseline value, model variance, rollout safety. Never gradients.

A reasoning benchmark runner for comparing LLMs as OpenClaw agents use them. 52 prompts, 3 eval sets, 11 traps, LLM-as-judge, tier-based leaderboard.

Archived and unsupported: parses transcript formats as of its last update (OpenCode: legacy JSON only; current SQLite storage unsupported). Transcript-first evaluation of Codex, Claude Code, and Pi sessions.

Vendor-neutral research umbrella for measuring AI plugin, agent, and MCP server quality across CLI runtimes (Claude Code, Gemini CLI, Copilot CLI, Codex CLI).

A curated list of benchmarks, harnesses, leaderboards, and tools for evaluating AI coding agents.

PandaProbe harness turns agent failures into fixes

检测 AI Agent 代理指标与真实业务结果背离的审计工具，面向增长与运营团队

开源通用 AI Agent 真实任务评测 · 同 Prompt、客观开奖、评分细则全公开 | Open-source evaluation of general-purpose AI Agents on real-world tasks with verifiable outcomes — by PingWest / 硅星人

A durable, long-running agent that improves and evaluates other agents.

Documented, reproducible finding: VulcanBench declarative grader mis-scored all functional tasks as 0.0 due to a repo-root pytest-cov addopts leak. Filed upstream issue #79.

Turn agent telemetry into eval jobs — outcome, quality, and spend — correlated with business outcomes per agent run.

Field guide to building production-grade evaluation infrastructure for LLM agent systems. Based on real production work at Airbnb (BPI Virtual Analyst), Shell (NLP), and contributions to LangChain, LiveKit, and Ragas.