✅ Testing & Quality

Eval-driven dev, mutation testing, visual regression, proving quality.

BlogHamel Husain, Shreya Shankar

The AI Evals FAQ

A comprehensive FAQ on building LLM evaluation systems, arguing that manual error analysis — not off-the-shelf metrics — should drive eval strategy.

• Start by manually reviewing 100+ traces to find real failure patterns before picking or building any evaluator.
• Begin with a single domain expert reviewing outputs by hand; only invest in custom annotation tooling once you understand your failure modes.
• Spend the bulk of your time (60–80%) on error analysis, not on chasing high pass rates on evals that don't catch real issues.

View ↗

✅ Testing & Quality

added by Adam Tomat • 17th Aug 2026

VideoNuno Maduro

Nuno Maduro on Pest 5's Test Impact Analysis and Time-Based Sharding

A Laracon US 2026 talk where Pest creator Nuno Maduro demos Pest 5's new Tia Engine (Test Impact Analysis) and time-based test sharding.

• The Tia Engine records a dependency graph on the first run, then reruns only the tests affected by recent changes — Pest's docs cite a typical 10-minute Laravel suite dropping to about 4 seconds on repeat runs.
• Time-based sharding distributes tests across CI jobs by actual recorded execution time (via `--update-shards`) rather than file count, so parallel shards finish at roughly the same time instead of one running long.
• Thread consensus: great for smashing through tests locally, but Tia's dependency graph isn't 100% equivalent to a full run, so full (sharded) suites still belong in CI.

💬

Community vibe Warm reception — several people said it's exactly what they need for current projects — but some pushed back that Test Impact Analysis isn't guaranteed equivalent to a full suite run, so CI should keep running full/sharded suites rather than relying on it alone.

View ↗

✅ Testing & Quality

added by Tom Harper • 28th Jul 2026

BlogNoumena • xjdr

AI, Rockets, and the Return of Hard Contracts

An essay arguing that once AI makes code generation cheap, human line-by-line review becomes the bottleneck — so quality has to be enforced mechanically, through strict types and schemas, instead.

• Proposes "disposable implementation, durable contracts": generated code is cheap and replaceable, but the types, schemas, and interfaces around it must be strict enough to catch errors before a human ever looks.
• Uses the Apollo-vs-Soviet rocket engine design contrast — exhaustive preflight analysis vs. rapid build-fire-inspect-modify iteration — to argue the best approach combines both: fast generation, severe mechanical selection.
• Calls for reversing the last two decades' drift toward loosely-specified, ergonomic systems — explicit contracts and state machines are what let generation scale from hundreds to thousands of implementations.

View ↗

✅ Testing & Quality🤖 Building with Agents

added by Adam Tomat • 24th Jul 2026

BlogOpenAI

OpenAI Retracts Its SWE-Bench Pro Recommendation, Calling ~30% of Tasks Broken

Confirmed via search (source page is bot-blocked) — verify before relying on it. OpenAI audited SWE-Bench Pro, the Scale AI-built benchmark meant to replace the deprecated SWE-bench Verified, and is retracting its recommendation that the research community treat it as a reliable coding-capability eval.

• Automated investigator agents flagged 200 of 731 public-split tasks (27.4%) as broken; five experienced engineers reviewing independently found 249 (34.1%) broken.
• Frontier model pass rates on the public split rose from 23.3% to 80.3% in eight months — a jump OpenAI attributes to benchmark gaming/saturation rather than genuine capability gains.
• OpenAI is calling on the industry to build new coding benchmarks with experienced developers that are harder to game and more trustworthy.

View ↗

✅ Testing & Quality🧠 Models

added by Radar • 9th Jul 2026

BlogPaul Kinlan

How Might a Modern Lighthouse Work With LLMs?

A proposal for a Lighthouse-style auditing tool where checks are plain-language descriptions of good practice that an LLM judges from screenshots and traces, instead of hand-coded rules.

• Combines a spec of good practices, machine-readable fix guidance, and evidence primitives (screenshots, heap snapshots, perf traces) as the model's "senses."
• Runs a continuous find-fix-test loop rather than a one-shot audit, and reports fixing several real site issues this way (still with human review).
• Because tests are descriptions of outcomes rather than code, anyone who can describe "good" can author a check — widening coverage into fuzzier, subjective quality dimensions.

View ↗

✅ Testing & Quality🤖 Building with Agents

added by Adam Tomat • 4th Jul 2026

Repoopenclaw

openclaw/crabbox

A CLI that leases cloud/self-hosted compute, syncs your local diff, and runs commands/tests remotely.

• ~950 stars; a Go CLI plus a TypeScript coordinator.
• 40+ infrastructure providers supported.
• Core loop: warm a box, sync the diff, run the suite — coordinator deployable on Cloudflare Workers.

💬

Community vibe The sharer's own first reaction was that he'd already solved the same problem himself, and replies broadened into scepticism about this kind of tool generally - one called a similar product buggy and token-hungry, another reckoned they're overkill for most use cases.

View ↗

🛠️ Tools & Apps✅ Testing & Quality🤖 Building with Agents

added by Adam Tomat • 24th Jun 2026

BlogDecoding AI • Alejandro Aboy

How Evaluation-Driven Development (EDD) Works for AI Agents

A pre-merge evaluation gate for AI agent changes, using simulated inputs run through the real agent.

• An offline gate answering "does it work / did anything regress".
• Simulate inputs (drawn from real traces), not outputs.
• Rejects always-on prod eval as too costly; runs targeted branch experiments with calibrated binary judges.

View ↗

✅ Testing & Quality🤖 Building with Agents

added by Adam Tomat • 23rd Jun 2026

BlogAddy Osmani

The New Software Lifecycle

Reframes the SDLC: agents are mostly harness, and verification moves to the centre.

• Agents are roughly 10% model, 90% harness.
• Static vs dynamic context load is a key design tradeoff.
• The vibe-coding → engineering spectrum is defined by depth of verification.

💬

Community vibe General agreement that fixing the harness beats waiting on a better model, backed by real benchmark gains people had seen elsewhere. One caveat surfaced though: constantly updating guardrails after every dumb agent mistake adds its own mental overhead, and can end up stalling other work while you build better QA tooling first.

View ↗

🤖 Building with Agents✅ Testing & Quality👥 Teams & Ways of Working

added by Adam Tomat • 23rd Jun 2026

BlogGarrett Lord

Evals: the strategic IP that will define the next era of AI

Argues that private evals built from workflow plus domain judgement become durable competitive IP.

• Evals encode hard-won domain knowledge competitors can't easily copy.
• They turn "is the AI good at our job?" into a measurable, owned asset.
• The next era of AI advantage is defined by proprietary evaluation, not just models.

View ↗

✅ Testing & Quality

added by Adam Tomat • 22nd Jun 2026

BlogMujahid Abbas

Evals in Laravel: How to Prove Your AI Output Is Actually Good

A practical guide to building AI-output evals (distinct from unit tests) in a Laravel app.

• Tests check "did it run"; evals check "is it good".
• Golden datasets for deterministic outputs, an LLM-judge for open-ended ones.
• Run on a schedule / before prompt changes, sampling prod traffic to catch model drift.

View ↗

✅ Testing & Quality🤖 Building with Agents

added by Adam Tomat • 18th Jun 2026

RepoPaul Duvall

AI Development Patterns

A curated catalog of 24+ patterns for building software with AI, organised by maturity and lifecycle.

• Foundation / Development / Operations tiers with a suggested adoption order.
• Tackles context-window limits, quality-left, and sandboxed security.
• Three lenses: Harness, Loop, and Lifecycle Engineering.

View ↗

🤖 Building with Agents✍️ Prompting & Context✅ Testing & Quality

added by Adam Tomat • 17th Jun 2026

RepoKai Aizen

Claude-Red

Security / red-team tooling for Claude.

• Aimed at probing and hardening Claude-based systems.
• Repo identity resolved via web search — see the source for current scope.

View ↗

🛠️ Tools & Apps✅ Testing & Quality

added by Adam Tomat • 17th Jun 2026