ellmer vs Snorkel AI

The output is a research and benchmarking program, not a release feed. Recent work argues that single-episode benchmarks measure the wrong thing: agents should be scored across dependent states, tool calls, simulated users, approval rules, and learning carried between tasks. Concrete artifacts back the argument — Senior SWE-Bench with 100 tasks from real pull requests and half the set held private, GDPval+ for professional reasoning, and collaboration on Agents' Last Exam with Berkeley RDI. Alongside these, Snorkel publishes head-to-head evaluations of frontier model releases and hosts a reading group that surfaces outside research.

Snorkel is moving from evaluation-as-scoring to evaluation-as-training signal: the milestone framing scores intermediate progress, the continual-learning thread treats improvement across a task sequence as the measured quantity, and the newest reading-group post pushes further upstream still, into how much a reasoning model should be trained before it is tested. Publishing benchmarks with private splits and running public model comparisons builds the position that Snorkel is the neutral scorer, which is what makes the enterprise environments business defensible. The through-line is that measurement, not model capability, is the bottleneck.

Expect the milestone and continual-learning threads to converge into a named benchmark or environment suite with the same public-private split as Senior SWE-Bench. The feed carries research, talks, and reading-group recaps rather than platform releases, so it does not indicate what ships in the product.