The CMU TheAgentCompany 2026 benchmark figure (30.3% task completion for best-in-class frontier model, up from 24% in 2024) is the current capability constraint for enterprise agentic AI. Capability trajectory projects to ~40% by late 2027, which does not cross the 95% production-readiness threshold within the 3-year TCO horizon enterprise business cases operate against. The Stanford DEL 12% durable cohort operates within the 30.3% (narrow scope + human-in-the-loop + GAUGE-dimensional governance discipline), not around it. Capability is not the variable that separates the 12% from the 88%.
Re-review 10 Jun 2026: capability legs verified. 30.3% full-completion top score (Gemini 2.5 Pro, 175-task set, 39.3% with partial credit) per TheAgentCompany paper v2 (arxiv 2412.14161v2); the 24% 2024 baseline (Claude 3.5 Sonnet) per v1. No frontier model found above 50% on the benchmark, so watch (1) has not fired. The Stanford DEL 12%-durable-cohort leg failed primary-source verification: the cited Enterprise AI Playbook contains no 12/88 cohort (see AM-029 correction, 10 Jun 2026). Extended same day after the full exposure-map investigation (docs/editorial/stanford-1288-exposure-map-2026-06-10.md): the 12/88 has no primary source anywhere as an ROI or cohort distribution — it is the IDC/Lenovo pilot-graduation finding (roughly 88% of AI proof-of-concepts never reach production, roughly 12% graduate; via CIO.com, Mar 2025) fused with the Stanford DEL's name and an invented methodology. Any future re-anchor of the cohort-behaviour leg must use the IDC graduation metric or drop the cohort framing. The capability-constraint reading stands; the cohort-behaviour leg is unanchored. Status Up -> Partial. Watches unchanged: (1) frontier model crossing 50%, (2) capability-wait vs governance-discipline equivalence studies, (3) benchmark refresh shifting task distribution.
Correction log
- 10 Jun 2026One leg unanchored on re-review. The CMU capability figures verify cleanly (30.3% full completion for Gemini 2.5 Pro and 39.3% partial-credit on the 175-task TheAgentCompany set per paper v2; 24% for Claude 3.5 Sonnet in the Dec 2024 v1). The Stanford DEL '12% durable cohort' referenced in the claim text does not exist in the cited source: the Enterprise AI Playbook (Pereira, Graylin, Brynjolfsson, Apr 2026) studies 51 successful deployments and contains no 12/88 ROI cohort (full finding at AM-029, correction of 10 Jun 2026). The claim's capability-constraint argument holds on its own evidence; the sentence tying the constraint to the 12%/88% cohort behaviour has no verifiable referent. Status Up -> Partial.
/holding/AM-031/Embed this claimiframe + oEmbed
The card auto-updates when the claim's status, last-reviewed date, or correction log changes. Embedders never need to refresh — the card is rendered live from the canonical record.
Email-me when AM-031's status, next review date, or correction log changes. One email per change. No newsletter subscription, no other mail.
The claim: The CMU TheAgentCompany 2026 benchmark figure (30.3% task completion for best-in-class frontier model, up from 24% in 2024) is the current capability constraint for enterprise agentic AI. Capability trajectory projects to ~40% by late 2027, which does not cross the 95% production-readiness threshold within the 3-year TCO horizon enterprise business cases operate against. The Stanford DEL 12% durable cohort operates within the 30.3% (narrow scope + human-in-the-loop + GAUGE-dimensional governance discipline), not around it. Capability is not the variable that separates the 12% from the 88%.
About this register
The Reporting register tracks claims published from articles addressed to senior enterprise IT leaders — CIOs, IT directors, heads of platform. Claims are reviewed on a 30–90 day cadence; each review either reaffirms the claim, marks one substantive part as Partial, or marks it Not holding once the underlying evidence has been overtaken.
Recent corrections in Reporting
- AM-132 · Partial · 10 Jun 2026
One of four legs unanchored on re-review. The claim text attributes '12% of deployments clearing 300%+ ROI with 88% at or below break-even at 12-18 months' to the Stanford DEL 2026 Enterprise AI Playbook. Full-text verification on 10 Jun 2026 found no such figure in that source: the playbook (Pereira, Graylin, Brynjolfsson, Apr 2026) studies 51 successful deployments by design and contains no ROI distribution, no 300%-plus cohort, and no break-even measurement point (full finding at AM-029, correction of 10 Jun 2026). The only verified figure carrying the same 12/88 numerals is IDC research with Lenovo (via CIO.com, Mar 2025): roughly 88% of AI proof-of-concepts never reach production and roughly 12% graduate — a pilot-to-production graduation metric, not an ROI distribution. The Gartner 28%, McKinsey 23%/17%, and MIT NANDA 95% legs verify; they support a small high-performing tail and a large struggling body, but none documents the two-peak bimodal shape the claim asserts. Status Up -> Partial.
- AM-129 · Partial · 10 Jun 2026
One of three read-against anchors unanchored on re-review. The claim text cites 'Stanford Digital Economy Lab Enterprise AI Playbook (12/88 bimodal ROI distribution at 12-18 months)' and frames the realistic ROI band around 'the highest-discipline 12% cohort'. Full-text verification on 10 Jun 2026 found the playbook contains no 12/88 distribution, no bimodal ROI shape, and no 12-18-month ROI measurement point (full finding at AM-029, correction of 10 Jun 2026). The claim's core negative finding — no mid-market enterprise has produced a documented +240% ROI in 90 days under audited conditions — is unaffected; the McKinsey State of AI 2025 and MIT NANDA legs verify and continue to support it. The '12% cohort' framing has no verifiable referent. The only verified figure carrying the 12/88 numerals is IDC's pilot-graduation finding (roughly 88% of AI proof-of-concepts never reach production; via CIO.com, Mar 2025), a different metric. Status Up -> Partial.
- AM-201 · Partial · 10 Jun 2026
One of four named datasets unanchored on review. The claim text names 'Stanford DEL's 12% clearing 300%+ ROI vs 88% at or below break-even' as one of four independent datasets. Full-text verification on 10 Jun 2026 found the Stanford DEL Enterprise AI Playbook contains no such distribution — it studies 51 successful deployments by design and carries no ROI-realisation failure data (full finding at AM-029, correction of 10 Jun 2026). The McKinsey (23% scaling, 17% EBIT-attribution), Gartner (28% fully paying off), and MIT NANDA (95% no measurable P&L impact) datasets verify; the claim's spine stands on three datasets rather than four. The only verified figure carrying the 12/88 numerals is IDC's pilot-graduation finding (roughly 88% of AI proof-of-concepts never reach production; via CIO.com, Mar 2025), a different metric from an ROI distribution. Status Up -> Partial.
Reviews coming up in Reporting
- AM-063 · Holding · next +15d (27 Jun 2026)
AI agents executing financial transactions need a four-control bundle (action-approval gates by blast radius, kill-swit…
- AM-061 · Holding · next +15d (27 Jun 2026)
Production agentic-AI costs at scale routinely run multiples of POC projections, and a layered optimisation programme c…
- AM-003 · Partial · next +15d (27 Jun 2026)
GPT-5 Pro's tiered-subscription model forces enterprises to classify problems by computational difficulty — $200/month…