Measure AI coding agents by shipped work, not code volume
More PRs and commits can be useful, but the available evidence points to review as the constraint small teams should watch first.
Short answerUse AI coding agents only if shipped work rises; code volume alone is the wrong success metric.
By JasonPublished Oct 14, 2026Last verified Oct 14, 20267 min read
Short answer: small teams should treat AI coding agents as code-volume multipliers until their own delivery metrics prove otherwise. The stakes are practical: if agents create more pull requests than your review process can absorb, you may pay for faster typing while shipping the same amount of software more slowly. The clearest signal is a mismatch between more code activity and no statistically significant change in Jira-tracked issue and epic resolution in the Harvard/Jellyfish findings reported by Ars Technica.
What changed
AI coding agents appear to increase the amount of software work entering the system. After agent introduction, total lines of code rose 30 percent, commits rose 20 percent, and pull requests rose 23 percent on average in the Harvard/Jellyfish findings reported by Ars Technica.
That is not the same as finished software. The same findings showed no statistically significant change in the Jira-tracked resolution rate for Issues and Epics after AI tools were introduced. They also found no compositional shift in the size or complexity of Jira-tracked issues, so the evidence does not support a simple explanation that teams merely took on larger work.
The pressure point is review. Average time from pull-request submission to merge increased 49 percent after AI agents were introduced, comments per pull request rose 35 percent, and the share of workers performing code reviews increased 14 percent. By March 2026, 80 percent of measured firms used some form of AI code review, but AI review still accounted for a minority of review work.
The metric that should worry small teams
The useful derived number is the ratio between review slowdown and pull-request growth: review-process time rose 49 percent while pull requests rose 23 percent, so the review slowdown was 2.13 times the PR increase. This does not prove why review slowed; it tells a small team where to inspect first.
| Measure after AI-agent introduction | Change | Reader decision |
|---|---|---|
| Total lines of code | +30% | Do not use this alone as a success metric |
| Total commits | +20% | Treat as activity, not delivery |
| Pull requests | +23% | Check reviewer capacity before expanding usage |
| Jira issue and epic resolution rate | No statistically significant change | Require shipped-work metrics before claiming productivity |
| Review-process time | +49% | Budget review as the limiting resource |
The separate NBER paper points in the same strategic direction, though it used different measures and should not be treated as a direct replication. It found that autonomous coding agents had a 240 percent cumulative effect on commits, then 80 percent for projects and 30 percent for actual releases. The release-level number is still positive, but it is much smaller than the commit-level number.
Before and after: what the evidence actually supports
| Area | Before | After AI coding-agent introduction | What to do with it |
|---|---|---|---|
| Lines of code | Pre-introduction baseline | 30 percent increase | Watch for code growth without delivered work |
| Commits | Pre-introduction baseline | 20 percent increase | Treat as throughput into the system |
| Pull requests | Pre-introduction baseline | 23 percent increase | Pair with review-time tracking |
| Issues and Epics | Pre-introduction baseline | No statistically significant resolution-rate change | Use as the delivery check |
| Review-process time | Pre-introduction baseline | 49 percent longer on average | Treat review as capacity planning |
This is the important distinction for indie builders and small software teams: an agent can remove friction from writing code while adding friction to accepting, merging and maintaining it. If the team has one main reviewer, review is not a side process. It is the production line.
Who should act
Small teams using agents for production code should inspect their review queue before expanding access. The reported pattern is not “more code, more output”; it is “more code, more review work, no statistically significant change in Jira-tracked resolution.”
Teams can treat this as lighter monitoring if agents are limited to throwaway prototypes, one-off scripts, documentation drafts or code that never enters the main release path. In those cases, merge-time delay is less likely to determine whether users receive value.
- If agent-created PRs wait longer than human-written PRs, then cap agent scope because the bottleneck is review, not typing.
- If resolved customer issues do not rise while commits rise, then stop reporting commit volume as productivity because it is measuring input.
- If senior engineers are reviewing more code, then protect their time with smaller PRs and stricter acceptance criteria because agent output can create hidden coordination work.
- If agents are mostly used for tests, migrations or internal tooling, then keep them in those lanes and track merge time because low-risk work is easier to review.
What we could not confirm
The complete Chen and Stratton methodology and tables are unconfirmed. Publication of the full paper would settle the exact model choices, firm-level controls and robustness checks.
The specific AI coding-agent products and vendors included are also unconfirmed. A product-level breakdown would settle whether the effect is broad across the category or concentrated in particular tools.
The 49 percent review-time increase is reported, but the mechanism is not isolated. It could reflect code quality, reviewer behavior, risk controls or other process changes; a design that separates those causes would settle it.
The dataset scope is not perfectly aligned across public descriptions. Ars describes 300 million work events, more than 700000 employees and over 700 firms through March 2026, while Jellyfish’s preliminary announcement describes 100000 software engineers across 500 companies. That may reflect a preliminary version versus a later dataset, but the public figures differ.
Security impact, cost, ROI, subgroup effects by company size or language, and outcomes after March 2026 remain unconfirmed. Those gaps matter if you are buying seats, setting policy or replacing review time with automation.
What to do next
Turn the agent rollout into a delivery measurement exercise.
- [ ] Track resolved issues, shipped releases or customer-visible changes, not only lines, commits and PRs.
- [ ] Compare agent-assisted PRs with human-written PRs on review time, change requests and comments per PR.
- [ ] Set a maximum PR size for agent work so reviewers can reject bloated changes quickly.
- [ ] Keep a separate queue for low-risk agent tasks such as tests, scaffolding and internal scripts.
- [ ] Revisit the policy when full methodology, product-level results or post-March 2026 outcomes are available.
We would not freeze AI coding-agent adoption because of these findings. We would freeze the dashboard most teams use to defend it. A weekly report that celebrates more commits, more pull requests or more lines is measuring the part of engineering that agents are expected to inflate. It is not measuring whether customers got fixes, whether releases moved, or whether senior engineers lost focus to review queues.
For a small team, the right move is narrower than either hype or rejection. Keep agents close to bounded work: tests, migration scaffolds, small refactors with acceptance criteria, internal scripts and draft implementations that a human can review quickly. Do not let agent output quietly expand the amount of production code that must be inspected by the same one or two senior people.
The overstated version is “AI agents do not improve software productivity.” The defensible version is more specific: the Harvard/Jellyfish findings reported by Ars show more code activity without a statistically significant lift in Jira-tracked issue and epic resolution, while review work became heavier. The separate NBER paper makes the picture less absolute by finding positive, attenuated gains at projects and releases. Our view would change if later evidence showed teams redesigning review, testing and product scoping around agents consistently shipping more customer-visible work without longer review cycles, higher defect load or hidden maintenance cost.
Use AI coding agents only if shipped work rises; code volume alone is the wrong success metric.
AI coding agents can increase throughput into the engineering system, but small teams should judge them by resolved work, releases and review load.
Teams using agents only for prototypes, disposable scripts or non-release work can treat this as monitoring guidance rather than an adoption warning.
Sources
- AI coding agents generate more code, but not more softwareHacker News (front page), Oct 12, 2026.
- Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding ToolsNBER, May 29, 2026.
- New Research from Harvard & Jellyfish: AI Is Making Developers Faster. So Where's the Business Impact? - JellyfishJellyfish, Jan 28, 2026.
- AI coding agents generate more code, but not more softwareWinzheng, Oct 10, 2026.
Jason, Founder & editor, Benchdict. About Benchdict