Design an Autonomous Coding Agent
Track
Agentic System Design
Inspired by
Claude Code / Devin
Scale
large repos · sandboxed execution · test-driven loop
Difficulty
Advanced
A coding agent is the agentic interview where the environment does the verification: the agent can run the code and read the result, so the design centers on a tight plan → edit → run tests → observe loop with a real feedback signal. It is also the domain with the clearest public evidence of what works: on SWE-bench — 2,294 real GitHub issues from 12 Python repositories — the best 2023 systems resolved under 2% of tasks (Jimenez et al.); by late 2024 a deliberately simple agent around Claude 3.5 Sonnet resolved 49% of SWE-bench Verified (Anthropic). Walk it through the agentic project loop — context and orchestration are where it's won, and tests give evaluation for free.
1. Clarify the task (spine: task)
- Goal: given an issue / feature request, produce a correct code change (edits + tests) in a real repository — ideally a passing PR.
- When it acts vs. stops: iterate until tests pass or a step/time budget is hit; escalate on ambiguous requirements or destructive operations.
- Success: tests green + task actually solved (not just tests deleted) + reasonable, reviewable diff.
- Scope: the agent loop over a sandboxed repo. Adjacent: the CI/review system it hands off to.
Interview signal
"The agent can execute code, so verification is grounded — tests are the reward signal." Naming the executable environment as the core asset separates a coding agent from a code-completion model.
Think firstSWE-bench solve rates jumped from under 2% to ~50% in about eighteen months — on the same benchmark. Models improved, but that's not the whole story. What else was being engineered?Commit to an answer, then revealHide
The agent–environment interface. SWE-agent showed that with the same model, redesigning how the agent views files, searches, and edits code lifted solve rates several-fold (to 12.5% pass@1) — the interface, not the model, was the bottleneck. Anthropic's 49% run pushed the opposite lesson to its limit: a minimal scaffold with two general-purpose tools, letting a stronger model drive. Both agree on the design principle: an agent is a model plus an environment you control, and the environment is your half of the work.
2. Requirements & constraints
| Dimension | Target | Consequence |
|---|---|---|
| Repo scale | Large, unfamiliar codebases | Retrieval over the repo, not whole-repo in context |
| Safety | Runs arbitrary code | Isolated sandbox, no prod creds, network limits |
| Latency/cost | Minutes/task, bounded | Step budget; model tiering; cache builds |
| Reliability | Doesn't wreck the repo | Diff review, reversible changes, test gate |
Think firstThe repo is millions of lines and the context window holds a fraction of it. How should the agent decide what to read?Commit to an answer, then revealHide
Retrieval, not stuffing: code search (lexical + embedding), symbol lookup, and the dependency graph surface a small relevant set, and the agent explores incrementally from there — open a file, follow references. Context is a budget to spend, not fill; the design question is what earns its place in it. The agent loop is exactly this discipline: gather context → take action → verify, repeated (Anthropic).
3. Context & tools (spine: context)
- Repo retrieval: the codebase is too large for the context window — retrieve relevant files via code search (lexical + embedding), the dependency graph, and symbol lookup. Feed targeted context, not the whole repo.
- Tools:
read_file,edit_file,run_shell/run_tests,search_code,git— typed and sandboxed. Execution is a first-class tool, not an afterthought. - Interface design matters as much as tool choice. The SWE-agent paper (Yang et al.) calls this the agent–computer interface (ACI): compact search results instead of pages of hits, a file viewer showing ~100 lines at a time, and an edit command that runs a linter and rejects malformed edits with feedback — each measurably moved the solve rate. Anthropic's SWE-bench scaffold went further in the minimal direction: essentially a bash tool and a string-replacement edit tool, with the model deciding everything else (Anthropic).
- Environment: an isolated container with the repo checked out; the agent's actions (edits, commands) mutate it.
Think firstThe agent has made three edits and the same test still fails with the same error. What should the harness do that a naive retry loop wouldn't?Commit to an answer, then revealHide
Detect the stuck state and force a change of strategy. Repeated identical failures mean the agent's model of the bug is wrong — the harness should feed the full failure history back, prompt an explicit re-diagnosis (reflection) rather than another blind edit, and if the loop persists, stop and escalate. Budgets exist precisely because an agent that can't tell it's stuck will happily burn the whole budget on one wrong theory.
4. Orchestration (spine: orchestration)
The loop is plan → edit → run → observe → reflect:
- Understand the task; locate the relevant code (retrieval).
- Make an edit.
- Run the tests / build: the grounded feedback signal.
- Read failures; reflect and fix; repeat until green or budget hit.
The ReAct loop — and why agents compound errors
An agent cycles Thought → Action → Observation, each Observation conditioning the next Thought. But errors multiply across steps: if a single step is p reliable, an n-step trajectory is only pⁿ reliable. Drag p and watch a 95%-per-step agent fall below 60% end-to-end over ten steps — the case for bounded loops, validation, and human-in-the-loop.
- Reflection on failure: feed the test output back so the model diagnoses, rather than blindly re-editing.
- Step budget & stop conditions: cap iterations; stop on green, on repeated identical failures (stuck), or on ambiguity → escalate.
- Keep the topology simple. Anthropic's guidance is to start with a single augmented loop and add structure only when it measurably helps (Anthropic). Cognition (Devin) goes further for code: parallel sub-agents making edits tend to make conflicting implicit decisions, because each holds a partial context, so keep write-actions single-threaded and share full context (Cognition). Sub-agents are safest for read-only work: exploring the repo, reproducing the bug.
Think firstThe agent reports “all tests pass.” In how many ways can that claim be hollow?Commit to an answer, then revealHide
At least three: it deleted or weakened the failing tests (reward hacking), it overfit to the visible tests without solving the actual issue, or it got lucky on a flaky suite. That's why SWE-bench grades against held-out fail-to-pass tests the agent never sees, and why your own eval needs protected tests plus trajectory checks: did it edit sensible files, run tests before claiming done?
5. Evaluation (spine: evaluation)
- Outcome: does the change pass the hidden/held-out tests and actually solve the issue? SWE-bench measures exactly this; pass@k captures the retry-until-solved reality.
- Benchmark hygiene: the original SWE-bench contained under-specified issues and broken tests that unfairly penalized correct patches — SWE-bench Verified is a 500-task subset screened by human annotators to fix this (OpenAI). The lesson generalizes: audit your eval set before trusting movement on it.
- Trajectory: did it edit the right files, run tests before claiming done, and avoid thrashing? Catches agents that "pass" by gaming or luck.
- Guard against reward hacking: an agent that deletes failing tests "passes" — eval must use protected tests it can't modify.
Think firstThis agent runs arbitrary generated shell commands. Sketch the threat model before reading on.Commit to an answer, then revealHide
Three distinct threats: destructive actions (rm -rf, force-push, dropping a database), exfiltration — a malicious file or issue comment prompt-injects the agent into leaking secrets, and runaway resource use. Each maps to a different control: sandbox + no prod credentials, treating all repo/issue text as untrusted input, and hard resource/step caps — plus every change landing as a reviewable PR, never a direct push.
6. Guardrails & safety (spine: guardrails)
- Sandbox everything: no production credentials, restricted network, resource limits — it runs arbitrary generated code.
- Reversibility: changes land as a branch/PR a human reviews, never a direct push to main.
- Prompt injection: repo content, issue text, and dependency output are untrusted — a malicious file must not exfiltrate secrets or run unsafe commands.
Think firstA full eval run is 500 tasks × dozens of model calls each, and two identical runs give different scores. How do you make the numbers trustworthy without unbounded spend?Commit to an answer, then revealHide
Report across multiple trials (pass@k or mean ± spread), never a single run — agent variance easily swamps a 2-point "improvement." Contain cost with a fixed cheap smoke subset for every change and the full suite only for candidate releases. And trace everything: when the score moves, per-task trajectories tell you which tasks flipped and why — without traces, a regression is just a number.
7. Operations (spine: operations)
- Tracing: log every tool call, edit, and test result — essential for debugging non-deterministic runs and for the eval set.
- Cost/latency: tier models (cheap for navigation, strong for hard reasoning), cache builds/deps, parallelize independent read-only sub-tasks.
- Feedback: failed/abandoned tasks become eval cases and prompt improvements.
8. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Retrieve repo context | Stuff whole repo in context | Repos exceed context; retrieval is cheaper and sharper |
| Test-driven loop | Generate-and-hope | Execution gives a real, groundable reward signal |
| Minimal general tools | Many specialized tools + rigid workflow | Simple scaffolds transfer; ACI quality beats tool count |
| Single-threaded writes | Parallel editing sub-agents | Partial-context workers make conflicting decisions |
| PR + human review | Auto-merge | Reversibility and trust; code is high-stakes |
| Protected test eval | Any tests pass | Prevents reward hacking (deleting tests) |
9. Interviewer follow-ups
The agent 'passed' by deleting the failing test. How do you prevent it?
Evaluate against protected, hidden tests the agent can't see or modify, and diff- check that it didn't weaken the test suite. Reward hacking is the defining failure mode of code agents — the eval, not the agent, must own the ground-truth tests. This is exactly how SWE-bench grades: held-out fail-to-pass tests applied after the agent finishes.
The repo is millions of lines. How does the agent find the right code?
Retrieval: code search (lexical + embedding), symbol/definition lookup, and the dependency graph surface a small relevant set. The agent explores incrementally (open a file, follow references) rather than loading everything — context is a budget to spend, not fill.
Would you use multiple agents in parallel to go faster?
Only for read-only work (exploration, reproduction, review). Parallel sub-agents that edit hold partial context and make conflicting implicit decisions — the failure mode Cognition documents from building Devin. Keep writes single-threaded with full shared context; buy speed with caching and model tiering instead.
How do you keep it from running something destructive?
Sandbox with no prod access, restricted network, and resource caps; require confirmation or block destructive shell patterns; land changes as a reviewable branch. Treat all repo/issue text as untrusted input that cannot escalate the agent's permissions.
Key takeaways
- A coding agent's superpower is a sandboxed executable environment — tests are a grounded reward signal for a plan → edit → run → reflect loop.
- The agent–computer interface is a measured, first-order lever: SWE-agent's ACI redesign and Anthropic's two-tool 49% SWE-bench Verified scaffold both show the environment matters as much as the model.
- Repo retrieval (not whole-repo context) and reflection on test failures drive success; step budgets and stuck-detection prevent thrashing.
- Evaluate on protected tests (pass@k, multiple trials) to stop reward hacking, and keep every change sandboxed and reversible.
References
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: Jimenez et al., ICLR 2024; the benchmark (2,294 real issues) and the under-2% 2023 baseline.
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering: Yang et al., NeurIPS 2024; ACI design and the 12.5% pass@1 result.
- Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet: Anthropic engineering; the 49% minimal two-tool scaffold.
- Introducing SWE-bench Verified: OpenAI; the human-validated 500-task subset and why eval sets need auditing.
- Building Effective AI Agents: Anthropic engineering; start simple, workflows vs. agents, add structure only when it pays.
- Don't Build Multi-Agents: Cognition (Devin); context engineering and why parallel editing sub-agents conflict.
- Building agents with the Claude Agent SDK: Anthropic engineering; the gather-context → take-action → verify-work loop.