How Berkeley CS Students Built Autonomous Agents (Step

1. The Agent-Computer Interface Paradigm

Most people think building an autonomous AI agent is about picking the right model. It’s not. In our team’s experience building automated test rigs, the bottleneck is always the interface — how the model talks to the computer.

The SWE-agent paper (Yang et al., NeurIPS 2024) from Princeton NLP showed that a well-designed Agent-Computer Interface (ACI) matters more than the model itself. On SWE-bench (2,294 real GitHub issues), SWE-agent resolved 12.5% pass@1 with GPT-4 Turbo — far exceeding the less than 2% achieved by giving the same model a raw Linux shell. On HumanEvalFix, it hit 87.7%.

The ACI insight: human-friendly interfaces (bash, CSS selectors, XPath) are brittle for LLMs. Agents need commands designed for their capabilities — simple, structured, with precise feedback loops.

2. Architecture: The ACI Pattern

An autonomous agent system has three layers:

[LLM (GPT-4, Claude, DeepSeek)] 
        ↕ structured commands + feedback
[Agent-Computer Interface (ACI)]
        ↕ executable actions  
[Environment (browser, shell, API)]
```bash

SWE-agent's ACI provides specialized commands that replace the raw Linux shell:

| Command | Shell Equivalent | Why for Agents |
|---------|-----------------|----------------|
| `open <file>:<line>` | `vim` / `less` | Shows 100-line window — models struggle with huge files |
| `edit <file> <start>-<end>` | `sed` | Precise line edits with syntax validation feedback |
| `search <term>` | `grep -r` | Results formatted as file:line:context triples |
| `submit` | — | Signals task completion to stop the loop |

The key design choice: **context window management**. The ACI limits file views to 100 lines with only the last 5 observations retained. The paper's hyperparameter sweep showed this outperformed larger windows — models confuse themselves with too much history.

---
## 3. Visual Grounding: Beyond Brittle Selectors

Web agents need to interact with UIs that change on every render. Standard CSS/XPath selectors break on React/NextJS SPAs where elements get dynamic IDs like `:r1q:`.

There are three approaches, in order of robustness:

### Approach A: DOM-based (fragile but fast)
```python
## 2.
await page.click("#submit-button-7f3a")
```bash

### Approach B: Semantic DOM (better, needs good identifiers)
```python
## 3. await page.get_by_role("button", name="Submit Order").click()
```bash

### Approach C: Visual grounding (most robust, most expensive)
```python
import base64
from openai import OpenAI

client = OpenAI()

async def click_visual_target(page, target_description: str, screenshot_path: str):
    screenshot = await page.screenshot(full_page=True)
    b64 = base64.b64encode(screenshot).decode()
    
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": f"Where is '{target_description}'? Return x,y coordinates."},
                {"type": "image_url", "image_url": f"data:image/png;base64,{b64}"}
            ]
        }]
    )

Parse “x,y” from response

x, y = map(int, resp.choices[0].message.content.strip().split(”,”)) await page.mouse.click(x, y)

Cost trade-off: Each screenshot-based action costs ~$0.01-0.03 in GPT-4o vision tokens. A 20-step workflow runs $0.20-0.60 just for grounding. The GroundingAgent paper (Luo et al., AAAI 2026) achieved 65.1% zero-shot accuracy on RefCOCO using iterative visual reasoning without fine-tuning, but at 3-5x latency of DOM-based methods.

When to use each: DOM for stable internal dashboards, semantic for well-maintained SPAs with aria attributes, visual for third-party SaaS where you control nothing.

4. Anti-Detection: The Cat-and-Mouse Game

Cloudflare, Akamai, and DataDome flag headless browsers within milliseconds. The Berkeley EECS technical report (Ozdarendeli, 2026) showed that prompt-injection attacks succeed on 63.1% of unprotected web agents, but a layered defense reduces that to 1.9%.

Common evasion techniques and their real trade-offs:

Stealth fingerprinting

## 4.

5. from playwright_stealth import stealth_sync

from playwright.sync_api import sync_playwright

with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() stealth_sync(page) # Patches ~25 browser detection signals

85% reduction in bot flags per EV-000007

Residential proxy rotation

Cost: $3-10/GB for residential proxies vs $0.50/GB for datacenter
Latency: +200-800ms per request due to multi-hop routing
Effectiveness: Avoids IP reputation blacklists entirely
```bash

### Cookie injection
```python
## 6. cookies = [
    {"name": "session", "value": "eyJ...", "domain": ".target.com", "path": "/"}
]
await page.context.add_cookies(cookies)
```bash

**The hard truth**: Anti-detection is a maintenance burden, not a one-time fix. Security vendors update signatures weekly. Expect 5-10% of your agent engineering time to go to keeping evasion working.

---
## 5. State Machines and Checkpointing

Multi-step agents fail. The question is how expensively.

Without checkpointing, a 12-step workflow that fails at step 11 burns all 11 steps of tokens on retry. With checkpointing, you resume from step 11 — saving ~80% of retry tokens.

### Production checkpoint pattern

```python
import json, os, time
from dataclasses import dataclass, asdict
from typing import Any

@dataclass
class AgentCheckpoint:
    state_id: str
    step: int
    payload: dict[str, Any]
    token_budget_remaining: int
    cost_budget_remaining: float
    timestamp: float

class CheckpointManager:
    def __init__(self, storage_path: str = "./checkpoints"):
        self.path = storage_path
        os.makedirs(storage_path, exist_ok=True)

    def save(self, cp: AgentCheckpoint):
        path = f"{self.path}/{cp.state_id}.json"
        with open(path, "w") as f:
            json.dump(asdict(cp), f)
    
    def load(self, state_id: str) -> AgentCheckpoint | None:
        path = f"{self.path}/{state_id}.json"
        if os.path.exists(path):
            with open(path) as f:
                return AgentCheckpoint(**json.load(f))
        return None

    def validate(self, cp: AgentCheckpoint) -> bool:
        """Check checkpoint integrity before resuming"""
        required = ["step", "payload", "token_budget_remaining"]
        return all(getattr(cp, f, None) is not None for f in required)
```powershell

### Token cost comparison

| Workflow length | Without checkpointing | With checkpointing | Savings |
|----------------|----------------------|-------------------|---------|
| 10 steps, fail at step 8 | 10 + 10 = 20 steps | 10 + 2 = 12 steps | 40% |
| 20 steps, fail at step 17 | 20 + 20 = 40 steps | 20 + 3 = 23 steps | 42.5% |
| 50 steps, fail at step 43 | 50 + 50 = 100 steps | 50 + 7 = 57 steps | 43% |

At GPT-4 pricing ($15/1M input, $60/1M output), a 100-step retry costs roughly $3-8. Checkpointing cuts that to $1.50-4 per recovery.

---
## 6. Budget Enforcement: Preventing Runaway Costs

The most expensive bug is an agent stuck in a loop. Production systems enforce both token and dollar caps:

```python
import os

class BudgetEnforcer:
    def __init__(self):
        self.token_cap = int(os.getenv("AGENT_TOKEN_CAP", "50000"))
        self.cost_cap = float(os.getenv("AGENT_COST_CAP", "0.50"))
        self.tokens_used = 0
        self.cost_used = 0.0

    def check(self, estimated_tokens: int = 0) -> bool:
        if self.tokens_used + estimated_tokens > self.token_cap:
            return False  # Hard stop
        return True

    def record(self, tokens: int, cost: float):
        self.tokens_used += tokens
        self.cost_used += cost

    def summary(self) -> dict:
        return {
            "tokens_used": self.tokens_used,
            "cost_used": self.cost_used,
            "token_remaining": self.token_cap - self.tokens_used,
            "cost_remaining": self.cost_cap - self.cost_used,
        }

The mini-swe-agent 2.0 (Feb 2026) hit 74% on SWE-bench Verified with aggressive budget caps — proof that constraints don’t necessarily hurt performance.

7. Benchmark Comparison

The jump from 12.5% to 74%+ comes mostly from better models and improved ACI design — not architectural breakthroughs.

8. Production Considerations

When not to use a web agent

• Tasks with reliable APIs (use the API instead — cheaper, faster, deterministic)
• High-security environments (banking, healthcare portals — terms-of-service risk)
• Workflows requiring greater than 99.9% reliability (agents fail 5-20% of the time even with all mitigations)

Stack recommendations

• Browser automation: Playwright (best API, cross-browser, CDP support)
• Stealth: playwright-stealth + fingerprint randomization
• LLM: GPT-4o for visual grounding, Claude 3.5+ for code tasks, DeepSeek for cost-sensitive
• State: Custom checkpoint manager (no existing framework does this well)
• Budget: Custom enforcer per above
• Monitoring: LangSmith or custom traces with step-level token/cost logging

Failure modes to design for

1. Captcha: Can’t solve most captchas. Route to human or skip the action.
2. Rate limiting: Exponential backoff with jitter. Budget for 429s.
3. DOM mutation mid-flight: Re-ground before each action, don’t cache selectors.
4. Session expiry: Re-authenticate via cookie refresh before each workflow.

9. References

• Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,” NeurIPS 2024. arxiv:2405.15793
• Luo et al., “Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning,” AAAI 2026. arxiv:2511.19516
• Ozdarendeli, “Evaluating and Improving Autonomous AI Agents for Secure and Private Deployment,” UC Berkeley EECS Technical Report UCB/EECS-2026-149, 2026.
• SWE-agent: github.com/SWE-agent/SWE-agent
• mini-swe-agent 2.0: 74% on SWE-bench Verified (Feb 2026)
• WebArena: webarena.dev

To see how we optimize agent memory windows, check out our guide on Why AI Coding Agents Truncate Context.

Related Guides

• How to Set Cost Caps & Retry Limits for LLM Pipelines — Prevent infinite agent loops and catastrophic API token spend.