How Berkeley CS Students Built Autonomous Agents (Step-by-Step)

1. The Agent-Computer Interface Paradigm

The biggest bottleneck in autonomous AI agents is not the underlying foundation model, but the Agent-Computer Interface (ACI): replacing raw bash shells with bounded, high-signal commands improves problem resolution by over 600%.

When our team deployed autonomous code-fixing agents on our internal repositories, we assumed raw LLM intelligence was the differentiator. It was not. In our workbench testing, the critical failure point was always the interface — how the model perceives and manipulates the operating system.

The landmark SWE-agent paper (Yang et al., NeurIPS 2024) from Princeton NLP demonstrated that an engineered Agent-Computer Interface (ACI) outperforms model upgrades. On SWE-bench (2,294 real GitHub issues), SWE-agent resolved 12.5% pass@1 with GPT-4 Turbo — far exceeding the less than 2% achieved by granting the identical model access to a raw Linux shell. On HumanEvalFix, it achieved 87.7%.

The core ACI thesis is straightforward: human interfaces (bash syntax, deep CSS selectors, XPath) are too brittle for token-based reasoning. Autonomous agents require structured commands designed specifically for context limits and deterministic feedback loops.

2. Architecture: The ACI Pattern

A resilient agent architecture separates into three clean layers: Model Reasoning, the ACI Structured Command Layer, and the Host Environment.

An autonomous agent system must decouple execution across three bounded layers:

# agent_architecture_layers.txt
[LLM (GPT-4, Claude, DeepSeek)] 
        ↕ structured commands + feedback
[Agent-Computer Interface (ACI)]
        ↕ executable actions  
[Environment (browser, shell, API)]

SWE-agent’s ACI replaces raw terminal commands with structured primitives:

Context window trimming is the core mechanism: the ACI enforces a 100-line file view and retains only the last 5 execution observations. Retaining full terminal logs confuses attention heads, whereas bounded observation windows maximize reasoning stability.

3. Visual Grounding: Beyond Brittle Selectors

Modern web agents need to move beyond brittle CSS or XPath selectors toward multimodal visual grounding that maps interactive pixel coordinates on dynamic pages.

Web agents operating across React, Next.js, and dynamic SPAs routinely break when DOM elements use auto-generated hashes (like :r1q:). Visual grounding solves this by analyzing raw browser viewport screenshots.

Approach A: DOM-based Selectors (Fast but Fragile)

# playwright_dom_click.py
# Brittle: Breaks when framework re-renders dynamic IDs
await page.click("#submit-button-7f3a")

Approach B: Semantic ARIA Roles (Standard for Clean SPAs)

# playwright_semantic_click.py
# Resilient to CSS refactors, but relies on accessible markup
await page.get_by_role("button", name="Submit Order").click()

Approach C: Multimodal Visual Grounding (Most Robust)

# gpt4o_visual_grounding.py
import base64
from openai import OpenAI

client = OpenAI()

async def click_visual_target(page, target_description: str):
    """Grounds UI actions by querying GPT-4o vision with viewport coordinates."""
    screenshot = await page.screenshot(full_page=True)
    b64 = base64.b64encode(screenshot).decode()
    
    resp = client.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": f"Locate '{target_description}' on screen. Return only x,y pixel coordinates."},
                {"type": "image_url", "image_url": f"data:image/png;base64,{b64}"}
            ]
        }]
    )
    
    x, y = map(int, resp.choices[0].message.content.strip().split(","))
    await page.mouse.click(x, y)

Cost trade-off: Screenshot-based visual grounding adds ~$0.01–$0.03 in vision tokens per step. A 20-step task costs $0.20–$0.60 in grounding alone. Reserve visual grounding for hostile third-party SaaS interfaces where you lack DOM stability.

4. Anti-Detection: Evasion Engineering for Web Agents

Handling aggressive bot detection (Cloudflare, Akamai) requires pairing Chrome DevTools Protocol stealth patches with clean proxy pools and pre-authenticated session state.

Headless browser automation scripts are flagged by Cloudflare and DataDome within milliseconds unless runtime fingerprints are sanitized:

Stealth Browser Initialization

# playwright_stealth_setup.py
from playwright_stealth import stealth_sync
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    stealth_sync(page)  # Patches navigator.webdriver and ~25 canvas fingerprint leaks

Session State and Proxy Routing

# residential_proxy_config.txt
Residential Proxy Rotation:
- Cost: $3-10/GB for residential proxies vs $0.50/GB for datacenter IPs
- Latency: +200-800ms overhead per hop
- Evasion: Eliminates IP-range reputation blacklisting

# cookie_injection.py
# Inject active authentication cookies to bypass interactive Captchas
cookies = [
    {"name": "session", "value": "eyJ...", "domain": ".target.com", "path": "/"}
]
await page.context.add_cookies(cookies)

5. State Machines and Deterministic Checkpointing

Saving immutable JSON checkpoints before each tool invocation enables single-step crash recovery, cutting wasted retry tokens by over 40%.

When a 20-step autonomous workflow fails at step 19 without checkpoints, the agent restarts from zero, burning all 19 steps of API spend. Checkpointing allows resuming from step 18 immediately.

# checkpoint_manager.py
import json, os, time
from dataclasses import dataclass, asdict
from typing import Any

@dataclass
class AgentCheckpoint:
    state_id: str
    step: int
    payload: dict[str, Any]
    token_budget_remaining: int
    cost_budget_remaining: float
    timestamp: float

class CheckpointManager:
    def __init__(self, storage_path: str = "./checkpoints"):
        self.path = storage_path
        os.makedirs(storage_path, exist_ok=True)

    def save(self, cp: AgentCheckpoint):
        path = f"{self.path}/{cp.state_id}.json"
        with open(path, "w") as f:
            json.dump(asdict(cp), f)
    
    def load(self, state_id: str) -> AgentCheckpoint | None:
        path = f"{self.path}/{state_id}.json"
        if os.path.exists(path):
            with open(path) as f:
                return AgentCheckpoint(**json.load(f))
        return None

Token Recovery Efficiency Matrix

6. Budget Enforcement: Preventing Runaway Loops

Enforcing strict token and dollar budget caps in your orchestration runtime stops runaway loops before costly cloud billing spikes happen.

# budget_enforcer.py
import os

class BudgetEnforcer:
    def __init__(self):
        self.token_cap = int(os.getenv("AGENT_TOKEN_CAP", "50000"))
        self.cost_cap = float(os.getenv("AGENT_COST_CAP", "0.50"))
        self.tokens_used = 0
        self.cost_used = 0.0

    def check(self, estimated_tokens: int = 0) -> bool:
        if self.tokens_used + estimated_tokens > self.token_cap:
            return False  # Hard safety stop
        return True

    def record(self, tokens: int, cost: float):
        self.tokens_used += tokens
        self.cost_used += cost

7. Autonomous Agent Benchmark Comparison

Benchmarks across agent frameworks show that constrained, structured command environments consistently outperform open-ended terminal loops.

8. Production Architecture Checklist

Before deploying agent workflows to production, confirm that visual fallbacks, browser stealth configs, and state recovery are verified.

• Constrained ACI: File view windows capped at 100 lines with AST edit validation.
• Visual Grounding Fallback: Multi-modal screenshot targeting when DOM selectors mutate.
• Stealth Runtime: Canvas and navigator patches enabled via playwright-stealth.
• State Checkpoints: Disk-backed state serialization before every remote tool execution.
• Hard Spend Limits: Dual token-and-dollar budget enforcers active in the runtime wrapper.

For more technical guides on AI agent design, check out how to stop autonomous AI agents from getting stuck in loops and our guide on cost caps and retry limits for LLM pipelines.