Plan, Search, Replan, Act: Lessons From Building AI Agents

"Agent" has become one of those words that means whatever the person saying it is selling. Stripped of the marketing, an agent is a language model inside a loop, with a few tools it is allowed to call and permission to change its plan when what it finds disagrees with what it expected.

I have built two very different ones: a research agent that plans, searches, replans and then answers, and a voice agent you talk to out loud. They taught me different lessons, and both taught me that the loop is the easy part.

Why a model needs a loop at all

A model answering from memory is confidently wrong at exactly the moments that matter: anything recent, anything niche, anything it half-remembers. Worse, the wrong answer looks identical to the right one. Same tone, same fluency, same confidence.

Giving it tools (a search engine, Wikipedia, a calculator) only helps if it uses them at the right time and then actually reads what comes back. That is what the loop is for.

Plan, search, replan, act

The research agent follows four steps:

• Plan. Break the question into the things it needs to find out.
• Search. Call the tools for the first unanswered item.
• Replan. Look at what came back and revise the plan. Maybe a question is answered, maybe a new one appeared, maybe the original assumption was wrong.
• Act. When the plan says enough is known, write the answer from the evidence gathered.

In code, the skeleton is short. Everything interesting is in the limits around it:

MAX_STEPS = 6

def research(question, llm, tools):
    plan = llm.plan(question)
    notes = []
    for step in range(MAX_STEPS):
        if plan.is_complete():
            break
        task = plan.next_task()
        tool = tools.get(task.tool)
        if tool is None:
            notes.append(f"Unknown tool requested: {task.tool}")
            continue
        # cap what goes back into context
        result = tool(task.query)[:4000]
        notes.append(result)
        plan = llm.replan(question, plan, notes)
    # answer from the evidence, not from memory
    return llm.answer(question, notes)

Where agents go wrong

Every failure I have seen falls into one of a handful of shapes:

• Wandering. Each search suggests another search, and the agent never decides it knows enough. A hard step limit is not optional. Neither is a sensible fallback when it is reached: answer with what you have and say what is missing.
• Context flooding. A full web page dumped back into the prompt drowns the question. Trim and summarise tool output before it goes back in.
• Ignoring the evidence. The agent searches diligently, then answers from memory anyway. Asking it explicitly to answer from the gathered notes, and to say when the notes do not cover something, makes a large difference.
• Inventing tools. Models will occasionally request a tool that does not exist, or call a real one with nonsense arguments. Validate every tool call before running it.

The voice agent: a different set of problems

Typing hides how good a model is. A conversation you have to type is form-filling with better grammar. So the second agent listens and speaks: speech recognition on the way in, text-to-speech on the way out.

The surprise was that the model was the least difficult part. The hard part is the join between the three pieces, because people judge spoken conversation on timing in a way they never judge text.

• Know when the person has finished. Wait too long after silence and the agent feels slow. Cut in too early and it interrupts people mid-thought.
• Start speaking early. Do not wait for the full reply. Send the first complete sentence to the speech engine while the rest is still being generated.
• Write for the ear. Bullet points, tables and code read aloud are painful. Tell the model it is speaking, and keep replies short.
• Let people interrupt. If the user starts talking, stop playback. Nothing feels more robotic than a machine that will not stop talking.

What I would do the same way again

• Log every step. The plan, each tool call, each result, each revision. When an agent gives a strange answer, the log is the only way to find out which step went wrong.
• Few tools, well described. Three tools with clear descriptions beat ten vague ones. The model chooses tools from their descriptions, so the descriptions are part of the program.
• Cap everything. Steps, tokens, tool output, time. Agents are loops, and loops without limits end up in your bill.
• Show the sources. An answer that cites what it found can be checked. That is worth more than any amount of confident prose.

Limits first

An agent is not magic and does not need to be. It is a model given a plan, real tools and the humility to revise. Build the limits first and the cleverness second, and it becomes something you can trust with a real question. Both agents are live on my projects page if you want to try to trip them up.