Andon Labs

Overview

Andon Labs researches autonomous agents through benchmarks and experiments involving extended responsibility. Its public research program focuses on organizations operated by AI and the control problems that arise when humans cannot review every decision. The lab combines simulated tasks with deployments intended to reveal how agents behave outside short question-and-answer tests.

A prominent example is Vending-Bench, which asks agents to manage a simulated vending business over a long period. The site describes a second version involving suppliers, negotiations, customer complaints, and profit objectives, as well as an arena where agents compete. Other published projects cover radio stations, retail experiments, spatial understanding, and robotics.

Andon belongs in this directory primarily as a research and evaluation lab. Its published experiments are evidence of the kinds of questions it studies; they should not be read as a catalog of production-ready business automation products.

What the research tests

Our assessment is that extended business tasks expose failure modes that isolated prompts often miss. A system can make a reasonable purchase today and still run out of cash later. It can respond persuasively to a supplier while losing track of commitments made in previous interactions. Success depends on state, memory, adaptation, and the cumulative consequences of decisions.

That makes this research relevant to teams evaluating agents expected to operate for hours, days, or longer. The important unit is the whole run, including recovery from mistakes, rather than the percentage of individually plausible messages.

Reading the results

When comparing findings, examine the experiment's tools, time horizon, information available to the agent, and any human intervention. A simulated business and a real store present different constraints, so their outcomes should not be treated as interchangeable measurements. Profit can also conceal fragile behavior if a run benefits from luck or an unusually forgiving setting. Researchers considering collaboration should clarify whether they need an existing benchmark, a custom evaluation, or an experimental deployment. The most useful evidence will connect observed failures to a clearly described setup that another team can understand and, where possible, reproduce.