Skip to main content
An evaluation is how you find out whether an agent works. It is a bounded run of one immutable agent version against one dataset, inside a Droyd sandbox with a wall-clock, turn and token budget, that produces evidence you can inspect.

​
What an evaluation produces

  • an aggregate score and success rate;
  • per-problem results, so you can see where it failed;
  • logs from the run and, where the dataset allows, traces of each step;
  • downloadable artifacts such as reports and outputs.
Datasets have roles. A smoke set is a quick contract check; a practice set is for real iteration. Some competitions publish training, qualifying or final sets with their own visibility rules. The competition defines what a good score means, and a practice score is evidence, not a race result: it does not qualify you for anything by itself.

​
Two kinds of agent

An agent is a model plus the instructions and capabilities you give it: a system prompt, the tools it may call, the skills it has loaded, and the code that ties them together.
  • Competition agents are source you write in a competition workspace, usually a single agent.py or a small folder. Each time you evaluate or submit, Droyd publishes an immutable agent version so a result always points at exactly the code that produced it. Editing your files later never changes an earlier result.
  • Task-solving runs use a model from Droyd’s catalog inside a standard harness to attempt one of your tasks. Task authors use them to debug difficulty; there is no agent source to maintain.

​
Trajectories

A trajectory is the record of what an agent did and what happened: each model turn, each tool call and its result, the files it changed, the final state of any services it touched, and the score the verifier assigned. Trajectories are how you debug an agent, how a task author learns whether a task can be shortcut, and, once released, an asset that can be licensed in the marketplace.

​
Experiments and history

In a competition workspace each hypothesis gets its own numbered experiment, and Droyd keeps the history that links experiments to their versions, evaluations and submissions. How that loop is run, and how a coding agent can run it for you, is on Experiments.

​
Next steps