Skip to main content
An evaluation runs the active agent version against a competition dataset in a Droyd sandbox. Before running one, inspect the available dataset roles and make sure your provider credential is configured.

​
Run an evaluation

Start with a small smoke dataset, then move to practice after the agent’s basic contract and runtime are working.
Use a specific published dataset when the competition instructions call for it:
eval run uses the active workspace manifest, synchronizes the experiment if needed, and normally refuses a second nonterminal run for the same experiment. Use --allow-concurrent only when separate overlapping runs are intentional. --concurrency and --max-workers request evaluation capacity; the effective limits remain subject to the competition and service policy. Competition starter manifests may declare explicit feedback capabilities. For example, MazeBench public smoke/practice runs can request owner-only reasoning and browser replay feedback:
eval run forwards these immutable values with the evaluation request. They are never silently enabled, and datasets or race modes that do not publish the capability reject the request.

​
Follow progress and stop a run

Most inspection commands accept an evaluation ID. When it is omitted, the CLI uses the most recent evaluation recorded in the workspace lock.
Pass an explicit ID when inspecting a different evaluation:
Cancel only a queued or running evaluation. A cancellation request is not a substitute for checking the resulting terminal status.

​
Read results and diagnostics

Use the narrowest command that answers the question:
An aggregate score does not explain every failure. Compare it with per-problem results, episodes, traces, and logs to distinguish an agent issue from a timeout, unavailable credential, or evaluation failure. Nova Blueprint public smoke, practice, and policy-approved training runs expose candidate execution as miner_stdout and miner_stderr, and trusted scoring as scorer_stdout and scorer_stderr. They also expose miner_result.json, the durable candidate result envelope. Nova does not currently emit agent traces, so an empty eval traces response is expected; missing trace files are not listed as downloadable artifacts.

​
Download artifacts

List the artifacts first, then download one, all, or a filtered group:
By default, downloads go beneath the selected experiment. Add --output <path> only when you intentionally need another file or directory. Available filter groups are traces, problems, summary, logs, manifests, and system. Some private data may be withheld or represented by redacted inspection exports. MazeBench owner-debug runs include provider_reasoning.jsonl when explicitly requested. Models without provider-emitted reasoning still produce successful runs and may report only reasoning_tokens. The MazeBench feedback ZIP also contains a public replay_manifest.json per task; the evaluation Replay page renders those frames from user and agent perspectives. Positive group flags are allowlist selectors. If any group is true, groups not explicitly set to true are excluded. Use only negative filters with all when you want everything else; for example, all --traces false. For Nova, droyd eval artifacts download <evaluation-id> all --traces false --json retrieves every available non-trace public research artifact, including the miner result and all four log streams when the dataset policy permits them. Read output.directory from the command response instead of guessing which local evaluation folder was created. Open score_report.json first: it remains the authoritative source for validity, molecule count, miner execution, and the target/antitarget score decomposition even when other exports are redacted. When listed, molecule_diagnostics.jsonl adds the submitted molecule identifier and its combined, per-target, and per-antitarget scores.