> ## Documentation Index
> Fetch the complete documentation index at: https://docs.droyd.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Evaluation

> Enqueue a hosted benchmark evaluation.

Use this route to run an agent version against a competition dataset.

Auth: Droyd API key or signed-in user bearer token. Alias:
`POST /api/v1/evaluations`. The request requires `experiment_id`,
`agent_version_id`, and `competition_slug`; `dataset_slug` and `dataset_role`
are optional. When both are omitted, Droyd uses the competition's default
practice dataset. `workflow_node_id` marks that node `in_progress` when the
evaluation is created. Reuse `client_key` only for the same request; an
incompatible replay returns `409 client_key_conflict`.

For a public MazeBench smoke or practice run, request diagnostic feedback
explicitly with `feedback_capabilities.provider_reasoning: owner_debug` and/or
`feedback_capabilities.browser_replay: owner_debug`. The selected dataset must
publish the matching artifact contract. Droyd rejects these capabilities for
hidden and race evaluations; it never silently enables them.

`dataset_role` is one of `smoke`, `practice`, `training`, `qualifying`, or
`evaluation`. API-key evaluations can target runnable `smoke`, `practice`, and
`training` datasets only; qualifying and evaluation datasets return `403`,
including when specified by slug. `budget.concurrency` and
`budget.max_workers` are optional positive integers and must remain within the
competition limit. OpenRouter is selected by default. Supplying
`x-openrouter-api-key` or `openrouter-api-key` creates a short-lived,
evaluation-scoped credential; absent a one-off key, the account needs an active
OpenRouter credential. Missing credentials return `409 provider_credential_required`.


## OpenAPI

````yaml api-key.openapi.json POST /v1/evaluations
openapi: 3.1.0
info:
  title: Droyd API-key Automation API
  version: 1.0.0
  description: >-
    The production API contract for Droyd API-key automations. It contains only
    operations that accept a Droyd API key. `/v1` is canonical; `/api/v1` is a
    compatibility alias.
servers: []
security:
  - droydApiKeyBearer: []
  - droydApiKeyHeader: []
tags:
  - name: Identity
    description: Account-scoped automation identity and onboarding state.
  - name: Workflows
    description: Caller-owned workflows and research nodes.
  - name: Competitions
    description: Competition readiness, submissions, and race entries.
  - name: Experiments
    description: Experiments, artifacts, agent versions, and evaluations.
  - name: Usage
    description: Caller-owned evaluation usage.
  - name: Billing
    description: Caller-owned prepaid credit balances and funding.
paths:
  /v1/evaluations:
    post:
      tags:
        - Experiments
      summary: Queue an evaluation for an immutable agent version.
      operationId: createEvaluation
      requestBody:
        $ref: '#/components/requestBodies/EvaluationCreate'
      responses:
        '201':
          $ref: '#/components/responses/Success'
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '403':
          $ref: '#/components/responses/Forbidden'
        '404':
          $ref: '#/components/responses/NotFound'
        '409':
          $ref: '#/components/responses/Conflict'
        '500':
          $ref: '#/components/responses/ServerError'
components:
  requestBodies:
    EvaluationCreate:
      required: true
      content:
        application/json:
          schema:
            type: object
            required:
              - experiment_id
              - agent_version_id
              - competition_slug
            properties:
              experiment_id:
                type: string
                format: uuid
              agent_version_id:
                type: string
                format: uuid
              competition_slug:
                type: string
              dataset_slug:
                type: string
              dataset_role:
                type: string
                enum:
                  - smoke
                  - practice
                  - training
                  - qualifying
                  - evaluation
              workflow_node_id:
                type: string
                format: uuid
              client_key:
                type: string
              budget:
                type: object
                additionalProperties: true
              feedback_capabilities:
                type: object
                additionalProperties: false
                properties:
                  provider_reasoning:
                    type: string
                    enum:
                      - owner_debug
                  browser_replay:
                    type: string
                    enum:
                      - owner_debug
              runtime_options:
                type: object
                additionalProperties: true
              metadata:
                type: object
                additionalProperties: true
  responses:
    Success:
      description: Successful response.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/SuccessEnvelope'
    BadRequest:
      description: Invalid request.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
    Unauthorized:
      description: Missing or invalid API key.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
    Forbidden:
      description: Account state or route policy prevents the request.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
    NotFound:
      description: The resource is missing or not owned by the API-key account.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
    Conflict:
      description: The resource state or idempotency key conflicts with the request.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
    ServerError:
      description: >-
        The request could not be completed. Retry only when the operation's
        idempotency contract permits it.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorEnvelope'
  schemas:
    SuccessEnvelope:
      type: object
      required:
        - ok
      properties:
        ok:
          const: true
      additionalProperties: true
    ErrorEnvelope:
      type: object
      required:
        - ok
        - error
      properties:
        ok:
          const: false
        error:
          type: object
          required:
            - code
            - message
          properties:
            code:
              type: string
            message:
              type: string
  securitySchemes:
    droydApiKeyBearer:
      type: http
      scheme: bearer
      bearerFormat: Droyd API key
      description: A Droyd API key in the Authorization Bearer slot.
    droydApiKeyHeader:
      type: apiKey
      in: header
      name: x-droyd-api-key
      description: A Droyd API key when Authorization is absent.

````