Skip to content

Trials' Session traces in the Experiment's Trace, and each Arm's tokens and cost #1023

Closed

Description

@JacobStephens2

Parent

#977

What to build

Once Session traces exist (#985), each Trial's and Grader's session joins its Experiment's Trace. The trace ID comes from the Experiment's Command, with the Trials and Graders as spans beneath it. Each Arm's tokens by Model, and its cost where the Harness reports one, are summed from its Trials' Session traces and shown beside its minutes.

Acceptance criteria

  • An Experiment's Trace holds its Trials and Graders as spans beneath it.
  • thirdshift experiments <id> --json shows each Arm's tokens by Model, and its cost where reported.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions