feat(bundle): leave a step's user-scoped fields out of an import, and name what a task needs beyond core

[earakely-scale]

Summary

A stored task carries values that only work in the setup it was stored from:

• the sandbox provider its run used, where a name like beta_scale exists only where a plugin registers it;
• the agent and models it ran with, including a custom agent another user doesn't have;
• endpoints, output locations and inputs held in the user's object store: a LiteLLM URL, a private trajectory bucket, a build context or a skill;
• billing tags (metadata.attribution).

agent-env bundle init copied all of them. A folder written for another machine could then fail on a provider plugin or an agent that has nothing to do with the task.

The principle: each field of a step describes either the task itself or the setup of the user who ran it. An import keeps the first kind and leaves out the second; a run elsewhere fills those in from its own setup.

• Field scopes. FieldScope (agent_env/entity/scopes.py) has two values. TASK is the default for any field not declared otherwise; USER marks setup fields. Each step type declares field_scopes, a path → scope mapping that merges its parent's. Paths are dotted keys, with [] for each element of a list. These core fields are user-scoped:

  • every step: metadata.attribution;
  • deploy_env: sandbox_type, env_state_type, env_state_instance_id;
  • deploy_agent: a2a_agent_id, a2a_agent_version, env_vars, sandbox_type, litellm_base_url;
  • install_agent: a2a_agent_id, a2a_agent_version;
  • deploy_sandbox: sandbox_type;
  • deploy_human_agent: a2a_url;
  • prompt_agent: model, model_params, effort, max_thinking_tokens, harness, user_model, trajectory_output_prefix;
  • rubrics_verifier: judge_a2a_agent_id, default_model, default_model_api_base, effort, max_thinking_tokens, judge_sandbox_type;
  • load_artifact: urls;
  • run_docker_container: docker_context_url;
  • add_skills: each skill's object_url (skills[].object_url);
  • snapshot_env: env_instance_id.

  Each one is optional. When it's left out, the run uses its own setup: the --sandbox flag or config default, the configured default agent and models, its own object store, and its own endpoints. install_agent with no agent now installs the configured default, as deploy_agent already did.

  Three of them have no default to fall back on: load_artifact.urls, docker_context_url and a skill's object_url can each be a step's only input. A step left without its only input still builds, so the import works. A new TaskStep.config_problems() names what's missing: problems in a step's own fields, found without a store. It's checked wherever preflight() is, including for a step a dry run can't preflight, and the step raises the same at run time. So a receiver's run stops before it starts and names the field to supply.

  These stay task-scoped:

  • deploy_agent's changelog (agent_changelog_object_url and its cutoff): without it the agent starts fresh instead of from the earlier run's state, and nothing left in the task says so;
  • a2a_agent_id on the verify_a2a_* steps: the agent is what the task verifies;
  • verify_a2a_skill_config.skill_bundle_object_url: without it the bundle probe scores as failed, which lowers the grade instead of stopping the run;
  • snapshot_env.gateway_url: with no deployed env, it's how the step finds the gateway;
  • prompt_agent.user_a2a_url: setting it is what tells a human counterpart from a simulator.

• Imports leave user-scoped fields out by default. Any set value is left out, 0 and False included, and listed under "Left out". A step left with no agent is listed under "Floating", as deploying the configured default. --include-scope user keeps the fields as stored, for a folder that runs against the same store.

• "Needs beyond core". The report and README list each step, env and artifact type that core doesn't ship. Each type is credited to the distribution whose entry point registered it, or to "configuration". These types are task-scoped: a run without the plugin fails, and the folder now says so up front.

• An agent_env.entity package for what an entity's fields say:

  • entity/refs.py holds the ref declarations, which were agent_env/entity_refs.py;
  • entity/scopes.py holds the field scopes and prune, moved out of the importer;
  • entity/paths.py holds the path syntax both use.

  agent_env/entity_refs.py stays as a re-export of its six public names, so plugins importing the old path, the sdk among them, keep working; the plugin-api check reports no break. git diff -C shows refs.py as entity_refs.py with 24 lines changed.

This is off the hot path: imports change, and steps gain a class attribute and a store-free check.

Tests

• Contract (tst/unit/task_step/test_field_scopes.py), for every registered step type:
  • each scoped field is an optional parameter, or a path the base scopes for every step;
  • each type keeps the base's scopes;
  • each built-in type that scopes fields of its own builds from a minimal stored step without them;
  • each step left without an input that has no default names it in config_problems().
• Importer:
  • user-scoped fields are left out and reported, including list paths and a field set to zero, and the rest is kept;
  • include_scopes=[FieldScope.USER] keeps them as stored;
  • each need is credited to its plugin or to configuration, including a type a config file replaces and a plugin the registry build skipped;
  • a core-only task needs nothing;
  • the CLI flag works.
• Integration (tst/integration/cli/bundle_init_scopes_test.py): two machines as two local stores. The source's config registers its own sandbox provider and step type; the receiver's registers nothing. A folder imported by default runs on the receiver, while an --include-scope user copy runs only at its source. An agent task takes the receiver's own default agent. A plugin step, and an input left out with no default, are refused on the receiver before anything runs, naming what's missing.
• Unit tier: 8,029 passed.

Checked on two machines

Two fresh Ubuntu VMs:

• A: core plus a stand-in plugin that registers a sandbox provider acme_vm and a step type acme_note;
• B: core only.

A stored three tasks that name A's own setup: its provider, a custom agent, models, a LiteLLM URL, env vars, a trajectory bucket and attribution. A imported each task by default and with --include-scope user, and B ran the folders.

Folder	On A	On B
sandbox task, default	(not run)	runs and passes, twice
sandbox task, --include-scope user	runs and passes	refused: Unknown sandbox backend: 'acme_vm'
agent + judge task, default	(not run)	dry run passes once B registers its own default agent; without one, it names the missing a2a-default
agent + judge task, --include-scope user	dry run passes	refused: no agent acme-solver in the store
plugin-step task, default	runs and passes	refused: 'acme_note' is not a known step type; its README names agentenv-acme

The default folders mention acme in only two places: the plugin step itself, and the README line naming the plugin it needs.

Real CoopBench tasks. I imported five CoopBench tasks (t01 to t05) from the prod store, each by default and with --include-scope user. All ten imports succeeded.

• The default folders hold no setup values. There's no beta_scale or modal provider, claude-code-cli agent, model, attribution, or s3:// trajectory prefix; the agents float to the configured default. The --include-scope user copies keep all of these.
• Core-only dry run. On a core-only VM, each default folder is refused only for its task material in the store: env coopbench-star-pm-v2, the universe, and the task's evidence-collector script. Bringing those along is the next step. Each user copy is also refused for claude-code-cli.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues remain.

Summary

Bundle imports now leave out user-specific setup by default and report types that need plugins or configuration.

• Bundle imports now leave machine-specific fields to each machine.
• Bundle reports name the types that need more than core.
• Entity references now live under the entity package.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Stored task or eval] --> B[Leave out user-scoped fields]
  B --> C[Pin retained references]
  C --> D[Report needs beyond core]
  D --> E[Check staged bundle]
  E --> F[Write bundle folder]
  F --> G[Check receiver setup and missing inputs]
  G --> H[Run with receiver defaults]

Loading

_{Reviews (6) · Last reviewed commit: "refactor(entity): keep agent_env.entity_..." · Reviewed by Greptile}