Repository navigation
Conversation
Migration 0004 keeps rules that need order or grouping as
{"type": "check", "check": "unsupported", "legacy": {...}} and says
the engine reports them as INCONCLUSIVE. validate_rules rejected that
shape, so validate_agentic_metadata raised for every such revision: the
agentic evaluation ended in ERROR and the scenario could not be saved.
Skip validation for exactly that shape.
…rules Revisions saved before SimulaMet#65 hold their controls in the sections tools, retrieval, budgets, trajectory, policy, rerank, handoffs, approvals, guardrails and state. validate_agentic_metadata now rejects those sections and migration 0004 converts only rules, so such a revision could neither be evaluated (the agentic evaluation ended in ERROR) nor saved. Move each section, unchanged, into an unsupported rule of the shape 0004 writes; the engine reports it as INCONCLUSIVE.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #67; merging #67 first keeps this diff to the migration and its test.
Builds on #67, whose commit is the first one here.
Before #65, scenario revisions stored agentic controls in the sections
tools,retrieval,budgets,trajectory,policy,rerank,handoffs,approvals,guardrailsandstate.validate_agentic_metadatanow rejects those keys ("Unknown metadata.agentic keys"), and migration 0004 converts onlyrules, so a revision saved before #65 keeps them. Its agentic evaluation ends in ERROR, and saving the scenario raises in_create_revision. All 12 agentic scenarios ininfra/fixtures/agentic_demo_catalog_v3.jsonas it was before #65 are rejected after migration 0004 onmain.Migration
0005_convert_legacy_agentic_controlsmoves each such section, unchanged, into anunsupportedrule of the shape 0004 writes ({"type": "check", "check": "unsupported", "legacy": {"type": <section>, <section>: <value>}}), after any rules already stored. The engine reports it as INCONCLUSIVE. Run through 0004 and 0005, all 12 of those scenarios validate, with every setting kept.The tests in
audits/agentic/tests/test_legacy_control_migration.pyuse scenario D04 from that catalog: it validates after the migration, each section is kept unchanged and evaluates INCONCLUSIVE, stored rules come first, and metadata in the current shape is unchanged.pytest -n 4 -m "not embedded_hatchet"(no API keys in the environment, the local-Ollama live test deselected) gave 1378 passed, 2 skipped onmainand 1384 passed, 2 skipped with both commits;ruff check .is clean.Design question: nothing is converted to a custom check, so these controls report INCONCLUSIVE until someone rewrites them as rules. If you would rather convert the sections whose meaning is exactly the same as a custom check (for example
tools.forbiddenby name), I can add that, section by section, with a test against the old check for each.AI disclosure: implemented with Claude Code.