Repository navigation
[bot] Merge master/1d27d3fa into rel/dev - #1898
Merged
Merged
Conversation
…keaway LX-3167 evaluates AI Publisher takeaways in gd-eval. Dataset items need a structured input matching the takeaway endpoint body, so the new TakeawayInput model carries it, Langfuse items read it from input or metadata, and an item carrying it resolves to the document_takeaway kind. Unknown keys are rejected so a dataset typo fails loudly. jira: LX-3167 risk: low Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The document_takeaway kind (LX-3167) needs a client for the AI Publisher takeaway endpoint. It maps TakeawayInput to the camelCase body, omitting every unset field because the server DTO forbids extra keys and nulls, and always sends filterContext. The whole response is kept on a new optional ChatResult.raw_response so the evaluator can tell noData apart from an empty takeaway; every other kind leaves it None. A non-2xx answer raises HTTPStatusError with the status and response body in its message, so an item error shows why the endpoint refused (flag off, no agent, failed visualization). jira: LX-3167 risk: low Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The document_takeaway kind (LX-3167) has to prove first that a takeaway is short and punchy: one finding for the page, not an executive summary, whatever maxCharacters allows. Deterministic checks gate word budget, headline length, sentence count, one paragraph, bold headline, no headings, lists, ids or meta openers, and the length limit; an item's format block switches a check off when its instruction asks for another shape. word_count and headline_word_count go into the detail so they can be published as trend scores. Default judge criteria (one finding, no executive summary, chart walk-through, background opener, figures, dates, causes outside the data, chart description) are added unless the item opts out. Their wording is calibrated against the judge with make calibrate-takeaway on fixed good and bad texts; the reference good example used to fail the cause criterion and an obvious executive summary used to pass. noData decides the item alone, so an expected takeaway that came back empty is a failure rather than an empty text to judge. The per-criterion grading loop moves out of the summary evaluator into a shared helper both evaluators use. jira: LX-3167 risk: low Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LX-3167 runs document_takeaway in the scheduled LLM eval CI through a Tavern shim, and by hand through the CLI. The CLI's routing backend becomes a test_kind -> client map, so document_takeaway items go to the takeaway endpoint next to dashboard_summary, and every client is closed. Items pulled with --langfuse-dataset carry takeaway_input in metadata with a plain label as input, which the loader already reads. evaluate_document_takeaway runs an item K times, applies the gate and writes the Langfuse experiment item through the same run context, gate scores and metadata as the agentic kinds, so the nightly report and PR comment read it unchanged. Every check is scored per run, with word_count and headline_word_count as numeric trend scores. HTTP errors and a missing takeaway_input surface as item errors, never as judged failures. The README documents the kind and its item shape. jira: LX-3167 risk: low Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
feat(gooddata-eval): evaluate Publisher document takeaways
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1898 +/- ##
===========================================
+ Coverage 85.10% 85.31% +0.20%
===========================================
Files 337 342 +5
Lines 24343 24670 +327
===========================================
+ Hits 20716 21046 +330
+ Misses 3627 3624 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 1d27d3f (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/39927258963).