Repository navigation
Parse optimization backtest statistics sent as an object - #683
Conversation
Optimization backtests now serialize their statistics as an object keyed
by the statistic's index ("0" to "24") or, for any other statistic, by
its name. QCOptimizationBacktest.statistics expected an array, so
`lean cloud optimize` failed with a validation error once the backtests
had statistics. The model now holds a dict and converts the older array
shape into index keys, so both parse.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
The fix is correct and well scoped. One thing to change in the description before this can go in.
Requested change
- Please remove the
#### Risksection from the description. A description says what changed, how it was tested and the proof. Judging the risk is the reviewer's job. The facts it lists (no CLI options or output change, the array shape still parses) can stay, worded as facts in the Description section.
What I checked
- The model and lookups match Lean's
OptimizationBacktestJsonConverter.WriteJsonkeys each of Lean's statistics by itsStatisticsIndicesposition (Sharpe Ratio 15, Compounding Annual Return 6, Probabilistic Sharpe Ratio 13, Drawdown 7) and any other statistic by its name. So"15","6","13"and"7"are the right keys. Thebeforevalidator maps the old array onto the same indices, the wayReadJsondoes. - A side benefit: the old array writer silently dropped any value it couldn't parse, which shifted every later index. With keys, a dropped statistic can no longer make the CLI read the wrong one. A missing key still raises a
KeyError, which is the same failure class as the oldIndexError(#570), not a regression. QCOptimizationBacktest.statisticshas no other readers in the CLI.lean optimize(local) reads the backtest result JSON, not this model.- Tests:
tests/commands/cloud/test_optimize.pygives 28 passed with the change. Withlean/reverted to master, 11 fail, so the new cases do cover the bug. CI is green on every platform and Python version. - End to end: I ran
lean cloud optimize ... --target "Sharpe Ratio" --target-direction max --parameter weight 0.5 1 0.5 --node O2-8 --parallel-nodes 2against an optimization whose backtests return the object shape. On master it fails with the issue'slist_typeerror. With this branch it printsOptimal parameters: weight: 1.0and the statistics table. I also ran it with--constraint "Drawdown < 50", which goes through_backtest_meets_constraints, and it found the same optimum.
Not blocking, existing behavior worth its own issue
- The statistic values are in percent units: Lean strips the
%sign before writing them, in both the old and the new shapes. So--constraint "Drawdown < 0.5"matched no backtest, while< 50did. The CLI's local constraint check therefore compares a percent value with whatever the user typed. This predates the change, and the test fixture (0.24 vs 0.25) hides it.
Other repositories: none need an update. The Lean side has already shipped.
|
Removed the |
There was a problem hiding this comment.
Approving. The one change I asked for is done: the description no longer has a Risk section, and its facts (no CLI options or output change, the array shape still parses into the same indices) are now in the Description section. There have been no code changes since my last review (head is still 9193d70), so what I checked there still holds:
- The
"15","6","13"and"7"keys match Lean'sOptimizationBacktestJsonConverterstatistic indices. Thebeforevalidator maps the older array shape onto the same keys. QCOptimizationBacktest.statisticshas no other readers in the CLI.- The new test cases fail without the source change (11 failures) and pass with it. CI is green on every platform and Python version.
- Tested end to end:
lean cloud optimizefails with the issue'slist_typeerror on master. With this branch it prints the optimal parameters and statistics, both without a constraint and with--constraint "Drawdown < 50".
Not blocking, existing behavior worth its own issue: the statistic values are in percent units, so the CLI's local constraint check compares a percent value against whatever the user typed (Drawdown < 0.5 matches nothing, < 50 does). This was already the case before this change.
Other repositories: none need an update.
Closes #682
Description
Since QuantConnect/Lean#8984, an optimization backtest's statistics serialize as an object rather than an array. Lean's own statistics are keyed by their former array index (
"0"to"24"), and any other statistic is keyed by its name (for examplePortfolio Turnover).QCOptimizationBacktest.statisticswasList[float], solean cloud optimizefailed with a pydanticlist_typevalidation error once the backtests had statistics, and printed no result.lean/models/api.py:QCOptimizationBacktest.statisticsis nowDict[str, float]. Abeforevalidator turns the older array shape into index keys, so both shapes parse. This mirrors Lean'sOptimizationBacktestJsonConverter, which still reads both.lean/commands/cloud/optimize.py:_get_backtest_statisticreads the keys"15","6","13"and"7"(Sharpe Ratio, Compounding Annual Return, Probabilistic Sharpe Ratio, Drawdown), the same indices as before, matchingStatisticsIndicesin Lean's converter._backtest_meets_constraintsgoes through this function too.QCOptimizationBacktest.statisticsis read nowhere else in the CLI. No CLI options or output change. The array shape still parses into the same indices, so both the old and the new responses resolve each lookup to the same statistic.Tests
test_cloud_optimize_displays_optimal_backtest_resultsnow runs every target/extremum case with the statistics as an array and as an object (index keys plus a named custom statistic).test_optimization_backtest_parses_statistics_as_list_or_objectparses the array, the object, and the object with a custom statistic.tests/commands/cloud/test_optimize.pypasses 28/28.python -m pytest -s -rsgives 2777 passed and 82 skipped.python static_analysis.pygives BUILD PASSED (the same 5 unused-import warnings as master).scripts/readme.pyleaves README unchanged.lean cloud optimize "<project>" --target "Sharpe Ratio" --target-direction max --parameter weight 0.5 1 0.5 --node O2-8 --parallel-nodes 2. The returned optimization's backtest statistics were an object ("0","1", … plusPortfolio Turnover). On master, the model rejects this response with theInput should be a valid listerror from the issue. With this change, the command finished and printedOptimal parameters: weight: 1.0and the optimal backtest's statistics table.🤖 Generated with Claude Code