Sum DynamoDB ConsumedCapacity across paginated pages (CLI-4199)

[GarrettBeatty]

Issue

aws dynamodb scan/query with auto-pagination (the default) reports only the final page's ConsumedCapacity instead of the sum across all pages, undercounting consumed capacity. (CLI-4199 / D103095602)

Concrete example

A scan that spans two pages against a table with a global secondary index, e.g. page 1 consumes 448.5 CU and page 2 consumes 63.5 CU (512.0 total):

aws dynamodb scan --table-name Orders --index-name byCustomer \
    --return-consumed-capacity INDEXES

Before — the aggregated result carries only the last page's numbers:

{
    "Items": [ /* all items from both pages */ ],
    "Count": 1000,
    "ScannedCount": 1000,
    "ConsumedCapacity": {
        "TableName": "Orders",
        "CapacityUnits": 63.5,                          // last page only
        "Table": { "CapacityUnits": 0.0 },
        "GlobalSecondaryIndexes": {
            "byCustomer": { "CapacityUnits": 63.5 }     // last page only
        }
    }
}

After — every numeric leaf is summed across pages, each index keeps its own counter, and TableName comes from the first page:

{
    "Items": [ /* all items from both pages */ ],
    "Count": 1000,
    "ScannedCount": 1000,
    "ConsumedCapacity": {
        "TableName": "Orders",
        "CapacityUnits": 512.0,                         // 448.5 + 63.5
        "Table": { "CapacityUnits": 0.0 },
        "GlobalSecondaryIndexes": {
            "byCustomer": { "CapacityUnits": 512.0 }    // summed per index
        }
    }
}

With --no-paginate (single page) output is unchanged. When --return-consumed-capacity is not requested, ConsumedCapacity is absent from both the responses and the result (no phantom {"TableName": null}).

Approach

The paginator config keys understand wildcard (*) path segments:

• result_key — a path may contain * segments. Each concrete leaf matched on every page is summed (numeric leaves only), and each distinct matched map-key keeps its own running counter. This is what reaches the runtime-keyed per-index maps (GlobalSecondaryIndexes, LocalSecondaryIndexes, VectorIndexes) that plain dotted paths can't express.
• non_aggregate_keys — a wildcard/dotted path takes its value from the first page (e.g. ConsumedCapacity.TableName).

A * is only honored when the service model confirms the *'s parent member is a map shape (and, for result keys, the leaf resolves to a numeric shape). Otherwise the wildcard path is dropped (it is never compiled as a plain jmespath key, where * would become a list projection that gets silently concatenated across pages) — so a service can't accidentally opt into wildcard aggregation just by shipping a * in its model. A @validates_models allowlist in the paginator-config test enforces which (service, operation) pairs may use wildcard result keys.

DynamoDB config (paginators-1.sdk-extras.json overlay)

For Scan and Query, result_key adds the top-level ConsumedCapacity.{CapacityUnits,ReadCapacityUnits,WriteCapacityUnits}, ConsumedCapacity.Table.*, and the per-index maps ConsumedCapacity.{GlobalSecondaryIndexes,LocalSecondaryIndexes,VectorIndexes}.*.*; non_aggregate_keys adds ConsumedCapacity.TableName.

Engine changes (awscli/botocore/paginate.py)

• _iter_wildcard_leaves(data, segments) — yields (concrete_path, value) for each leaf present in a page matching the segment pattern (* matches every dict key at that level).
• _walk_to_leaf_container / _is_summable_number — helpers to accumulate into the result and to guard against booleans / non-numerics.
• build_full_result sums wildcard numeric leaves per page; _record_non_aggregate_key_values records wildcard leaves from the first page.
• A nested non_aggregate leaf that is None on the first page is skipped only when its parent object is itself absent from the response (recording it would fabricate a phantom parent like {"TableName": null} implying the member is present). If the parent is present, a genuinely-null leaf is still surfaced as null — so services with nullable nested non_aggregate keys (e.g. kinesis DescribeStream's StreamDescription.KeyId, athena GetQueryResults) keep their exact prior output.
• Paginator._is_map_wildcard_path performs the model-shape gate; result/non-aggregate keys are split into plain (compiled jmespath, unchanged) vs. validated-wildcard lists. An all-wildcard result_key (no plain primary key) raises a clear PaginationError instead of a later IndexError.

Tests

• Unit (tests/unit/botocore/test_paginate.py): wildcard leaf iteration, per-key summation across pages, the map-shape gate + fallback (built against a real ServiceModel), nested null-leaf preserved when parent present, phantom parent skipped when parent absent, gate-failing wildcard dropped, and all-wildcard result_key raises.
• Functional (tests/functional/dynamodb/test_pagination.py): scan/query total sum, per-GSI wildcard sum with distinct counters, VectorIndexes sum, and no ConsumedCapacity (not even a phantom {"TableName": null}) when capacity isn't requested.
• Paginator-config validation: first result_key may not be a wildcard; wildcard numeric map leaves require the allowlist.
• Verified end-to-end against live DynamoDB (on-demand table + GSI, multi-page scan).

[aemous]

Some minor changes requested. Otherwise looks good overall.

[aemous]

If the AttributeError catch is specifically for certain lines like shape = shape.members.get(segment), I prefer we wrap those lines only or check for None if they don't exist. I don't want to suppress a different error that should fail loudly.

[GarrettBeatty]

6a8e9c8?diff=split

[aemous]

test_missing_on_a_page_treated_as_zero tests the case where a specific index key (i1) is absent on page 2 but the parent Idx map is still present (with i2). There's no test for the case where the entire CC or CC.Idx parent dict is absent on a page (e.g., a page where ConsumedCapacity is missing entirely but other pages have it). The code handles this correctly (_iter_wildcard_leaves returns nothing when data isn't a dict or the segment is missing), but a test would document this edge case.

[GarrettBeatty]

6a8e9c8?diff=split