Build a DynamoDB Cost Agent: On-Demand vs Provisioned Math, Unused GSIs, Standard-IA Table Class, and Idle Tables Without Throttling Production

What This Agent Does

A DynamoDB cost agent pulls 30 days of consumed and provisioned capacity for every table and every global secondary index in the account, computes the break-even utilization between on-demand and provisioned billing from the real price list, finds the indexes nothing has read in 90 days, finds the tables where storage is the dominant line and Standard-IA would halve it, and finds the tables that have served zero requests all quarter. The math is deterministic boto3 and CloudWatch. The LLM does one job: it reads the table's Terraform block plus a grep of the codebases that reference the table and index names, and produces a PR whose diff touches the right module and whose description names the specific risk a reviewer should check. The agent never calls delete-table, never deletes a backup, and never flips billing mode on a table that has throttled in the last 30 days.

The reason this needs an agent rather than a dashboard is that DynamoDB's pricing has two cliffs and one trap. The cliffs are the on-demand versus provisioned crossover and the Standard versus Standard-IA crossover, and both depend on numbers a dashboard does not show side by side. The trap is that a global secondary index charges for every write to the base table whether or not anyone ever queries it, so an unused GSI on a write-heavy table can cost more than the table itself.

Where DynamoDB Spend Actually Hides

Prices in this post are us-east-1 list prices. Since the on-demand price cut in late 2024, on-demand reads cost $0.125 per million read request units and writes $0.625 per million write request units. Provisioned capacity is $0.00013 per RCU-hour and $0.00065 per WCU-hour. Storage is $0.25 per GB-month for Standard and $0.10 for Standard-IA, which charges about 25% more per unit of throughput in exchange. Put the prices in a config file, not in code, because the agent will outlive the next price change.

Step 1: The Break-Even Math Everyone Gets Wrong

One provisioned WCU delivers up to 3,600 writes per hour and costs $0.00065. Those same 3,600 writes on on-demand cost 3,600 times $0.625 per million, which is $0.00225. On-demand is therefore about 3.5 times the price of a fully used provisioned unit, and the ratio is identical for reads. Invert it and you get the only number that matters:

break-even utilization = 0.00065 / 0.00225 = 0.289

If a table would run its provisioned capacity above roughly 29% average utilization, provisioned is cheaper. Below that, on-demand is cheaper, and the gap widens fast: a table at 5% utilization pays six times more on provisioned than it would on on-demand. Before the 2024 price cut the break-even was about 14%, which is why a lot of "always use provisioned" advice is now wrong.

The catch is that "average utilization" is not what a provisioned table achieves. Autoscaling targets 70% utilization by default and scales on a lagging signal, so the realistic ceiling for a bursty table is 40 to 55%, while a flat table can sit at 65%. The agent computes the expected provisioned cost from the hourly consumption series rather than from the monthly mean:

# breakeven.py — per-table cost under both billing modes from 30d of hourly data
OD_READ, OD_WRITE = 0.125 / 1e6, 0.625 / 1e6        # per request unit
PROV_RCU_H, PROV_WCU_H = 0.00013, 0.00065           # per unit-hour
TARGET_UTIL = 0.70                                  # autoscaling target
AUTOSCALE_LAG_FACTOR = 1.25                         # observed overshoot on scale-up

def on_demand_cost(hourly_reads, hourly_writes):
    return sum(hourly_reads) * OD_READ + sum(hourly_writes) * OD_WRITE

def provisioned_cost(hourly_reads, hourly_writes, min_rcu=1, min_wcu=1):
    cost = 0.0
    for r, w in zip(hourly_reads, hourly_writes):
        # units needed this hour if autoscaling held exactly the target
        rcu = max(min_rcu, (r / 3600) / TARGET_UTIL * AUTOSCALE_LAG_FACTOR)
        wcu = max(min_wcu, (w / 3600) / TARGET_UTIL * AUTOSCALE_LAG_FACTOR)
        cost += rcu * PROV_RCU_H + wcu * PROV_WCU_H
    return cost

def recommend(hourly_reads, hourly_writes, throttled_last_30d):
    od, prov = on_demand_cost(hourly_reads, hourly_writes), provisioned_cost(hourly_reads, hourly_writes)
    peak_to_mean = max(hourly_writes + hourly_reads) / (sum(hourly_writes + hourly_reads) / len(hourly_writes) + 1e-9)
    if prov < od * 0.7 and peak_to_mean < 4 and not throttled_last_30d:
        return "switch_to_provisioned", od, prov
    if od < prov * 0.7:
        return "switch_to_on_demand", od, prov
    return "keep", od, prov

Two guards in that function are deliberate. The 30% margin means the agent never proposes a billing-mode change that saves less than that, because every switch costs a reviewer's attention and the mode can only be changed once every 24 hours. The peak-to-mean ceiling of four means a table that does nothing all day and then absorbs a batch job at midnight stays on on-demand even if the arithmetic says provisioned is cheaper, because autoscaling will throttle the first minutes of that batch every night and the on-call engineer will rightly blame the agent.

Step 2: Collect Capacity, Indexes, Class, TTL, and Backups

The IAM role needs dynamodb:ListTables, dynamodb:DescribeTable, dynamodb:DescribeTimeToLive, dynamodb:DescribeContinuousBackups, dynamodb:ListBackups, dynamodb:ListTagsOfResource, cloudwatch:GetMetricData, application-autoscaling:DescribeScalableTargets, and ce:GetCostAndUsage. The same IAM shape as the RDS cost agent: read everything, write nothing, with an explicit Deny on dynamodb:DeleteTable, dynamodb:DeleteBackup, and dynamodb:UpdateTable so that a future "small improvement" cannot quietly make the agent a writer.

# collect_dynamodb.py — read-only inventory, one dict per table
import boto3
from datetime import datetime, timedelta, timezone

ddb, cw = boto3.client("dynamodb"), boto3.client("cloudwatch")
NOW = datetime.now(timezone.utc)

def hourly(table, metric, index=None, days=30, stat="Sum"):
    dims = [{"Name": "TableName", "Value": table}]
    if index:
        dims.append({"Name": "GlobalSecondaryIndexName", "Value": index})
    r = cw.get_metric_data(
        MetricDataQueries=[{"Id": "m", "MetricStat": {
            "Metric": {"Namespace": "AWS/DynamoDB", "MetricName": metric, "Dimensions": dims},
            "Period": 3600, "Stat": stat}}],
        StartTime=NOW - timedelta(days=days), EndTime=NOW)
    return r["MetricDataResults"][0]["Values"]   # empty list = no traffic at all

def table_facts(name):
    t = ddb.describe_table(TableName=name)["Table"]
    facts = {
        "name": name,
        "billing": t.get("BillingModeSummary", {}).get("BillingMode", "PROVISIONED"),
        "table_class": t.get("TableClassSummary", {}).get("TableClass", "STANDARD"),
        "size_gb": round(t["TableSizeBytes"] / 1e9, 2),
        "items": t["ItemCount"],
        "prov_rcu": t["ProvisionedThroughput"]["ReadCapacityUnits"],
        "prov_wcu": t["ProvisionedThroughput"]["WriteCapacityUnits"],
        "reads_h": hourly(name, "ConsumedReadCapacityUnits"),
        "writes_h": hourly(name, "ConsumedWriteCapacityUnits"),
        "throttled": sum(hourly(name, "ThrottledRequests")) > 0,
        "ttl": ddb.describe_time_to_live(TableName=name)["TimeToLiveDescription"]["TimeToLiveStatus"],
        "pitr": ddb.describe_continuous_backups(TableName=name)
                   ["ContinuousBackupsDescription"]["PointInTimeRecoveryDescription"]["PointInTimeRecoveryStatus"],
        "backups": ddb.list_backups(TableName=name).get("BackupSummaries", []),
        "gsis": [],
    }
    for g in t.get("GlobalSecondaryIndexes", []):
        facts["gsis"].append({
            "name": g["IndexName"],
            "size_gb": round(g["IndexSizeBytes"] / 1e9, 2),
            "reads_90d": sum(hourly(name, "ConsumedReadCapacityUnits", g["IndexName"], days=90)),
            "writes_30d": sum(hourly(name, "ConsumedWriteCapacityUnits", g["IndexName"])),
            "prov_rcu": g.get("ProvisionedThroughput", {}).get("ReadCapacityUnits", 0),
            "prov_wcu": g.get("ProvisionedThroughput", {}).get("WriteCapacityUnits", 0),
        })
    return facts

Three things the collector has to get right, because each one produces a wrong recommendation if ignored:

• ConsumedReadCapacityUnits is a Sum over the period, not a rate. Divide by 3,600 to get the per-second number you compare against ProvisionedReadCapacityUnits. Mixing the two is how people conclude a table is at 3,600% utilization.
• TableSizeBytes and ItemCount are updated roughly every six hours. Fine for cost work, wrong for anything that needs to know whether a migration finished.
• A GSI has its own capacity on a provisioned table and its own storage on every table. The 90-day read window on the index is the proof of non-use; the 30-day write total is what you are paying to keep it alive.

Autoscaling minimums come from application-autoscaling describe-scalable-targets --service-namespace dynamodb. The MinCapacity on each table/<name> and table/<name>/index/<gsi> resource is the single most common waste in provisioned accounts: a minimum of 100 WCU on a table that consumes 2 is $47 a month per table, forever, and it hides inside a perfectly green utilization graph.

Step 3: Classify With a Fixed Vocabulary

Classification is a lookup over the facts, in priority order, and each label maps to exactly one action:

def classify(f, cost_explorer_lines):
    reads, writes = sum(f["reads_h"]), sum(f["writes_h"])
    if reads + writes == 0 and f["items"] > 0:
        return "idle_table"                      # 30d zero traffic, has data
    storage_share = cost_explorer_lines["storage"] / max(cost_explorer_lines["total"], 1e-9)
    if f["table_class"] == "STANDARD" and storage_share > 0.5 and f["size_gb"] > 10:
        return "standard_ia_candidate"
    action, od, prov = recommend(f["reads_h"], f["writes_h"], f["throttled"])
    if action != "keep":
        return action
    return "healthy"

def classify_gsi(g, code_refs):
    if g["reads_90d"] == 0 and g["writes_30d"] > 0 and not code_refs:
        return "unused_gsi"
    return "healthy"

The storage share comes from Cost Explorer, not from TableSizeBytes times a price. Grouping by USAGE_TYPE with a filter on the DynamoDB service returns lines that end in TimedStorage-ByteHrs, ReadCapacityUnit-Hrs, WriteCapacityUnit-Hrs, ReadRequestUnits, WriteRequestUnits, TimedPITRStorage-ByteHrs, and TimedBackupStorage-ByteHrs, each with a region prefix. Cost Explorer does not group by table, so the agent allocates the account-level storage line across tables by TableSizeBytes and labels the result as an estimate. The cost anomaly agent uses the same usage-type grouping, so the two agents share one Cost Explorer client and one cache.

Standard-IA is worth a sentence of honesty. It cuts storage from $0.25 to $0.10 per GB-month and raises throughput prices by roughly 25%, so it only wins when storage is more than about half the table's bill. The agent's threshold of 50% plus a 10 GB floor reflects that. A table class change is limited to two per 30 days, which is another reason the recommendation should be right the first time.

The code_refs argument is the part that keeps unused-GSI deletion safe. CloudWatch proves nobody has queried the index in 90 days; it cannot prove nobody will next quarter-end. A grep across the services' repositories for the index name, plus the IndexName= string in any Lambda or Step Functions definition pulled from the account, is the second signal. If either signal is non-zero, the index stays.

Step 4: Let the Model Write the PR, Then Validate It Like Hostile Input

Every finding carries a label, the numbers behind it, the Terraform resource that owns the table, and the grep results. The model turns that into a diff and a review note:

You are proposing a DynamoDB cost change. You will be given one finding with
its label, the 30-day cost under each billing mode, the autoscaling targets,
the table's Terraform block, and grep results for the table and index names.

Produce JSON only: {"files": [{"path": ..., "patch": ...}],
"pr_title": ..., "pr_body": ..., "risk": ..., "rollback": ...}

Rules:
- Edit only the Terraform resource that owns this table. Do not touch other
  resources in the same file.
- For switch_to_on_demand: set billing_mode = "PAY_PER_REQUEST", remove
  read_capacity and write_capacity, and remove the matching
  aws_appautoscaling_target and aws_appautoscaling_policy resources.
- For switch_to_provisioned: set billing_mode = "PROVISIONED" with
  read_capacity and write_capacity equal to the p95 hourly consumption
  divided by 3600, and add autoscaling targets with max at 4x that value.
- For unused_gsi: delete only the global_secondary_index block whose name
  matches, and state in pr_body that this cannot be undone without a
  backfill that costs one write unit per item.
- For standard_ia_candidate: set table_class = "STANDARD_INFREQUENT_ACCESS".
- pr_body must quote the dollar figures you were given. Do not compute new ones.

The validator does not trust the output. It runs terraform plan on the patched module and checks the plan against the label: a switch_to_on_demand plan must show exactly one update in-place on the table and only destroy operations on autoscaling resources; an unused_gsi plan must show one GSI removal and nothing else; any plan containing replace or delete on an aws_dynamodb_table fails closed. The model's pr_body dollar figures are compared to the finding's numbers by string match. This is the same discipline the Terraform plan review agent applies to human PRs, pointed at the agent's own work.

Routing follows the blast radius, the way the approval gates post lays out:

• Autoscaling minimum reductions and Standard-IA switches are reversible within the day and auto-merge after plan passes, with a Slack note.
• Billing-mode switches require one approval. The PR body carries the 24-hour cooldown warning and the exact aws dynamodb update-table --billing-mode rollback command for the on-call engineer.
• GSI deletions require an approval from the owning team's CODEOWNERS entry and sit for seven days before merge, during which a CloudWatch alarm on the index's ConsumedReadCapacityUnits at any value above zero closes the PR automatically.
• Idle tables never get a deletion PR. The agent moves them to on-demand so they cost storage only, enables an on-demand backup through AWS Backup, and opens a ticket with the owner tag. Deletion is a human decision, informed by the waste reclamation agent's 30-day quarantine pattern.

What It Found, and What It Got Wrong

On an account with 212 tables across 14 services, the first run produced the following, with the dollar figures taken from Cost Explorer after the changes merged rather than from the agent's estimates:

The two mistakes are more useful than the savings. First, the provisioned recommendation for one of the three high-volume tables was correct on average and wrong on the first of every month, when a billing job quadrupled writes for 40 minutes. The peak-to-mean guard did not catch it because the spike sat inside a single hour and was diluted by the hourly period. The fix was to run the spike check on one-minute data for the top 20 tables by cost, which is cheap because it is only 20 tables. Second, one "unused" GSI was queried only by a quarterly compliance export whose code lived in a repository the grep did not cover. The seven-day alarm window would not have saved it either, because the quarter-end was five weeks out. It is now in the allowlist, and the grep covers every repository with a dynamodb dependency rather than the ones someone remembered to list.

Limits Worth Stating in the PR Template

• Reserved capacity is out of scope. Provisioned tables with steady load can buy reserved capacity for a further discount, but Savings Plans do not cover DynamoDB, and the commitment math belongs with the Savings Plans coverage agent's sizing logic, not here.
• Global tables double every write. A replicated table pays replicated write units in every region. The agent reports them but does not propose removing a replica, because that is an availability decision.
• DynamoDB Streams and Kinesis consumers are invisible to the capacity metrics. A table that nobody reads directly may feed a stream that a Lambda depends on. The idle check includes describe-table's LatestStreamArn and refuses to label a table idle while a stream has an active event source mapping.
• Cost Explorer lags by about a day and does not split by table. Every per-table storage figure is an allocation, and the PR body says so.

Run it weekly, keep the IAM deny on every write path, and let the model draft while the plan output decides. DynamoDB's bill is one of the few in AWS that a read-only agent can cut by a third without anyone noticing a change in latency, provided the arithmetic uses the current price list and the guards stay in place.