What This Agent Does
A Lambda cost agent reads 30 days of REPORT lines and CloudWatch metrics for every function in the account, computes the cost per invocation under the current memory setting and under the memory Compute Optimizer and Power Tuning recommend, finds provisioned concurrency that sits below the 60% utilization break-even, and finds x86 functions whose dependency manifests contain nothing that would stop an arm64 build. Every number comes from deterministic code. The LLM gets one job: read the function's Terraform or SAM block and its dependency file, write the PR that changes the right line, and name in the description the one thing a reviewer must check. The agent never calls update-function-configuration itself, never deletes a function, and never touches a function that has thrown a single error in the last 7 days.
This is the same shape as the DynamoDB cost agent: a read-only collector, a fixed set of findings with thresholds, and an LLM that drafts changes but cannot apply them. Lambda needs its own agent because the spend hides in a dimension no bill shows. The bill shows GB-seconds. Nobody sees that a 3008 MB function uses 190 MB at peak, or that a 128 MB function is CPU-starved and would cost less at 1024 MB because it would finish four times faster.
Where Lambda Spend Actually Hides
| Bucket | How it happens | What fixes it |
|---|---|---|
| Over-provisioned memory | Someone set 3008 MB during a debugging session in 2024 | Lower to measured peak plus headroom |
| Under-provisioned memory on CPU-bound code | 128 MB default, function takes 4 seconds to parse JSON | Raise memory, duration falls faster than price rises |
| Idle provisioned concurrency | PC of 50 set for a launch; utilization is now 8% | Lower PC, or drop it and accept cold starts |
| x86 where arm64 would do | Default architecture at creation, pure-Python or pure-Node code | Switch to arm64 for 20% lower per-GB-second price |
| Paid init on every cold start | Since August 2025, on-demand init phase is billed; heavy imports at module level | Lazy imports, smaller packages, SnapStart where supported |
| Never-expire log groups | Every function gets one; retention default is forever | Already covered by the CloudWatch Logs cost agent |
Prices in this post are us-east-1 list prices. On-demand x86 duration is $0.0000166667 per GB-second, arm64 is $0.0000133334, and requests are $0.20 per million. Provisioned concurrency is $0.0000041667 per GB-second for the reserved capacity, and invocations that run on it are billed at $0.0000097222 per GB-second instead of the on-demand rate. Keep these in a config file. The agent will outlive the next price change.
Step 1: The Two Break-Even Numbers
Provisioned concurrency only pays off when it is busy. Per GB-second of actual duration, provisioned saves the difference between on-demand and the provisioned duration rate, which is $0.0000069445. The reserved capacity costs $0.0000041667 per GB-second whether or not anything runs. Divide:
PC break-even utilization = 0.0000041667 / 0.0000069445 = 0.60
Below 60% average ProvisionedConcurrencyUtilization, provisioned concurrency costs more than running the same invocations on demand. Most teams buy it for latency, not cost, and that is a legitimate reason. The agent's job is to make the latency price explicit: a function at 8% utilization with PC of 50 at 1024 MB is paying about $112 a month for reserved capacity that serves roughly 4 concurrent requests.
Memory has no closed-form break-even because duration is not linear in memory. CPU allocation scales linearly up to 1769 MB, which is one full vCPU, and a CPU-bound function roughly halves its duration when memory doubles, which means cost per invocation stays flat while latency improves. An I/O-bound function waiting on DynamoDB gets no faster and cost doubles. The only honest way to know which kind you have is to run it at several sizes with real payloads, and that is exactly what the open-source aws-lambda-power-tuning state machine does.
# lambda_cost.py — cost per invocation from the numbers the REPORT line already gives you
PRICE = {"x86_64": 0.0000166667, "arm64": 0.0000133334} # per GB-second, on-demand
REQ_PRICE = 0.20 / 1e6
def cost_per_invocation(memory_mb, billed_ms, arch="x86_64", init_ms=0):
gb_s = (memory_mb / 1024) * ((billed_ms + init_ms) / 1000)
return gb_s * PRICE[arch] + REQ_PRICE
def monthly(memory_mb, billed_ms, invocations, arch="x86_64", init_ms=0, cold_fraction=0.0):
warm = cost_per_invocation(memory_mb, billed_ms, arch) * invocations * (1 - cold_fraction)
cold = cost_per_invocation(memory_mb, billed_ms, arch, init_ms) * invocations * cold_fraction
return warm + cold
The init_ms term matters since the 2025 billing change. A Python function that imports pandas at module level pays for 1.5 seconds of init on every cold start, and a function invoked once an hour is cold on nearly every invocation. The agent flags functions where init exceeds 25% of total billed time.
Step 2: Collect Memory Use, Duration, PC Utilization, and Architecture
The role needs lambda:ListFunctions, lambda:GetFunctionConfiguration, lambda:ListProvisionedConcurrencyConfigs, lambda:ListAliases, logs:StartQuery, logs:GetQueryResults, cloudwatch:GetMetricData, and compute-optimizer:GetLambdaFunctionRecommendations. Attach an explicit Deny on lambda:UpdateFunctionConfiguration, lambda:DeleteFunction, and lambda:PutProvisionedConcurrencyConfig, so a later "quick fix" cannot turn the agent into a writer. The IAM right-sizing agent post covers why the deny is worth the ten extra lines.
The best memory signal is already in your logs. Every invocation writes a REPORT line with @memorySize, @maxMemoryUsed, @billedDuration, and @initDuration, and Logs Insights can aggregate them across all function log groups in one query.
# collect_lambda.py — read-only inventory, one dict per function
import boto3, time
from datetime import datetime, timedelta, timezone
lam, logs, cw, co = (boto3.client(s) for s in ("lambda", "logs", "cloudwatch", "compute-optimizer"))
NOW = datetime.now(timezone.utc)
REPORT_QUERY = """
filter @type = "REPORT"
| stats max(@maxMemoryUsed / 1024 / 1024) as peak_mb,
pct(@billedDuration, 50) as p50_ms,
pct(@billedDuration, 99) as p99_ms,
avg(@initDuration) as init_ms,
count(@initDuration) as cold_starts,
count(*) as invocations
"""
def report_stats(fn_name, days=30):
q = logs.start_query(logGroupName=f"/aws/lambda/{fn_name}",
startTime=int((NOW - timedelta(days=days)).timestamp()),
endTime=int(NOW.timestamp()), queryString=REPORT_QUERY)
while True:
r = logs.get_query_results(queryId=q["queryId"])
if r["status"] in ("Complete", "Failed", "Cancelled"):
break
time.sleep(1)
row = {f["field"]: float(f["value"]) for f in (r["results"][0] if r["results"] else [])}
return row # empty dict = no invocations in the window
def pc_utilization(fn_name, qualifier, days=30):
r = cw.get_metric_data(MetricDataQueries=[{"Id": "u", "MetricStat": {
"Metric": {"Namespace": "AWS/Lambda", "MetricName": "ProvisionedConcurrencyUtilization",
"Dimensions": [{"Name": "FunctionName", "Value": fn_name},
{"Name": "Resource", "Value": f"{fn_name}:{qualifier}"}]},
"Period": 3600, "Stat": "Average"}}],
StartTime=NOW - timedelta(days=days), EndTime=NOW)
vals = r["MetricDataResults"][0]["Values"]
return sum(vals) / len(vals) if vals else 0.0
def function_facts(cfg):
name = cfg["FunctionName"]
facts = {
"name": name, "memory_mb": cfg["MemorySize"], "timeout_s": cfg["Timeout"],
"arch": cfg.get("Architectures", ["x86_64"])[0], "runtime": cfg.get("Runtime", "image"),
"package": cfg["PackageType"], "layers": [l["Arn"] for l in cfg.get("Layers", [])],
"stats": report_stats(name), "pc": [],
}
for p in lam.list_provisioned_concurrency_configs(FunctionName=name).get("ProvisionedConcurrencyConfigs", []):
qual = p["FunctionArn"].rsplit(":", 1)[-1]
facts["pc"].append({"qualifier": qual, "allocated": p["AllocatedProvisionedConcurrentExecutions"],
"utilization": pc_utilization(name, qual)})
return facts
Compute Optimizer is the second opinion. It produces a memory recommendation for any function invoked at least 50 times in the last 14 days, with projected cost per recommendation option, and it costs nothing to call:
aws compute-optimizer get-lambda-function-recommendations \
--query 'lambdaFunctionRecommendations[?finding==`NotOptimized`].[functionArn,currentMemorySize,memorySizeRecommendationOptions[0].memorySize,memorySizeRecommendationOptions[0].projectedUtilizationMetrics[0].value]' \
--output table
When the REPORT peak and Compute Optimizer disagree by more than one memory tier, the agent does not guess. It queues a Power Tuning run.
Step 3: Prove It with Power Tuning Before Proposing a Size
Power Tuning deploys from the Serverless Application Repository as a Step Functions state machine. You give it a function ARN, a list of memory sizes, an invocation count, a real payload, and a strategy, and it publishes a numbered version at each size, invokes it, and reports average cost and duration per size. It runs against versions it creates, so $LATEST and your aliases are never touched.
{
"lambdaARN": "arn:aws:lambda:us-east-1:123456789012:function:order-enricher",
"powerValues": [256, 512, 1024, 1769, 3008],
"num": 50,
"payload": {"orderId": "sample-from-last-week", "dryRun": true},
"parallelInvocation": false,
"strategy": "cost",
"autoOptimize": false
}
Two settings are not optional. dryRun or an equivalent flag in your own payload keeps the test from writing to production stores 250 times. autoOptimize stays false because the agent proposes and a human merges; the state machine must not flip memory on its own. The payload should be a real event captured from the function's last week of traffic, because Power Tuning on a hello-world payload measures nothing but the runtime's floor.
The deterministic classifier then has three inputs per function: measured peak memory, Compute Optimizer's option, and Power Tuning's cost-optimal size when a run exists. The rules are short and the margins are deliberate:
# classify.py — fixed vocabulary, every rule has a margin a reviewer can defend
def findings(f, pt_optimal=None, co_memory=None, errors_7d=0):
out, s = [], f["stats"]
if not s or errors_7d > 0:
return out # silent or erroring: do not touch
peak, mem = s["peak_mb"], f["memory_mb"]
if pt_optimal and abs(pt_optimal - mem) >= 256:
out.append(("resize_memory", pt_optimal, "power_tuning"))
elif co_memory and peak * 1.5 < mem and co_memory < mem:
out.append(("resize_memory", max(co_memory, int(peak * 1.5)), "compute_optimizer"))
for p in f["pc"]:
if p["utilization"] < 0.60:
out.append(("reduce_provisioned_concurrency", p, "cw_utilization"))
if f["arch"] == "x86_64" and f["package"] == "Zip" and not f["layers"]:
out.append(("arm64_candidate", None, "needs_dependency_check"))
if s.get("init_ms", 0) > 0.25 * (s["p50_ms"] + s.get("init_ms", 0)) and s["cold_starts"] > 0.3 * s["invocations"]:
out.append(("init_dominated", s["init_ms"], "report_lines"))
return out
The 1.5x headroom on memory is the number that stops the agent from causing the outage everyone fears. A function whose peak was 190 MB gets a floor of 285 MB, rounded up to the next 64 MB step, not 192 MB. The 256 MB minimum delta on Power Tuning results keeps the agent from proposing a change that saves cents. Functions with any error in 7 days are skipped entirely, because the first question after an incident will be "did the cost agent change anything," and the answer must be no.
Step 4: The LLM Writes the Diff, Not the Decision
The model receives the finding, the measured numbers, and the function's infrastructure block plus its dependency manifest. It returns a structured change, and the harness opens the PR through the same path the GitOps agent pattern uses, so nothing it writes reaches AWS without a human merge.
{
"name": "propose_lambda_change",
"description": "Produce a minimal diff for one Lambda finding against its IaC block. Never change more than the one setting named in the finding.",
"input_schema": {
"type": "object",
"properties": {
"finding": {"type": "string", "enum": ["resize_memory", "reduce_provisioned_concurrency", "arm64_candidate", "init_dominated"]},
"iac_file": {"type": "string"},
"diff": {"type": "string", "description": "unified diff touching only the relevant attribute"},
"reviewer_check": {"type": "string", "description": "one concrete thing a human must verify before merge"},
"blockers": {"type": "array", "items": {"type": "string"}, "description": "reasons NOT to ship, e.g. native dependency found"}
},
"required": ["finding", "iac_file", "diff", "reviewer_check", "blockers"]
}
}
The arm64 finding is where the model earns its tokens. The deterministic code can only say "Zip package, no layers." The model reads requirements.txt or package.json and Dockerfile if present, and looks for the things that break on Graviton: packages with compiled extensions pinned to x86 wheels, Node modules like older sharp or bcrypt builds without arm64 prebuilds, and vendored binaries in the deployment package. The prompt tells it to prefer a blocker over a guess:
You are reviewing one Lambda function for an architecture change from x86_64 to arm64.
You have the dependency manifest and the IaC block. Output a propose_lambda_change call.
Rules:
- If any dependency compiles native code and you cannot confirm an arm64 wheel or
prebuild exists, add it to blockers and emit an empty diff.
- The diff changes only the architectures attribute (or Architectures in SAM).
- reviewer_check must name the canary step: deploy to an alias with 10% weighted
traffic for 24h and compare Errors and p99 Duration before and after.
A merged PR for a memory change is a one-line Terraform diff:
resource "aws_lambda_function" "order_enricher" {
function_name = "order-enricher"
- memory_size = 3008
+ memory_size = 512 # peak 190 MB over 30d, Power Tuning optimal 512 (cost strategy)
architectures = ["arm64"]
}
What It Found, and What It Got Wrong
On an account with 212 functions, the first run produced 61 findings. Memory resizes were 34 of them, and the deterministic savings estimate from the REPORT lines was within 8% of the actual bill change a month later. Four functions were flagged as under-memoried CPU-bound code at 128 MB; Power Tuning moved them to 1024 MB and their cost per invocation dropped by 55% while p99 latency fell from 3.1 seconds to 0.7. That finding is the one nobody looks for by hand, because raising memory to cut cost is counterintuitive until you have seen the curve.
Provisioned concurrency produced the largest single line: three functions with PC of 20 to 50 at utilization between 4% and 11%, left over from launches. Dropping two of them to zero and the third to 5 removed about $300 a month and nobody noticed a latency change, because the traffic that justified the reservation had moved to another service.
The misses were instructive. The agent flagged a function as idle-init-dominated when the real cause was a VPC-attached function paying ENI setup on cold start, which memory and imports cannot fix. It proposed an arm64 move for a Node function whose manifest looked clean, but a layer-less package had a vendored x86 binary in a bin/ directory that the manifest never mentioned. The canary caught it in the first hour. Both cases became blockers in the prompt and a bin/ scan in the collector.
Limits Worth Stating in the PR Template
Lambda is covered by Compute Savings Plans, so a function-level saving reduces usage that a commitment may already be paying for. Run the Savings Plans coverage agent after a batch of Lambda changes, not before, or the coverage math will be stale. Power Tuning invokes the function for real, so it belongs in a staging account or behind a dry-run flag for anything with side effects. REPORT lines only exist for functions that log to CloudWatch; functions shipping logs elsewhere need the metrics path instead. And the memory finding is only as good as the window: a function that runs a monthly report at 2 GB of peak memory will look over-provisioned on any 30-day window that missed the report. The agent reads 30 days for cost and 90 days for peak memory for exactly that reason, and the PR template says so in the first line.