cloud

Build an AWS Service Quota Agent: Find the Limits You're About to Hit, Count the Quotas CloudWatch Doesn't Track, and File Increases Before a Scale-Up Fails

Build an AWS service quota agent: track quota utilization with CloudWatch, count untracked limits, forecast exhaustion, and request increases safely via PR.

November 19, 2026·12 min read·
#ai#sre#aws#cloud#automation#reliability#terraform

What This Agent Does

An AWS service quota agent walks every account and region you run in, computes utilization for each quota that matters, forecasts when the growing ones run out, and checks whether your own autoscaling ceilings even fit under the limits. For each quota at risk it produces one of four verdicts: clean up first, add an alarm, request an increase, or ask a human. Alarms and increases ship as a Terraform pull request. The only API write it can make, RequestServiceQuotaIncrease, sits behind an approval and a hard cap.

The reason to build it is that quota failures arrive at the worst possible moment. Nobody hits the On-Demand vCPU limit on a quiet Tuesday. You hit it during the traffic spike, the regional failover, or the first full-size load test, when the autoscaler asks for forty nodes and EC2 answers VcpuLimitExceeded. Then the fix is a support request that takes anywhere from a few minutes to a few days, and no amount of on-call skill makes it faster. Quotas are the one capacity problem you can only solve in advance.

Step 1: Read Utilization for the Quotas AWS Already Meters

The Service Quotas API returns the applied value for every quota and, for a subset, a pointer to the CloudWatch metric that tracks usage against it. Start there, because it is free data:

aws service-quotas list-service-quotas --service-code ec2 \
  --query 'Quotas[?UsageMetric!=`null`].[QuotaCode,QuotaName,Value,Adjustable]' \
  --output table

Each quota with a UsageMetric carries the namespace (almost always AWS/Usage), the metric name, the dimensions, and a recommended statistic. The collector does not hardcode any of that. It reads it from the quota and queries what it is told to query:

# quotas/metered.py
import boto3, datetime as dt

def metered(region: str, service: str, days: int = 30):
    sq = boto3.client("service-quotas", region_name=region)
    cw = boto3.client("cloudwatch", region_name=region)
    end = dt.datetime.now(dt.timezone.utc)
    for page in sq.get_paginator("list_service_quotas").paginate(ServiceCode=service):
        for q in page["Quotas"]:
            um = q.get("UsageMetric")
            if not um or q["Value"] == 0:
                continue
            stat = um.get("MetricStatisticRecommendation", "Maximum")
            pts = cw.get_metric_statistics(
                Namespace=um["MetricNamespace"], MetricName=um["MetricName"],
                Dimensions=[{"Name": k, "Value": v} for k, v in um["MetricDimensions"].items()],
                StartTime=end - dt.timedelta(days=days), EndTime=end,
                Period=86400, Statistics=[stat])["Datapoints"]
            series = [p[stat] for p in sorted(pts, key=lambda p: p["Timestamp"])]
            if series:
                yield {"region": region, "service": service, "code": q["QuotaCode"],
                       "name": q["QuotaName"], "quota": q["Value"],
                       "adjustable": q["Adjustable"], "daily_max": series,
                       "util": max(series[-7:]) / q["Value"]}

Two details matter. Quotas are per account and per region, so the outer loop is every enabled region in every account, and global services such as IAM and CloudFront answer only in us-east-1. And Value is the applied quota, which may be the default or may be an increase somebody requested in 2022 and forgot. list-aws-default-service-quotas returns the defaults; the agent diffs the two so that a new account missing an increase its sibling accounts have is itself a finding.

Step 2: Count the Quotas That Have No Usage Metric

This is where console dashboards stop and where most real outages come from. A large share of the quotas that bite have no UsageMetric at all. Nothing in CloudWatch tells you how many VPCs, Elastic IPs, or IAM roles you have relative to the limit. The agent keeps a small registry mapping quota codes to a counting function:

# quotas/counted.py
def _count(client, op, key, **kw):
    return sum(len(p[key]) for p in client.get_paginator(op).paginate(**kw))

COUNTERS = {
    # (service, quota code): (description, counter)
    ("vpc", "L-F678F1CE"): ("VPCs per Region",
        lambda s: _count(s.client("ec2"), "describe_vpcs", "Vpcs")),
    ("ec2", "L-0263D0A3"): ("EC2-VPC Elastic IPs",
        lambda s: len(s.client("ec2").describe_addresses()["Addresses"])),
    ("vpc", "L-DF5E4CA3"): ("Network interfaces per Region",
        lambda s: _count(s.client("ec2"), "describe_network_interfaces", "NetworkInterfaces")),
    ("iam", "L-FE177D64"): ("Roles per account",
        lambda s: s.client("iam").get_account_summary()["SummaryMap"]["Roles"]),
    ("vpc", "L-0EA8095F"): ("Inbound or outbound rules per security group",
        lambda s: max_rules_in_any_group(s)),
}

Quota codes are stable identifiers, but verify each one in your account with aws service-quotas list-service-quotas --service-code vpc before trusting a registry copied from a blog post, this one included.

The last entry shows the trap in this category: some quotas are per resource, not per account. "Rules per security group" is not exhausted when the account has many rules. It is exhausted when one group has sixty. The counter for a per-resource quota returns the maximum across all resources and names the offender, the same way the security group audit agent names a specific group instead of a count. Routes per route table and policies per role behave the same way.

IAM is kind here: get_account_summary returns both the count and the quota for roles, policies, and instance profiles in a single call.

Step 3: Forecast Exhaustion and Check Burst Headroom

Current utilization answers the wrong question. A quota at 85% that has been flat for a year is fine. A quota at 55% that grows two points a week is a ticket to file now. The forecast is deliberately boring, a linear fit over the daily maxima:

# quotas/forecast.py
import numpy as np

def days_to_exhaustion(daily_max: list[float], quota: float) -> float | None:
    if len(daily_max) < 14:
        return None
    slope, _ = np.polyfit(range(len(daily_max)), daily_max, 1)
    if slope <= 0:
        return None
    return max(0.0, (quota - daily_max[-1]) / slope)

The more valuable check does not look at history at all. It asks: if every autoscaler in this region went to its configured maximum at once, would the request fit? For each Auto Scaling group the agent multiplies MaxSize by the vCPU count of its largest instance type, and for each Karpenter NodePool it reads spec.limits.cpu. The sum is the burst ceiling you have promised your workloads. If that ceiling exceeds the Standard On-Demand vCPU quota (L-1216C47A), your maxSize is fiction, and you will learn so during the event the headroom was configured for.

QuotaApplied7d peakTrendDays leftBurst ceiling
Standard On-Demand vCPUs (us-east-1)1,9201,104+9/day902,640
Elastic IPs (us-east-1)2019flatn/an/a
Lambda concurrent executions (eu-west-1)1,000870+11/day12n/a
Rules per security group (sg-0a41)6057+1/week21n/a
IAM roles1,000612+3/day129n/a

The first row looks comfortable at 57% utilization and is actually the worst finding on the page: a full scale-out needs 720 more vCPUs than the account can launch. This is the same shape of forecast the capacity planning agent runs for nodes and PVCs, one layer further down, at the provider limit that the cluster cannot see.

Step 4: Use CloudTrail Limit Errors as Ground Truth

Counting can miss quotas you never registered. CloudTrail cannot, because every refused call is logged with an error code. lookup-events does not filter on error code, so this runs in Athena against the trail's table:

SELECT eventsource, eventname, errorcode, awsregion,
       count(*) AS hits, max(eventtime) AS last_seen
FROM cloudtrail_logs
WHERE eventtime > date_format(date_add('day', -14, now()), '%Y-%m-%dT%H:%i:%sZ')
  AND (errorcode LIKE '%LimitExceeded%' OR errorcode LIKE '%QuotaExceeded%')
GROUP BY 1, 2, 3, 4
ORDER BY hits DESC;

A row such as ec2.amazonaws.com / RunInstances / VcpuLimitExceeded / 214 is not a prediction. It already happened, usually silently, inside an autoscaler that retried until capacity freed up elsewhere. If the pods behind it sat unschedulable in the meantime, that is the cloud-side cause of the symptom in the Pod Pending guide. Any error code the registry does not know becomes a needs_human finding with the raw event attached, which is how the registry grows.

Keep throttling separate. ThrottlingException and RequestLimitExceeded are API rate quotas, a different mechanism with a different fix (backoff, caching, fewer describe calls), and mixing them into this report buries the resource limits.

Step 5: The Model Decides Cleanup Versus Increase

Everything above is deterministic. The model gets the table rows, the CloudTrail hits, and an inventory of reclaimable resources, and its job is the judgement call a tired engineer gets wrong: asking AWS for more of something you are wasting.

SYSTEM = (
    "You triage AWS service quota findings. You are given computed facts only. "
    "Return one verdict per finding, as JSON.\n"
    "cleanup_first: reclaimable resources exist that would bring utilization "
    "under 70%. You may NOT choose request_increase while this is true.\n"
    "request_increase: adjustable quota, and either under 30 days to exhaustion, "
    "a burst ceiling above the quota, or CloudTrail limit errors in the window. "
    "Propose a value: the larger of burst ceiling x 1.2 and the 90-day forecast, "
    "never more than 2x the applied value. Write a three-sentence justification "
    "citing the numbers.\n"
    "alarm_only: above 60% but flat, or not adjustable. Recommend the threshold.\n"
    "needs_human: unknown error code, a non-adjustable quota under 30 days, or "
    "growth that looks like a leak (one resource type, one creator, linear).\n"
    "Do not invent usage numbers. Do not recommend a value you were not given "
    "the inputs to compute."
)

The Elastic IP row is the canonical cleanup_first. Nineteen of twenty in use, and the waste inventory shows seven unassociated addresses left behind by deleted NAT gateways. The right action is the waste reclamation agent's release queue, not a support case. The IAM roles row is the leak pattern: three new roles a day, all created by one CI principal, almost certainly a per-branch role that nothing deletes. Raising the quota there makes the leak someone else's problem in four months.

The justification text matters more than it seems. Large increases route to a human at AWS, and requests that say why ("burst ceiling of 2,640 vCPUs across 6 node groups, current quota 1,920, failover test scheduled") get approved faster than a bare number.

Step 6: Ship Alarms and Increases as a Terraform PR

For metered quotas, the alarm uses the SERVICE_QUOTA() metric math function, which resolves the current limit at evaluation time. The threshold is a percentage, so the alarm stays correct after an increase lands:

resource "aws_cloudwatch_metric_alarm" "vcpu_quota" {
  alarm_name          = "quota-ec2-standard-ondemand-vcpu-80pct"
  comparison_operator = "GreaterThanThreshold"
  threshold           = 80
  evaluation_periods  = 3
  alarm_actions       = [aws_sns_topic.platform_alerts.arn]

  metric_query {
    id = "usage"
    metric {
      namespace   = "AWS/Usage"
      metric_name = "ResourceCount"
      period      = 300
      stat        = "Maximum"
      dimensions = {
        Service  = "EC2"
        Resource = "vCPU"
        Type     = "Resource"
        Class    = "Standard/OnDemand"
      }
    }
  }
  metric_query {
    id          = "pct"
    expression  = "(usage / SERVICE_QUOTA(usage)) * 100"
    label       = "Standard On-Demand vCPU quota utilization"
    return_data = true
  }
}

For the counted quotas there is no metric to alarm on, so the agent publishes its own counts as a custom metric on each run and the alarm compares against a static threshold the PR sets.

The increase itself is one resource. Applying it files the request:

resource "aws_servicequotas_service_quota" "ec2_standard_vcpu" {
  service_code = "ec2"
  quota_code   = "L-1216C47A"
  value        = 3200   # burst ceiling 2,640 x 1.2, rounded; was 1,920
}

Putting the quota in Terraform has a second benefit beyond review. It records the intended value next to the node groups that depend on it, so the next account built from the same module requests the same limit on day one instead of discovering the default during its first load test. For AWS Organizations, a quota request template does the same job for every newly created account.

Small increases are often approved automatically within minutes. Larger ones open a support case, and the apply can sit waiting. The agent follows up by polling:

aws service-quotas list-requested-service-quota-change-history-by-quota \
  --service-code ec2 --quota-code L-1216C47A \
  --query 'RequestedQuotas[0].[Status,DesiredValue,CaseId]'

PENDING and CASE_OPENED mean wait. DENIED and NOT_APPROVED go to a human with the case ID, because the next step is a conversation with AWS support, not a retry.

Guardrails That Live in Code

  • Read-only by default. The agent's role has servicequotas:List*, servicequotas:Get*, cloudwatch:GetMetricStatistics, autoscaling:Describe*, and the specific describe calls the counters need. servicequotas:RequestServiceQuotaIncrease exists only on the CI role that applies merged Terraform, scoped with the servicequotas:service condition key to the services you allow.
  • One open request per quota. Before proposing an increase the wrapper checks the change history and drops the proposal if a request is PENDING or CASE_OPENED. Duplicate requests slow the first one down.
  • A cap the model cannot argue with. The proposed value is clamped to 2x the applied quota in code, regardless of what the verdict says. Anything larger needs a human who can explain it to AWS.
  • Cleanup beats increase, enforced. If the waste inventory holds reclaimable resources of the same type, the wrapper downgrades request_increase to cleanup_first, even when the model disagrees.
  • Approval on the PR. The merge is the gate, following the same approval pattern as every other write-capable agent on this site, and the applied change shows up attributed to a run and a reviewer.

Honest Limits

A quota is permission, not capacity. An approved 3,200 vCPU limit does not guarantee that c7g.4xlarge exists in us-east-1b at 3 a.m. InsufficientInstanceCapacity is a separate failure with a separate fix, mainly instance-type diversification of the kind the Karpenter Spot setup already forces on you.

The registry is never complete. AWS has thousands of quotas, the agent counts the few dozen you registered, and the CloudTrail query is the only thing that catches the rest, after the first refusal rather than before it.

Linear forecasts miss step changes. A launch, a migration, or a new tenant doubles usage overnight, and thirty days of flat history says nothing about it. The burst-ceiling check covers the compute case. For everything else, the honest answer is that planned events need a human to run the numbers ahead of time.

Some quotas cannot be raised at all, and some that claim Adjustable: true are refused in practice past a certain size. When a non-adjustable quota is trending toward its limit, the fix is architectural: more accounts, more regions, or a different design. That is a decision for people, and the agent's job is only to make sure they hear about it with ninety days of notice instead of none.

#ai#sre#aws#cloud#automation#reliability#terraform
D
DevToCashAuthor

Senior DevOps/SRE Engineer · 10+ years · Professional Trader (IDX, Crypto, US Equities)

I write about real infrastructure patterns and trading strategies I use in production and in live markets. No courses, no affiliate hype — just documentation of what actually works.

More about me →