Build a Security Group Audit Agent: Find 0.0.0.0/0 Ingress and Unused Groups, Prove Nothing Needs Them with ENIs and VPC Flow Logs, and Tighten via Terraform PR

What This Agent Does

A security group audit agent inventories every security group in an account, splits the findings into three piles (groups attached to nothing, rules open to 0.0.0.0/0 on ports that should never be public, and rules open to the world that are probably fine because they front a load balancer), and then does the part Security Hub never does: it proves whether anything still needs each rule. The proof is deterministic. Network interfaces tell you what the group is attached to. Launch templates, ECS services, and Lambda VPC configs tell you what will attach to it. VPC Flow Logs tell you whether a single packet from outside the VPC was accepted on that port in the last 30 days. The LLM gets the pre-chewed evidence and writes the classification and the pull request text. The only write path is a branch in the Terraform repo. The agent's IAM role explicitly denies ec2:RevokeSecurityGroup* and ec2:DeleteSecurityGroup, so it cannot fix anything directly even if it tries.

It is the network sibling of the IAM right-sizing agent: same shape, same read-only posture, same "narrow it, prove the narrowing is safe, let a human merge" loop. Security groups are easier in one way and harder in another. Easier, because attachment is a hard fact you can enumerate. Harder, because "nobody connected in 30 days" is evidence, not proof, and the agent has to say so.

Where the Findings Pile Up

In a 400-group account the first row is usually a third of all groups. The second and third rows are a dozen groups that have been red in Security Hub for a year because nobody can say for certain the rule is unused. That certainty is what the agent manufactures.

Step 1: A Role That Cannot Fix Anything

Every permission is a describe or a Logs Insights query. The explicit Deny is the real guardrail, for the same reason it was in the IAM post: an Allow added to this role later, by anyone, cannot turn the agent into a thing that revokes rules in production.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadNetworkInventory",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSecurityGroupRules",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeVpcs",
        "ec2:DescribeFlowLogs",
        "ecs:ListClusters", "ecs:ListServices", "ecs:DescribeServices",
        "lambda:ListFunctions",
        "elasticloadbalancing:DescribeLoadBalancers"
      ],
      "Resource": "*"
    },
    {
      "Sid": "QueryFlowLogs",
      "Effect": "Allow",
      "Action": [
        "logs:StartQuery", "logs:GetQueryResults", "logs:StopQuery"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/vpc/flow-logs/*"
    },
    {
      "Sid": "NeverChangeNetworkSecurity",
      "Effect": "Deny",
      "Action": [
        "ec2:AuthorizeSecurityGroup*", "ec2:RevokeSecurityGroup*",
        "ec2:ModifySecurityGroupRules", "ec2:DeleteSecurityGroup",
        "ec2:CreateSecurityGroup", "ec2:ModifyNetworkInterfaceAttribute"
      ],
      "Resource": "*"
    }
  ]
}

Verify the boundary before trusting it. The second command must fail:

aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0a1b2c3d4e5f --max-results 5
aws ec2 revoke-security-group-ingress --group-id sg-0a1b2c3d4e5f --protocol tcp --port 22 --cidr 0.0.0.0/0
# An error occurred (UnauthorizedOperation) ... explicit deny

Step 2: Inventory Groups, Rules, and Everything That References Them

The collector answers two questions per group. What is attached right now, and what is configured to attach later. The second question is the one most "unused security group" scripts skip, and it is why those scripts delete the group behind an autoscaling group that happens to be scaled to zero at 3 a.m.

# collect_sgs.py — read-only inventory of groups, rules, and references
import boto3
from collections import defaultdict

ec2, ecs, lam = boto3.client("ec2"), boto3.client("ecs"), boto3.client("lambda")

HIGH_RISK_PORTS = {20, 21, 22, 23, 25, 110, 135, 143, 445, 1433, 1434, 3000,
                   3306, 3389, 4333, 5000, 5432, 5500, 5601, 6379, 8080,
                   8088, 8888, 9200, 9300, 27017, 27018, 2379, 2380, 10250}

def groups():
    for page in ec2.get_paginator("describe_security_groups").paginate():
        for g in page["SecurityGroups"]:
            yield {"id": g["GroupId"], "name": g["GroupName"], "vpc": g.get("VpcId"),
                   "is_default": g["GroupName"] == "default",
                   "tags": {t["Key"]: t["Value"] for t in g.get("Tags", [])}}

def open_ingress_rules():
    out = defaultdict(list)
    for page in ec2.get_paginator("describe_security_group_rules").paginate():
        for r in page["SecurityGroupRules"]:
            if r["IsEgress"]:
                continue
            cidr = r.get("CidrIpv4") or r.get("CidrIpv6")
            if cidr not in ("0.0.0.0/0", "::/0"):
                continue
            lo, hi = r.get("FromPort", -1), r.get("ToPort", -1)
            out[r["GroupId"]].append({
                "rule_id": r["SecurityGroupRuleId"], "proto": r["IpProtocol"],
                "from": lo, "to": hi, "cidr": cidr,
                "all_ports": lo == -1,
                "high_risk": lo == -1 or any(lo <= p <= hi for p in HIGH_RISK_PORTS),
            })
    return out

def attachments():
    """Live ENIs per group, with what they belong to."""
    out = defaultdict(list)
    for page in ec2.get_paginator("describe_network_interfaces").paginate():
        for eni in page["NetworkInterfaces"]:
            kind = eni.get("Description", "")
            owner = (eni.get("Attachment") or {}).get("InstanceId") or kind[:60]
            for g in eni["Groups"]:
                out[g["GroupId"]].append({
                    "eni": eni["NetworkInterfaceId"], "ip": eni["PrivateIpAddress"],
                    "owner": owner, "public_ip": bool(eni.get("Association", {}).get("PublicIp")),
                    "is_lb": kind.startswith("ELB "), "is_lambda": kind.startswith("AWS Lambda VPC"),
                })
    return out

def future_references():
    """Things that will attach the group later, or reference it from another rule."""
    refs = defaultdict(set)
    for page in ec2.get_paginator("describe_security_group_rules").paginate():
        for r in page["SecurityGroupRules"]:
            ref = r.get("ReferencedGroupInfo", {}).get("GroupId")
            if ref:
                refs[ref].add(f"rule:{r['GroupId']}")
    for lt in ec2.get_paginator("describe_launch_templates").paginate():
        for t in lt["LaunchTemplates"]:
            v = ec2.describe_launch_template_versions(
                LaunchTemplateId=t["LaunchTemplateId"], Versions=["$Default", "$Latest"])
            for ver in v["LaunchTemplateVersions"]:
                d = ver["LaunchTemplateData"]
                for sg in d.get("SecurityGroupIds", []):
                    refs[sg].add(f"launch-template:{t['LaunchTemplateName']}")
                for ni in d.get("NetworkInterfaces", []):
                    for sg in ni.get("Groups", []):
                        refs[sg].add(f"launch-template:{t['LaunchTemplateName']}")
    for cluster in ecs.list_clusters()["clusterArns"]:
        arns = ecs.list_services(cluster=cluster)["serviceArns"]
        for i in range(0, len(arns), 10):
            for s in ecs.describe_services(cluster=cluster, services=arns[i:i+10])["services"]:
                cfg = s.get("networkConfiguration", {}).get("awsvpcConfiguration", {})
                for sg in cfg.get("securityGroups", []):
                    refs[sg].add(f"ecs:{s['serviceName']}")
    for page in lam.get_paginator("list_functions").paginate():
        for f in page["Functions"]:
            for sg in f.get("VpcConfig", {}).get("SecurityGroupIds", []):
                refs[sg].add(f"lambda:{f['FunctionName']}")
    return refs

A group with no live ENIs and no entry in future_references is genuinely unused, and the agent can say so without flow logs. A group with zero ENIs but a launch template reference is "idle", not "unused", and goes to the hold pile. A group referenced by a rule in another group (ReferencedGroupInfo) cannot be deleted at all until that rule is changed, which Terraform will tell you loudly if the agent forgets.

Step 3: Prove a Public Rule Is Unused with Flow Logs

For open rules on attached groups the question changes from "is anything attached" to "did anything from outside the VPC ever get through this rule." Flow logs answer that, with one catch that trips up every first attempt: flow log records carry an interfaceId, not a security group ID. The join goes through the ENI list from Step 2. The second catch is direction. Every connection produces records in both directions on the same ENI, so filter on dstAddr equal to the ENI's private IP to count only ingress.

# flow_evidence.py — accepted ingress from outside the VPC on one rule, last 30 days
import time, boto3

logs = boto3.client("logs")

def accepted_external_ingress(log_group, enis, vpc_cidr, port, days=30):
    ids = ", ".join(f'"{e["eni"]}"' for e in enis)
    ips = ", ".join(f'"{e["ip"]}"' for e in enis)
    port_filter = "" if port == -1 else f"and dstPort = {port}"
    q = f"""
    filter interfaceId in [{ids}] and dstAddr in [{ips}]
      and action = "ACCEPT" {port_filter}
      and not isIpv4InSubnet(srcAddr, "{vpc_cidr}")
    | stats count(*) as flows, sum(bytes) as bytes,
            min(@timestamp) as first_seen, max(@timestamp) as last_seen by srcAddr
    | sort flows desc | limit 50
    """
    end = int(time.time()); start = end - days * 86400
    qid = logs.start_query(logGroupName=log_group, startTime=start, endTime=end, queryString=q)["queryId"]
    while True:
        r = logs.get_query_results(queryId=qid)
        if r["status"] in ("Complete", "Failed", "Cancelled", "Timeout"):
            return r
        time.sleep(1)

Three refinements worth the extra lines. First, check describe_flow_logs for the VPC before running anything. If flow logs are off, or were turned on 9 days ago, the agent must report "insufficient evidence" rather than "no traffic". Second, exclude your own corporate egress CIDRs and your CI runners' NAT IPs from the "external" definition, otherwise every legitimately used bastion looks like it is being hammered from the internet. Third, treat REJECT counts on the same port as a separate signal. Thousands of rejected SYNs on 22 with zero accepts is the normal background radiation of a public IP. Zero rejects usually means the ENI is not actually reachable on that port, which is itself worth noting.

The same flow-log plumbing serves the network cost agent, and the same economics apply: on a large VPC, query logs delivered to S3 with Athena instead of paying Logs Insights scan fees for a monthly audit.

Step 4: What the Model Decides

The model never sees the raw API output. It sees one JSON object per group: open rules with their high-risk flag, live attachments with owner type, future references, and the flow-log summary per rule. Its job is classification and prose, bounded by rules it is not allowed to reinterpret.

You are reviewing AWS security group findings. For each group, output one of:
  delete_group        — zero ENIs and zero references. Flow logs are irrelevant.
  remove_rule         — rule open to 0.0.0.0/0 on a high-risk port, attached ENIs are
                        not load balancers, flow logs cover >= 30 days, accepted
                        external ingress = 0. Name the alternative (SSM, group reference,
                        corp CIDR variable).
  restrict_rule       — accepted external ingress comes from <= 3 stable source CIDRs
                        that are known corporate or partner ranges. Propose those CIDRs.
  authorized_public   — 80/443 (or a port tagged public-port) on an ENI whose owner is an
                        internet-facing load balancer. No change. One line of justification.
  hold                — anything else. Say exactly what evidence would resolve it.

Rules:
- A load balancer ENI is never a candidate for remove_rule or restrict_rule.
- Never propose restricting a public web port to the IPs you observed. Observed
  sources are a sample of the internet, not an allowlist.
- Groups tagged audit:exempt, aws:eks:cluster-name, kubernetes.io/cluster/*, or
  elbv2.k8s.aws/cluster are managed by controllers, not Terraform. Output hold.
- The default group gets remove_rule for every rule and is never delete_group.
- Evidence windows shorter than 30 days always yield hold.

A remove_rule finding comes back shaped for a reviewer who has 30 seconds:

{
  "group": "sg-0a1b2c3d4e5f (legacy-bastion)",
  "rule": "sgr-07e... tcp/22 from 0.0.0.0/0",
  "verdict": "remove_rule",
  "evidence": "2 ENIs (i-0f3a..., i-09c1...), flow logs 30d: 0 ACCEPT from outside 10.0.0.0/16, 41,230 REJECT on dstPort 22, instances have SSM agent online",
  "alternative": "aws ssm start-session; or restrict to var.corp_cidr if SSM is not viable",
  "confidence": 0.9
}

Confidence drops below 0.7 and the finding goes into the PR description as a note, not a diff. That is the same threshold the Kubernetes NetworkPolicy agent uses when Hubble flows are thin, and for the same reason: a false "unused" on a network rule is an outage, not a cost bug.

Step 5: The Fix Is a Pull Request, Gated by Its Own Plan

The agent writes to one branch in the Terraform repo and nothing else, following the pull-requests-not-API-calls rule. Two conventions make the diffs reviewable. Use standalone aws_vpc_security_group_ingress_rule resources rather than inline ingress blocks, so a removal is a deleted resource instead of a rewritten group. And express corporate ranges as a variable, so a restrict_rule finding is a one-line change.

# Before: the finding
resource "aws_vpc_security_group_ingress_rule" "bastion_ssh_world" {
  security_group_id = aws_security_group.legacy_bastion.id
  ip_protocol       = "tcp"
  from_port         = 22
  to_port           = 22
  cidr_ipv4         = "0.0.0.0/0"
}

# After: removed. Access is via SSM Session Manager (see PR body for the 30-day
# flow-log evidence). If a corporate allowlist is required instead:
#
# resource "aws_vpc_security_group_ingress_rule" "bastion_ssh_corp" {
#   security_group_id = aws_security_group.legacy_bastion.id
#   ip_protocol       = "tcp"
#   from_port         = 22
#   to_port           = 22
#   cidr_ipv4         = var.corp_cidr
# }

Before the PR opens, the agent runs terraform plan -out=tfplan in CI and inspects the JSON. The gate is strict: every resource change must be a delete of aws_vpc_security_group_ingress_rule or aws_security_group, or an update of an ingress rule's cidr_ipv4. A plan that contains anything else, including a replacement of an instance because a group ID changed, aborts the PR.

terraform show -json tfplan | jq -e '
  [.resource_changes[]
    | select(.change.actions != ["no-op"])
    | select(
        (.type == "aws_vpc_security_group_ingress_rule" and (.change.actions == ["delete"] or .change.actions == ["update"])) or
        (.type == "aws_security_group" and .change.actions == ["delete"])
      ) | not
  ] | length == 0' >/dev/null || { echo "plan touches something outside SG rules; aborting"; exit 1; }

That is a narrow, hand-written version of the Terraform plan review agent. If you already run the general one, give it this allowlist as policy instead of duplicating the check.

Groups Terraform does not own are a separate pile. The agent cannot write a diff for a ClickOps group, so it opens an issue with an import block proposal and the finding attached. Deleting it by hand is a human decision with a ticket, not an agent action.

After Merge: Watch for the Thing You Broke

A removed SSH rule that someone needed will show up as REJECT records on port 22 from a source that previously had ACCEPTs, within hours. For seven days after each merged PR the agent re-runs the Step 3 query against the same ENIs with action = "REJECT" and compares sources against the pre-merge accept list. A match from a known corporate range opens a revert PR with the evidence attached and pages the on-call channel. A match from a random residential IP is the internet doing what it does and gets a line in the weekly summary. The symmetry with the IAM agent's post-merge AccessDenied watch is deliberate: the shrink is only done when the watch window closes clean.

Where It Will Be Wrong

Run it in shadow mode for a month first and expect these:

• Flow logs are not everywhere. Peered VPCs, Transit Gateway attachments, and accounts that turned flow logs on last quarter all produce confident-looking zeros. The describe_flow_logs check and the 30-day minimum are the defense, and "hold: flow logs enabled 12 days ago" is a legitimate finding.
• Cross-account references are invisible. A rule in a peered account's group can reference your group by ID. describe_security_group_rules in your account will not show it. Peered VPCs get a mandatory hold on delete_group until a human confirms with the other account.
• Zero ENIs is not zero future ENIs. The reference scan covers launch templates, ECS services, and Lambda. It does not cover a Terraform module that creates instances on demand, or a CloudFormation stack in another region. If a group's tags say Terraform owns it, let Terraform's plan be the final word.
• Observed sources are not an allowlist. The prompt forbids narrowing public web ports to observed CIDRs. Keep forbidding it. The one time the model rationalizes an exception, you lock out every customer not in last month's sample.
• Controller-managed groups churn. The AWS Load Balancer Controller and EKS create and delete groups constantly. Tag-based exclusion is cheaper than teaching the agent their lifecycle, and the waste reclamation agent earlier in this series made the same choice for the same reason.
• Security Hub will disagree on timing. EC2.18 and EC2.19 re-evaluate on a schedule, so a merged fix can show as failed for a day. Reconcile against the console, not against the agent's own report.

Takeaway

The reason world-open security group rules survive for years is not that anyone thinks they are fine. It is that proving a rule is unused requires joining five data sources (groups, rules, ENIs, launch configs, flow logs) that were never designed to be joined, and nobody has an afternoon for it. All of that joining is deterministic code. The judgment layer, telling an ALB's public 443 apart from a forgotten database port and writing a PR description a reviewer can trust, is a narrow task the model does well when the evidence is pinned and the rules it cannot override are written down. Keep the role's explicit Deny in place, make the Terraform plan the final gate, and watch the rejects after merge. The agent earns its keep the first time it closes a Security Hub finding that has been red since the person who opened the port left the company.