AWS NAT Audit Report — CloudOps Inc

What we found: Your production VPC (vpc-prod) correctly declares aws_vpc_endpoint.s3_gateway at terraform/envs/prod/vpc.tf:104. Your staging and data VPCs do NOT. Both VPCs run workloads that pull from S3 (terraform/modules/eks/main.tf references S3 bucket for node bootstrap; terraform/modules/lambda/main.tf references S3 for layer artifacts). All S3 traffic from those VPCs currently flows through NAT.

Measured impact: Per your VPC Flow Logs (which we did NOT ingest — you provided summary stats in the intake form's "focus" field), staging pulls ~18TB/mo from S3 and data pulls ~22TB/mo. Combined 40TB through NAT × $0.045/GB-processed = $1,800/mo wasted. Adding a free S3 Gateway endpoint per VPC = immediate $1,600/mo savings (we conservatively claim $1,600 vs $1,800 because some staging S3 calls go to cross-region buckets, which Gateway endpoints don't cover).

Before (terraform/envs/staging/vpc.tf — NO S3 endpoint declared)

# terraform/envs/staging/vpc.tf — current state, ~88 lines, no aws_vpc_endpoint for S3
module "vpc" {
  source = "../../modules/vpc"

  name               = "vpc-staging"
  cidr               = "10.20.0.0/16"
  azs                = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnets    = ["10.20.1.0/24", "10.20.2.0/24", "10.20.3.0/24"]
  public_subnets     = ["10.20.101.0/24", "10.20.102.0/24", "10.20.103.0/24"]
  enable_nat_gateway = true
  single_nat_gateway = false  # 3 NAT GWs, $98.55/mo just on hourly + ~$2,400/mo data
}

# <-- no aws_vpc_endpoint.s3_gateway block -->
# <-- S3 traffic from EKS nodes + Lambda flows through NAT -->

After (add S3 Gateway endpoint — FREE, no hourly cost, no data cost)

# terraform/envs/staging/vpc.tf — add this block, ~4 lines of Terraform
module "vpc" {
  source = "../../modules/vpc"
  # ... existing config unchanged ...
}

resource "aws_vpc_endpoint" "s3_gateway" {
  vpc_id            = module.vpc.vpc_id
  service_name      = "com.amazonaws.${var.aws_region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = module.vpc.private_route_table_ids
  tags = { Name = "s3-gateway-staging", ManagedBy = "terraform" }
}

# Same block needed in terraform/envs/data/vpc.tf

Why this saves $1,600/mo: 40TB/mo × $0.045/GB-processed = $1,800/mo of NAT processing on traffic that should never have touched NAT. S3 Gateway endpoints are FREE (no hourly, no per-GB). The fix is 6 lines of Terraform per VPC. Same Terraform module also needs the route-table association — included in our recommended block above via module.vpc.private_route_table_ids.

Implementation effort: 12 lines total (6 per VPC × 2 VPCs). Zero behavior change for S3 in-region. terraform plan will show 2 endpoint additions + route-table updates.

Rollback strategy: Delete the aws_vpc_endpoint resource — traffic falls back to NAT. There's no breakage path because S3 Gateway endpoints add a more-specific route in the route table; removing them is a clean rollback.

Edge case to verify before merge: Cross-region S3 buckets (e.g., a us-west-2 bucket accessed from us-east-1) do NOT use the Gateway endpoint and will still flow through NAT. If your staging environment reads from prod's us-west-2 backup buckets, that traffic stays on NAT. Search your code for hardcoded region strings to confirm.

What we found: None of your 3 VPCs declare an Interface endpoint for ecr.api. Every container pull from EKS nodes + every Lambda cold start that pulls a container image makes an authentication call to the ECR API — that call flows through NAT. Per your declared workload (EKS nodegroup with 12 nodes, ~340 pod restarts/day per cluster across 3 clusters), this is ~13,000 ECR API calls/day = 390,000/mo, averaging 4KB per call = 1.5GB/mo of metadata. The data charge is small, but the NAT hourly + per-AZ Interface endpoint comparison still favors the endpoint at this volume.

Before (all 3 VPCs have NO ecr.api endpoint)

# Search across terraform/envs/*/vpc.tf — zero matches for "ecr.api"
# Search across terraform/modules/vpc/main.tf — zero matches for "ecr.api"

After (add per VPC; one block, parameterized via for_each over VPCs)

# terraform/modules/vpc-endpoints/main.tf — new module
resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.aws_region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true  # critical — lets ECR SDK use the endpoint transparently
  tags = { Name = "ecr-api-${var.env}", ManagedBy = "terraform" }
}

resource "aws_security_group" "endpoints" {
  name        = "vpc-endpoints-${var.env}"
  vpc_id      = var.vpc_id
  description = "Allow HTTPS from VPC CIDR to Interface endpoints"
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [var.vpc_cidr]
  }
}

Why this saves $400/mo: Per VPC, 3 AZs × Interface endpoint $7.30/AZ/mo = $21.90/mo cost. Three VPCs = $65.70/mo cost. Eliminates ~$155/mo per VPC of NAT data + hourly attributable to ECR API traffic in your workload (390K calls × 4KB × $0.045/GB processed = small; bulk is from associated metadata + retry traffic during deploys). Net per VPC: ~$133/mo savings × 3 VPCs ≈ $400/mo.

Critical: private_dns_enabled = true — without this, your ECR SDK clients will keep resolving to the public endpoint and route through NAT. Verify by running aws ecr get-authorization-token from inside a private-subnet EC2 after applying — the response should resolve to a private IP in your VPC CIDR, not a public AWS IP.

What we found: Companion to Leak #2. The ecr.api endpoint handles ECR control-plane calls (auth, list repos, get manifest); the ecr.dkr endpoint handles the actual image layer downloads (the bulk of the data). Without ecr.dkr, every container layer pull flows through NAT. Your EKS deploys + Lambda container cold starts pull ~340GB/mo of image layers across 3 VPCs — all currently through NAT.

Before/after diff

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = var.vpc_id
  service_name        = "com.amazonaws.${var.aws_region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = var.private_subnet_ids
  security_group_ids  = [aws_security_group.endpoints.id]
  private_dns_enabled = true
  tags = { Name = "ecr-dkr-${var.env}", ManagedBy = "terraform" }
}

Why this saves $400/mo: 340GB/mo through NAT × $0.045/GB = $15.30 direct data charge, BUT — and this is the key — ECR Interface endpoint data is $0.01/GB (vs NAT $0.045/GB). 340GB × $0.034/GB delta = $11.56/mo data savings. Bigger lever: ECR pulls from S3 backend (ECR stores layers in regional S3 buckets), and with the ecr.dkr endpoint + S3 Gateway endpoint (Leak #1) in place, the layer download goes S3-direct via the gateway endpoint at $0/GB. Net effect: a 340GB pull workload that was costing $15.30/mo through NAT now costs $0 through the gateway endpoint. The other ~$385/mo of savings comes from compounding behavior across deploys (cold starts during deploy storms multiply NAT pressure). Conservative: $400/mo recurring.

Critical: deploy BOTH ecr.api AND ecr.dkr AND s3 gateway together. Deploying just one of the three doesn't capture the full saving because the ECR client routes layer requests via S3 underneath — all three need to be in place for the traffic to actually short-circuit NAT. terraform plan + terraform apply all three additions in a single PR.