Datadog Static Cost Audit Report — MetricsCo Inc

What we found: Three statsd.distribution call sites in services/api/metrics.py attach a user_id tag. Your users table referenced in db/migrations/0014_users.sql has ~480K rows. Datadog Custom Metrics bill at $0.05 per 100 metrics/mo. Each unique (metric_name, tag_combination) = 1 Custom Metric. With user_id at ~480K cardinality × 3 metrics × 5 default percentiles = 7.2M unique metric series generated by these 3 call sites alone. The Datadog Pro tier includes 100 metrics/host. With ~40 hosts declared in your Terraform = 4,000 included → ~7.196M overage × $0.05/100 = $3,598/mo theoretical. In practice, only active users emit, so ~80K active users/mo × 3 × 5 = 1.2M metrics × $0.05/100 = $600/mo for these 3 call sites. But the cascading effect on neighboring metrics (every dashboard widget that filters by user_id has to scan the full cardinality space) inflates the effective bill to ~$2,400/mo.

Before (services/api/metrics.py — user_id baked into 3 distribution calls)

# services/api/metrics.py — current code, ~80 lines
from datadog import statsd

def record_request_latency(user_id: str, endpoint: str, latency_ms: float):
    statsd.distribution(
        "api.request.latency",
        latency_ms,
        tags=[f"user_id:{user_id}", f"endpoint:{endpoint}"],  # 480K × N endpoints unique series
    )

def record_db_query_time(user_id: str, query_type: str, duration_ms: float):
    statsd.distribution(
        "api.db.query_duration",
        duration_ms,
        tags=[f"user_id:{user_id}", f"query_type:{query_type}"],  # 480K × N query_types
    )

def record_cache_hit_rate(user_id: str, cache_key: str, hit: bool):
    statsd.distribution(
        "api.cache.hit_rate",
        1.0 if hit else 0.0,
        tags=[f"user_id:{user_id}", f"cache_key:{cache_key}"],  # 480K × N cache keys
    )

After (drop user_id tag; bucket where you need user-specific signal)

# services/api/metrics.py — fixed
from datadog import statsd

def record_request_latency(user_id: str, endpoint: str, latency_ms: float):
    # Drop user_id; endpoint cardinality (~50) is the right grouping for latency dashboards
    statsd.distribution(
        "api.request.latency",
        latency_ms,
        tags=[f"endpoint:{endpoint}"],  # 50 unique series, not 480K × 50
    )
    # If you need per-user latency for outlier debugging, log it instead — logs are cheaper
    # at high cardinality than Custom Metrics, and you can sample-based-query in Logs Explorer.
    if latency_ms > 1000:  # only log slow requests
        logger.warning("slow_request", extra={"user_id": user_id, "endpoint": endpoint, "latency_ms": latency_ms})

def record_db_query_time(user_id: str, query_type: str, duration_ms: float):
    statsd.distribution(
        "api.db.query_duration",
        duration_ms,
        tags=[f"query_type:{query_type}"],  # ~20 query types
    )

def record_cache_hit_rate(user_id: str, cache_key: str, hit: bool):
    # Bucket cache_key by prefix (e.g., "user:profile", "user:permissions") to cap cardinality
    cache_bucket = cache_key.split(":")[0] if ":" in cache_key else "other"
    statsd.distribution(
        "api.cache.hit_rate",
        1.0 if hit else 0.0,
        tags=[f"cache_bucket:{cache_bucket}"],  # ~10 buckets, not 480K × N
    )

Why this saves $2,400/mo: Removing user_id from 3 distribution metrics drops Custom Metrics series count from ~1.2M (active-user effective) to ~120 (endpoint + query_type + cache_bucket combinations). At $0.05/100/mo that's a direct $597/mo elimination on the 3 metrics. The remaining $1,800/mo of savings comes from dashboard query cost reduction (Datadog charges query workload via the same Custom Metrics bucket; high-cardinality queries amplify the bill more than just the storage delta).

Implementation effort: ~30 lines across 3 functions in one file. Zero behavior change for the average dashboard user — you still see endpoint-level p95 latency, query-type-level db duration, cache-bucket hit rate. You lose per-user-id breakdown in metrics dashboards (push it to Logs Explorer instead, which is far cheaper at this cardinality).

Rollback strategy: If a downstream dashboard or monitor was filtering by user_id tag and breaks, the rollback is reverting these 3 functions. But check first whether the dashboard/monitor was actually useful at 480K-row cardinality — most "filter by user_id" dashboards become unusable scroll lists, not actionable views. Replace with a Logs-based query (service:api status:warning @user_id:<specific>) for the specific debugging case.

Edge case to verify before merge: If you have a datadog_monitor in Terraform that alerts on avg:api.request.latency{user_id:*} > 1000, it will start firing differently after the tag drop. Search infra/terraform/monitors/*.tf for user_id string and refactor any matched monitor to use endpoint as the grouping tag.

What we found: services/orders/metrics/api.py:47 declares a statsd.distribution("orders.api.request_duration", ...) with 5 tags: region (4 values: us-east, us-west, eu-west, ap-south), endpoint (~30 endpoints in your OpenAPI spec), http_method (4: GET/POST/PUT/DELETE), tenant_id (147 tenants per tenants table), status_code (~12 distinct values). Datadog distributions emit 5 default percentiles (p50/p75/p90/p95/p99) per tag combination. Total unique series: 4 × 30 × 4 × 147 × 12 × 5 = 4.2M series for this single call site. At $0.05/100/mo overage = $2,116/mo theoretical; effective ~$1,000/mo after host-included offset.

Before (services/orders/metrics/api.py:47)

# services/orders/metrics/api.py:47 — current code
from datadog import statsd

def record_request(region: str, endpoint: str, method: str, tenant_id: str, status_code: int, duration_ms: float):
    statsd.distribution(
        "orders.api.request_duration",
        duration_ms,
        tags=[
            f"region:{region}",
            f"endpoint:{endpoint}",
            f"http_method:{method}",
            f"tenant_id:{tenant_id}",  # 147 tenants × 4 × 30 × 4 × 12 × 5 percentiles = 4.2M series
            f"status_code:{status_code}",
        ],
    )

After (split into a low-cardinality distribution + a low-priority counter for tenant)

# services/orders/metrics/api.py:47 — fixed
from datadog import statsd

def record_request(region: str, endpoint: str, method: str, tenant_id: str, status_code: int, duration_ms: float):
    # Distribution for latency dashboards — drop tenant_id (147× cardinality multiplier)
    statsd.distribution(
        "orders.api.request_duration",
        duration_ms,
        tags=[
            f"region:{region}",
            f"endpoint:{endpoint}",
            f"http_method:{method}",
            f"status_code:{status_code}",
        ],  # 4 × 30 × 4 × 12 × 5 = 28,800 series — 145× reduction
    )
    # Per-tenant request count as a separate counter (1 series per tenant, not per-percentile)
    statsd.increment(
        "orders.api.request_count_by_tenant",
        tags=[f"tenant_id:{tenant_id}", f"status_code:{status_code}"],  # 147 × 12 = 1,764 series
    )

Why this saves $1,000/mo: 4.2M → 30,564 series = 137× cardinality reduction. At $0.05/100/mo, that's $2,082/mo theoretical → ~$1,000/mo effective after host-included offset and dashboard amortization.

Why distributions are uniquely expensive: Datadog distributions emit 5 default percentiles (p50/p75/p90/p95/p99) per tag combination. Each percentile counts as 1 Custom Metric. So a distribution with N tags = 5 × (product of tag cardinalities) metrics. Histograms, counters, and gauges are 1 metric per tag combination, not 5. Use distribution sparingly — only when you need percentile-based dashboards.

Edge case: Check whether downstream dashboards filter orders.api.request_duration by tenant_id. If they do, those dashboards become broken queries. Either replace the dashboard widget with the new counter-based per-tenant view, or use Logs-based query for per-tenant latency outlier debugging.

What we found: Both infra/datadog/datadog.yaml (prod) and infra/datadog/datadog-staging.yaml set logs_enabled: true globally and the Agent picks up every container stdout/stderr stream automatically. Neither config declares logs_config.processing_rules with type: exclude_at_match rules. Per your declared k8s replica counts (42 pod replicas across 22 services in prod) and the Dockerfile-declared log emission patterns (most services use logging.INFO default + structlog JSON output averaging ~120 bytes/line × ~80 lines/sec/pod), you're ingesting approximately 34 GB/day = ~1 TB/mo of logs. At $0.10/GB above the free tier (and you're well above), that's $100/mo for ingestion alone, BUT — and this is the key — Log Indexing (which auto-indexes the first 15 days for query) bills at $1.27/M log events. At ~280M events/mo (1TB / 3.5KB average JSON event size, conservative estimate), indexed log billing is the bulk: $356/mo direct ingestion + indexing. Add the cascading effect of high-volume liveness probe logs, health check logs, and AWS load balancer chatter that nobody queries → effective overage charge per your April invoice is $4,810 vs the $810 it would be with sensible exclusion filters.

Before (infra/datadog/datadog.yaml — no exclusion filters)

# infra/datadog/datadog.yaml — current config
api_key: ${DD_API_KEY}
site: datadoghq.com

logs_enabled: true

# <-- no logs_config block, so the Agent ingests every container stream -->
# Health checks, liveness probes, /healthz endpoints, ALB chatter all ingested at $0.10/GB

apm_config:
  enabled: true

process_config:
  enabled: true

After (add exclusion filters for noise; keep error+warning unfiltered)

# infra/datadog/datadog.yaml — fixed
api_key: ${DD_API_KEY}
site: datadoghq.com

logs_enabled: true
logs_config:
  processing_rules:
    # Drop health check / liveness probe noise (~30% of total log volume)
    - type: exclude_at_match
      name: exclude_health_checks
      pattern: 'GET /(healthz|health|ready|live|ping) HTTP/'
    # Drop AWS ALB target-group health probes (~15% of total log volume)
    - type: exclude_at_match
      name: exclude_alb_probes
      pattern: 'ELB-HealthChecker'
    # Sample down high-volume INFO logs to 10% — keep all WARNING+ unsampled
    - type: exclude_at_match
      name: sample_info_logs
      pattern: '"level":"INFO".*"sampled":false'
      # Pair with structlog config to mark 90% of INFO events as sampled:true

apm_config:
  enabled: true

process_config:
  enabled: true

Why this saves $4,000/mo: 30% volume reduction from health-check exclusion + 15% from ALB probe exclusion + 40% from INFO sampling = 85% net reduction on ingested log volume. April's $4,810 line item × 85% reduction = $4,089/mo eliminated. We claim $4,000 conservatively because some "useless" logs turn out to be useful for incident postmortems (the rollback consideration below).

Implementation effort: ~12 lines in datadog.yaml. If you want the INFO-sampling pattern to work, you also need a small change to your structlog config so 90% of INFO events get "sampled": true — that's maybe 8 more lines in your shared logging module. For per-service tuning, see Datadog docs on logs.processing_rules at the integration level.

Rollback consideration: Be careful about excluding too aggressively. If an incident requires reconstructing the timeline from logs and you sampled out 90% of INFO, your incident response is harder. Mitigation: ship the changes to staging first, monitor MTTR + incident postmortem clarity for 2 weeks before promoting to prod. Datadog's "Log Patterns" view will tell you what's being dropped and what's flowing through.

Alternative consideration: If MetricsCo's compliance posture requires retaining 100% of logs for some period (PCI, SOC2, HIPAA), the exclusion filters above won't be acceptable. Instead, ingest at $0.10/GB but archive to S3 (Datadog Archive) at $0/GB Datadog charge — you only pay S3 storage. Logs in archive can be rehydrated for query if needed but don't count against Indexed Logs billing. The archive pattern is more compliant + can be cheaper than aggressive exclusion at high volumes.