The WAF Incident: How Our Own Firewall Took Down Production

Security controls are supposed to stop attacks, not customers. Ours stopped everything, at once. This is the post-mortem of the morning the web application firewall in front of our API took production down, written blameless and in full, because the lessons are more useful than the embarrassment is painful.

Summary

• Impact: the API was unreachable in staging and production, so the mobile app couldn't sign users in or move money until the firewall layer was switched off.
• Trigger: nothing we deployed. No release, no config change.
• Root cause: libmodsecurity3, the engine inside our self-managed ModSecurity firewall, was incompatible with the Ubuntu 24.04 base it ran on, and corrupted its own memory on real requests.
• Contributing factors: the OWASP Core Rule Set was cloned unpinned at build time; the firewall ran inside the same container path as every request; and no alert told us before we found it ourselves.
• Resolution: ModSecurity disabled the same morning. The firewall moved to AWS WAF on the load balancer, validated on staging in Count mode before blocking anything, and ModSecurity was removed from the image. It hasn't recurred.

What we were running

Internet
  → Application Load Balancer (TLS)
  → ECS service: Nginx + ModSecurity (libmodsecurity3 + OWASP CRS)
  → app container

ModSecurity was compiled into the Nginx reverse proxy that sat in front of the app, with the OWASP Core Rule Set (CRS) loaded from the image. Every request passed through it. That's what made it valuable as a filter, and what made it a single point of failure.

Timeline

• Morning: the API is unreachable. ECS reports every task as running.
• Triage, outside in: a curl -Iv against the API shows the TLS handshake completing with a valid certificate, which rules out the classic silent killer, an expired certificate, and proves the load balancer is answering. The failure is behind it.
• The clue: the proxy container's logs show ModSecurity asking to allocate malloc(97757110034913), roughly 97 TB. No request is that large. That's a corrupted size, and it points at the firewall rather than the app.
• Mitigation: two lines in the Nginx config are commented out to switch ModSecurity off. The API comes straight back.
• Investigation: the Dockerfile clones the CRS from its main branch with no tag, and a new CRS release landed that week. Hypothesis one: the rules changed underneath us.
• Root cause: the evidence doesn't hold up for hypothesis one. The allocation value is the same every time, which is a deterministic memory corruption in the library, not a misbehaving rule. libmodsecurity3 on Ubuntu 24.04 is the fault.
• Same day: an AWS WAF web ACL is created for staging in Count mode, with logging to CloudWatch, and leadership gets a plain-language incident email.
• After 48 hours: with staging traffic reviewed, production gets its own web ACL, rules move from Count to Block, and ModSecurity is removed from the image.

# The whole mitigation: switch the engine off and reload.
# modsecurity on;
# modsecurity_rules_file /etc/nginx/modsec/main.conf;

“Nothing changed”

The first thing everyone said, me included, was that nothing had changed. It was true of our code and false of our system. Two parts of the security layer could move without a commit from us: the rule set, cloned from whatever main was at build time, and the engine, running on an OS release it couldn't handle. Container images feel immutable, but an image built from moving parts is only as reproducible as its least-pinned input.

The unpinned rule set turned out not to be the cause, and it's worth saying so plainly, because the version of this story that blames the CRS update is neater and wrong. But it was the same class of risk, and it would have bitten us eventually. Pinning it was never optional.

# Pin the base image by digest, not just by tag
FROM ubuntu:22.04@sha256:<digest-you-tested>

# Pin the engine to a version you tested on that base
RUN apt-get update && apt-get install -y --no-install-recommends \
      libmodsecurity3=<tested-version> && rm -rf /var/lib/apt/lists/*

# Pin the rule set to an exact tag; bump it in a reviewed pull request
ARG CRS_VERSION=vX.Y.Z
RUN git clone --branch ${CRS_VERSION} --depth 1 \
      https://github.com/coreruleset/coreruleset.git /usr/share/modsecurity-crs

Why we moved to AWS WAF instead of fixing ModSecurity

We could have pinned the base image, found a compatible build of the library, pinned the CRS and switched ModSecurity back on. We chose not to, for four reasons:

• We didn't want to own a C library in the request path. A small team shipping a remittance product shouldn't be debugging memory corruption in its firewall.
• A managed firewall fails outside our containers. If AWS WAF has a bad day, it isn't our Nginx process that crashes.
• It covers the whole load balancer, not one container image, so every service behind it is protected the same way.
• It comes with the visibility we lacked: metrics, sampled requests and logs in CloudWatch, which also give us audit evidence for our compliance programme.

The trade-offs are real and we accepted them knowingly: managed rules are less tunable than your own, AWS can update a rule group's behaviour unless you pin a static version, and the managed rule sets can false-positive on fintech payloads. Base64-encoded KYC uploads and free-text names or addresses are classic triggers for SQL injection rules. The cost was a few dollars a month per web ACL plus per-rule and per-request charges, which is nothing next to one more morning like that one.

How we rolled it out

• One web ACL per environment. Sharing one between staging and production means a rule you loosen or an IP you block while debugging staging changes production too, and on shared mobile-network IPs that can lock out real customers.
• Managed rule groups: the AWS core rule set, known bad inputs, SQL injection and the Amazon IP reputation list, plus rate limits per IP. Writes got more headroom than reads, because mobile users on flaky connections retry.
• Count before Block. Every rule started in Count mode on staging. We ran the full customer flow (sign-up, KYC upload, transfer, status checks) and read the sampled requests for 48 hours before anything was allowed to block.
• Logs that survive a bad day: a CloudWatch log group with the aws-waf-logs- prefix AWS requires, 30-day retention and deletion protection on, because firewall logs are evidence.
• Then production, then ModSecurity removed from the Dockerfile in its own pull request, so we weren't carrying dead, broken code in the image.

We built it in the console under pressure. Here's the same shape as Terraform, so the next environment is a review rather than a repeat:

resource "aws_wafv2_web_acl" "api" {
  name  = "api-prod"
  scope = "REGIONAL" # regional web ACLs attach to load balancers

  default_action {
    allow {}
  }

  rule {
    name     = "aws-core-rule-set"
    priority = 1

    override_action {
      count {} # validate first; switch to none {} to let the group block
    }

    statement {
      managed_rule_group_statement {
        vendor_name = "AWS"
        name        = "AWSManagedRulesCommonRuleSet"
        version     = "<static-version>" # pin it; leave unset to take AWS's default version
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "aws-core-rule-set"
      sampled_requests_enabled   = true
    }
  }

  # ...known bad inputs, SQL injection, IP reputation and rate limits follow the same pattern

  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "api-prod"
    sampled_requests_enabled   = true
  }
}

resource "aws_cloudwatch_log_group" "waf" {
  name              = "aws-waf-logs-api-prod" # the prefix is required
  retention_in_days = 30
}

resource "aws_wafv2_web_acl_logging_configuration" "api" {
  resource_arn            = aws_wafv2_web_acl.api.arn
  log_destination_configs = [aws_cloudwatch_log_group.waf.arn]
}

resource "aws_wafv2_web_acl_association" "alb" {
  resource_arn = aws_lb.api.arn
  web_acl_arn  = aws_wafv2_web_acl.api.arn
}

The detection gap

The uncomfortable part: we found this by trying to use the product. No alarm fired. Every ECS task was running, which is exactly what a container health view will tell you while a proxy inside it is failing every request. Health that matters is measured from where the customer stands.

• Alarm on the load balancer's own 5xx count and on healthy target count, not just on whether tasks are running.
• Run an external uptime check against a real API path, and page a person when it fails.
• Ship the proxy's error logs somewhere searchable, so a clue like that 97 TB allocation is one query away.
• Export database logs to CloudWatch too. We did it as part of the same clean-up, so the next investigation starts with more evidence.

Telling leadership

The executive update was short: what happened, in plain language; that it was not caused by a release or by anyone's change, stated early, because the first question is always who broke it; a one-line explanation of what a web application firewall does; and a numbered list of what was fixed and what would change next. No mallocs, no library names. The technical detail lived in the internal post-mortem, and this is the public version of it.

What I'd tell anyone running ModSecurity today

• Pin the base image by digest, the engine by version and the CRS by tag, and bump each one deliberately in a reviewed pull request.
• Test every upgrade against real traffic on staging before it reaches production.
• Know your kill switch, write it down, and make sure someone other than its author can use it in minutes.
• Alarm on what customers experience, not on whether containers are running.
• If you don't have the capacity to own a firewall engine, don't. A managed WAF at the edge, validated in Count mode and pinned to static rule versions, is a perfectly respectable answer.

Security tooling is still software. If it can change underneath you, it can take you down.