Building a 34-Rule WAF: From 15 Signatures to Full OWASP Coverage

When we first deployed our WAF, it had 15 rules — enough to catch the most obvious SQL injection attempts and a few known scanner paths. Within a week of production traffic analysis, we realized that attackers were sailing through with double-encoded payloads, disguised bot user agents, and path traversal attempts that our rules simply didn't cover. We needed a complete overhaul.

This post covers how we expanded from 15 basic signatures to 34 detection rules across 9 OWASP categories, added multi-layer URL decoding, built an anomaly scoring system, and deployed honeypot traps that permanently ban automated scanners.

WHY 15 RULES WASN'T ENOUGH

Our original ruleset covered three categories: SQL injection (5 rules), XSS (4 rules), and known scanner paths (6 rules). The gaps were glaring:

• No Remote Code Execution detection — attackers could attempt command injection via parameters
• No file inclusion protection — ../../etc/passwd attempts passed through
• No protocol violation detection — malformed requests weren't flagged
• No bot detection — headless browsers and automated scanners looked like normal traffic
• No encoding evasion prevention — double-encoding bypassed every rule

We analyzed 30 days of Nginx access logs (2.1 million requests) and categorized every blocked and unblocked suspicious request. The result was a comprehensive rule taxonomy based on OWASP attack categories.

THE 9 OWASP CATEGORIES

We organized all 34 rules into 9 categories, each targeting a specific attack surface:

MULTI-LAYER URL DECODING: deepDecode()

The most critical improvement was handling encoding evasion. Attackers commonly double- or triple-encode payloads to bypass signature-based detection. A simple %2527 (double-encoded single quote) would sail through a rule looking for ' because the WAF only decoded once.

Our deepDecode() function recursively decodes URLs up to 3 layers deep:

/**
 * Multi-layer URL decoder to prevent double/triple encoding evasion.
 * Decodes recursively until the string stabilizes or maxDepth is reached.
 */
function deepDecode(input: string, maxDepth: number = 3): string {
  let decoded = input;
  for (let i = 0; i < maxDepth; i++) {
    try {
      const next = decodeURIComponent(decoded);
      if (next === decoded) break; // Stable — no more encoding layers
      decoded = next;
    } catch {
      break; // Invalid encoding — stop decoding
    }
  }
  return decoded;
}

// Example:
// Input:  "%25253Cscript%25253E"  (triple-encoded <script>)
// Pass 1: "%253Cscript%253E"
// Pass 2: "%3Cscript%3E"
// Pass 3: "<script>"           <-- Now the XSS rule catches it

Every incoming request URL, query string, and Referer header is passed through deepDecode() before rule matching. This single function eliminated an entire class of evasion techniques.

ANOMALY SCORING SYSTEM

Not every suspicious request warrants an immediate ban. A single probe attempt might be a curious developer, while a sequence of injection attempts is clearly an attack. We implemented anomaly scoring to accumulate threat signals per IP:

// Rule severity determines score contribution
const RULE_SEVERITY: Record<string, number> = {
  // Critical: instant ban (score >= 100)
  "rce-jndi-lookup":       100,
  "rce-shell-injection":   100,
  "injection-union-select": 100,

  // High: two hits trigger a ban (50-60 each)
  "xss-script-tag":         60,
  "xss-event-handler":      50,
  "injection-sqli-basic":   50,
  "file-inclusion-traversal": 50,

  // Medium: needs 2-3 hits (35-40 each)
  "probe-wordpress":        40,
  "probe-env-file":         40,
  "bot-scanner-ua":         35,

  // Low: noise accumulator (10-20 each)
  "probe-common-paths":     20,
  "rate-burst-requests":    15,
  "protocol-null-byte":     10,
};

// Ban threshold: accumulated score >= 50 triggers a ban
const BAN_THRESHOLD = 50;

When a request triggers a rule, the IP's score is incremented by the rule's severity value. Critical rules (score 100) cause an instant ban. High-severity rules need just one more violation. Low-severity rules need several accumulations before reaching the threshold.

IP REPUTATION TRACKING

To persist reputation data across WAF scan cycles, we built an IP reputation system with two layers:

• Database: waf_ip_reputation table stores long-term threat data — total score, first seen, last seen, ban status, top rule triggered
• Redis cache: waf:rep:<ip> keys with 7-day TTL for fast lookup during request processing

// src/lib/server/ip-reputation.ts
export async function updateIPReputation(
  ip: string,
  ruleId: string,
  severity: number
): Promise<void> {
  // Update database record
  await prisma.waf_ip_reputation.upsert({
    where: { ipAddress: ip },
    update: {
      totalScore: { increment: severity },
      hitCount: { increment: 1 },
      lastSeenAt: new Date(),
      topRule: ruleId,
    },
    create: {
      id: crypto.randomUUID(),
      ipAddress: ip,
      totalScore: severity,
      hitCount: 1,
      topRule: ruleId,
      firstSeenAt: new Date(),
      lastSeenAt: new Date(),
    },
  });

  // Update Redis cache for fast path
  const cacheKey = `waf:rep:${ip}`;
  await redis.hincrby(cacheKey, "score", severity);
  await redis.expire(cacheKey, 604800); // 7-day TTL
}

The WAF cron job runs every 2 minutes (*/2 * * * *), parsing new Nginx log entries, matching them against all 34 rules, and updating IP reputation in both the database and Redis.

HONEYPOT RULES: THE SCANNER TRAP

Honeypot rules are our favorite addition. These are decoy paths that legitimate users will never visit, but automated scanners always check:

• /wp-admin and /wp-login.php — WordPress admin (we don't run WordPress)
• /phpmyadmin and /adminer.php — database management tools (not installed)
• /.env and /.git/config — configuration file probes

Any request to a honeypot path triggers an immediate 365-day ban. There is zero legitimate reason for a visitor to request /wp-admin on a Next.js platform. In the first week after deployment, honeypot rules caught 847 unique IPs — all automated scanners.

THREAT FEED INTEGRATION

Beyond reactive detection, we proactively block known-bad IPs using threat intelligence feeds. We expanded from 7 to 11 threat feeds:

• Spamhaus EDROP — hijacked IP ranges used for spam and attacks
• IPsum Level 5 — IPs flagged by 5+ independent blacklists
• Tor Exit Nodes — anonymous network exit points (common attack source)
• Firehol Level 1 — aggregated threat intelligence from multiple sources
• Plus 7 existing feeds (AbuseIPDB, Spamhaus DROP, Emerging Threats, etc.)

The seedDefaultSources() function was fixed to upsert by name instead of skipping when any source exists, ensuring new feeds are always added during updates.

WAF UI OVERHAUL

The admin WAF dashboard was completely rebuilt to surface actionable intelligence:

• 4 stat cards at the top: total attacks (24h), unique attacker IPs, rules triggered, IPs currently banned
• 24-hour attack timeline — hour-by-hour bar chart showing attack volume patterns
• Categorized collapsible rule sections — rules grouped by OWASP category with toggle for enable/disable
• IP intelligence modal — click any IP to see full reputation history, all rules triggered, geographic location, and manual ban/unban controls

RESULTS AFTER 30 DAYS

• 34 active rules across 9 OWASP categories (up from 15 in 3 categories)
• 2,341 unique IPs banned — 847 by honeypot, 1,494 by anomaly scoring
• Zero false positives on legitimate traffic (honeypots are zero-risk by design)
• deepDecode() catches 12% more attacks that previously bypassed single-decode matching
• Anomaly scoring reduced noise by 60% — low-severity probes accumulate silently instead of generating alerts
• 11 threat feeds blocking 23,000+ known-bad IP ranges preemptively

The WAF is never "done" — new attack patterns emerge constantly. But the architecture we built — category-based rules, multi-layer decoding, anomaly scoring, and honeypot traps — gives us a framework that's easy to extend. Adding a new rule is a single object in the rules array. The scoring, reputation tracking, and ban logic handle the rest automatically.