Install ModSecurity WAF on Nginx VPS

June 2026 · 9 min read

ModSecurity is the most powerful Web Application Firewall (WAF) for Nginx — protecting against SQL injection, XSS, Remote Code Execution, and other web attacks using the OWASP Core Rule Set (CRS) maintained by the OWASP team and regularly updated. This tutorial shows how to compile Nginx with ModSecurity module, install OWASP CRS, configure it in two modes (detection and prevention), and test blocking of SQL injection and XSS attacks.

Understanding WAF vs Network Firewall

A network firewall inspects at layers 3/4 (IP, port), while a WAF inspects at layer 7 (HTTP request body) — it can detect SQL injection statements by reading and parsing HTTP content. ModSecurity operates in two modes:

Detection mode (logging only): Does not block; only records warning logs and allows the request through — used for testing and tuning rules.
Prevention mode (blocking): Returns HTTP 403 Forbidden when malicious requests are detected — used when you're confident rules don't block legitimate traffic.

Install Nginx with ModSecurity Module

Unlike Apache, Nginx doesn't support dynamic modules for ModSecurity, so you must compile Nginx with the ModSecurity module:

# Update packages
apt update && apt upgrade -y

# Install build dependencies
apt install -y build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev libgd-dev libgeoip-dev libjemalloc-dev wget curl git

# Download Nginx source
cd /tmp
wget http://nginx.org/download/nginx-1.26.0.tar.gz
tar xzf nginx-1.26.0.tar.gz

# Download ModSecurity connector for Nginx
git clone --depth 1 https://github.com/SpiderLabs/ModSecurity-nginx.git

# Download ModSecurity standalone library
git clone --depth 1 https://github.com/SpiderLabs/ModSecurity.git

# Build ModSecurity library
cd ModSecurity
./build.sh
./configure --prefix=/usr/local/modsecurity
make && make install

# Verify ModSecurity library installation
ls -la /usr/local/modsecurity/lib/

# Compile Nginx with ModSecurity module
cd /tmp/nginx-1.26.0
./configure \
  --prefix=/etc/nginx \
  --sbin-path=/usr/sbin/nginx \
  --modules-path=/usr/lib64/nginx/modules \
  --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log \
  --pid-path=/var/run/nginx.pid \
  --lock-path=/var/run/nginx.lock \
  --with-http_ssl_module \
  --with-http_v2_module \
  --with-http_realip_module \
  --with-http_gzip_static_module \
  --with-http_secure_link_module \
  --with-http_auth_request_module \
  --add-dynamic-module=/tmp/ModSecurity-nginx

make && make install

# Verify Nginx build
nginx -v
nginx -t

Install OWASP Core Rule Set (CRS)

CRS is OWASP's rule set for detecting common attacks: SQL injection, XSS, Remote File Inclusion, etc. It's highly customizable:

# Download OWASP CRS
cd /opt
git clone https://github.com/coreruleset/coreruleset.git crs

# Navigate to CRS
cd crs

# Copy configuration template
cp crs-setup.conf.example crs-setup.conf

# Copy active rules
cp rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf.example rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
cp rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf.example rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

# Verify CRS installation
ls -la /opt/crs/rules/ | head -20

Configure ModSecurity in Nginx

Edit nginx.conf to add ModSecurity directives:

sudo nano /etc/nginx/nginx.conf

# In the http block, add (before "include /etc/nginx/sites-enabled/*")
load_module modules/ngx_http_modsecurity_module.so;

http {
    # ... other config ...

    # ModSecurity setup
    modsecurity on;
    modsecurity_rules_file /opt/crs/crs-setup.conf;
    modsecurity_rules_file /opt/crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf;
    modsecurity_rules_file /opt/crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf;

    # Include all .conf files in rules/
    include /opt/crs/rules/*.conf;

    # ... other server blocks ...
    include /etc/nginx/sites-enabled/*;
}

Create a main ModSecurity configuration file:

sudo cat > /opt/crs/modsecurity.conf << 'EOF'
# ModSecurity setup (default detection mode)
SecRuleEngine DetectionOnly

# Configure audit log
SecAuditEngine On
SecAuditLog /var/log/modsecurity/audit.log
SecAuditLogFormat JSON
SecAuditLogType Concurrent

# Set debug log
SecDebugLog /var/log/modsecurity/debug.log
SecDebugLogLevel 3

# Max HTTP body size to scan
SecRequestBodyLimit 13107200
SecRequestBodyInMemoryLimit 131072

# Max response body size to scan
SecResponseBodyLimit 1048576

# Disable for CONNECT requests (VPN/tunnel)
SecAllowList 2000 id:2000 phase:1 t:none nolog noauditlog "CONNECT" "@rx ^CONNECT$"
EOF

# Create log directory
mkdir -p /var/log/modsecurity
chmod 755 /var/log/modsecurity

# Test Nginx config
sudo nginx -t

Switch from Detection to Prevention Mode

When confident the rules don't block legitimate traffic, switch to prevention mode:

# Edit modsecurity.conf
sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' /opt/crs/modsecurity.conf

# Reload Nginx
sudo nginx -t && sudo systemctl reload nginx

# Check for errors
tail -f /var/log/nginx/error.log

Test ModSecurity Blocks SQL Injection and XSS

Create a test rule to verify ModSecurity protection:

# Create test rules temporarily
cat > /opt/crs/test-rules.conf << 'EOF'
# Simple test rules
SecRule ARGS "@rx (?i:union.*select)" "id:1000,phase:2,deny,log,msg:'SQL Injection attempt'"
SecRule ARGS "@rx (?i:alert('xss')"
# Should get HTTP 403 Forbidden

# View logs
sudo tail -f /var/log/modsecurity/audit.log

Per-Vhost ModSecurity Configuration

To enable ModSecurity only on specific vhosts:

sudo cat > /etc/nginx/sites-available/myapp << 'EOF'
server {
    listen 80;
    server_name myapp.com;

    # Enable ModSecurity for this vhost
    modsecurity on;
    modsecurity_rules_file /opt/crs/crs-setup.conf;
    modsecurity_rules_file /opt/crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf;

    location / {
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    # Exclude rules for certain endpoints
    location /api/upload {
        modsecurity_rules "SecRequestBodyLimit 52428800";  # 50 MB
        proxy_pass http://127.0.0.1:3000;
    }
}
EOF

sudo ln -s /etc/nginx/sites-available/myapp /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

Exclude Rules for False Positives

If ModSecurity blocks legitimate requests, find the rule ID and exclude it:

# Check audit log for the rule ID that blocked
sudo tail -f /var/log/modsecurity/audit.log | grep "id:"

# Example: rule 930100 triggers on PHP code in search query
# Whitelist rule 930100 for /admin path
cat >> /opt/crs/modsecurity.conf << 'EOF'

# Whitelist rules with false positives
SecRule REQUEST_URI "@beginsWith /admin/upload" \
  "id:10000,phase:1,nolog,pass,skipAfter:END_WHITELIST"

# Or disable specific rule globally
SecRuleRemoveById 930100

SecMarker "END_WHITELIST"
EOF

# Reload Nginx
sudo systemctl reload nginx

Verify ModSecurity is Working Correctly

# Test Nginx configuration
sudo nginx -t
# Should see: "nginx: configuration file /etc/nginx/nginx.conf test is successful"

# Verify module is loaded
ps aux | grep nginx

# Check ModSecurity error logs
sudo tail -20 /var/log/nginx/error.log | grep -i modsecurity

# View detailed audit log
sudo tail -50 /var/log/modsecurity/audit.log | python3 -m json.tool

# Test normal request (should be 200 OK)
curl -v "http://127.0.0.1/" --output /dev/null -s -w "%{http_code}\n"
# Should return 200 or 301/302 if redirect

# Test request with malicious pattern
curl -v "http://127.0.0.1/?id=1' or '1'='1" 2>&1 | grep -i "403\|denied"

Tip: If ModSecurity blocks legitimate traffic when switching to prevention mode, switch back to detection mode and find the false positive rule ID in audit.log, then whitelist it — don't disable WAF entirely as you'd lose protection.

Common Troubleshooting

Error: "nginx: [emerg] module not found"

The ModSecurity module didn't compile successfully or Nginx can't find the .so file

# Verify the .so file exists
ls -la /usr/lib64/nginx/modules/ngx_http_modsecurity_module.so
ls -la /etc/nginx/modules/ngx_http_modsecurity_module.so

# If file exists but path is wrong, fix nginx.conf
sudo nano /etc/nginx/nginx.conf
# Check line: load_module modules/ngx_http_modsecurity_module.so;
# With prefix /etc/nginx, modules path should be /etc/nginx/modules/

# If .so file doesn't exist, recompile Nginx
cd /tmp/nginx-1.26.0
./configure --add-dynamic-module=/tmp/ModSecurity-nginx ...
make
# Don't use make install; just copy the .so
sudo cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/

Error: "can't allocate memory" from ModSecurity

ModSecurity is scanning large requests and running out of buffer memory

# Reduce SecRequestBodyInMemoryLimit
sudo nano /opt/crs/modsecurity.conf
# Change to:
SecRequestBodyInMemoryLimit 65536     # Reduced from 131072

# Or increase VPS RAM and adjust limit higher
SecRequestBodyInMemoryLimit 262144    # 256 KB

# Restart services
sudo systemctl restart nginx

ERROR 403 Forbidden on legitimate requests

WAF is blocking legitimate traffic — whitelist those rules

# Check which rule blocked it in audit log
sudo grep "\"severity\":\"CRITICAL\"" /var/log/modsecurity/audit.log | tail -5

# Find rule ID (e.g., id:930100) and disable it
echo "SecRuleRemoveById 930100" | sudo tee -a /opt/crs/modsecurity.conf

# Or disable rule for specific path
echo "SecRule REQUEST_URI \"@beginsWith /api\" \"id:10001,phase:1,nolog,pass,skipAfter:SKIP_API\"" | sudo tee -a /opt/crs/modsecurity.conf
echo "SecMarker \"SKIP_API\"" | sudo tee -a /opt/crs/modsecurity.conf

# Reload
sudo systemctl reload nginx

# Test again
curl -v "http://127.0.0.1/path/to/resource"

ModSecurity blocks everything over HTTPS

HTTPS decryption requires trusted certificates or ModSecurity sees SSL errors

# Check error log
sudo tail -50 /var/log/nginx/error.log | grep -i ssl

# If SSL certificate is invalid, renew it
sudo certbot renew --force-renewal

# Fix proxy_pass for HTTPS backends
location / {
    proxy_pass https://backend.example.com:443;  # Use HTTPS
    proxy_ssl_verify off;                         # If self-signed
    proxy_ssl_trusted_certificate /etc/ssl/certs/ca-bundle.crt;
}

Frequently Asked Questions

How much CPU/memory does ModSecurity consume?

ModSecurity uses roughly 2-5% CPU per request when scanning large payloads; detection mode is more efficient than prevention mode since it doesn't need to re-evaluate requests. On a 2-core VPS, run detection mode on production first, then scale infrastructure if prevention mode is needed.

Does ModSecurity support HTTP/3?

The Nginx ModSecurity module is designed for HTTP/1.1 and HTTP/2 only. HTTP/3 (QUIC) is still experimental in Nginx and uses a different protocol (not TCP), which ModSecurity wasn't designed for. For HTTP/3 WAF, consider Cloudflare WAF or AWS WAF instead.

Which OWASP CRS rules should I disable if they're too strict?

Some organizations disable rules like id:942251 (PHP code in parameters) if their app legitimately sends PHP code (e.g., template engine). However, recommendation is to start with detection mode and test the default CRS first, then whitelist actual false positives rather than disabling rules blindly — disabling too many rules reduces WAF protection significantly.