The Mitigation Dilemma for Proxy Operators
When a DDoS attack hits a web server, the mitigation playbook is relatively straightforward. Rate-limit by source IP, challenge suspicious requests with CAPTCHAs, drop traffic on unused ports, and let a scrubbing center separate attack traffic from legitimate visitors. The scrubbing center knows what legitimate HTTP traffic looks like and can make accurate filtering decisions.
Proxy gateways break every part of this playbook. Legitimate traffic arrives from thousands of different customer IPs, many of which have never connected before. Rate limiting by source IP penalizes customers who send high volumes of legitimate proxy requests. The gateway must accept connections on its service ports from any authenticated source, so port-based blocking is off the table. And scrubbing centers designed for HTTP inspection cannot meaningfully process SOCKS5 tunnels or TCP CONNECT sessions.
The result is that many proxy providers either accept periodic downtime during attacks or deploy mitigation rules so aggressive that they degrade service quality for paying customers. Neither is sustainable. This guide covers the techniques that actually work.
Kernel-Level Filtering vs. Application-Level
The first decision in proxy gateway defense is where in the network stack to apply filtering. The two options have dramatically different performance characteristics.
Application-Level Filtering
Application-level filtering processes packets in userspace, typically through the proxy application itself or a reverse proxy sitting in front of it. This approach has the advantage of full protocol context: the filter knows whether a connection is authenticated, what URLs are being requested, and whether the session behavior matches normal customer patterns.
The disadvantage is throughput. Userspace packet processing on a typical proxy gateway tops out at roughly 200,000 to 500,000 packets per second before CPU saturation causes latency spikes. A moderately sized UDP flood can push 2 to 5 million packets per second. By the time attack packets reach the application layer, the damage is already done.
Kernel-Level Filtering
Kernel-level filtering through iptables, nftables, or XDP/eBPF processes packets before they reach the application. XDP in particular can process packets at the network driver level, achieving throughput of 10 million or more packets per second on commodity hardware. This is where volumetric attacks must be handled.
The tradeoff is reduced context. At the kernel level, the filter sees raw packet headers but not application-layer session state. It can filter by source IP, destination port, packet size, protocol flags, and rate, but cannot distinguish an authenticated customer connection from an unauthenticated one.
The Layered Approach
Effective proxy gateway defense uses both layers. Kernel-level rules handle volumetric attacks by dropping traffic that matches attack signatures (specific source subnets, abnormal packet sizes, non-TCP protocols on proxy ports). Application-level filtering handles the residual traffic that passes kernel rules, identifying and throttling connections that are authenticated but behaving abnormally.
# Kernel-level: drop UDP floods targeting SOCKS5 port iptables -A INPUT -p udp --dport 1080 -j DROP # Kernel-level: rate limit new TCP connections per source iptables -A INPUT -p tcp --dport 1080 --syn \ -m connlimit --connlimit-above 50 -j DROP # Kernel-level: drop oversized packets (likely amplification) iptables -A INPUT -p udp -m length --length 1400:65535 -j DROP # Application-level: handled by proxy auth + session monitoring
Port-Specific Detection
A proxy gateway typically exposes three to five ports: SOCKS5 (1080), HTTP CONNECT (3128 or 8080), an API port for customer management, and possibly a health check endpoint. Each port has its own traffic profile, and each requires its own detection baseline.
Consider a gateway where port 1080 normally handles 30,000 connections per second and port 3128 handles 8,000. An attack that pushes port 3128 to 25,000 connections per second is clearly anomalous for that port, even though 25,000 is below the normal rate for port 1080. A single aggregate baseline across all ports would miss this because the total connection rate (55,000) might be within normal bounds for the gateway overall.
Flowtriq builds independent baselines for each monitored port on each node. This per-port granularity catches targeted attacks that aggregate monitoring overlooks. When the baseline for port 3128 detects a deviation, the auto-mitigation response targets only that port, leaving port 1080 traffic completely unaffected.
Preserving IP Reputation During Attacks
For proxy providers, IP reputation is inventory. The value of the business is directly tied to the quality and cleanliness of its IP pool. A DDoS attack threatens IP reputation in several ways:
- Upstream null-routing: If the attack volume exceeds what the provider's upstream can absorb, the transit provider may null-route the attacked IP prefix. Even after the attack ends, the null-route may persist for hours while the upstream evaluates the situation.
- Blocklist entries: High-volume outbound traffic from the provider's IPs (caused by reflection attacks using spoofed source addresses) triggers entries on Spamhaus, AbuseIPDB, and similar blocklists. These entries take days or weeks to clear.
- BGP instability: Repeated attacks and mitigations can cause BGP route flapping, which degrades reachability across the internet and further damages IP reputation.
The defense against reputation damage requires fast detection and equally fast upstream communication. If the provider can detect an attack within 30 seconds and push a BGP FlowSpec rule to its upstream within another 30 seconds, the attack traffic gets filtered at the upstream edge before it reaches the provider's network. The gateway IP never gets overwhelmed, the upstream never considers null-routing, and no blocklist entry is generated.
Auto-Mitigation: From Detection to Rule Deployment
Manual mitigation cannot keep pace with attacks on proxy infrastructure. The time from detection to mitigation must be measured in seconds, not minutes. Here is the pipeline that works:
- Detection (0-30 seconds): The Flowtriq agent on the gateway node detects a traffic anomaly against the per-port baseline. The alert includes attack characteristics: source subnets, protocol, packet sizes, and rate.
- Rule generation (immediate): Based on the attack fingerprint, the system generates targeted mitigation rules. For a UDP flood on port 1080, this might be an iptables rule dropping UDP on that port from the top contributing source subnets. For a SYN flood, it might be a connlimit rule capping new connections per source.
- Local deployment (immediate): Rules are applied to the local kernel firewall on the attacked node. This stops the attack from consuming application resources.
- Upstream propagation (0-60 seconds): For volumetric attacks that threaten to saturate the inbound link, the system pushes BGP FlowSpec rules to the upstream router. This moves the filtering upstream of the provider's network, protecting the link itself.
- Monitoring and adjustment: The agent continues monitoring. If the attack vector changes (switches from UDP to SYN flood), the system generates new rules targeting the updated attack profile. Old rules are automatically expired when the attack subsides.
Flowtriq's iptables rule generator can also produce rules manually for operators who want to pre-stage defenses for specific attack patterns they have encountered before.
BGP FlowSpec for Upstream Filtering
BGP FlowSpec is the mechanism that makes upstream filtering possible without a phone call to your transit provider's NOC. It allows the provider to announce traffic filtering rules as BGP updates, which the upstream router applies at its edge. This is the difference between filtering attack traffic at your gateway (where your inbound link may already be saturated) and filtering it three hops upstream (where the upstream's multi-terabit backbone can absorb it without impact).
For proxy providers, FlowSpec rules need to be precise. A rule that drops all UDP traffic to the gateway prefix will also drop legitimate DNS responses and other UDP-based protocols that the proxy network depends on. Effective FlowSpec rules combine destination prefix, destination port, source prefix, protocol, packet length, and TCP flags to target specifically the attack traffic.
The key metric for proxy providers is not time-to-detect. It is time-to-upstream-filter. Every second between detection and FlowSpec deployment is a second where the attack is consuming your inbound link capacity and threatening IP reputation.
Practical Checklist for Proxy Gateway Defense
- Deploy per-node, per-port monitoring on every gateway. Aggregate network monitoring is too coarse to catch port-targeted attacks.
- Layer kernel and application filtering. Use XDP or iptables for volumetric attack handling and application-level logic for abuse detection on authenticated sessions.
- Pre-negotiate BGP FlowSpec with your upstream transit providers. When the attack arrives is not the time to discover your upstream does not support FlowSpec.
- Separate gateway IPs from pool IPs. Ensure that an attack on a gateway IP does not cause collateral damage to the IP pool that customers are using for outbound traffic.
- Automate mitigation end-to-end. Manual rule creation during an attack means minutes of downtime for every customer connected to the targeted gateway.
- Monitor IP reputation proactively. Track blocklist status for your gateway and pool IPs continuously so that reputation damage from reflected traffic is caught and remediated before customers report connectivity issues.
Detect and mitigate in seconds, not minutes. Flowtriq's agent deploys on your gateway nodes, builds per-port baselines, and auto-generates mitigation rules the moment an attack begins. See the proxy provider solution or generate rules now with the iptables generator.