How Detection Works

The full pipeline from raw packet to confirmed incident

Detection Pipeline Overview

Every second, the Flowtriq agent executes one tick. Each tick reads traffic counters, feeds them into the baseline, evaluates threshold conditions, and decides whether an attack is starting, continuing, or ending. The pipeline has five stages:

1. Traffic sampling - Read PPS (packets per second) and BPS (bytes per second) from /proc/net/dev, flow collector, or Scapy ring buffer
2. Baseline feeding - Add the sample to the sliding window (only during clean traffic, never during attacks)
3. Threshold evaluation - Compare current PPS/BPS against the dynamic threshold
4. Sustained confirmation - Require 3 consecutive ticks above threshold before opening an incident
5. Incident lifecycle - Open, update (every 5s), or resolve the incident via the Flowtriq API

PPS vs BPS Detection

The engine runs two independent detection paths. An attack is flagged if either one triggers.

BPS-based detection only fires after the baseline is ready. This prevents false positives on nodes with naturally high bandwidth (backup servers, CDN edges).

Baseline Calculation

The baseline is a sliding window of the most recent 300 PPS/BPS samples (configurable via baseline_window in /etc/ftagent/config.json). Percentiles are recalculated every 10 ticks using a full sort of the window.

Sliding Window Mechanics

Time-of-Day Awareness

The baseline also maintains per-hour buckets (24 deques, one for each hour of the day). Once the current hour has at least 60 samples, the engine uses that hour's p99 instead of the flat p99. This prevents false positives from predictable daily traffic patterns (morning ramp-up, evening peak, overnight lull).

The hourly p99 is clamped to never drop below 50% of the flat p99, which prevents dangerously low thresholds during quiet hours.

Baseline Persistence

When the agent shuts down (service restart, reboot), it serializes the baseline state to /var/lib/ftagent/baseline.json. On next startup, it restores from disk if the file contains at least 50 valid samples. This means a systemctl restart ftagent does not lose 5 minutes of learned traffic history.

Attack Traffic Exclusion

Attack traffic is never fed into the baseline. This prevents threshold inflation where a long attack gradually raises the baseline, making it harder to detect the next attack. Specifically, samples are skipped when:

• An attack is currently active (self.attacking == True)
• The sustained-confirmation counter is above zero (traffic is being evaluated for attack)
• The baseline is not ready AND PPS exceeds the absolute floor (a new node under attack should not build a baseline from attack data)

Startup Grace Period

For the first 90 seconds after the agent starts, attack detection is suppressed. This prevents false positives caused by legitimate startup spikes:

• Cache warming (Redis, Memcached, Varnish)
• Log rotation (rsyslog, logrotate)
• Backup flush (rsync, Borg, Restic)
• Service reconnection storms

The only exception: if PPS exceeds 5x the absolute floor (50,000 PPS by default), the agent fires immediately. Traffic at that level is unambiguously an attack regardless of startup conditions.

Sustained Confirmation

When PPS first crosses the threshold, the agent does not immediately open an incident. Instead, it requires 3 consecutive ticks (3 seconds) above the threshold before confirming an attack. This filters out:

• VoIP registration storms (1-2 second bursts)
• CDN cache refills (brief spikes when upstream pulls content)
• Game server heartbeat aggregation (periodic bursts from player sync)
• BGP reconvergence (short-lived traffic rerouting)

The exception is the same as the grace period: traffic above 5x the floor triggers immediately without waiting for sustained confirmation.

Cooldown Between Incidents

After an attack resolves, the engine enters a 60-second cooldown where re-detection is suppressed. This prevents rapid-fire micro-burst incidents (10s attack, 10s calm, 10s attack) that cause alert fatigue. The attack must sustain past the cooldown to open a new incident.

As with other safeguards, traffic above 5x the floor bypasses the cooldown immediately.

Attack Resolution

An active attack resolves when PPS drops below the threshold for 10 consecutive ticks (10 seconds). This hysteresis prevents premature resolution during brief lulls in multi-wave attacks.

Multi-Vector Detection

Multi-vector attacks (e.g. SYN flood + DNS amplification + ICMP flood arriving simultaneously) are handled within a single incident. The agent does not open separate incidents per vector. Instead:

• The initial classification is computed from protocol breakdown at attack start
• Every 5-second update re-evaluates the classification using a voting system
• The final classification at attack end reflects the dominant protocol and flag patterns across the full duration
• If no single protocol exceeds 65% of traffic, the family is classified as multi_vector

Spoofing Detection via TTL Analysis

The agent tracks TTL values from every sampled packet and computes TTL entropy (Shannon entropy over the TTL distribution). Spoofed attacks produce high TTL entropy because randomly forged source IPs traverse different network paths with different hop counts.

Botnet Detection via Source IP Counting

Botnet detection uses two signals:

• Source IP count: More than 5,000 unique source IPs in the attack window (raised from 300 to avoid flagging legitimate flash crowds from viral events or product launches)
• IOC pattern matching: Any packet payload matching a known botnet family signature (Mirai, Gafgyt, Mozi, XorDDoS, Muhstik, Tsunami, Hajime, Meris, Mantis, Fodcha)

Source IP cardinality is tracked using a HyperLogLog estimator for memory efficiency. The detailed IP table is capped at 100,000 entries with LRU eviction of one-hit-wonder IPs.

IOC Matching and Payload Signatures

The agent supports two layers of payload inspection:

IOC Patterns (Threat Intelligence)

IOC patterns are pushed from the Flowtriq API to the agent during the config sync cycle. Each pattern is a byte string matched against raw packet payloads. When a match is found, the agent records the attack name and family (e.g. mirai_variant:mirai). Up to 5,000 IOC hits are stored per incident.

Amplification Payload Signatures

The agent recognizes amplification responses by their first bytes:

When at least 5 packets match a signature, the agent overrides the attack subtype classification to the specific amplification vector.

Server-Side Threshold Override

The dashboard can push a custom threshold to any node, overriding the dynamic baseline. When server_threshold is set, it replaces baseline.threshold for all detection decisions. This is useful for nodes where you know exactly what traffic level constitutes an anomaly.

Threat Intel Threshold Adjustment

If more than 30% of source IPs in the current traffic match the threat intelligence blocklist, the effective threshold is reduced to 70% of normal. This means attacks from known-bad sources are caught earlier, even if they have not yet reached the full baseline threshold.

Velocity Detection (Opt-In)

An optional detection mode that triggers on extreme rate-of-change rather than absolute threshold. Enabled via "velocity_detection": true in the config. Fires when:

• PPS delta between ticks exceeds 5x the average PPS, AND
• Current PPS exceeds 2x the average PPS

Disabled by default because it can produce false positives on bursty workloads (game servers, streaming events).

FAQ: Tuning Detection

How do I reduce false positives?

• Wait 24 hours after deploying the agent. Most false positives resolve as the baseline learns your traffic patterns.
• Increase the multiplier from 3x to 4x or 5x in Dashboard → Nodes → [Node] → Detection Settings.
• Set a static threshold if you know your maximum legitimate PPS. Switch the detection mode to "Static" and enter the value.
• Schedule maintenance windows around known traffic events (backups, deployments, game launches) to suppress alerting.

How do I increase sensitivity (catch smaller attacks)?

• Lower the multiplier from 3x to 2x. This makes the threshold tighter.
• Reduce baseline_window from 300 to 120 in config.json. A shorter window reacts faster to traffic changes but is more sensitive to noise.
• Enable velocity detection to catch rapid ramp-up attacks that have not yet crossed the absolute threshold.

Why did the agent not detect a known attack?

• The attack PPS may have been below the threshold. Check the incident timeline vs the node's threshold at that time.
• The agent may have been in the startup grace period (first 90 seconds).
• The attack may have been a 1-2 second microburst that did not pass sustained confirmation (3 consecutive ticks).
• The agent may have been in post-attack cooldown (60 seconds after the previous attack resolved).