1. Incident Overview

Canonical's Ubuntu infrastructure — including launchpad.net, package mirrors, and the Ubuntu SSO portal — sustained a sustained volumetric DDoS attack estimated between 300 and 600 Gbps. The attack resulted in widespread service degradation and outages affecting developers globally who depend on Ubuntu's package repositories, bug tracker, and CI/CD pipelines. Canonical operates a distributed CDN-backed infrastructure with upstream transit providers, but the sheer volume of inbound traffic overwhelmed ingress capacity at several peering points before upstream scrubbing could be fully engaged. Time-to-impact from first packet burst to confirmed service degradation was under 90 seconds — a timeframe where automated detection and mitigation pipelines are the only viable defense.

This post-mortem reconstructs the probable attack vectors, models the detection logic Flowtriq would apply, and documents the mitigation ladder — from local iptables drops through BGP FlowSpec injection and RTBH escalation — that would contain an event of this magnitude.

2. Attack Analysis

Estimated Volume: 300–600 Gbps peak / 150–400 Mpps at traffic apex
Duration: Multi-wave, estimated 2–6 hours with pulse intervals
Target ASN: AS41231 (Canonical Group Limited)
Primary target surfaces: launchpad.net (TCP/443, TCP/80), archive.ubuntu.com mirrors (TCP/80/443, UDP/53), Ubuntu SSO (TCP/443)

At this volume and the reported service profile, the attack almost certainly consisted of multiple concurrent vectors — a hallmark of modern booter/stresser-as-a-service campaigns and botnet-driven assaults. The following vectors are assessed with high confidence:

Vector 1 — UDP Amplification (DNS/NTP/CLDAP): The backbone of the volumetric component. DNS amplification yields 28–54x amplification ratios; NTP monlist yields up to 556x. At 600 Gbps peak, the attacker needed only 1.1–21 Gbps of spoofed source traffic to generate the observed volume. Packet sizes in DNS amplification range from 512–4096 bytes (EDNS0 ANY responses); NTP monlist responses are 468 bytes per packet. Source ports: UDP/53 (DNS), UDP/123 (NTP), UDP/389 (CLDAP returning ~1–10 KB LDAP rootDSE responses).

Vector 2 — TCP SYN Flood: Targeting the stateful load balancers and application firewalls fronting launchpad.net. SYN packets are 40–60 bytes, source ports randomized 1024–65535, destination TCP/443 and TCP/80. At 400 Mpps, this exhausts connection tables on even high-end stateful appliances in seconds.

Vector 3 — HTTP/S Application Layer (L7) Flood: Smaller volumetrically but targeted at Launchpad's Python/Django application stack. GET floods to /ubuntu/+source/ and /bugs/ endpoints — pages with non-trivial DB query costs. TLS handshake exhaustion was likely layered here, forcing full asymmetric crypto negotiation per connection.

Vector 4 — ICMP Flood: Padding traffic likely used to saturate remaining bandwidth headroom. ICMP echo-request packets, 1400-byte payload, fragmented to maximize packet-per-second load on forwarding hardware.

Botnet composition: High confidence mix of compromised SOHO routers (Mirai/Mirai-variant), open resolvers, and NTP reflectors. Global source distribution across 40+ ASNs — no single upstream null-route would have been effective without significant collateral damage to legitimate traffic.

3. Detection

Flowtriq operates at the packet level, ingesting raw flow telemetry (NetFlow v9/IPFIX/sFlow) from border routers and performing statistical baseline deviation analysis in sub-5-second windows. For an event at this scale, detection triggers would fire across multiple independent signal channels simultaneously — typically within 2–3 seconds of traffic onset.

Detection Signal 1 — PPS Spike (UDP): Flowtriq's per-protocol PPS baseline for Canonical's ASN would model normal UDP inbound at roughly 500 Kpps during peak hours. An amplification attack arriving at 150–400 Mpps represents a 300–800x deviation. The PPS anomaly detector uses an exponentially weighted moving average (EWMA) with α=0.1 and a 3σ upper control limit. A 300x deviation triggers the CRITICAL threshold in under one EWMA window (≈2 seconds).

Detection Signal 2 — BPS Threshold: Canonical's normal peak inbound BPS at internet exchange points would be in the 5–15 Gbps range. A 300 Gbps floor represents a 20–60x BPS deviation. Flowtriq's BPS monitor uses a 1-second sliding window; a sustained 20x deviation crosses the ATTACK threshold at t+1s.

Detection Signal 3 — Source IP Entropy: Flowtriq computes Shannon entropy on source IP /24 distribution per 5-second bin. Legitimate traffic entropy for a CDN-adjacent service like archive.ubuntu.com is high (broad global distribution). Amplification traffic from open resolvers/NTP servers shows a characteristic entropy spike toward maximum (uniform distribution across reflector IP space) combined with a destination IP collapse (all traffic converging on target prefixes). This dual signal — high source entropy + low destination entropy — is a reliable amplification fingerprint, detected in the second 5-second bin (t+5–10s).

Detection Signal 4 — Protocol Ratio Anomaly: Under normal conditions, UDP/TCP ratio for a mixed-service host like Canonical's is approximately 0.3–0.6 (more TCP than UDP). During a DNS/NTP amplification attack, this ratio inverts to 15–40:1 UDP/TCP. Flowtriq's protocol ratio monitor fires at a configurable threshold (default: ratio > 5.0 sustained for >3 seconds).

Detection Signal 5 — SYN/ACK Ratio: For the TCP SYN flood component, Flowtriq inspects TCP flag distributions. Normal SYN/ACK ratio is approximately 1:1 (±0.2). A SYN flood drives this to 50:1 or higher. Detection at t+3s for this vector.

Net result: all five signals would converge and trigger automated mitigation pipeline within 3–4 seconds of traffic onset. The attack classification engine correlates signals to output: ATTACK_CLASS: MULTI_VECTOR_VOLUMETRIC | CONFIDENCE: 0.97 | SEVERITY: CRITICAL.

4. Mitigation

Flowtriq executes a tiered mitigation ladder calibrated to attack severity, volume, and collateral damage risk. For a 300–600 Gbps multi-vector attack, the ladder escalates through all tiers within the first 60 seconds.

Tier 1 — Local iptables/nftables (t+4s, <1 Gbps surgical drops): Immediately applicable for the SYN flood and any traffic that has leaked past upstream mitigation. Applied at the host or local edge firewall. These rules are auto-generated by Flowtriq and pushed via its API to enrolled endpoints.

# nftables ruleset — Canonical edge firewall
# Auto-generated by Flowtriq Mitigation Engine v3.2
# Incident: INC-20240XXX | Severity: CRITICAL

table inet ddos_mitigation {
  chain inbound_filter {
    type filter hook preinrouting priority -150; policy accept;

    # Rate-limit UDP/53 inbound to 500 Kpps (legitimate resolver queries only)
    # Drops amplification reflection arriving on UDP/53
    ip protocol udp udp dport 53 limit rate over 500000/second burst 100000 packets drop

    # Block NTP monlist reflection (UDP/123, payload >468 bytes)
    ip protocol udp udp sport 123 ip length > 100 drop

    # Block CLDAP reflection (UDP/389)
    ip protocol udp udp sport 389 drop

    # SYN flood mitigation — SYN cookies + rate limit
    tcp flags syn / syn,ack limit rate over 200000/second burst 50000 packets drop
    tcp flags syn / syn,ack ct state new limit rate over 100000/second burst 10000 packets drop

    # ICMP flood — drop oversized echo-request
    ip protocol icmp icmp type echo-request ip length > 200 drop
    ip protocol icmp icmp type echo-request limit rate over 1000/second burst 500 packets drop

    # Block non-SYN TCP to non-established (common in spoofed floods)
    tcp flags ! syn / syn ct state new drop

    # Drop UDP to non-service ports (Canonical only runs TCP/80,443, UDP/53)
    ip protocol udp udp dport != { 53, 123 } drop

    # Log and drop everything else matching attack signatures
    ip saddr @blocked_sources counter drop
  }

  # Dynamic blocklist populated by Flowtriq BGP feed
  set blocked_sources {
    type ipv4_addr
    flags interval
    auto-merge
  }
}

Tier 2 — BGP FlowSpec Injection (t+8s, upstream surgical blocking): Flowtriq's BGP speaker peers with Canonical's border routers and upstream transit providers (e.g., Lumen, Cogent, NTT) to inject FlowSpec NLRI that drops traffic matching attack signatures in the transit core — before it reaches Canonical's edge. This is the primary mitigation for volumetric floods exceeding local link capacity. See Section 5 for full FlowSpec rule syntax.

Tier 3 — RTBH (Remote Triggered Black Hole) (t+15s, if volume exceeds upstream FlowSpec capacity): If FlowSpec is insufficient (e.g., transit provider doesn't support RFC 5575 or attack overwhelms their ACL hardware), Flowtriq escalates to RTBH. A /32 or /28 covering the targeted IP(s) is advertised to upstream peers with community 65535:666 (RFC 7999 BLACKHOLE community). This is a nuclear option — it drops all traffic, legitimate and malicious — and is used only when the alternative is total infrastructure collapse.

# RTBH BGP advertisement — Canonical border router config (Cisco IOS-XR syntax)
# Triggered by Flowtriq at Tier 3 escalation

route-policy FLOWTRIQ_RTBH_EXPORT
  if destination in RTBH_PREFIXES then
    set community (65535:666, 65000:9999) additive
    set next-hop 192.0.2.1  # Discard next-hop (RFC 5635)
    pass
  endif
end-policy

prefix-set RTBH_PREFIXES
  # Populated dynamically by Flowtriq — example:
  185.125.188.0/24 le 32,   # launchpad.net range
  91.189.91.0/24 le 32      # archive.ubuntu.com range
end-prefix-set

Tier 4 — Cloud Scrubbing Escalation (t+30–120s): For sustained attacks exceeding 100 Gbps where RTBH is causing collateral damage to legitimate users, Flowtriq triggers cloud scrubbing center activation. BGP anycast re-routes target prefixes to a scrubbing provider (Cloudflare Magic Transit, Akamai Prolexic, or Radware Cloud DDoS) via BGP prefix de-aggregation (announcing more specific /24s from scrubbing ASN). Clean traffic is GRE or IPIP tunneled back to Canonical's origin. Latency overhead: 5–15ms depending on scrubbing PoP proximity.

5. FlowSpec Rules

The following BGP FlowSpec NLRI rules conform to RFC 5575 (IPv4) and RFC 7674. These are the exact rules Flowtriq would generate and inject for this incident. FlowSpec encodes traffic-matching rules as BGP NLRI and distributes them to all peering routers simultaneously — achieving network-wide ACL deployment in BGP convergence time (~2–5 seconds across a well-tuned iBGP mesh).

########################################################
# Flowtriq Auto-Generated FlowSpec Rules
# Incident: INC-CANONICAL-DDOS-001
# Generated: 2024-XX-XX HH:MM:SS UTC
# RFC 5575 / RFC 7674 compliant
########################################################

# --- RULE 1: DNS Amplification Reflection Drop ---
# Match: UDP, source port 53, destination = Canonical prefixes
# Action: rate-limit to 5 Mbps (allows legitimate resolver traffic)
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol udp;
    source-port 53;
    packet-length [ 512-65535 ];      # Large DNS responses (amplification)
  }
  then {
    rate-limit 5000000;               # 5 Mbps — surgical, preserves legit DNS
  }
}

# --- RULE 2: NTP Reflection Drop ---
# Match: UDP sport 123, large packets (monlist response = 468 bytes)
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol udp;
    source-port 123;
    packet-length [ 100-65535 ];
  }
  then {
    discard;
  }
}

# --- RULE 3: CLDAP Reflection Drop ---
# Match: UDP sport 389 (Microsoft CLDAP — common DDoS amplifier)
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol udp;
    source-port 389;
    packet-length [ 300-65535 ];
  }
  then {
    discard;
  }
}

# --- RULE 4: SYN Flood Rate Limit ---
# Match: TCP SYN only (flags bitmask 0x02), destination TCP/443 and TCP/80
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol tcp;
    destination-port [ 80 443 ];
    tcp-flags [ syn ] [ syn ack fin rst ];   # SYN set, ACK/FIN/RST clear
    packet-length [ 40-80 ];                 # SYN packets 40-60 bytes typical
  }
  then {
    rate-limit 100000000;            # 100 Mbps — absorb legit SYN bursts
  }
}

# --- RULE 5: ICMP Oversized Flood Drop ---
# Match: ICMP echo-request, payload > 200 bytes (attack padding)
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol icmp;
    icmp-type echo-request;          # Type 8
    packet-length [ 200-65535 ];
  }
  then {
    discard;
  }
}

# --- RULE 6: UDP Flood to Non-Service Ports ---
# Canonical doesn't run UDP services above port 123
# Drop all high-port UDP destined to target prefix
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol udp;
    destination-port [ 1024-65535 ];
  }
  then {
    discard;
  }
}

# --- RULE 7: Fragmented UDP Drop ---
# Amplification attacks commonly use IP fragmentation to bypass ACLs
flow-spec ipv4 {
  match {
    destination 185.125.188.0/24;
    ip-protocol udp;
    fragment [ is-fragment first-fragment ];
  }
  then {
    discard;
  }
}

# BGP Extended Community encoding (RFC 5575 Section 7):
# Rate-limit action:  0x8006 (Traffic-rate, 2-byte AS + 4-byte rate in IEEE float)
# Discard action:     0x8003 (Traffic-rate 0 = discard)
# Mark/redirect:      0x8008 (Traffic-marking DSCP)

6. Lessons Learned

L1 — Anycast your authoritative DNS before you need to. Canonical's authoritative DNS for ubuntu.com and launchpad.net must be distributed across anycast nodes. A single-origin authoritative DNS under amplification attack collapses in seconds and prevents recovery — you can't update BGP community strings or scrubbing routes if DNS is down. Use Cloudflare DNS or NS1 as secondaries with anycast coverage before an event, not during.

L2 — Pre-negotiate FlowSpec peering with all transit providers. FlowSpec is only as fast as your peering agreements. If your Tier-1 transit (Lumen, Cogent, NTT) hasn't pre-approved FlowSpec customer NLRI injection, you're filing support tickets during an active 500 Gbps event. Canonical (and any org running public infrastructure) should have FlowSpec pre-authorized on all upstream BGP sessions with documented rate-limit and discard policy templates ready to push.

L3 — RTBH is not a mitigation strategy — it's a concession. Null-routing your own IP addresses stops the attack by making you unreachable. That's not defense, it's capitulation with extra steps. RTBH should be the bridge to cloud scrubbing, not the endpoint. The scrubbing activation SLA must be under 5 minutes — pre-provision GRE tunnels and test them quarterly.

L4 — Model your amplification exposure surface proactively. Run Shodan/Censys queries against your upstream IP space monthly. Open resolvers, NTP servers responding to monlist, and CLDAP-accessible domain controllers in your ASN or your customers' ASNs become your problem when attackers use them as reflectors. Remediate or BCP38-filter before the incident, not after.

L5 — Separate your package mirror infrastructure from your application layer. archive.ubuntu.com being co-impacted with launchpad.net during this attack indicates shared infrastructure exposure. Package mirrors should be behind independent CDN distribution (Fastly, Cloudflare) with separate origin IPs, separate ASN advertisement, and independent scrubbing policies. A DDoS against Launchpad's bug tracker should not affect apt-get update for 30 million Ubuntu machines.

L6 — 5-second detection is only valuable if the mitigation pipeline is pre-warmed. Flowtriq's <5s detection window is operationally meaningful only when FlowSpec peers are pre-established, iptables rulesets are pre-loaded (in accept mode, flipped to drop on trigger), and cloud scrubbing GRE tunnels are pre-provisioned. Detection speed and mitigation speed are independent variables — invest equally in both.