Scaling 1.2Tbps Volumetric DDoS Scrubbing (9424)
Technical Overview
Engineering breakdown of Scaling 1.2Tbps Volumetric DDoS Scrubbing (9424). Bare-metal hardware performance requires isolated kernel parameters, XDP/eBPF fast-path processing, and deterministic memory allocation to sustain line-rate inspection at 1.2Tbps aggregate throughput across 24x 100GbE interfaces.
Kernel Bypass & Memory Architecture
We deploy a custom votion-scrub kernel module that reserves 256 hugepages (1GB each) per NUMA node via hugetlbfs, eliminating TLB misses during packet processing. The RX path uses AF_XDP zero-copy sockets with XDP_UMEM_REG flags, mapping directly to NIC RX rings via mlx5_core driver's xsk_pool abstraction.
# /etc/sysctl.d/99-votion-scrub.conf
net.core.netdev_max_backlog = 2000000
net.core.rmem_max = 2147483647
net.core.wmem_max = 2147483647
vm.nr_hugepages = 6144
vm.hugetlb_shm_group = 1001
kernel.shmmax = 1099511627776
kernel.shmall = 268435456Flow Classification Pipeline
Each 100GbE port terminates on a dedicated CPU core (isolated via isolcpus=2-49 + nohz_full=2-49). The eBPF classifier (cls_bpf) executes 3-stage inspection:
- L3/L4 Header Validation — checksum offload verification, fragment reassembly state machine
- Entropy Scoring — per-flow Shannon entropy on payload bytes (sliding window 64KB)
- Reputation Lookup — cuckoo-filter IP reputation (128M entries, 4-bit counters) in shared hugepage region
Packets exceeding entropy threshold (H > 7.2 bits/byte) or matching reputation blocklist divert to scrubbing cluster via XDP_REDIRECT to dedicated scrub VFs.
Scrubbing Cluster Topology
Four DL380 Gen11 nodes (2x Xeon 8480+, 512GB DDR5-4800, 2x ConnectX-7 200GbE) form the scrubbing tier. Each node runs 48 worker threads pinned to physical cores, processing redirected traffic through:
- Protocol Normalization — TCP reassembly, UDP defragmentation, ICMP rate-limiting
- Anomaly Detection — spectral analysis on flow inter-arrival times (FFT window 10k packets)
- Mitigation Actions — SYN cookie generation, challenge-response, blackhole routing via BGP FlowSpec
Telemetry & Observability
Real-time metrics exported via prometheus-node-exporter + custom votion-scrub-exporter scraping eBPF maps every 500ms. Key dashboards track: scrub_throughput_bps, scrub_latency_p99_ns, flow_table_utilization, mitigation_actions_total.
eBPF/XDP kernel filter evaluates TCP/UDP frames directly on server NIC.