Deep Dive: WireGuard Mesh Networking for Clusters (4543)

Introduction

WireGuard has become the de facto standard for high-performance, cryptographically secure overlay networks. When operating at cluster scale on bare metal — hundreds of nodes across multiple racks — the devil is in the details: MTU management, kernel bypass, key lifecycle automation, and failure-domain isolation. This article walks through the architecture, tuning knobs, and operational tooling we use at Votion Cloud to run WireGuard mesh at line rate on 100 GbE fabrics.

Architecture Overview

The mesh is built as a full-mesh overlay where each node runs a single WireGuard interface (wg0) with a /24 IPv6 ULA prefix (fd00:4543::/64). Peers are discovered via a lightweight gRPC control plane that distributes public keys, allowed IPs, and endpoint metadata. The data plane remains pure kernel WireGuard; we avoid userspace proxies to keep latency sub-50 µs.

Key Design Decisions

• Single interface per node — reduces routing table complexity and simplifies BPF-based observability.
• Static endpoint IPs — each node advertises its physical NIC IP; no NAT traversal needed in controlled bare-metal environments.
• Automated key rotation — 24-hour key TTL with zero-downtime rekey using WireGuard's REKEY_AFTER_TIME and REKEY_AFTER_MESSAGES.
• MTU 8921 — accommodates 9000-byte jumbo frames minus 79 bytes of WireGuard overhead (IPv6 + UDP + WireGuard header).

Kernel Tuning for 100 GbE Line Rate

Default kernel parameters bottleneck WireGuard at ~25 Gbps per core. The following sysctl settings (applied via systemd-sysctl) are mandatory for saturating 100 GbE NICs:

# /etc/sysctl.d/99-wireguard-4543.conf
net.core.netdev_max_backlog = 250000
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.udp_mem = 65536 131072 262144
net.ipv6.udp_mem = 65536 131072 262144
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# Enable busy-poll for RX path (requires NIC driver support)
net.core.busy_read = 50
net.core.busy_poll = 50
# Increase conntrack table for NAT-heavy workloads
net.netfilter.nf_conntrack_max = 2000000
net.netfilter.nf_conntrack_buckets = 500000

Additionally, pin WireGuard's NAPI processing to dedicated CPU cores using ethtool -L and irqbalance banlists. We reserve cores 0-3 for interrupt handling and cores 4-15 for WireGuard encryption/decryption (via wg set wg0 peer ... with napi_id on kernels ≥ 6.6).

Operational Tooling & Observability

We ship a wg-meshctl CLI that wraps the control-plane API for day-2 operations:

• wg-meshctl peer list --health — shows handshake age, RX/TX bytes, and last-seen timestamp.
• wg-meshctl rekey --node — triggers immediate key rotation for a single peer.
• wg-meshctl topology export --format dot — emits GraphViz DOT for the network-topology block above.

Metrics are scraped by Prometheus via the wireguard_exporter (custom fork exposing per-peer wg_peer_rx_bytes, wg_peer_tx_bytes, wg_peer_handshake_age_seconds). Alerts fire on handshake age > 180s or peer RX rate drop > 80% over 5m.

Failure Domain Isolation

Each rack runs its own /64 ULA subnet (fd00:4543:rackX::/64) with a rack-gateway node performing inter-subnet routing via BGP (FRR). This limits blast radius: a compromised key in rack 3 cannot decrypt traffic in rack 7. The control plane enforces this by signing peer allowed-IPs with per-rack CA certificates.

Conclusion

WireGuard mesh on bare metal delivers IPsec-grade security with kernel-bypass performance — but only when you treat the kernel as a first-class tuning surface. The patterns above (single interface, automated rekey, MTU discipline, CPU pinning, rack-scoped addressing) are battle-tested across 2,000+ nodes at Votion Cloud. Clone the reference implementation and adapt to your fabric.