Scaling NVMe Storage Arrays with RAID-10 (3381)
Executive Summary
At Votion Cloud, we operate one of the largest NVMe-backed storage fabrics in the industry. This article distills our production learnings from scaling RAID-10 arrays across 3,381 NVMe devices (hence the "3381" designation) to deliver sub-millisecond latency at 14M IOPS per rack. We cover kernel bypass techniques, mdadm vs. hardware RAID trade-offs, and a reproducible benchmarking methodology you can run in our interactive sandbox.
Architecture: The 3381 Topology
Each storage node houses 24× U.2 NVMe drives (3.84 TB each) connected via a dual-port PCIe 4.0 x16 switch to two AMD EPYC 9654 CPUs. Drives are grouped into 12-drive RAID-10 sets (6 mirrors, striped). The 3381 figure represents the total drive count across a 141-node cluster (141 × 24 = 3384, minus 3 hot spares).
Key Design Decisions
- mdadm software RAID over hardware RAID: eliminates controller bottleneck, enables per-drive telemetry.
- XFS with `inode64,noatime,nodiratime,logbufs=8,logbsize=256k` for metadata-heavy workloads.
- IRQ affinity: each NVMe queue pinned to a dedicated CPU core (via `irqbalance` disabled, `set_irq_affinity.sh`).
- I/O scheduler: `none` (no-op) for NVMe; `mq-deadline` for rotational fallbacks.
Benchmark Methodology & Results
We use fio with a custom job file that mimics our production mix: 70% 4K random read, 20% 128K sequential write, 10% 4K random write (70/20/10). Tests run for 30 minutes with 1-minute ramp. All numbers are steady-state after cache warm-up.
Single-Node Results (24 drives, RAID-10)
| Metric | Value | vs. Single Drive |
|---|---|---|
| 4K Rand Read IOPS | 2.14M | 11.9× |
| 4K Rand Write IOPS | 1.02M | 5.7× |
| 128K Seq Read BW | 48.7 GiB/s | 12.1× |
| 128K Seq Write BW | 24.3 GiB/s | 6.0× |
| 99th %ile Latency (read) | 112 µs | — |
| 99.99th %ile Latency (write) | 487 µs | — |
Scaling is near-linear up to 12 drives per RAID-10 set; beyond that, CPU becomes the limiter (see CPU Analysis).
CPU Analysis: The Hidden Bottleneck
At 2.14M IOPS, the 96-core EPYC 9654 spends ~68% of cycles in `nvme_queue_rq` and `blk_mq_complete_request`. We mitigated this with:
- IO_URING + `IORING_SETUP_IOPOLL` (busy-poll) reduces syscall overhead by 42%.
- Per-CPU submit/completion queues (mdadm `--write-behind` disabled).
- Hugepages for fio buffers (`--hugepage-size=1G`).
After tuning, CPU utilization at peak IOPS drops to 41%, leaving headroom for compression (ZSTD) and encryption (AES-XTS) offload.
Operational Tooling: CLI Builder for Array Provisioning
We automate array creation with a Go-based CLI that validates topology, applies kernel params, and registers with our control plane. The sandbox below lets you generate a ready-to-run provisioning script for any Votion region.
Network Topology: Storage Fabric Integration
NVMe-oF (TCP) connects compute nodes to storage nodes via a dedicated 200 GbE Clos fabric (4× 50 GbE per storage node). We use `nvmetcli` for target configuration and `nvme-cli` for initiator multipathing (round-robin, `queue_depth=1024`). The diagram below shows a single rack's connectivity.
eBPF/XDP kernel filter evaluates TCP/UDP frames directly on server NIC.
Failure Domain & Rebuild Strategy
With 3381 drives, annualized failure rate (AFR) of 0.44% translates to ~15 drive failures/year. Our rebuild policy:
- Hot spare per 12-drive RAID-10 set (automatic `mdadm --manage --add`).
- Rebuild throttle: `sysctl -w dev.raid.speed_limit_max=200000` (200 MB/s) to avoid impacting latency SLOs.
- Typical 3.84 TB rebuild completes in 5.2 hours; zero data-loss events in 18 months.
Conclusion & Next Steps
RAID-10 on NVMe scales predictably when you eliminate kernel bottlenecks and align CPU, IRQ, and queue topologies. The 3381-array architecture delivers 14M IOPS/rack at <200 µs p99 latency, with linear cost scaling. Try the benchmark sandbox, generate a provisioning script for your region, and explore the cost estimator to model your workload.
Further reading: NVMe/TCP Offload with DPU, RAID-10 vs. Erasure Coding for Hot Data.