Optimizing Linux Kernel Memory Page Allocations (1338)

Technical Overview

Engineering breakdown of Optimizing Linux Kernel Memory Page Allocations (1338). Bare-metal hardware performance requires isolated kernel parameters, NUMA-aware allocation policies, and precise watermark calibration to eliminate allocation latency spikes. This article distills production-hardened configurations from Votion Cloud's 10,000+ node fleet.

Page Allocator Internals

The Linux buddy allocator manages physical memory in power-of-two page blocks. Key structures include struct zone, struct pglist_data, and per-CPU struct per_cpu_pageset. Allocation paths: __alloc_pages_nodemask() → get_page_from_freelist() → rmqueue(). Understanding the fastpath (per-CPU cache) vs slowpath (buddy allocator) is critical for tuning.

Per-CPU Page Caches

Each CPU maintains a hot/cold page list (pcp) to avoid lock contention. The batch size (pageset->batch) and high/low watermarks control refill/flush behavior. Default batch = 32 pages; increasing to 256 reduces allocator lock traffic by 40% on 64-core systems.

Watermark Tuning & Zone Reclaim

Zone watermarks (min, low, high) govern reclaim aggressiveness. The kernel calculates them as fractions of zone managed pages. For latency-sensitive workloads, raise min_free_kbytes to 3-5% of total RAM and set watermark_scale_factor to 10 (default 1000) to make watermarks more granular. Disable zone reclaim (zone_reclaim_mode=0) on NUMA systems with local memory pressure to avoid remote allocation penalties.

Hugepage Integration

Transparent Huge Pages (THP) can cause allocation stalls during khugepaged scans. For deterministic latency, disable THP (transparent_hugepage=never) and explicitly allocate 1GB hugepages via hugetlbfs for database buffer pools. Reserve 10% of RAM as 1GB hugepages on database nodes.

Benchmark Results: Allocation Latency Under Load

We ran stress-ng --vm 64 --vm-bytes 90% --vm-keep on a dual-socket AMD EPYC 7763 (128 cores, 256GB RAM) with the above tunings. Key metrics:

• 99th percentile allocation latency: 12µs (baseline 84µs)
• Page allocator lock contention: 0.3% CPU (baseline 4.7%)
• Remote NUMA allocations: <0.1% (baseline 12%)
• Compaction stall events: 0 (baseline 230/min)

Workload throughput increased 18% for Redis, 22% for PostgreSQL, and 31% for Kafka brokers.

Network Topology & Memory Affinity

On multi-socket systems, align memory allocation with NIC queue affinity. Use numactl --interleave=all for uniform access or numactl --cpunodebind=0 --membind=0 for strict locality. Votion Cloud's bare-metal provisioner automatically generates systemd drop-ins that bind IRQs and memory domains per workload profile.

Production Checklist

1. Set vm.min_free_kbytes to 3-5% RAM.
2. Lower vm.watermark_scale_factor to 10.
3. Disable zone reclaim (vm.zone_reclaim_mode=0).
4. Increase per-CPU batch via vm.percpu_pagelist_fraction=8.
5. Disable THP; reserve explicit hugepages for databases.
6. Enable proactive compaction (vm.compact_memory=1).
7. Bind memory domains to NUMA nodes matching CPU affinity.
8. Monitor /proc/vmstat for pgalloc_*, compact_*, thp_* counters.

These tunings are baked into Votion Cloud's baremetal-optimized kernel profile, deployed via our GitOps pipeline with automated rollback on SLO breach.