Memory Latency on a Cloud Server: Measured, Not Guessed
Short answer (checked 2026-11-01): A post that reached the Hacker News front page this weekend argues that a CPU showing 100% busy is often just waiting for RAM, and shows a GROUP BY that got faster by doing more work while touching memory less. We ran the same idea on the cloud server that writes this blog: a load costs about 2 ns when the data fits in L1 and about 200 ns at a 256 MB working set, roughly a hundred times more. For anyone renting servers that gap matters more than the core count on the order form. Below are the script, our numbers and what they change when you choose between cloud and dedicated.
Source. "It's the memory latency, stupid" by Maor Kern (Pivot / Epsio Labs, published 2026-10-28), discussed on Hacker News with 175 points by 2026-11-01. Their figures below are as the post prints them; ours we measured ourselves.
What the Pivot post measured
The test is one query, GROUP BY UserID over the ClickBench "hits" table (about 99 million rows, about 17 million distinct users), pinned to one core of an AMD EPYC 9124 bare-metal server with 128 GB DDR5. A plain hash table with 17 million keys fits in no cache, so almost every lookup goes to RAM. Splitting the rows first into 4096 partitions, each with a small hash table, adds work but keeps each table in cache:
| Metric (one core) | Plain hash table | Radix partitioned |
|---|---|---|
| Wall time, 1st / 2nd run | 3.9 s / 3.6 s | 3.3 s / 2.7 s |
| Slots stalled on memory | 50.1% | 23.5% |
| Slots retiring work | 16.1% | 32.2% |
| Demand loads from DRAM | 76.2 million | 25.7 million |
The partitioned version retires more operations (19.4 billion vs 11.4 billion) and still finishes first. The author's tuned version ran in 1.85 s on one core and 0.17 s on all 16. The lesson is not about one database: "CPU 100%" in top tells you the core is not idle, not that it is computing.
Measure it yourself: memlat.js
The classic way to see latency is a pointer chase: an array where each slot holds the index of the next one, in random order, one cache line (64 bytes) apart. Every load depends on the previous one, so the CPU can neither prefetch nor overlap them, and the time per hop is the latency of wherever the working set lives. There is no C compiler on our blog machine, so this is JavaScript for Bun: the JIT turns the loop into a few machine instructions, which adds a little to every figure. Compare the steps, not the absolute values. Save it as memlat.js:
// memlat.js - pointer-chase memory latency by working-set size (bun memlat.js)
import { readFileSync, readdirSync } from 'node:fs'
const cpu = readFileSync('/proc/cpuinfo', 'utf8').match(/model name\s*:\s*(.+)/)?.[1] ?? '?'
const base = '/sys/devices/system/cpu/cpu0/cache'
const caches = readdirSync(base).filter(d => d.startsWith('index')).map(d => {
const r = f => readFileSync(`${base}/${d}/${f}`, 'utf8').trim()
return `L${r('level')}${r('type') === 'Data' ? 'd' : r('type') === 'Instruction' ? 'i' : ''} ${r('size')}`
})
console.log(`CPU: ${cpu}\nCaches: ${caches.join(', ')}`)
// one random cycle through n slots, 64 bytes (16 x int32) apart: one cache line per hop
function ring(bytes) {
const n = Math.max(2, Math.floor(bytes / 64))
const order = new Uint32Array(n)
for (let i = 0; i < n; i++) order[i] = i
for (let i = n - 1; i > 0; i--) {
const j = Math.floor(Math.random() * (i + 1))
const t = order[i]; order[i] = order[j]; order[j] = t
}
const a = new Int32Array(n * 16)
for (let i = 0; i < n; i++) a[order[i] * 16] = order[(i + 1) % n] * 16
return a
}
function chase(a, hops) {
let p = 0
for (let i = 0; i < hops; i++) p = a[p]
return p
}
const hops = 20_000_000
console.log('working set ns/load')
for (const kb of [16, 32, 256, 1024, 4096, 16384, 65536, 262144]) {
const a = ring(kb * 1024)
chase(a, hops / 4) // warm up: JIT and caches
const best = []
for (let r = 0; r < 3; r++) {
const t = Bun.nanoseconds()
const sink = chase(a, hops)
best.push((Bun.nanoseconds() - t) / hops)
if (sink < 0) console.log(sink)
}
const label = kb >= 1024 ? `${kb / 1024} MB` : `${kb} KB`
console.log(`${label.padStart(10)} ${Math.min(...best).toFixed(1).padStart(7)}`)
}
bun memlat.js
Output, run on a DCXV Cloud server with Bun 1.4.4 on 2026-11-01:
CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
Caches: L2 4096K, L1d 32K, L3 16384K, L1i 32K
working set ns/load
16 KB 2.0
32 KB 2.2
256 KB 12.5
1 MB 24.3
4 MB 122.2
16 MB 138.4
64 MB 164.7
256 MB 205.1
A second run the same morning differed by at most 2 ns up to 1 MB and at most 12 ns above it:
| Working set | ns per load, run 1 / run 2 | Where it lives |
|---|---|---|
| 32 KB | 2.2 / 2.4 | L1 data cache |
| 256 KB | 12.5 / 10.5 | L2, spilling into L3 |
| 1 MB | 24.3 / 24.3 | L3 |
| 16 MB | 138.4 / 137.2 | page walks, L3 and DRAM |
| 256 MB | 205.1 / 193.5 | DRAM |
Three things the numbers say
- The guest does not see the real caches. The VM reports a 4 MB L2 and a 16 MB L3; Intel lists 50 MB of cache for the Xeon E5-2698 v4 (Intel ark, checked 2026-11-01). The hypervisor presents a topology, and
lscpuinside a cloud server reads that, not the silicon. Time the hardware instead of trusting the labels. - The cliff comes long before the cache is full. Latency jumps five times between 1 MB and 4 MB, far below 50 MB. With 4 KB pages a random walk over 4 MB touches 1,024 pages, more than the TLB holds, and inside a VM a TLB miss walks both the guest's and the host's page tables; the L3 is also shared with the other cores on the socket. We did not separate these causes. The practical result is the same: in a VM, plan as if your fast working set is a few megabytes.
- Bigger RAM is not faster RAM. From 1 MB to 256 MB the cost per load grows about eight times. More gigabytes add capacity, not speed; what speeds up a latency-bound job is a smaller hot set, as the Pivot partitioning shows.
What it means if you rent servers
- Cloud servers suit most web apps, APIs and small databases: their hot data fits in cache, and they wait on the network or the disk long before they wait on RAM. Size them by measured memory use, as in our cloud server sizing guide.
- Dedicated servers pay off when a large working set is hit at random all day: analytics, big hash joins, in-memory caches, a PostgreSQL server whose indexes no longer fit in RAM. You get the whole L3 and every memory channel with no neighbours, and you can turn on huge pages, which cut TLB misses. Configure one on the dedicated server page.
- Before upgrading either, run the chase on the box you have and check whether your slow job is latency-bound. Doubling the cores does nothing for one thread that spends half its time stalled. Memory itself got dearer this year, see server RAM in 2026, so the cheapest fix is often code that touches less of it.
Test before you buy
Our Tier III data centres in Prague and Covilha run cloud servers and dedicated servers. Start a cloud server, run memlat.js next to your own workload and send the output to sales@dcxv.com: we will help you decide whether a dedicated server would change the numbers.
