Harness Findings for Weekend

Everything the research agents surfaced, vetted and de-duplicated, in one place. Every URL was fetched and confirmed to resolve. Anything dated, dormant or thin on evidence is flagged inline rather than quietly dropped.

How to use it: the first section is the short list. The rest is a reference, not a queue — read by whatever you are actually working on, not top to bottom.

Start here

GitHub, Oct 2018 — 43 seconds, 24 hours — 25 min. A 43-second network partition caused 24h11m of degradation — roughly 2000:1. The single best argument that recovery paths, not faults, set outage length.
Slack, 22 Feb 2022 — metastable failure — 20 min. Read load rose superlinearly with cache misses because group-DM membership is sharded by user ID. The new control plane made it worse by being better at its job.
AWS S3, Feb 2017 — the untested recovery path — 12 min. One mistyped parameter; hours of outage because the index subsystem had not been fully restarted in large regions 'for many years'. Restart time grew with scale and nobody measured it.
Cloudflare, July 2019 — one regex — 10 min. Catastrophic backtracking took CPU to ~100% network-wide. The real lesson is process: WAF rules deliberately skipped the staged rollout.
How NOT to Measure Latency — Gil Tene — 55 min talk. Coordinated omission. Watch before trusting any latency number, including your own.
Timeouts, Retries and Backoff with Jitter — AWS — 20 min. Retries turn a partial outage into a total one; jitter exists because synchronised backoff just moves the stampede.
The Part of PostgreSQL We Hate the Most — Pavlo — 20 min. Append-only MVCC is the root of four coupled problems. Reframes bloat as an architecture choice, not a vacuum-tuning failure.
How we scaled nginx and saved 54 years a day — Cloudflare — 15 min. 6x p99 improvement by moving read() and — the overlooked one — open() off the event loop. Tail latency from one blocking syscall nobody suspected.

Outages and postmortems

AWS Kinesis, Nov 2020 — O(N^2) threads — 15 min. Every front-end server made threads for every other server, so a small capacity addition blew past an OS thread limit. They chased memory pressure for 4h24m first.
Meta, Oct 2021 — the failure that ate its own tooling — 10 min. Not really the BGP story you remember: a capacity-audit command took down the backbone and a bug in the audit tool failed to stop it.
Metastable Failures in Distributed Systems (HotOS'21) — 30 min, 5 pages. Names the thing: a system that stays broken after the trigger is gone because the load it generates sustains itself.
Metastable Failures in the Wild (OSDI'22) — Follow-up with real measurements from production incidents.
Surfing Complexity — Lorin Hochstein — Ex-Netflix. Applies resilience-engineering and safety science to real incidents. The antidote to five-whys postmortem culture.
SRE Weekly — Weekly since 2015. Curated incident reports and postmortems with a one-line take on each. Zero interview-prep content.

Database migrations and sharding

How Discord Stores Trillions of Messages — 20 min. Cassandra to ScyllaDB: 177 to 72 nodes, read p99 40-125ms to 15ms, 9-day migration at 3.2M msg/sec. The transferable part is the Rust request-coalescing tier, not the database swap.
Figma — how the databases team lived to tell the scale — 25 min. ~100x growth and they did NOT shard until 2023. Vertical partitioning bought four years; what ended it was one table too big to split further.
Sharding Postgres at Notion — 20 min. The mechanics of resharding a live system: partition key, double-write, backfill, verify, cut over.
MyRocks: space- and write-optimized MySQL — Meta — 15 min. They were space-bound on flash with CPU to spare, which is why LSM was free for them and may not be for you. Engine choice by which resource is actually saturated.
Scaling Memcache at Facebook (NSDI'13) — 1 hr. Still the best single document on caching at scale: leases for stampedes, invalidation, regional replication.

Storage engines and the write path

Universal Compaction in RocksDB and me — Mark Callaghan — 10 min. Leveled wrote 6.3GB vs universal's 32.5GB — but with trivial-move enabled universal drops back to 6.3GB. The 'universal has lower write amp' folklore is really about one config flag.
RocksDB Tuning Guide — 40 min. Treats read/write/space amplification as a three-way budget you explicitly spend, not a tradeoff to 'balance'.
RocksDB Compaction (wiki) — 20 min. Primary source for leveled vs universal vs FIFO. Read before any blog retelling.
Torn Write Detection and Protection — Alex Miller — 15 min. Your WAL's durability assumption rests on whether a partially-written page is detectable, and most systems get this via checksums never designed for it.
What Does a Database for SSDs Look Like? — Marc Brooker — 15 min. NVMe moves the 5-minute caching rule to ~30 seconds and makes cross-AZ replication cheaper than a local fsync WAL. Argues single-machine durability is now the wrong default.
Ben Stopford — Log Structured Merge Trees — The clearest LSM explanation there is. Pairs with your LSM trees note.

Postgres internals

Measuring the Memory Overhead of a Postgres Connection — Andres Freund — 12 min. Measured properly a connection costs ~1.3MiB with huge pages, not the 7-10MB ps implies. The real limit is snapshot/procarray contention — the honest case for PgBouncer.
The Internals of PostgreSQL — Suzuki, ch. 5-7 — 90 min. Concurrency control, vacuum, HOT and index-only scans, with the actual tuple headers and page layouts drawn. Maintained, not dated.
How Postgres Chooses Which Index to Use — pganalyze — 15 min. Walks real cost numbers through planner index selection, including why correlation stats make it prefer a seemingly worse index.
Implementing Stripe-like Idempotency Keys in Postgres — Brandur — The definitive practitioner writeup on idempotency under real consistency constraints. Directly relevant to the payment-processor project.
Jepsen: PostgreSQL 12.3 — 30 min. Real G2-item anomalies under serializable. Also nails the permanent caveat: Postgres 'repeatable read' is snapshot isolation. Fixed — read for the reasoning.

Isolation, consistency and consensus

Strong Consistency Models — Kyle Kingsbury — 25 min. Linearizability is a single-object real-time property, serializability a multi-object one, so they are incomparable. Still the clearest statement of it.
Jepsen: Consistency Models (reference map) — 20 min. The lattice showing which models are totally-available, sticky-available, or unavailable under partition. The most useful single artifact for reading vendor claims.
Snapshot Isolation vs Serializability — Marc Brooker — 8 min. Under SI you compare write sets; under serializability, read sets. Read sets are far larger in OLTP — that one difference explains most of the throughput gap.
What Fekete's Anomaly Teaches About Isolation — Marc Brooker — 10 min. A read-only transaction exposing a constraint violation under SI, plus the surgical fixes instead of escalating everything to serializable.
Jepsen: MySQL 8.0.34 — 30 min. MySQL's default Repeatable Read permits write skew, read skew and lost updates — i.e. the level most MySQL apps run on is weaker than its own docs claim.
Hermitage — Martin Kleppmann — 20 min. SQL test scripts showing each vendor's isolation levels diverge from the standard and from each other. The empirical companion to the theory.
Data Replication Design Spectrum — Alex Miller — 20 min. Replication on one axis: failure masking (quorums) vs failure detection (reconfiguration). Lands Raft as the unwaveringly moderate midpoint.
Enough With All The Raft — Alex Miller (talk) — 35 min. Raft has no advantage on any clean theoretical dimension; the interesting design space is under-explored because everyone reaches for Raft reflexively.
Living Without Atomic Clocks — CockroachDB — 20 min. Single-key linearizability on NTP-grade clocks via uncertainty intervals and read restarts, and exactly what you give up vs TrueTime.
Deterministic Simulation Testing — Alex Miller — 25 min. From the FoundationDB lineage: finding consensus bugs requires making the entire system deterministic, including time and disk.

Reliability primitives

Using Load Shedding to Avoid Overload — AWS — 20 min. Past capacity, serving fewer requests well beats serving all of them badly. Shedding is what happens when the rate limiter says no.
Reliability and Constant Work — AWS — 15 min. Systems that do the same work regardless of input cannot have a bad day. Counterintuitive and beautiful.
Workload Isolation Using Shuffle Sharding — AWS — 15 min. How one abusive tenant is made unable to take down everyone. Combinatorics doing real work.
Avoiding Fallback in Distributed Systems — AWS — 15 min. Your fallback path is the least-tested code you own and it runs only during emergencies.
Caching Challenges and Strategies — AWS — 20 min. Modality, stampedes, and why a cache that improves the average makes the tail worse.
Exponential Backoff and Jitter — AWS — 10 min. The original post with the simulation showing full jitter wins.
Addressing Cascading Failures — Google SRE — 30 min. Where the failure modes are laid out properly, including load-shedding and graceful degradation.
Handling Overload — Google SRE — 30 min. Criticality-based rejection and adaptive throttling from the client side.
Performance Under Load — Netflix — 15 min. Adaptive concurrency limits: stop hand-tuning pool sizes, derive the limit from observed latency. The basis of the adaptive-limiter project.
Keeping Netflix Reliable with Prioritized Load Shedding — 15 min. Not all requests are equal: drop playback telemetry before you drop playback.
Rubbing Control Theory on the Go Scheduler — Cockroach Labs — 30 min. Go scheduler latency is a leading indicator of end-to-end p99 with ~22x amplification; admission control targeting 1ms scheduler p99 took backup-induced p99 from 10s to 60ms.
Load Balancing — Sam Rose — Interactive simulation: drag sliders and watch each algorithm fail. Far better than any static explainer.

Benchmarking and measuring

The USE Method — Brendan Gregg — 15 min. Utilization, Saturation, Errors for every resource. A checklist that finds the bottleneck instead of guessing.
Benchmarking Checklist — Brendan Gregg — 10 min. The questions that invalidate most benchmarks. Read before any measurement run.
Flame Graphs — Brendan Gregg — How to read them. go tool pprof -http gives you these for free.
wrk2 — Gil Tene — The open-loop fix for wrk, by the person who named coordinated omission.
High Performance Go Workshop — Dave Cheney — 2-3 hr. The benchmarking-methodology half is the durable part: how the compiler silently deletes your benchmark, why benchstat and not eyeballing ns/op.
Branch predictor: how many ifs are too many? — Cloudflare — 18 min. Microbenchmarks pin the BTB at ~4096 entries; predicted-taken ~2-3.5 cycles, unpredicted ~10.5. Never-taken branches are essentially free.
Thread Count Scaling — Denis Bakhvalov (Easyperf) — PMU counters, measurement bias, and why your benchmark is lying. Blog slowing since 2024 but the back catalogue is the practical guide to CPU measurement on Linux.
Queueing theory on a cocktail napkin — Dan Slimmon — Queueing theory at a level you can actually apply to capacity questions. Blog going quiet — read while it is up.

Go: runtime, GC, scheduler

A Guide to the Go Garbage Collector — 45 min. The only doc explaining the GC pacer as a control problem: GC CPU bounded at 25%, live-heap/GOGC ratio is what you tune, and why GOGC alone cannot bound peak memory.
Getting to Go: The Journey of Go's GC — Rick Hudson — 40 min. Why Go chose a non-generational, non-compacting concurrent collector and traded throughput for latency — 300ms pauses to under 500us. Dated in numbers, not in design.
Go runtime: 4 years later — Michael Knyszek — 15 min. Why the memory-ballast trick should die: GOMEMLIMIT is a soft limit that trades CPU to stay under a ceiling, and avoids the death spiral a hard limit creates.
Go Scheduler — Nghia Nguyen — 90 min. The best current GMP writeup outside the source: syscall handoff (P detaches, sysmon retakes after ~10ms), runnext starvation, work stealing, async preemption via signals.
On concurrency in Go HTTP servers — Eli Bendersky — 15 min. Goroutine-per-connection over the netpoller/epoll, with the load numbers that justify it. Directly relevant to the stdlib HTTP server build.
How Stacks are Handled in Go — Cloudflare — 15 min. Why goroutine stacks start tiny and grow by copying, and why that forces the runtime to rewrite every pointer into the stack. Mechanism current, version details historical.

Go: profiling and performance

Go Profiler Notes — Felix Geisendorfer (Datadog) — 60 min. Measured overhead for every profiler, and the warning that block-profiler rates are a sampling period in nanoseconds — which is why most block profiles are wrong.
Inside the Go CPU profiler — Sumer Cip — 20 min. Driven by setitimer SIGPROF at 100Hz, which caps resolution and biases against short-lived goroutines. Makes you stop over-trusting a flat CPU profile.
Connecting Go Profiling With Tracing — Felix Geisendorfer — 15 min. pprof labels are inherited by child goroutines, so you can slice a production CPU profile by which customer burned the CPU.
More powerful Go execution traces — 15 min. Tracing overhead fell from ~10-20% to ~1-2%, and the flight recorder dumps a rolling trace only when a request exceeds a latency threshold. Production tail-latency hunting became viable.
Language Mechanics on Escape Analysis — Ardan Labs — 20 min. Why returning a pointer is often cheaper than returning a value: sharing up escapes to the heap, sharing down stays on the stack.
sync.Pool and the mechanics behind it — VictoriaMetrics — 25 min. The per-P private/shared victim cache, and therefore why pooling small or variable-size objects loses to the allocator.
Profile-guided optimization in Go 1.21 — 12 min. A measured 3.83% CPU win with the mechanism shown: PGO-driven inlining eliminated 4,974,135 allocations, plus interface devirtualization with a type-guard fallback.
Save up to 14% CPU with continuous PGO — Datadog — 12 min. On already-tuned services a 3.4% CPU cut was worth ~$250k/yr. Production profiles merged into default.pgo at build time.

Go: production war stories

How We Saved 70K Cores Across 30 Services — Uber — 20 min. GOGCTuner computes GOGC from the cgroup memory limit instead of a static value, cutting p99 CPU ~65% on one service. Neat hack: a self-referencing finalizer to run once per GC cycle.
LeakProf: In-Production Goroutine Leak Detection — Uber — 20 min. Samples goroutine stacks and flags concentrations blocked at the same channel op: 10 real leaks, 1 false positive, fixes gave 2.5x and 5x peak-memory reductions. Best taxonomy of leak shapes.
Never start a goroutine without knowing how it will stop — Dave Cheney — 10 min. The single rule that prevents most goroutine leaks. The discipline behind the LeakProf findings.
Go channels are bad and you should feel bad — JT Olio — 20 min. The counterweight to channel-first design: channels do not compose with context cancellation cleanly, and a mutex is often both faster and easier to reason about.
uber-go/automaxprocs — 10 min. Why GOMAXPROCS defaulting to host CPU count inside a CPU-quota'd container causes throttling and scheduler thrash. Becoming legacy — Go 1.25 made the runtime container-aware.
Why Discord is switching from Go to Rust — 12 min. Latency spiked every 2 minutes because Go's GC force-runs on that interval and had to scan a huge LRU cache. Read as a GC case study; it predates GOMEMLIMIT. Weakest on hard numbers — the evidence is a graph.

Optimization case studies with real numbers

How we built Pingora — Cloudflare — 15 min. nginx to Rust: 1T+ req/day on 70% less CPU and 67% less memory. The win is work-stealing allowing cross-thread connection-pool sharing — reuse 87.1% to 99.92%, i.e. 160x fewer handshakes.
Goodbye Microservices: from 100s of problem children to 1 superstar — Segment — 10 min. 140+ services made shared-library changes so expensive that 32 improvements shipped during the microservice era vs 46 in the year after collapsing to a monolith.
Prime Video: 90% cost reduction (archive copy) — 6 min. Step Functions hit a wall at 5% of expected load, paying per state transition plus S3 calls to shuttle frames. Collapsing to one ECS process moved transfer into memory. NOTE: Amazon pulled the original; only the archive resolves.
Cost Efficiency at Scale in Big Data File Format — Uber — 12 min. Parquet at exabyte scale: SNAPPY to ZSTD cut storage 39%, and reordering columns by type similarity gave another 32% — adjacency determines how well the encoder works.
Squeezing the firehose: Kafka compression — Cloudflare — 10 min. On a 100Gbps-peak topic, Snappy gave 2.25x and Zstandard 4.5x — enough to cancel a hardware order. Trading a network bottleneck for a CPU one, deliberately.
Lepton: saving 22% losslessly from images — Dropbox — 15 min. A bit-exact reversible JPEG re-encoder, flat 22% across 16 billion images. The load-bearing constraint is round-trip speed, so it runs inline on downloads.
Broccoli: syncing faster by syncing less — Dropbox — 15 min. Brotli fork cut wire data ~33%, p50 request 3.5MiB to 1.6MiB, median latency down 30%+. The insight is per-file-type compression level — video is already incompressible.
How Shopify Reduced Storefront Response Times with a Rewrite — 12 min. 4-6x faster server response. The transferable part is the verifier that ran both implementations against live traffic and routed by comparison result — zero-downtime cutover.
Ruby 3.3 YJIT: faster while using less memory — Shopify — 12 min. ~65% faster railsbench while cutting JIT memory ~20%, via a cold threshold that stops compiling rarely-executed methods.
Byte Down: making Netflix's data infra cost-effective — 10 min. Cost visibility plus ownership, not imposed quotas, cut warehouse storage 10%+.
Building Resilient Payment Systems — Shopify — 25 min. Idempotency and reconciliation where getting it wrong means real money.
Scaling with Common Sense — Zerodha — 15 min. Enormous market-open bursts on deliberately boring infrastructure. A useful antidote to assuming scale requires exotic tooling.

Low-level: SIMD, cache, bit tricks

Parsing Gigabytes of JSON per Second (simdjson) — 45 min, 20pp. First standard-compliant JSON parser past 1GB/s/core: branchlessly locate structural characters across 64-byte SIMD blocks, then parse only those.
Faster remainders when the divisor is a constant — Lemire — 7 min. Computing the remainder from the low bits instead of deriving it from the quotient: 4.5 to 1.9 cycles/integer. Go's GC adopted it for ~1.5% whole-program gain.
Fast float parsing in practice — Lemire — 6 min. 660MB/s vs strtod's 70MB/s by optimistically taking an exact fast path for the ~99% case and falling back only when exactness cannot be proven.
Ridiculously fast base64 encoding and decoding — Lemire — 4 min. ~2 cycles/byte to 0.2 with AVX2, including full validation, not a stripped benchmark.
Parsing JSON faster with AVX-512 — Lemire — 5 min. Minify 4.3 to 12GB/s but full DOM parsing only 2.8 to 3.6 — an honest case study in which stages actually scale with vector width.
ripgrep is faster than grep, ag, git grep — BurntSushi — 50 min. 25 reproducible benchmarks with the why: literal extraction from the regex, a lazy DFA, Teddy/memchr prefilters. Unusually honest about where ripgrep loses.
Designing a fast, cache-friendly hash table — Matt Kulukundis — 60 min talk. The canonical Swiss Table talk and ancestor of Rust's hashbrown: open addressing plus SSE2-scanned control bytes, 16 slots probed per instruction.
Sorting: speed is found in the minds of people — Alexandrescu — 89 min talk. Asymptotic reasoning actively misleads on modern hardware; rebuilds sorting around branch-free partitioning and ILP.
Arena allocator tips and tricks — Chris Wellons — Changed how a lot of people write C. Uncompromising low-level allocation design.
Extreme HTTP performance tuning: 1.2M req/s on 4 vCPU — Talawah — Canonical reference on Linux network-stack tuning, layer by layer with measurements. Blog dormant since 2022; this one post is why it is here.

OLAP and query execution

Asynchronous Data Inserts in ClickHouse — 20 min. 'Too many parts' fires at 300 active parts per partition; their benchmark hit it in 5 minutes at 200 inserts/10s, producing ~30,000 parts. Why MergeTree pushes batching to the server.
ClickHouse Joins Part 2: Hash Joins — 30 min. The build phase is single-threaded, so putting the larger table on the right cost 5.06s vs 0.82s — a 6x penalty from join-side ordering alone.
External Aggregation in DuckDB — 20 min. A 1-billion-group aggregation in 264s on a 16GB laptop vs Spark's 603s on a 256GB instance. Graceful degradation beats raw memory.
Querying Parquet with Precision — DuckDB — 15 min. How row-group zone maps and projection pushdown let you skip most of a file, and when they silently do not help.
CMU 15-721 Advanced Database Systems — Andy Pavlo — Full semester, video + papers. In-memory OLTP, vectorised vs compiled execution, why analytical and transactional engines diverge.

The papers

Dynamo (SOSP'07) — read in full — Quorums, vector clocks, and an explicit availability-over-consistency argument. Still worth reading cover to cover.
Raft — read in full — Written to be understood. Read this, not Paxos.
Spanner — read in full — TrueTime: buying consistency with hardware clocks. The idea is still not widely copied, which makes it worth understanding.
Gorilla (VLDB'15) — read in full — Why everyone built their own TSDB. The compression scheme is the whole answer.
Bigtable — skim — Foundational; the ideas are now common knowledge.
Dapper — skim — The origin of distributed tracing. Sampling strategy is the durable part.
Borg — skim — Where Kubernetes came from, and what Kubernetes left behind.
Maglev — skim — Consistent hashing for load balancers, with the connection-tracking detail.
Kafka (original paper) — skim — Historical interest; the log abstraction is better explained elsewhere now.
TAO — readable blog version — The graph store behind Facebook. Blog version suffices over the paper.
Papers ranked by PageRank over citations — rmarcus — If you want an ordering rather than a pile.

Blogs worth following

Marc Brooker — AWS principal engineer. Monthly-ish, near-zero noise. The best working writer on distributed systems trade-offs.
Alex Miller — Transactional Blog — Ex-FoundationDB. The most rigorous public writing on transactional storage design — decomposes systems by axis rather than describing products.
Jepsen analyses — Kyle Kingsbury — Breaking databases that claim consistency. The best training in reading vendor claims sceptically.
Brendan Gregg — Performance and observability. Irregular, always substantive.
Justin Jaffray — Weekly, reliable. Query-engine and incremental-computation internals from a Materialize/Cockroach background. Database theory made operational.
Small Datum — Mark Callaghan — MyRocks author. Rigorous, unglamorous storage-engine benchmarking by someone who cares about being right.
Nelson Elhage — Debugging methodology, performance archaeology, and reasoning about systems you did not build.
Russ Cox — research!rsc — Rare, but when he publishes it is definitive: memory models, dependency math, float formatting. With proofs.
Chris Wellons — null program — Monthly. Uncompromising low-level C: arenas, atomics, hand-written hash tables.
Sam Rose — Interactive simulation explainers of infra primitives. You drag sliders and watch the algorithm fail.
Brandur Leach — Ex-Stripe. Postgres-as-queue, idempotency, API design under real consistency constraints.
Hillel Wayne — Formal methods for working engineers, plus software-history research where he actually interviews people instead of repeating folklore.
Oren Eini (Ayende) — RavenDB's founder narrating storage-engine and query-optimizer work in public, including the dead ends. 20+ years, multiple posts a month.
Jamie Brandon — Scattered Thoughts — Builds his own databases and languages and writes up the failures. Publishes benchmarks that contradict his own thesis.
Fred Hebert — ferd.ca — Erlang/OTP veteran turned SRE. Operability and human factors with real theoretical grounding.
Colin Breck — Streaming and industrial systems at scale, plus engineering-leadership writing that is not management fluff.
Xe Iaso — Infra and protocol writing with a willingness to publish findings others sit on.
Chris Siebenmann — A short specific post almost every day for 20 years on Linux, ZFS, DNS, Prometheus. Enormous corpus, zero audience-chasing.
Murat Demirbas — Distributed systems paper reviews. How to get papers digested rather than read cold.
Aleksey Charapko — Paper reviews and a long-running distributed systems reading group.
Dan Luu — Irregular, sometimes tangential, consistently makes you think.
TigerBeetle — Deterministic simulation testing, VSR consensus, static allocation. Building a database that assumes hardware lies.
Antithesis — Deterministic simulation testing company. Autonomous bug hunting and causality analysis. High originality per post.
Oxide Computer — Hardware/firmware/OS co-design by people who ship all three. Pair with the Oxide and Friends podcast.
WarpStream — Diskless Kafka on S3: object-storage economics, metadata design, zone affinity, with real numbers.
Restate — Ex-Flink/Kafka people on log-centric architecture and why checkpointing is not durability.
Geoff Langdale — branchfree — Co-author of Hyperscan and simdjson. Deepest public writing on SIMD string matching. Slowing since late 2024.
Micah Lerner — Systems paper reviews of OSDI/SOSP/NSDI. Dormant since Jan 2025, but the back catalogue is the best Morning Paper successor.