What Is Caching?

Caching is one of those foundational ideas in system design that shows up everywhere, from the CPU in your laptop to the largest distributed systems on the planet. The core concept is simple: store the result of an expensive operation somewhere fast and cheap to access, so the next time you need it, you skip the expensive work entirely. Understanding why that matters, and where the interesting design decisions live, is what separates a surface-level answer from a strong one.

The Motivation: Why Do We Need Caching?

Let's start with the fundamental problem. In any system, you have a hierarchy of storage and compute resources, each with wildly different speed and cost characteristics:

CPU registers and L1/L2/L3 cache: nanosecond access, tiny capacity (kilobytes to a few megabytes)
RAM (main memory): ~100 nanoseconds, gigabytes of capacity
SSD: ~100 microseconds, terabytes of capacity
HDD (spinning disk): ~10 milliseconds, very cheap per byte
Network call to another service or database: ~1 to 100+ milliseconds, depends on load
External API or cross-region call: hundreds of milliseconds or more

The gap between RAM and a database query across the network can be five or six orders of magnitude. If your application is re-running the same expensive database query hundreds of times per second, you are essentially choosing to pay that high latency cost repeatedly, when you could pay it once and cache the result. Caching is about moving frequently-accessed data closer to where it is consumed, in exchange for some complexity around keeping it accurate.

Cache Hits, Misses, and Hit Rate

The vocabulary here matters for any design discussion. When a request asks for data and that data is already in the cache, that is a cache hit. When it is not, that is a cache miss, and we fall through to the underlying source of truth (usually a database). The ratio of hits to total requests is the hit rate, and it is the single most important metric for a cache's effectiveness.

A cache with a 99% hit rate means only 1 in 100 requests touches the database. A cache with a 50% hit rate barely helps at all, and depending on overhead, might even hurt. This is why caching works best when access patterns follow a power law or Zipf distribution, which they very often do in practice. A small percentage of your data gets the vast majority of reads. Think about Twitter: a handful of celebrity accounts generate far more traffic than millions of ordinary accounts. Caching those hot rows is enormously effective.

Where Caches Live in a System

There is no single place a cache lives. In a real system, caching happens at multiple layers simultaneously:

Client-side caching: The browser caches static assets (HTML, CSS, JavaScript, images) using HTTP Cache-Control headers. This avoids a network round-trip entirely.
CDN (Content Delivery Network): Services like Cloudflare or Akamai cache responses at edge nodes geographically close to users. Great for static content and even some dynamic content.
Application-level (in-process) cache: A simple in-memory dictionary or library like Guava Cache inside your application server. Fast, but not shared across multiple server instances, and lost on restart.
Distributed cache: A dedicated caching tier like Redis or Memcached that all application servers share. This is what most people mean when they say "add a cache" in a system design interview.
Database query cache: Some databases (MySQL historically) cache query results internally, though this is often disabled in modern setups because it does not scale well under write-heavy workloads.
CPU and OS caches: Below the application level, but worth knowing exist. The OS page cache means repeated reads of the same file from disk are often served from RAM transparently.

Caching Strategies: Read and Write Patterns

How you populate and update a cache is one of the most important design decisions. There are a few canonical patterns:

Cache-Aside (Lazy Loading): This is the most common pattern. The application checks the cache first. On a miss, it reads from the database, stores the result in the cache, and returns it. The cache only contains data that has actually been requested, which is efficient. The downside is that the first request for any piece of data always misses, and if many requests arrive simultaneously for the same missing key (called a thundering herd or cache stampede), they all hit the database at once.

def get_user(user_id):
    user = cache.get(f"user:{user_id}")
    if user is None:
        user = db.query("SELECT * FROM users WHERE id = ?", user_id)
        cache.set(f"user:{user_id}", user, ttl=300)
    return user

Write-Through: Every write goes to both the cache and the database synchronously. The cache is always warm and consistent. The tradeoff is write latency increases slightly, and you may cache data that is never read (wasted memory).

Write-Behind (Write-Back): Writes go to the cache immediately and to the database asynchronously, in the background. This gives very low write latency but introduces a window where data in the cache has not yet been persisted. If the cache node fails in that window, you lose data. This pattern appears in storage systems where durability can be traded for throughput, but it is risky for most web application data.

Read-Through: Similar to cache-aside, but the cache itself is responsible for loading from the database on a miss, rather than the application. The application only ever talks to the cache. This simplifies application code but requires a smarter cache layer.

Eviction Policies: What Leaves the Cache?

Caches have finite memory, so when they fill up, something has to go. The eviction policy determines what. The most common choices are:

LRU (Least Recently Used): Evict the item that has not been accessed for the longest time. This is the default intuition and works well for most workloads because recently-accessed data is likely to be accessed again soon.
LFU (Least Frequently Used): Evict the item accessed the fewest times. Better for workloads where popularity is stable over time, but older popular items can accumulate high counts and crowd out newer popular data.
FIFO (First In, First Out): Evict the oldest item regardless of access pattern. Simple but often poor in practice.
TTL (Time To Live): Items expire after a fixed time regardless of access frequency. This is not strictly an eviction policy but is almost always used alongside one to limit stale data.

Redis, for example, supports LRU and LFU as eviction policies and lets you configure TTLs per key. In practice, LRU with a TTL is the combination you will reach for most often.

Cache Invalidation: The Hard Problem

Phil Karlton's famous quote goes: "There are only two hard things in computer science: cache invalidation and naming things." It is funny because it is genuinely true. Once you have cached data, keeping it consistent with the source of truth is the hardest part of caching.

The approaches are:

TTL-based expiry: Accept that cached data may be stale for up to N seconds. Simple and sufficient for many use cases where slight staleness is acceptable (e.g., a product description, a user's follower count).
Event-driven invalidation: When a write happens, explicitly delete or update the cached entry. This keeps the cache more current but couples your write path to cache management. Also, in a distributed system, this notification can fail or arrive out of order.
Cache versioning: Instead of invalidating, change the cache key when the underlying data changes. Old keys naturally expire via TTL. This sidesteps race conditions around invalidation but can lead to key sprawl.

The trickiest scenario is a write-then-read race condition. Imagine: a user updates their profile. The write path deletes the cache entry. A concurrent read comes in, misses the cache, reads the old value from a database replica (which has not yet caught up due to replication lag), and writes that stale value back into the cache. Now the cache has stale data, and no one knows. This class of problem is subtle and is why some teams accept eventual consistency in their caches rather than fighting it.

Scalability and Distributed Caching

A single Redis node can handle hundreds of thousands of operations per second and store many gigabytes of data. For many applications, that is enough. But when it is not, you need to think about distributing the cache across multiple nodes.

The standard approach is consistent hashing. Each cache key is mapped to a node via a hash function. The consistent hashing ring ensures that when a node is added or removed, only a fraction of keys need to be remapped, rather than all of them. Redis Cluster uses a similar concept (with 16,384 hash slots) to shard data across multiple nodes automatically.

Another consideration is replication. Redis supports primary-replica setups where reads can be served from replicas for higher throughput, and the primary handles writes. If a primary fails, a replica can be promoted. This gives you both read scalability and fault tolerance, at the cost of some replication lag.

When Caching Hurts: Knowing When Not to Use It

Caching is not free. Let's be honest about when it is the wrong tool:

Highly unique or low-repetition reads: If every request is for different data, your cache hit rate will be near zero and you've added latency for the miss path with no benefit.
Strong consistency requirements: If your system cannot tolerate any stale data (financial transactions, inventory that must be exact), caching is dangerous without very careful invalidation.
Write-heavy workloads with few reads: Caching is a read optimization. If you write data and never read it again (or rarely), there is no benefit.
Security-sensitive data: Cached data may be accessible to more parts of the system than intended. Caching access control decisions or sensitive user data requires careful scoping.

Putting It All Together

In a system design interview, caching usually surfaces as the answer to "how do you handle read-heavy load" or "how do you keep latency low at scale." The strong answer is not just "add Redis." It is explaining which data to cache (hot, frequently-read, expensive-to-compute), where in the stack to cache it (CDN, distributed cache, in-process), how to populate it (cache-aside is usually the right default), how long to keep it (a TTL appropriate to how stale the data can be), and what happens when it fails (cache stampede mitigation, graceful degradation to the database).

The instinct to reach for a cache is usually right. The discipline is in thinking through what breaks when you do.

2172. Caching Overview

Similar Questions