Design image and video delivery

Media systems split into three problems that share almost no machinery: getting bytes in, turning them into the shapes clients actually need, and getting them out to the whole world. The objects are large — a photo is 2 MB, a video is gigabytes — so the design is dominated by bandwidth and storage cost rather than by request rate, which inverts most of the instincts built up on CRUD services. Two moves define the category: keep the bytes off your own servers on the way in, and never invalidate a cache on the way out.

Step 1 — Requirements

In scope:

• A user uploads a photo (up to 20 MB) or a video (up to 4 GB).
• Anyone permitted to see it views it, worldwide, quickly, on connections from 1 Mbps to fibre.
• Uploaded bytes are never lost.

Out of scope: the social graph and feed, comments, search, rights management, and moderation beyond the hook where it attaches.

Non-functional targets, stated as numbers in the style of gathering requirements:

• Durability of 11 nines on stored objects. At 36.5B objects added a year that is 36.5e9 × 1e-11 ≈ 0.4 expected losses per year, roughly one object every three years.
• Delivery availability 99.99%. Media bytes are immutable, which makes this much easier than it sounds.
• Time to first byte for a cached image, p99 under 100 ms globally. A round trip India to US East is ~200 ms, so a single-region origin cannot meet this target at all. That one number forces edge delivery before any other argument.
• Video starts playing within 2 s of pressing play.
• Upload durability is confirmed synchronously. Derivatives are async: photos ready within ~30 s at p95, video within ~2× its own duration.

The consistency posture: the media bytes are immutable, so CAP has nothing to say about them. It bites on the metadata row that says an object exists and is ready. When a partition occurs we choose consistency there — showing a broken image because a row claimed ready too early is worse than showing nothing — and PACELC's Else branch, the 99.9% of the time with no partition, is where we choose latency and serve every byte from the nearest edge. Background in consistency models.

Step 2 — Estimates

Assume 100M photos uploaded per day at 2 MB each, and 100 views per photo over its life.

Ingest.

100M uploads/day ÷ 86,400 s   ≈ 1,160 uploads/s average
peak at 3×                    ≈ 3,500 uploads/s
100M × 2 MB                   = 200 TB/day
200 TB ÷ 86,400 s             ≈ 2.3 GB/s ≈ 19 Gbps average
peak                          ≈ 7 GB/s  ≈ 56 Gbps

Storage. 200 TB/day × 365 = 73 PB/year of originals, before derivatives and before any durability overhead. Each photo also gets a thumbnail (~20 KB), a feed rendition (~200 KB) and a re-encoded full size (~500 KB): 720 KB on top of 2 MB, about 36% more, so ~100 PB/year of logical bytes.

Egress.

100M × 100 views              = 10B views/day
10B/day                       ≈ 120,000 QPS average (1B/day ≈ 12,000 QPS)
bytes per view, mixed sizes   ≈ 150 KB
10B × 150 KB                  = 1.5 PB/day
1.5 PB ÷ 86,400 s             ≈ 17 GB/s ≈ 140 Gbps average, ~420 Gbps peak

At an assumed CDN price near $0.02/GB, 1.5M GB/day × $0.02 ≈ $30,000/day, ~$11M/year in egress alone. That is the largest line item in the system, and most delivery decisions below are arguments about it. The habit comes from back-of-envelope estimation: size in bytes first, boxes second.

Step 3 — The upload path

The naive design has the client POST the file to an API server, which validates it and writes it to the object store. Take a service at 10,000 uploads/s to see why that fails:

10,000 uploads/s × 2 MB = 20 GB/s = 160 Gbps through the API tier
160 Gbps ÷ (10 Gbps NIC × 40% sustained) ≈ 40 machines

Forty machines whose entire job is copying bytes from one socket to another, adding nothing. Connections are worse than bandwidth. A 2 MB upload over a 5 Mbps mobile uplink takes 2 MB × 8 = 16 Mb ÷ 5 Mbps ≈ 3.2 s, and Little's Law turns that into 10,000/s × 3.2 s = 32,000 connections held open simultaneously — threads, TLS buffers and load balancer slots unavailable to the request path. At our own 3,500/s peak it is 7 GB/s and 11,200 pinned connections. Smaller, identical shape.

Pre-signed URLs. The client asks the API for an upload slot. The API authorises the user, writes a metadata row in state pending, and mints a pre-signed PUT URL: an object key, an expiry, a maximum size and an allowed content type, signed with a key only the API holds. The client then PUTs the bytes directly to the object store. The API tier moves about 1 KB per upload instead of 2 MB — a 2,000× reduction, turning 20 GB/s into 10 MB/s.

What you give up is the chance to inspect bytes before they land, and you get it back in three places: the signature constrains size, type, expiry (~15 minutes) and the exact key, so a client cannot overwrite someone else's object or upload a 40 GB file; the object stays private until processing passes; and scanning and moderation run in the pipeline below, gated on the same ready flag the reader path checks.

Multipart and resumability. A 4 GB video over a 20 Mbps uplink is 4 GB × 8 = 32 Gb ÷ 20 Mbps = 1,600 s ≈ 27 minutes on one connection. Mobile networks do not reliably hold a single TCP connection for 27 minutes, and with a single PUT one failure costs all of it.

Split it into 16 MB parts: 4 GB ÷ 16 MB = 256 parts, each separately signed, uploaded, checksummed and retried. A failed part costs 16 MB, about 6 s of re-upload, not 27 minutes. Eight parts in flight also lift aggregate throughput above what one connection's window allows. The client finishes by sending a manifest of part numbers and ETags, and the store assembles the object. Resume is: ask which part numbers already exist, upload the rest. Add a lifecycle rule that aborts incomplete multipart uploads after 7 days — abandoned parts do not appear in object listings and bill forever.

Step 3b — Processing is asynchronous, and has to be

The object store emits an object-created event. That event goes onto a queue, and a worker fleet consumes it.

Synchronous derivative generation is not merely slower, it is unimplementable. A 10-minute video transcodes in roughly 600 s of CPU work; at even 50 video uploads/s, Little's Law gives 50 × 600 = 30,000 requests held open, each an idle socket for ten minutes. Photos are cheaper — 200 ms to 1 s — but the same event pipeline handles both.

Media pipeline: pre-signed direct upload, asynchronous transcoding, and CDN delivery100%−+FitSVGEsc

Scroll to zoom · drag to pan · 0 fits · Esc closes

While processing runs, the uploading client already holds the bytes and renders its own local preview immediately. Other viewers get either nothing until status flips to ready, or a placeholder from a tiny blurhash string the client computed and sent with the metadata at slot-request time. For photos, ready typically lands before anyone else's feed request arrives.

Queue delivery is at-least-once, so duplicates are guaranteed and workers must be idempotent — the chain worked through in message queues and idempotency. Here idempotency is free if you take it: derive every output key deterministically from the content hash plus the rendition name, and a replayed job rewrites identical bytes to the same key. No dedupe table, no locking.

One consistency trap: if metadata reads come from a follower replica, replication lag means the uploader refreshes and their own photo is missing. Pin the uploader's reads to the leader for a short window after the write, and let everyone else read followers.

The bitrate ladder. A single MP4 at source bitrate is file hosting, not video delivery. Transcode each video into a ladder:

Rendition	Bitrate
240p	0.3 Mbps
360p	0.6 Mbps
480p	1.2 Mbps
720p	2.5 Mbps
1080p	5.0 Mbps

The ladder sums to 9.6 Mbps against a 5.0 Mbps source, so storage per video is about 1.9× the original. What that buys is adaptive bitrate streaming: HLS and DASH cut every rendition into 4-second segments and publish a manifest, and the player measures its own throughput per segment and picks the rendition for the next one. A phone on a 1 Mbps link plays 240p instead of buffering forever, and startup costs one segment rather than a full 5 Mbps buffer, which is how you hit a 2 s start.

Segmentation also parallelises the transcode. Ten minutes across five renditions is ~50 core-minutes serially. Cut into 600 ÷ 4 = 150 segments and each worker handles 4 s × 5 renditions ≈ 20 s of work, so 150 workers finish in well under a minute of wall clock. That is why transcoding is a queue-fed fleet and not a service with an endpoint. As a bonus, each segment is an ordinary immutable object the CDN caches like any image.

Step 4 — Delivery, and the insight that matters

Origin load is (1 − hit ratio) × total, and the leverage is nonlinear:

total: 120,000 view QPS, 17 GB/s
90% hit → origin 12,000 QPS, 1.7 GB/s
95% hit → origin  6,000 QPS, 0.85 GB/s   (20× smaller than serving everything)
99% hit → origin  1,200 QPS, 0.17 GB/s

Slipping from 95% to 90% doubles the origin. Climbing to 99% cuts it fivefold. Hit ratio is not a dashboard number, it is the input you size the origin with — and it is set by two things you control: the cache key and the TTL. Mechanics of the edge itself are in CDNs.

Cache keys. The key is what the edge hashes to find an object; anything you vary on multiplies entries for identical bytes. Varying on the whole query string is the standard self-inflicted wound: a URL decorated with ten campaign parameters becomes ten independent cache entries, each with its own cold start, repeated at every PoP. Strip query parameters from the key except an explicit allowlist that genuinely changes the bytes (w, h, format), and prefer encoding the format in the path so the key stays one string. Never vary on cookies or Authorization for media: if access control is needed, validate a signed token at the edge and exclude the token from the key, or every viewer gets a private copy and the hit ratio collapses to zero.

TTLs and the invalidation problem. A mutable URL such as /users/42/avatar.jpg forces a short TTL, because you cannot predict when the bytes behind it change. Set TTL to 60 s and do the arithmetic: an object viewed 20 times a day is viewed once every 86,400 ÷ 20 = 4,320 s, seventy times longer than the TTL, so almost every view is a miss. For the long tail — which is most objects — the hit ratio is effectively zero, and the 95% assumption above evaporates.

The alternative, purging on edit, is worse. A global purge fans out across hundreds of PoPs, takes seconds to minutes, and purge APIs are rate-limited in the hundreds-to-thousands per day. At 100M uploads/day, even a 1% edit rate is 1M purges/day ≈ 12/s sustained — three orders of magnitude past what those APIs exist to do. During propagation, different users hold different bytes and you cannot say who saw what.

Content-addressed URLs. Put a hash of the bytes in the path: /i/9f2c8a1b4d/feed.jpg, where the prefix is the leading hex of the SHA-256 of that derivative. Then:

• The URL and the bytes are the same fact. One URL can never map to two byte strings, so a cached copy can never be stale.
• Serve Cache-Control: public, max-age=31536000, immutable — one year, with revalidation on reload suppressed.
• An edit produces different bytes, therefore a different hash, therefore a different URL. Nothing at the edge needs to change.
• You never purge. Old objects age out of the edge by LRU and out of the origin by a lifecycle rule.

What changes on an edit is a pointer in the metadata row, which is small, lives in your own cache, and can be invalidated in milliseconds. You have traded an invalidation problem you cannot solve at 300 PoPs for one you solve in a database — the same trade a hashed JavaScript bundle filename makes, and the general answer to the cache invalidation question in caching.

Under immutability the earlier sizing becomes true: an object viewed 20 times misses once per PoP and hits nineteen times, and popular objects sit above 99%. Content addressing also deduplicates for free — the same image uploaded 50,000 times hashes to one key and is stored once. The one cost is ordering: you cannot name bytes you have not seen, so the delivery URL is assigned after processing. That is precisely why the metadata row starts pending and the client cannot construct the URL itself.

Storage tiering

Access is heavily front-loaded: assume ~90% of views land on objects under 30 days old, and year-old objects get a handful of views a year. Take list prices of roughly $0.023/GB-month hot, $0.010 infrequent-access plus a retrieval fee, $0.001 archive with retrieval in minutes to hours. On 73 PB (73M GB) of originals:

all hot                       73M × 0.023            ≈ $1.7M/month
30 days hot (6 PB) + rest IA  6M × 0.023 + 67M × 0.010 ≈ $0.8M/month
originals over 1 year archived 67M × 0.001           ≈ $67k/month for that slice

Tier the originals, never the derivatives. The original is the source of truth, kept so you can re-encode when the ladder changes; the derivatives are what viewers request, and archive retrieval measured in hours cannot sit on a user-visible path. Lifecycle rules do the moves automatically by prefix and age.

Durability: replication versus erasure coding

Three-way replication keeps three full copies in three failure domains: 3× storage, survives 2 simultaneous losses, and repair is a plain copy.

Reed-Solomon (10, 4) splits an object into 10 data fragments, computes 4 parity fragments, and spreads all 14 across 14 failure domains. Any 10 reconstruct the object. Overhead is 14/10 = 1.4×, and it survives 4 simultaneous losses. Better durability at less than half the storage.

73 PB of originals, 3× replication  = 219 PB
73 PB, RS(10,4)                     = 102 PB
difference 117 PB × $0.023/GB-month ≈ $2.7M/month

Erasure coding is not free. A degraded read, when a fragment's host is down, fetches 10 fragments from 10 machines and reconstructs, so its latency is the maximum of ten calls rather than one — the effect quantified in tail latency. Repairing a single lost fragment reads 10 fragments, a 10× read amplification, and at petabyte scale repair traffic never stops. Small objects fragment badly: a 20 KB thumbnail becomes fourteen ~1.4 KB pieces whose metadata and per-request overhead swamp the payload.

So: erasure-code large, cool objects; replicate small, hot ones. A 2 MB photo yields comfortable 200 KB fragments. Thumbnails get replicated, or packed many-to-a-container with the container erasure-coded.

Trade-offs

Decision	The naive option	What we ship	The number that decides it
Upload path	POST bytes through the API tier	Pre-signed PUT direct to the object store	10,000/s × 2 MB = 20 GB/s and 32,000 held connections
Large files	One PUT, retry from zero	Multipart, 16 MB parts, resumable	4 GB at 20 Mbps = 27 min per attempt; a failed part costs 6 s
Derivatives	Generate inline in the request	Event to queue to worker fleet	600 s of transcode work would hold 30,000 sockets at 50 videos/s
Video format	One file at source bitrate	Five-rung ABR ladder, 4 s segments	1.9× storage buys playback on 1 Mbps and a 2 s start
Invalidation	Short TTL, or purge on edit	Content-addressed URLs, one-year immutable TTL	60 s TTL vs a view every 4,320 s; 12 purges/s vs APIs built for hundreds a day
Cache key	Whole query string	Host, path, allowlisted params	10 campaign params = 10 cold entries per PoP for identical bytes
Origin sizing	Provision for full read traffic	Provision for (1 − hit ratio)	95% hit means 6,000 QPS at origin, 20× smaller
Storage class	Everything hot	Age-tier originals, derivatives stay hot	$1.7M/month falls to ~$0.8M/month at 73 PB
Durability	3× replication everywhere	RS(10,4) on large objects, replicate small ones	219 PB vs 102 PB, about $2.7M/month

In an interview

What is being tested is whether you recognise a bandwidth-and-storage problem rather than a QPS problem, and whether you reach for the two moves that define it: get the bytes off your servers, and stop invalidating caches.

Say it in this order:

1. Size in bytes before boxes: "100M × 2 MB is 200 TB/day, 73 PB/year, and about 1.5 PB/day of egress at 100 views each."
2. Propose pre-signed URLs and justify with the 20 GB/s figure, then volunteer what you lose and how the constrained signature plus async scanning gets it back.
3. Draw upload and delivery as two paths that meet only at the object store.
4. Give origin = (1 − h) × total and use it to size the origin, not as trivia.
5. Say "we never invalidate; the URL contains a hash of the content," then set the TTL to a year and explain that an edit is a new URL.
6. Name erasure coding with its overhead number. 1.4× against 3× is the entire argument.

The mistakes that lose points:

• Proxying uploads through the app tier and not noticing. This is the first thing an interviewer checks in this question.
• Answering cache invalidation with "we purge the CDN." Do the purge-rate arithmetic out loud, then give versioned URLs.
• Synchronous transcoding, or a transcoding service rather than a queue-fed fleet that parallelises by segment.
• Storing one video file. If you cannot explain adaptive bitrate, you have designed file hosting.
• Claiming 11 nines without saying how. Durability is a mechanism with a storage overhead, not an adjective.
• Never mentioning cost. This is one of the few designs where dollars are a stated requirement, and egress is what the real team optimises hardest.

Check yourself

1. A release appends a per-user tracking parameter to every image URL and the CDN hit ratio falls from 95% to 88%. Quantify the damage and name the fix.

Origin share goes from 5% to 12%. Origin requests go 6,000 to 14,400 QPS and origin egress 0.85 to 2.0 GB/s: 2.4× the origin capacity for byte-identical content. The cause is cache-key fragmentation — a per-user parameter makes every user's first view a miss, at every PoP. Fix it at the CDN, not in the app: drop the parameter from the cache key via an allowlist so all variants collapse to one entry. The bytes never depended on it, so the key should not either.

2. A product manager wants users to crop an already-published photo and have the change visible everywhere within 5 seconds. Your URLs are content-addressed with a one-year TTL. What do you do?

Not a purge. The crop is different bytes, so it hashes to a different URL; the work is making the pointer swap fast. Update the metadata row, invalidate your own small pointer cache — set its TTL to a few seconds — and the next render emits the new URL. Time-to-visible is transcode time plus that pointer TTL, both of which you control, and both well inside 5 s for a photo. The old URL keeps serving the old crop to anyone still holding it, which is correct rather than a bug, and ages out by LRU. Clients holding long-lived HTML need a push to re-render, but that is a client refresh problem, not a CDN one.

3. Halve the storage cost of a 73 PB/year library without weakening durability. Show the arithmetic and say where each move does not apply.

Two independent moves, either close to sufficient. Encoding: 3× replication is 219 PB, RS(10,4) is 73 × 1.4 = 102 PB, a 53% cut that also raises tolerance from 2 simultaneous losses to 4. Tiering: keep 30 days hot (30 × 200 TB = 6 PB) at $0.023/GB-month and move the other 67 PB to infrequent access at $0.010, so $1.7M/month becomes about $0.8M/month. Where they do not apply: do not erasure-code 20 KB thumbnails, where fourteen ~1.4 KB fragments cost more in metadata and requests than they save — replicate those or pack them into containers. And do not tier derivatives, only originals: archive retrieval is minutes to hours and nothing on the viewing path can wait that long.