Encryption
Match each encryption mechanism to the threat it actually stops, price a TLS handshake in round trips, and rotate keys without rewriting the data.
Encryption encodes information (plaintext) into a secret code (ciphertext) to protect its confidentiality; only a party holding the correct key can decode it back into plaintext (decryption). Two states of data matter in system design — in transit and at rest — and because they defend against two different attackers, one mechanism never covers both.
Encryption in transit
This secures data as it travels over a network against eavesdropping, tampering and "man-in-the-middle" (MITM) attacks. The mechanism is TLS (Transport Layer Security), the successor to SSL; HTTPS is simply HTTP running over a TLS connection. TLS uses asymmetric (public-key) cryptography for the handshake and key exchange, then faster symmetric cryptography for the data itself, because asymmetric operations cost orders of magnitude more CPU per byte.
The handshake is the part that changes a design, and it is priced in round trips rather than CPU. A round trip from India to US East is about 200 ms on the latency ladder, so a cold connection from Mumbai to a Virginia origin pays:
TCP handshake 1 RTT = 200 ms
TLS 1.3 handshake 1 RTT = 200 ms
---------------------------------------------------
before the first request byte moves = 400 ms
That is setup only. Time to first byte adds one more round trip — the request out to the origin, the first byte of the response back — so a cold cross-continent request cannot beat 600 ms; TLS 1.2, which spends two round trips on its handshake instead of one, cannot beat 800 ms. Choosing 1.3 over 1.2 is a version flag, not an architecture change, and it moves the floor by one round trip rather than under a budget. The rest is amortisation: pool connections so the setup is paid once per connection rather than once per request, which the HTTP lesson covers, and terminate TLS at a CDN edge so the handshake runs against a nearby machine.
mTLS (Mutual TLS) extends this. In standard TLS only the client verifies the server's identity; in mTLS both sides present and verify certificates, so the certificate becomes a service identity — the transport-level counterpart to the tokens in authentication and authorization. Encrypt internal traffic as well as external: with AES-NI, symmetric encryption runs near a gigabyte per second per core, so on a 0.5 ms datacenter hop the bytes are not the cost — handshakes and certificate rotation are.
Encryption at rest
This secures data stored on non-volatile media — disks, SSDs, databases, object storage — against physical theft of hardware or unauthorised access to files.
- Transparent Data Encryption (TDE): the database engine (MySQL, Oracle, SQL Server) encrypts data and log files on disk automatically, and the encryption and decryption are "transparent" to the application reading and writing.
- Filesystem or disk-level encryption: a whole volume encrypted by the operating system, with BitLocker on Windows or LUKS on Linux.
- Application-level encryption: the application encrypts specific sensitive fields — a national ID, a bank account — before they reach the database, so plaintext never exists there. It is the strongest and the most expensive to operate: the application now owns the keys, and the encrypted columns become hard to query.
The mistake is treating at-rest encryption as a defence against application compromise. TDE decrypts for any query the database authenticates, so a leaked read-only credential — or an SQL injection — returns plaintext rows out of a fully encrypted database. Disk encryption protects exactly one thing: media that leaves your control, such as a decommissioned drive or a snapshot copied into the wrong account.
Application-level encryption is the answer when the database itself is inside the threat model, and its price is queryability. With a random IV the ciphertext differs on every write, so equality, ordering, LIKE and range predicates all stop working: a report over an encrypted date of birth degrades from an index range scan into a full scan that decrypts every row. Two ways out, and choosing between them is a real trade-off:
- Deterministic encryption — the same plaintext always yields the same ciphertext, so equality works on the column directly. It leaks duplicates: a reader sees which rows share an email, and on a low-cardinality field frequency analysis recovers the values.
- A blind index — leave the column randomised and store
HMAC-SHA256(email, pepper)in a second column to look up on. Same equality, and the leak is confined to that one column. Ranges still fail, so keep a coarse plaintext bucket such as an age band if a report needs one.
Key management
Encryption moves the secret rather than removing it, so the question becomes who holds the key. Never hardcode keys in source code or configuration files. Use a Key Management Service (KMS) — AWS KMS, Google Cloud KMS, Azure Key Vault, HashiCorp Vault — which keeps keys in hardware-backed storage, controls who may call them, and logs every use.
Envelope encryption is what keeps that affordable. The KMS holds a key-encryption key (KEK) that never leaves it. Per object, the application asks for a data key and receives it twice — once in plaintext, once wrapped under the KEK. It encrypts the object with the plaintext key, stores the wrapped copy beside the ciphertext, and discards the plaintext. What lands on disk per object is that 32-byte key wrapped under the KEK plus the KEK id, the IV and the authentication tag needed to unwrap it — call it 200 bytes.
Two numbers pay for that extra hop, and the second bounds the first. Call volume: a KMS call leaves the VPC and takes single-digit milliseconds against an account quota in the low thousands of operations per second — take 2,000/s. Unwrapping on every read at 2,000 reads/s puts 2,000 × 0.005 s = 10 calls in flight by Little's Law — nothing for concurrency, but the whole quota. Cache the unwrapped data key under a short TTL. Rotation: with 100 million 1 MB objects, rotating a key that encrypted the data directly means rewriting 100M × 1 MB = 100 TB, roughly 100,000 s at a sustained 1 GB/s — nearly 28 hours of I/O competing with live traffic. Envelope encryption offers two different answers here, and running them together is the common error. Eager re-wrap is Θ(n): the bytes collapse to 100M × 32 B = 3.2 GB, some 31,000× less I/O, but the KEK never leaves the KMS, so every re-wrap is an API call and the 2,000/s quota sets the clock — 100M ÷ 2,000 = 50,000 s, about 14 hours. That is half the direct rewrite, not a thirty-thousandth of it: the binding constraint moved from disk bandwidth to call rate, and only the byte count fell by four orders of magnitude. Lazy rotation is the O(1) one: cut a new KEK version, re-wrap nothing, point new writes at it, and let old data keys stay decryptable under the versions the KMS retains. Its residue is that the old version can never be retired until an eager pass has drained behind it, so a rule forbidding live key material under a KEK older than a year buys back the 14-hour job as a rate-limited background task.
In an interview
What is being tested is whether you attach each mechanism to a threat, not whether you can list the acronyms; "we encrypt everything" scores nothing. Say instead: "TLS 1.3 on all traffic including service-to-service, mTLS inside the cluster so a compromised pod cannot impersonate a service; TDE on the database, which covers a stolen snapshot; application-level encryption on the two PII columns, with a blind index so email lookup still works; keys in a KMS under envelope encryption, so rotation does not mean re-encrypting the corpus."
Then price it, which is the step most candidates skip: give the handshake as round trips — one for TCP, one for TLS 1.3, about 200 ms each cross-continent — then add the request round trip if the budget is time to first byte, which puts a cold request at 600 ms before the origin does any work, and say where you terminate TLS and why.
The mistake that loses points is offering at-rest encryption as a defence against application compromise, answering a question about a leaked credential or SQL injection with "the database is encrypted". TDE hands plaintext to any query the database authenticates. Second-most common is naming a KMS and then keeping the key beside the data it protects.
Check yourself
1. Users are in India, the origin is in US East, and the p99 time-to-first-byte budget is 500 ms. A colleague proposes upgrading TLS 1.2 to TLS 1.3. Does that fix it?
No. Count to the first byte, not to the end of the handshake. TLS 1.2 costs 1 RTT for TCP plus 2 for the handshake, and time to first byte adds one more for the request to reach the origin and the first response byte to return: 4 × 200 = 800 ms, over budget before the server has done anything. TLS 1.3 removes one handshake round trip, which lands at 3 × 200 = 600 ms — still over the 500 ms budget, still with zero server time spent. Take the upgrade, because it is a version flag and costs nothing, but say plainly that it does not buy the SLO. The fix is shape: terminate TLS at an edge near the user so TCP and the handshake cost a few milliseconds locally, and the long haul becomes one pooled warm round trip of ~200 ms, leaving ~300 ms of the budget for the origin.
2. Compliance requires the users table encrypted. Product needs login by email and a report over "signed up in the last 30 days". What do you encrypt, and at which layer?
Not everything at the application layer, or both features break. Email needs equality only: randomised application-level ciphertext plus a blind index on
HMAC-SHA256(email, pepper). The rejected alternative is deterministic encryption of the column itself — one column fewer, but it shows any reader which accounts share an address. The signup timestamp needs a range scan, which no encrypted column supports, so leave it under TDE and defend it with access control. Say the residue out loud: TDE stops a stolen snapshot, not a stolen credential.
3. 100 million 1 MB objects, annual key rotation required. Estimate the cost with and without envelope encryption, and name the input that would change your answer.
Directly under one key, every object is rewritten: 100M × 1 MB = 100 TB, about 100,000 s at 1 GB/s sustained, nearly 28 hours competing with production traffic. Envelope encryption has two answers and a good one gives both. Re-wrapping every data key is Θ(n): 100M × 32 B = 3.2 GB, some 31,000× fewer bytes, but the KEK never leaves the KMS, so each re-wrap is an API call and a 2,000 ops/s quota puts the job at 100M ÷ 2,000 = 50,000 s ≈ 14 hours — the same order as the direct rewrite, because the constraint is call rate, not I/O. The O(1) answer is to re-wrap nothing: cut a new KEK version, send new writes to it, and let the KMS keep the old versions for decryption, with the eager pass trailing as a rate-limited background job. The input that changes the answer is object size. A wrapped data key costs about 200 B beside the ciphertext, so a 1 MB object carries 0.02% overhead while a 1 KB object carries ~20%, on top of one KMS call per object at write time — group objects under a shared data key and both costs divide by the group size.