Steam-like Game Store & Launcher

Requirements#

Clarify the scope first#

The prompt offers a choice — "focus on one or two of commerce, entitlement, or distribution" — and the real interview punishes candidates who take it too literally: design one area in isolation and the interviewer will ask, forty minutes in, how the whole thing fits together. Cover all four systems at a high level, then go deep where the physics is hardest. The answers below are the ones an OpenAI-style interviewer gives when asked:

Out of scope (say this out loud): the payment provider's internals, tax, the social graph, chat, anti-cheat, DRM internals, and the game engine. Each is a good extension question; none should eat the first ten minutes.

Functional Requirements#

Four FRs, each mapping to one phase in the High Level Design.

FR1 — Browse and purchase games; grant and revoke ownership

A user browses a catalog that changes rarely and buys a game with POST /purchases carrying an idempotency key. The purchase is charged once, recorded durably, and turned into an entitlement that the user's library shows on every device. A refund revokes exactly that entitlement.

FR2 — Publishers upload and release builds

A publisher uploads a 100 GB build straight to object storage, the platform chunks and hashes it, deduplicates it against earlier builds, publishes an immutable manifest, and releases the build at a chosen time. A 2 GB patch is a new manifest, not a new 100 GB upload.

FR3 — Download, install, resume, and incrementally update games

An authorized launcher receives a manifest plus short-lived signed CDN URLs, downloads only the chunks it lacks, verifies every chunk by hash, survives interruptions and CDN failures without restarting, and installs atomically.

FR4 — Verify ownership before launch, including offline grace periods

The launcher obtains a signed 7-day license when online, validates it locally when offline, renews it against the authoritative entitlement state, and refuses to run when it has expired or been tampered with.

Non-Functional Requirements#

NFR1 — Scale. 10M DAU; 100K purchases and 1M update downloads per day normally, 100x in a sale; a launch that moves 400 PB in 24 hours. The last number dominates every other load in the system combined.

NFR2 — Latency. Ownership verification p99 under 50 ms for a warm lookup. Download initiation p95 under 2 s. Full downloads are best-effort and measured in minutes to hours; the backend's job is to start them fast and never slow them down.

NFR3 — Correctness and durability. Exactly one charge per checkout; one grant per purchase; a refund always revokes; 99.999% durability for purchases, grants, and builds. A player who is told they do not own a game they paid for will be loud about it.

NFR4 — Availability and recovery. Entitlement 99.99%+ (it sits on the launch path), store 99.9% and degradable (browsing may serve stale catalog data; checkout may queue). Downloads resume after network hiccups and CDN outages; entitlement grants survive a store crash and an entitlement-service outage.

NFR5 — Cross-device consistency and offline access. The same account sees the same library on every machine. An installed, previously authorized game launches offline for a bounded period, with the revocation lag that implies stated explicitly.

Scale & Back-of-Envelope Estimations#

The number that identifies the real system is not 10M DAU — it is the asymmetry between checkout and distribution:

checkout at sale peak      ~120 orders / s          (one PostgreSQL transaction each)
launch day egress          ~37,000,000,000,000 bits / s

Commerce and entitlement are ordinary web systems with a hard correctness story; content distribution is a capacity-planning problem whose levers are time (preload), deduplication (content-addressed chunks and manifest diffs), and edge capacity (multi-CDN with an origin shield) — not anything that happens in the origin. Every number above is an illustrative assumption to be replaced by measured download completion rates, cache hit ratios, and per-region throughput.

Interviewer signal: candidates who multiply 5M by 80 GB and stop have done arithmetic. Candidates who notice that a 7-day preload turns 37 Tbps into 5 Tbps and the unlock into a 5 GB key-distribution event, and then name content-addressed chunks as the reason a 2 GB patch moves 2 GB, have found the system.

Follow-up Questions to Expect#

These nine questions give the rest of the design its purpose. Raise them before drawing the final architecture — every component below exists because one of them forces it.

• [DD1] Incremental updates and chunking: version 1.1 changes 2 GB of a 100 GB game, and a 1 MB insert near the start of a 20 GB file. How do players download ~2 GB and not 100 GB?
• [DD2] Launch-day scalability: 5M players, 80 GB, 24 hours, 400 PB. How do you serve it without melting the CDN or the budget?
• [DD3] Preload and simultaneous unlock: the publisher wants midnight UTC everywhere but allows a 7-day preload. How can players download early and not play early?
• [DD4] CDN failure mid-download: one provider degrades for 30% of players at 70% progress. How do you detect it, move them, and resume without restarting?
• [DD5] Payment timeout and idempotency: the provider charged 60 USD and the response was lost; the user retries. No double charge, no lost purchase — how?
• [DD6] Crash-safe event delivery: the purchase is PAID but the event never reached Kafka. How does the entitlement still arrive? And what do Alice, Bob, and Charlie see during a 30-minute entitlement outage?
• [DD7] Refund and repurchase race: GRANT #100, GRANT #200, then a late REVOKE #100. How do you avoid revoking the new purchase?
• [DD8] Entitlement scale and cache consistency: partitioning, caching, and the race where a refund commits and a stale cache issues a 7-day license.
• [DD9] Offline launch and revocation: what is in the signed license, how does the launcher verify it offline, what happens after a refund, and what about a clock set back to day 3?

Core Thinking & Design Pattern#

Before drawing boxes, orient around three tensions that shape everything:

1. Four systems with four different physics#

Commerce is money: exactly once, a third party that can time out after charging, auditable. Entitlement is a small, read-dominated dataset where availability and correctness matter far more than throughput. Distribution is petabytes: a capacity-planning and scheduling problem where the backend must stay out of the data path. Runtime access is an adversarial, sometimes-offline client. Say this asymmetry out loud early; a candidate who gives all four an equal, shallow treatment has not noticed it.

2. Separate the purchase, the grant, and the license#

A Purchase is a financial transaction with a provider reference. An EntitlementGrant is the durable right that one purchase created — one row per purchase, so a refund revokes exactly its own grant and "owns the game" means "any ACTIVE grant exists". A LicenseToken is a short-lived, signed capability the launcher holds so it can start the game offline. Three objects, three lifetimes, three failure modes; collapsing them is the root of most wrong answers (DD5, DD7, DD9).

3. Control plane authorizes, data plane carries, the client remembers#

The backend decides whether and what: it checks the grant, picks the RELEASED build, returns a manifest and short-lived signed URLs. The CDN carries bytes that are immutable and named by their hash, so any edge of any provider can serve them and the launcher can verify them. The launcher keeps its own progress on disk, so 5M clients times 12,500 chunks is never a write to PostgreSQL and a CDN failover is a URL refresh, not a restart.

Core Entities & API#

Core Entities#

Five entities carry the whole design. Spend your explanation on Purchase, EntitlementGrant, and GameBuild — their consistency is the architectural problem. LicenseToken is signed, not stored.

Game

The catalog entry: what is for sale and which build is current.

Purchase

One checkout attempt: the financial transaction, owned by the store.

EntitlementGrant

The durable right that one purchase created, owned by the entitlement service.

GameBuild

One version of a game's content and where its manifest lives.

LicenseToken

Temporary, signed authorization to launch; held by the launcher, not persisted as a row.

Three state machines — deliberately small:

Purchase     PENDING -> PAYMENT_PROCESSING -> PAID -> FULFILLED
             PAYMENT_PROCESSING -> PAYMENT_UNKNOWN (timeout) -> PAID | FAILED   (after reconciliation)
             FULFILLED -> REFUNDED

Grant        ACTIVE -> REVOKED                      (one grant per purchase; never back to ACTIVE)

GameBuild    UPLOADING -> PROCESSING -> READY -> RELEASED

There is no transition from a timeout to FAILED without asking the provider, and no transition that extends an existing license. Every path from PAID to a grant passes through the outbox (DD6), and every license is issued from an authoritative read (DD8).

Storage split

API Design#

External (FR1, FR3, FR4)

Three groups of calls, one per FR. Every call carries a bearer token, and the user is identified from that token, never from a body field.

Submission of money is synchronous up to PAID; everything after it is asynchronous and durable. The download endpoints return pointers and capabilities, never bytes; that is how 400 PB never touches a service.

Publisher (FR2)

Internal (events and service calls)

Internal calls go over gRPC; events are keyed by user_id so one user's grant and revoke stay ordered on one partition. No service writes another service's tables.

High Level Design#

The best way to present this design is not to draw the final architecture immediately. Start with the two flows the interviewer asked about — buy → own, and upload → download → launch — add one component only when a real failure forces it, and keep every box's position stable. Narrate it in four phases, each in its own colour on the whiteboard: Phase 1 purchase (black), Phase 2 grant (blue), Phase 3 publish (green), Phase 4 download and launch (purple). Every phase changes a few rows, so the schema blocks next to the database carry example values and show exactly what each phase writes. Each phase below is an interactive walkthrough: it plays on its own as it comes into view, and you can pause, step with the arrows or keyboard, click any component for its role, switch scenarios, and expand it.

Walk the interviewer through the picture phase by phase:

Phase 1  purchase  User A -> API-GW -> store service -> PostgreSQL (Purchase P100: PENDING)
                   -> payment provider (charge $60, key k1) -> confirmed pay_9f3
                   -> ONE transaction: P100 PAID + outbox GrantEntitlement ev-501 -> 201 {P100, PAID}
Phase 2  grant     outbox relay -> Kafka -> entitlement service
                   -> ONE transaction: grant E1 ACTIVE (user_a, game_1, P100) + processed_events [ev-501]
                   -> invalidate Redis entitlements:user_a -> EntitlementGranted -> P100 FULFILLED
Phase 3  publish   Publisher B -> API-GW -> Build Service -> GameBuilds 102 UPLOADING -> signed upload URLs
                   -> direct upload 100 GB to Blob Storage -> CDC chunks, SHA-256, dedup, manifest last
                   -> READY -> RELEASED at release_at -> prewarm hot chunks on the CDN
Phase 4  play      launcher -> API-GW -> download service -> entitlement check (Redis fast path)
                   -> license L1 (authoritative read, signed) -> manifest + signed CDN URLs (15 min)
                   -> chunks by hash from the CDN, verify SHA-256 -> launch after a local license check

A refund does not add a fifth phase: it is Phase 1 and Phase 2 again in reverse — REFUNDED plus a RevokeEntitlement event in one transaction, one grant revoked, one cache key deleted. It is covered at the end of this section and in DD7 and DD9.

Phase 1 — Purchase (FR1): one charge, one row, one durable promise#

POST /purchases  ->  validate price  ->  INSERT purchases (P100, PENDING)   [unique (user_id, idempotency_key)]
                 ->  charge $60 at the provider with the SAME key k1
                 ->  confirmed pay_9f3  ->  ONE transaction: status PAID, payment_reference, outbox ev-501
                 ->  201 { purchase_id: P100, status: PAID, library: pending }

timeout from the provider  ->  PAYMENT_UNKNOWN (never FAILED)  ->  503 Retry-After
retry with k1              ->  unique index -> existing row -> reconcile by key / webhook -> PAID

Three design points worth narrating here:

First, the row exists before the money moves. The purchase is inserted PENDING under a unique (user_id, idempotency_key) index before the provider is called. A client retry after a timeout hits the index, reads the existing row, and continues from wherever it stopped. The uniqueness is enforced by the database, not by application code that checks and then inserts (DD5).

Second, a timeout is unknown, never failed. The provider may have charged the card before the response was lost. The row goes to PAYMENT_UNKNOWN, the client gets a retryable error, and the store reconciles with the provider by key (or from the provider's webhook) before it does anything else. Marking it FAILED and letting the retry open a new charge is the bug every payment team has lived through.

Third, PAID and the grant event are one write. The status change and the GrantEntitlement outbox row are committed together. A purchase can therefore never be PAID without a durable event that will eventually put the game in the library, no matter what crashes next (DD6). The response says library: pending, because payment success is not ownership yet.

Phase 2 — Grant the entitlement (FR1): at-least-once delivery, once-applied grant#

outbox relay:  SELECT ... WHERE published_at IS NULL  ->  produce ev-501 (key user_a)  ->  mark published after ack
consumer:      BEGIN
                 INSERT INTO processed_events (ev-501) ON CONFLICT DO NOTHING;        -- 0 rows -> skip, ack
                 INSERT INTO entitlement_grants (E1, user_a, game_1, P100, ACTIVE, version 1);
               COMMIT;  ack offset
               DEL entitlements:user_a
               emit EntitlementGranted (ev-510)  ->  store: UPDATE purchases SET status='FULFILLED' WHERE P100 AND status='PAID'

The relay, not the store, publishes. The store's only job was to commit the row. A separate relay reads unpublished rows in order and marks them published only after the broker acknowledges; if it crashes in between it publishes again, which is fine. Shared durable state, not an in-memory queue in the store process, is what the next relay instance reads.

One grant per purchase, written once. The consumer inserts the event_id into processed_events in the same transaction as the grant, and acknowledges the Kafka offset only after the commit. A redelivery finds the event already processed and does nothing. The grant row is E1 for P100 — not a mutable "owns game_1" flag — so a later refund revokes exactly E1 and a second purchase is a second, independent row (DD7).

Services share events, not tables. The entitlement service never writes purchases; it emits EntitlementGranted and the store moves P100 to FULFILLED. A reconciliation job that looks for PAID rows without a grant after a few minutes is the safety net. It is not the mechanism.

Phase 3 — Publish a build (FR2): the Build Service is a control plane#

1  POST /builds (game_1, 1.1, 100 GB)        ->  INSERT game_builds (102, UPLOADING)
2  signed multipart upload URLs               ->  publisher PUTs 100 GB straight to staging/102/  (never through a service)
3  POST /builds/102/complete                  ->  UPLOADING -> PROCESSING (idempotent)
4  workers: content-defined chunking (rolling hash, 4-16 MB)  ->  SHA-256 per chunk
            HEAD chunks/<sha256>: exists? skip : PUT          ->  251 new chunks (2 GB), 12,249 reused (98 GB)
            PUT builds/game_1/1.1/manifest.json  LAST         ->  { files: [ { path, size, chunks: [sha256...] } ] }
5  READY (manifest_url, size, changed_bytes)  ->  POST /builds/102/release { release_at }  ->  RELEASED at T-0
6  prewarm the 251 new chunks in every region through the origin shield; Game.latest_build 101 -> 102

Bytes never pass through the Build Service. It issues signed upload URLs and the publisher's build farm uploads in parallel to object storage. The service tracks the row, orchestrates workers, and flips statuses; it could be restarted mid-upload without affecting a single byte.

A build is a manifest over immutable chunks. Chunk boundaries come from the content (a rolling hash), so an insert near the start of a file does not shift every boundary after it; SHA-256 names each chunk, so identical content across versions and across games is stored and transferred once. Version 1.1 writes 251 new objects and reuses 12,249 (DD1). The manifest is written last, after every chunk exists, so a manifest's existence is proof of a complete build.

Release is a status flip at a time. READY means "everything is in place"; RELEASED means "launchers may have it", and it happens at release_at on the server clock. For a launch, prewarming the new chunks on every CDN region before the flip is what keeps the first million requests from all missing to the origin (DD2).

Phase 4 — Download and launch (FR3, FR4): authorize, point, carry, verify#

1  launcher -> POST /downloads { game_1, installed_build: 101, device_id }
2  download service -> entitlement service: CheckEntitlement (Redis fast path) + IssueLicense
3  entitlement service: authoritative read on the shard -> E1 ACTIVE -> sign L1 { user_a, game_1, dev-7, iat, exp +7 d }
4  download service: RELEASED build for game_1 = 102 -> manifest + signed CDN URL templates (15 min, chunks/ prefix)
5  launcher: diff manifest 102 against local hashes -> need 251 of 12,500 -> GET by hash, 8-16 concurrent
             verify sha256(chunk) == name -> write -> checkpoint on disk; Range requests on resume
6  Play: verify_public(L1), game_id / device match, now < expires_at, now >= last_seen_time -> start the game

The server never trusts the client's view of ownership. The launcher's library screen may say "owned" from a cached read; the download service still asks the entitlement service, and the entitlement service reads the authoritative shard before it signs anything (DD8). Download authorization may take the Redis fast path because an unauthorized download of encrypted, unlaunchable bytes is cheap; a 7-day license is not.

URLs are short-lived capabilities. The CDN serves anyone with a valid signature for the chunks/ prefix for 15 minutes; it does not know what an entitlement is. The launcher refreshes URLs through the same endpoint when they expire or when a provider fails, and the content is the same hash on every provider, so failover is a URL change and a Range request (DD4).

Launch is local. With a valid license on disk the launcher needs no network to start the game: signature with the public key, identity, expiry, and a persisted last_seen_time that catches a clock set backwards. Renewal, when online, goes back to the authoritative read; a refund makes the next renewal fail, and that is the bound on offline revocation (DD9).

Refund and revoke (FR1): the same machinery in reverse#

POST /purchases/P100/refund  ->  provider refund (idempotent by purchase)
                             ->  ONE transaction: P100 FULFILLED -> REFUNDED + outbox RevokeEntitlement ev-502
relay -> Kafka -> entitlement service:
                 UPDATE entitlement_grants SET status='REVOKED', version=version+1, revoked_at=now()
                  WHERE purchase_id='P100' AND status='ACTIVE';          -- touches E1 only
                 INSERT processed_events (ev-502);  COMMIT;  DEL entitlements:user_a
online:   next CheckEntitlement / IssueLicense -> no ACTIVE grant -> denied
offline:  the launcher's L1 stays valid until its expires_at; it is never extended

A revoke names its purchase. WHERE purchase_id = 'P100' is what makes a late revoke harmless to a later repurchase (DD7). A revoke never touches a token already on a disk. The backend can stop issuing and renewing; it cannot reach a laptop on a plane. Say that trade-off before the interviewer asks (DD9).

Deep Dives#

DD1Incremental updates and chunking#

Scenario: Version 1.1 changes 2 GB of a 100 GB game, and a 1 MB insert lands near the start of a 20 GB file. How do players download about 2 GB, not 100 GB?

build 1.0  asset.pak -> [aaa, bbb, ccc, ddd, eee, ...]            launcher has all of them

fixed 4 MB chunks + 1 MB inserted after A:
           [A' a2f][X+B 7c1][B+C 0e9][C+D 5b3][D+E 91d] ...       every window shifted -> every hash new -> ~20 GB

content-defined chunks (rolling hash over the bytes, 4-16 MB):
           [A' a2f][X x91][B bbb][C ccc][D ddd][E eee] ...        boundaries resynchronise -> 2 new hashes -> ~5 MB

launcher:  need = manifest(1.1).hashes - local.hashes             = {a2f, x91}     (whole build: ~2 GB of 100)

Content-defined, not fixed-size. A rolling hash over the content decides where a chunk ends, with a minimum, target, and maximum size (4, 8, 16 MB). Because the boundary depends on the bytes near it and not on the offset from the start of the file, an insert changes the chunk it lands in and the one after, and then the old boundaries reappear. Fixed-size chunking is fine for parallel upload; it is the wrong tool for deltas.

The hash is the name. Each chunk is stored at chunks/<sha256> and the manifest maps each file to an ordered list of hashes and sizes. That one decision buys deduplication across versions and across games, integrity verification on the client, immutable CDN caching with a one-year TTL, and a trivially correct update algorithm: download the hashes you do not have. Chunk version counters buy none of those and have to be maintained by hand.

Pick the size with numbers. 2 GB chunks make small changes expensive and resume coarse. 64 MB (about 1,600 per build) is a reasonable middle. 4-16 MB (about 12,500 per build) gives fine deltas and resume at the cost of a ~1 MB manifest and more requests, which an HTTP/2 connection to a CDN absorbs easily. Separate the chunk count from download concurrency: 12,500 chunks, 8-16 in flight.

what CDC solves        inserts and shifts do not invalidate the rest of a file
what hashing solves    dedup, integrity, immutability, and "download only what is missing"
what neither solves    a publisher who re-encodes every asset; measure changed_bytes per release and tell them

DD2Launch-day scalability#

Scenario: 5 million players download an 80 GB build within 24 hours, 400 PB in total. How do you serve it without melting the CDN or the budget?

no preload        400 PB / 86,400 s                     = 37 Tbps for 24 h
7-day preload     400 PB / 604,800 s                    = 5.3 Tbps average, encrypted chunks, off-peak friendly
T-0               5M clients x ~1 KB release key        = 5 GB on one small endpoint

levers, by impact:  1 preload (time)  >  2 multi-CDN + prewarm + origin shield (edge capacity)  >  3 admission control (shape)

Preload is the lever that changes the problem. Let entitled players (pre-orders) download the build during the week before launch as encrypted chunks they cannot run. Launch day then has to deliver a key, not a game. The 37 Tbps spike becomes a 5 Tbps average that can be scheduled into off-peak hours per region, and the unlock is a brief spike on a tiny endpoint (DD3).

Then buy edge capacity, in more than one place. Multiple CDN providers, each pre-warmed with the new build's chunks in every region, with a regional origin shield in front of Blob Storage so that a cold edge fetches a chunk from the shield and the shield fetches it from the origin once. Request coalescing at the shield means a thousand simultaneous misses on the same chunk become one upstream request. The origin's job on launch day is to serve each chunk a few dozen times, not a few million.

Shape what is left, regionally. A traffic director hands launchers a provider and a rate based on measured per-provider, per-region capacity and health; randomise start times and let clients back off and resume rather than fail; cap by region, never globally. Peer-assisted transfer on LANs and ISPs is a legitimate further egress reducer because hashed chunks are verifiable without trusting the peer — mention it with its NAT and privacy caveats.

DD3Preload and simultaneous unlock#

Scenario: The publisher wants midnight UTC everywhere but allows a 7-day preload. How do players download early without playing early?

T-7 d   POST /downloads (preload)  ->  entitlement OK (pre-order)  ->  signed URLs for ENCRYPTED chunks
        launcher: 12,500 / 12,500 chunks on disk, verified by hash, unreadable without rk-7
T-1 h   GET /release-key (L1, build 102)  ->  403 NOT_RELEASED_YET      (server: now < release_at)
T-0     GET /release-key                  ->  grant ACTIVE?  now >= release_at?  ->  rk-7
        launcher: decrypt, verify, launch               5M x ~1 KB on one endpoint, not 400 PB on the CDN

Encrypt the build, not the transport. The chunks on the CDN are ciphertext under a per-build release key; a chunk's hash is the hash of the ciphertext, so integrity checks and deduplication work unchanged. Preloaded bytes are useless until the key arrives, which is the only thing the embargo needs.

The backend owns the clock. The key endpoint checks two facts the client cannot forge: an ACTIVE grant, and now >= release_at on the server's clock. A launcher with its system time set to tomorrow gets the same 403 as everyone else. The launcher's local clock plays no role in the unlock, which is why this problem is easy and offline expiry (DD9) is hard.

Size the unlock, not the download. At T-0 the key endpoint sees 5M requests in a few minutes — a few thousand per second against a tiny, cacheable-per-user response. Pre-scale it, put it behind the same rate limiter as checkout, and let launchers retry with jitter; that is a normal web spike, which is the whole point.

DD4CDN failure mid-download#

Scenario: One CDN provider degrades for 30% of players at 70% progress. How do you detect it, move them, and resume without restarting?

detect    client-reported chunk error rate, timeouts, throughput and p99 per provider per region
          -> circuit breaker: CDN-1 out for EU; new URL grants go to CDN-2
move      launcher: POST /downloads/refresh { build 102 }  ->  signed URLs on CDN-2 (same chunks/<sha256> names)
resume    local checkpoint: done 8,750 / 12,500 by hash; ccc partial at 2.1 MB of 8 MB
          GET cdn-2/chunks/ccc   Range: bytes=2100000-   ->  verify sha256 -> continue with the remaining 3,749

Measure downloads, not heartbeats. A provider can be reachable and useless. Launchers already know their per-chunk error rate and throughput; have them report it, aggregate per provider per region, and open a circuit breaker when the p99 or error rate crosses a threshold. New URL grants stop naming the degraded provider within a minute; nobody waits for a status page.

Progress belongs on the launcher. The checkpoint on disk is the manifest hash, the set of verified chunk hashes, and the byte offset of any partial chunk. It is written locally every few seconds and costs the backend nothing. The GameBuilds row describes the build; it never describes one user's download.

Content-addressed chunks make failover free. The chunk is chunks/<sha256> on every provider, backed by the same origin. Switching providers changes the host in the URL and nothing else; a half-downloaded chunk resumes with an HTTP Range request and is verified by hash before it is trusted. The URL refresh is the same call the launcher already makes every 15 minutes when signatures expire.

DD5Payment timeout and idempotency#

Scenario: The provider charged 60 USD, the response timed out, and the user clicks Buy again. No double charge and no lost purchase. How?

1  POST /purchases  Idempotency-Key: k1   ->  INSERT purchases (P100, user_a, game_1, 60.00, k1, h1, PENDING)
                                               unique (user_id, idempotency_key)     -- the dedup mechanism
2  charge $60 at the provider with provider idempotency key k1 -> provider charges pay_9f3
3  response lost  ->  P100 PAYMENT_UNKNOWN (NOT FAILED)  ->  503 Retry-After to the client
4  retry, same k1  ->  INSERT ... ON CONFLICT DO NOTHING -> 0 rows -> read P100: h1 matches, PAYMENT_UNKNOWN
5  reconcile: GET provider/charges?idempotency_key=k1 (or the webhook)  ->  succeeded, pay_9f3
6  ONE transaction: P100 PAID, payment_reference pay_9f3, outbox ev-501  ->  201 { P100, PAID }
   same k1, different payload  ->  409          never: a second provider attempt under a new key

The row before the money. Inserting P100 under a unique (user_id, idempotency_key) index before the provider call is what turns a retry into a lookup. Enforce uniqueness in the database; "select, then insert if absent" in application code races against itself under a double click.

Bind the key to the payload. k1 means "this checkout": game, price, currency. A retry with the same key and a different payload is a different intent and gets a 409, never a silent reuse of the old purchase. Store the payload hash on the row.

Unknown is a state. A timeout tells you nothing about the charge. PAYMENT_UNKNOWN says so, and the only valid next steps are asking the provider by key or receiving its webhook — never a new charge. The provider's idempotency key means even a duplicated provider call during the first attempt cannot double-charge. After PAID, the outbox event guarantees the game arrives (DD6).

DD6Crash-safe event delivery#

Scenario: The purchase is PAID and the store crashes before Kafka hears about it. Does the game ever arrive? And what do Alice, Bob, and Charlie see during a 30-minute entitlement outage?

BEGIN;
UPDATE purchases SET status = 'PAID', payment_reference = 'pay_9f3' WHERE purchase_id = 'P100';
INSERT INTO outbox_events (event_id, type, purchase_id, payload) VALUES ('ev-501', 'GrantEntitlement', 'P100', {...});
COMMIT;                                     -- both rows or neither; the store may die right here

relay:   SELECT ... WHERE published_at IS NULL  ->  produce  ->  UPDATE ... SET published_at = now()   (after broker ack)
         crash between produce and mark  ->  produced twice  ->  consumer: processed_events dedups (Phase 2)

30 min entitlement outage:
  Alice   license on disk valid 3 more days                 -> plays
  Bob     bought 5 min before: PAID, event waiting in Kafka -> cannot launch yet, "adding to your library"
  Charlie buys during the outage: store + provider are up    -> PAID, FULFILLED pending; library catches up after recovery

One transaction, two tables, same database. The outbox row is written by the same COMMIT as the status change. That is the entire guarantee: there is no moment in which the purchase is PAID and the event does not exist durably. The relay is a separate process that reads durable rows, so a store crash at any instant changes nothing except when the event is published.

Duplicates are the consumer's problem, by design. The relay marks a row published only after the broker acknowledges, so a relay crash republishes. Kafka redelivers on consumer crashes. Neither matters, because the entitlement consumer records event_id in processed_events in the same transaction as the grant (Phase 2). At-least-once everywhere, applied once at the end.

Decouple checkout from fulfillment, and say what the user sees. During an entitlement outage the store, the provider, and the outbox are all up: Charlie's checkout succeeds and his event waits in Kafka. Bob's does too. Alice plays on her cached license. The UI says "purchased, adding to your library" rather than failing commerce because a downstream service is down. On recovery: drain the backlog idempotently, refresh affected users' cached libraries, reconcile PAID against grants, and alert on the oldest unfulfilled purchase age.

DD7Refund and repurchase race#

Scenario: GRANT #100, GRANT #200, then a late REVOKE #100. How do you avoid revoking the new purchase?

entitlement_grants   E1 (user_a, game_1, purchase P100, ACTIVE)     <- GRANT P100
                     E2 (user_a, game_1, purchase P200, ACTIVE)     <- GRANT P200 (independent row)
REVOKE P100 (late):  UPDATE entitlement_grants SET status='REVOKED', version=version+1
                      WHERE purchase_id='P100' AND status='ACTIVE';        -- 1 row: E1 only
owns game_1?         SELECT 1 FROM entitlement_grants WHERE user_id='user_a' AND game_id='game_1' AND status='ACTIVE'  -> E2

revoke arrives BEFORE its grant:  keep a durable revocation for P100 (tombstone);  a late GRANT P100 inserts E1 already REVOKED

Model the fact, not the flag. Each purchase creates its own grant row; a refund revokes the grant that its purchase created and nothing else. Ownership is a derived question — does any ACTIVE grant exist for this user and game — not a stored boolean. Out-of-order delivery stops mattering because the two purchases never touch the same row.

Keep the cheaper guards too. event_id in processed_events handles redelivery of the same event. A per-grant version rejects a stale transition within one purchase's lifecycle. Neither requires global ordering or buffering across purchases.

Handle revoke-before-grant explicitly. If a refund event outruns its grant event (rare, but partitions rebalance), write the revocation durably against P100 so that when GRANT P100 finally arrives it is inserted already REVOKED, never resurrected. The same one-row-per-source model carries gifts, bundles, family sharing, and promotional grants: several sources, one question.

DD8Entitlement scale and cache consistency#

Scenario: 5 billion grants, 10M DAU opening their libraries, 99.99% availability. How do you partition and cache, and how do you stop a stale cache from minting a 7-day license after a refund?

partition     hash(user_id) -> 64 shards;  index (user_id, game_id, status);  ~50 rows per user on one shard
cache         entitlements:{user_id} = { game_ids with an ACTIVE grant }   LRU + TTL 1 h
              GET /library  -> SMEMBERS;  "owned" badge -> SISMEMBER;  miss -> one shard query -> fill
invalidate    after the grant / revoke transaction commits: DEL entitlements:{user_id}   (retried if Redis is down)

race          T1 UPDATE E1 REVOKED commits   T2 DEL not yet sent   T3 POST /licenses reads Redis -> ACTIVE -> signs 7 days
rule          display reads may be stale;  anything that SIGNS reads the shard:   SELECT ... WHERE status='ACTIVE'  -> then sign

Partition by the access pattern. Every hot query is "everything for one user": the library, one game's ownership, a license request. Hashing user_id keeps a user's grants on one shard and spreads blockbusters across all of them; "who owns game_1?" is an analytics question for a derived store, not a production path.

Cache the user's library, not the game's owners. entitlements:{user_id} as a set makes the library one SMEMBERS and an ownership badge one SISMEMBER. Libraries change a handful of times a year, so the hit rate is excellent, and a purchase busts the key so the game appears immediately. LRU decides which users' sets to keep.

Split reads by what they are allowed to get wrong. A stale library screen for a few hundred milliseconds is harmless. A 7-day signed license issued from a stale set is a week of free play after a refund, and a second Redis key for "recently revoked" has the same commit-then-write window. So the rule is simple: display and download fast paths may read the cache; issuing or renewing a license reads the authoritative shard. It costs one indexed read on a path that runs a few times a week per user.

DD9Offline launch and revocation#

Scenario: A 7-day signed license, a refund on day 2, an offline launch on day 4. What happens, and what about a clock set back to day 3?

issue (online)   authoritative read: ACTIVE grant for (user_a, game_1)  ->  L1 = sign_private{ license_id, user_id, game_id,
                                                                                               device_id?, issued_at d0, expires_at d7 }
verify (offline) verify_public(L1.sig)  AND  L1.user/game/device match  AND  now < expires_at  AND  now >= last_seen_time
                 then last_seen_time = now

day 0  purchase, L1 issued                  -> plays
day 2  refund -> E1 REVOKED (DD7)           -> online launch / renewal: denied         (L1 on disk is NOT extended, NOT revoked)
day 4  offline launch                        -> allowed by L1 until day 7              (the bound)
day 7  L1 expires                            -> denied until renewed
day 8  renewal -> no ACTIVE grant            -> 403
day 8  clock set back to day 3               -> last_seen_time (d8) > clock (d3)  ->  rollback  ->  require online verification

Signature, not just expiry. The license is signed with the platform's private key; the launcher holds only the public key and verifies the signature, the identities, and the expiry before every launch. A token with an edited expires_at fails verification. Device binding is optional: it limits sharing and costs friction when hardware changes.

Renewal issues, it never extends. Each online renewal does an authoritative read (DD8) and, if a grant is ACTIVE, signs a new token with a new expiry; the grant's granted_at is untouched. After a refund, renewal fails and the old token simply runs out. The backend's promise is "no new access after revocation plus a bounded tail", and the tail is the license lifetime: a product knob between offline convenience and enforcement speed.

An offline device is not trusted; make tampering expensive, not impossible. Persist the latest observed time (last_seen_time) on disk and refuse to launch when the clock is earlier than it, demanding an online check. Use a monotonic clock during a session. Where available, keep the checkpoint in hardware-backed storage. Say plainly that a determined attacker with full control of the machine can defeat any purely local check; the goal is reasonable tamper resistance, and DRM internals are out of scope.

Common Pitfalls#

Each of these reveals a different misunderstanding of the problem. Avoiding them is not about memorizing the "correct architecture" — it is about showing where ordinary web guarantees stop and explicit protocols must begin.

Final Thoughts#

The shape of a strong answer: name the asymmetry between an ordinary transactional store, a tiny but critical ownership service, and a petabyte-scale delivery problem; separate the purchase from the grant from the license; size the launch and notice that preload and content addressing, not servers, are the levers; then start from two simple flows and let each failure force exactly one mechanism into the design — a purchase row before the money and the same key at the provider, unknown-not-failed, an outbox in the same transaction, an idempotent consumer with one grant per purchase, user-partitioned entitlement with a per-user cache and authoritative reads for anything that signs, content-defined chunks under immutable manifests, direct uploads and signed CDN URLs with progress on the client, an encrypted preload unlocked by the server clock, and a signed 7-day license with a stated revocation bound. The FR→phase mapping keeps the HLD legible; the nine deep dives are the same nine questions you raised at the start, now answered.

Common failure modes to avoid: proxying bytes, fixed-size chunks, FAILED on timeout, two writes instead of an outbox, one ownership row per game, a license minted from a cache, and a token "fixed" after a refund.

These aren't toy problems — they reflect real infrastructure challenges at companies running digital storefronts and content delivery for hundreds of millions of players. If you want to walk through this design in a mock interview or get feedback on your own approach, book a session with us. We're here to help you sharpen your thinking, build your confidence, and land your next offer.

Most of the design exists to keep those four distinctions honest.