You're asked to design Sora: a product where users type a prompt and receive a generated video. This problem looks like "yet another async job system" on the surface, but the staff-level signal comes from recognizing what's actually scarce. It isn't API throughput or storage — it's **GPU-minutes**. Every design decision downstream of that observation (queueing, scheduling, progress UX, cost controls) is where this interview is won or lost. The candidate must design a system that: - Accepts a text prompt (optionally with an image or video as conditioning input) and generates a video up to 20s at up to 1080p - Shows honest status and progress, including a low-resolution preview before the final render completes - Lets users watch their finished videos and browse a public explore feed - Supports remixing an existing video with a modified prompt, preserving lineage back to the original - Never delivers a video — to the user or the public feed — without passing prompt-level and frame-level moderation Before diving in, ask yourself: - What is the scarce resource here, and what number proves it? - How do you keep submissions fast and the control plane up when the GPU fleet is saturated? - Why is a preview after 30–60 seconds worth more than a progress bar? - Where do the safety gates sit, and what happens when the classifiers are down? - What breaks first when demand spikes 3× — and what degrades gracefully instead? You don't need to answer all of these right away. A **Senior/Staff+ Engineer** doesn't stop at basic functionality — they **anticipate edge cases**, **design for resilience**, and **push for production-grade reliability**. But **great systems start with great questions.** What would *you* ask next? ## Requirements ### Functional Requirements We'll scope to four FRs. Each one maps to exactly one flow in the High Level Design. **FR1 — Generate** A user submits a text prompt (optionally with an image or video as conditioning input) and generation parameters (duration up to 20s, resolution up to 1080p, aspect ratio) to create a video. **FR2 — Track** A user can see the status and progress of their generation, including a low-resolution preview before the final render completes. **FR3 — View & browse** A user can watch their finished videos and browse a public explore feed of videos shared by others. **FR4 — Remix** A user can take an existing video (theirs or a public one) and re-generate it with a modified prompt, preserving lineage back to the original. > **Out of scope (say this out loud):** model training and fine-tuning pipelines, billing/credits, mobile capture, editing timelines (storyboards), and recommendation ranking for the feed beyond a simple recency/popularity ordering. Each of these is its own interview. ### Non-Functional Requirements **NFR1 — Latency is asymmetric.** Submission acknowledgment must be fast (< 500ms p99), but generation itself is minutes-scale and users accept that — *if* we give honest progress and a preview. Target: draft-quality preview within 30–60s; final 1080p render within 2–10 min depending on tier. **NFR2 — Cost efficiency over raw latency.** GPU fleet utilization is the dominant cost line. We optimize for high utilization (target > 60–70%) and fair queueing over minimizing any single job's wait. **NFR3 — Availability with graceful degradation.** The control plane (submit, status, view) targets 99.9%. The GPU data plane can degrade — under capacity crunch, queue times grow and free-tier resolution drops, but submissions never fail outright. **NFR4 — Durability.** Generated videos are user-created content; once a generation succeeds, the asset must not be lost (11 nines object-store durability). **NFR5 — Safety is a hard gate, not best-effort.** No video reaches the user or the public feed without passing prompt-level and frame-level moderation. This is a correctness requirement, not an optimization. **NFR6 — Provenance.** Every output is watermarked and carries C2PA metadata identifying it as AI-generated. ### Scale & Back-of-Envelope Estimations Assume a successful consumer launch: **The GPU fleet math — the number that identifies the real system:** ```jsx Daily demand = 5M generations × 3 GPU-min = 15M GPU-min/day = 250K GPU-hours/day → ~10,400 GPUs running continuously At 60% achievable utilization → provision ~17K H100s ``` This single number justifies the entire scheduling deep dive: a 5% utilization improvement is ~850 GPUs, i.e., tens of millions of dollars a year. The API tier, by contrast, is trivially small — a handful of stateless servers. > **Interviewer signal:** candidates who compute the GPU fleet size have identified the real system; candidates who dwell on API QPS have not. ### Follow-up Questions to Expect - **[DD1] Scheduling:** How do you keep ~17K GPUs highly utilized with jobs of wildly different sizes and three user tiers? - **[DD2] Inference:** What happens inside a generation, and how does that shape what the scheduler can do (preemption, batching, checkpointing)? - **[DD3] Safety:** Where do the moderation gates sit? Fail open or fail closed when classifiers are down? - **[DD4] Storage & delivery:** How do you store 55PB/year and serve 1.5PB/day of egress without melting the origin? ## Core Thinking & Design Pattern Before drawing boxes, orient around three tensions that shape everything: ### 1. The queue *is* the product With minutes-long jobs and a fixed GPU fleet, you cannot buy your way out of a spike in real time (GPU capacity has procurement lead times of months). So the system's job is to *shape demand*: priority tiers (paid vs. free), fair scheduling per user, preview-first generation so users get value early, and honest progress so waiting feels acceptable. This is closer to designing a build farm or a render farm than a CRUD service. ### 2. Separate the control plane from the data plane Submissions, status reads, and feed browsing are cheap, high-QPS, stateless-friendly operations that should stay up even when the GPU fleet is saturated or partially down. Generation is expensive, stateful, and failure-prone. Different availability targets, different scaling models, different teams. The boundary between them is a durable queue. ### 3. Safety and provenance are in the critical path Unlike a photo-upload product where moderation can be async-after-publish, a generative system *creates* the risky content itself. The moderation gates (prompt-time and frame-time) sit inline before delivery, which means their latency and availability budgets are part of the generation SLO. **The core recognition** This is a **GPU-bound async job system wearing a consumer product's clothes.** Size the fleet in the estimations, and let that number pull the interview toward scheduling, two-pass generation, and cost — while never forgetting that safety gates sit in the critical path and that immutable assets make delivery the easy part. ## Core Entities & API ### Core Entities > Note what's *not* here: no mutable "Video" row that jobs update in place. Jobs are the mutable workflow record; assets are immutable artifacts. This separation makes retries, dedup, and lineage trivially safe. #### User Identity, tier, and priority attributes. #### GenerationJob The central entity — the mutable workflow record for one generation. #### VideoAsset Immutable output artifact. #### ModerationRecord Per-job safety audit trail. #### RemixEdge Forms the lineage DAG behind FR4. ### API Design All under `/v1`, authenticated, rate-limited per user tier. #### Submit a generation (FR1) ```bash POST /v1/generations Body: { "prompt": "...", "conditioning_asset_id": null, "duration_s": 10, "resolution": "1080p", "aspect_ratio": "16:9", "remix_of_asset_id": null } Response 202: { "job_id": "...", "status": "QUEUED", "est_wait_s": 90 } ``` > Returns 202 immediately — generation is always async. `est_wait_s` comes from the scheduler's live queue model (DD1). An `Idempotency-Key` header dedupes double-taps. #### Track a generation (FR2) ```bash GET /v1/generations/{job_id} Response 200: { "status": "RUNNING", "progress_pct": 45, "preview_url": "https://cdn...", "output": null, "failure_reason": null } ``` ```bash GET /v1/generations/{job_id}/events # SSE → stream of { "status", "progress_pct", "preview_url" } updates ``` > Polling for background tabs; SSE for the active client. Progress events are pushed by workers (FR2 flow). #### View & browse (FR3) ```bash GET /v1/feed?cursor=&limit= Response 200: { "items": [ { "asset_id", "playback_urls", "author", "remix_count" } ], "next_cursor": "..." } ``` ```bash GET /v1/assets/{asset_id} Response 200: { "playback_urls": [...], "metadata": {...}, "lineage_summary": {...} } ``` > Playback URLs are CDN-tokenized: long-lived for public assets (cacheability), 5-minute signed for private. #### Remix (FR4) ```bash POST /v1/generations # with remix_of_asset_id set — reuses FR1's endpoint GET /v1/assets/{asset_id}/lineage Response 200: { "ancestors": [...], "descendants": [...] } ``` > Remix is deliberately *not* a separate creation endpoint — it's a generation with a conditioning input and a lineage edge, which keeps the write path singular. ## High Level Design > The system splits along one boundary: a stateless **control plane** (API tier, Job Service backed by a Postgres Jobs DB, Feed Service, SSE gateways) and a **GPU data plane** (Scheduler + Worker fleet). The connective tissue between them is a durable **priority queue**. A **Safety Service** gates delivery, an **Asset Service** fronts object storage + CDN, and a **Notification/Progress hub** (Redis pub/sub → SSE gateways) streams progress. One flow per FR. > **Durable queue = the boundary between planes.** Control plane: 99.9%, stateless, trivially scaled. Data plane: cost-optimized, degrades gracefully (queue grows, free tier caps) — submissions never fail. ### FR1 — Generate: the submission-to-render flow ``` Client → API Gateway → Generation API 1. AuthN/Z, rate limit by tier 2. Prompt Safety check (synchronous, <100ms): text classifier + policy rules → REJECT fast if disallowed 3. Write GenerationJob (status=QUEUED) to Jobs DB [source of truth] 4. Enqueue job_id into priority queue (per-tier lanes) 5. Return 202 + est_wait_s GPU Scheduler (control loop): 6. Pulls from queue lanes by weighted policy → assigns to a worker/node Worker (8×H100 node running the DiT inference server): 7. Fetch prompt + conditioning latents 8. Draft pass → upload preview → emit progress events 9. Final denoising pass → VAE decode → raw frames 10. Post-process: encode renditions, watermark + C2PA sign 11. Hand off to Safety Service (frame-level moderation) 12. On PASS: register VideoAsset, set job SUCCEEDED On FAIL: set job REJECTED with user-safe reason ``` Two design points worth narrating: **First, DB write before enqueue.** The Jobs DB is the source of truth; the queue is a delivery mechanism. If the queue loses a message, a sweeper re-enqueues any QUEUED job older than N seconds — so the queue only needs at-least-once semantics, and workers are idempotent per `job_id`. **Second, prompt safety is synchronous but frame safety is not.** Rejecting a bad prompt in 100ms saves 3 GPU-minutes; you cannot afford to discover it at step 11. ### FR2 — Track: status & progress flow ``` Worker → emits {job_id, progress_pct, stage, preview_url} → Redis pub/sub channel per job + periodic Jobs DB update SSE Gateway ← subscribes on client connect → streams to client GET /generations/{id} ← reads Jobs DB (covers reconnects/polling) ``` Progress for a diffusion model is honest and cheap: the denoising loop has a fixed step count, so `progress_pct = step/total_steps` per pass, with stage weights (draft 20%, final 70%, post 10%). The **preview after the draft pass** is the single highest-leverage UX decision in the system: it converts a 6-minute opaque wait into a 40-second wait plus an "it's working on my thing" confirmation — and it lets users cancel bad prompts early, refunding GPU-minutes to the fleet (a real utilization win, not just UX). ### FR3 — View & browse: the delivery flow ``` Client → GET /feed → Feed Service → reads Feed index (public assets, recency/popularity ordered, fan-out-on-read from an assets-by-visibility index; cached in Redis) Client → playback_urls → CDN (signed, TTL'd URLs) → CDN cache miss → Object Store origin (rendition ladder: 480/720/1080) ``` Videos are immutable once published, which makes them the perfect CDN workload: long TTLs, no invalidation problem. Origin egress stays under 5% because the feed concentrates views on a small hot set. Private videos use short-lived signed URLs; public ones use longer-lived tokens for cacheability. ### FR4 — Remix: lineage-preserving regeneration ``` Client → POST /generations { prompt', remix_of_asset_id } 1. Resolve parent asset → verify visibility (public or owned) 2. Fetch parent's stored latents (if retained) or re-encode from video 3. Create GenerationJob with parent_job_id + RemixEdge 4. → identical path to FR1 from the queue onward, with the parent latents as conditioning input to the DiT Lineage reads: GET /assets/{id}/lineage → walk RemixEdge DAG ``` The one interesting decision: **retain latents for public assets** (hot tier, ~2× the video's size) so remixes skip the video→latent re-encode and start from a cleaner conditioning signal. For private assets past 30 days, drop latents and re-derive on demand — a classic storage/compute trade you should name explicitly. ## Deep Dives Each deep dive below resolves a tension named in Core Thinking rather than introducing new machinery for its own sake. ### DD1 — GPU Scheduling & Queueing: The Heart of the System ~17K GPUs, jobs of wildly different cost (30 GPU-sec previews vs. 10 GPU-min 1080p renders), three user tiers, diurnal peaks at 3× average, and a hard constraint: you cannot autoscale H100s on demand. The scheduler is where the dominant cost line of the entire business is won or lost. **Queue structure.** Not one queue — a small matrix: tier (free/plus/pro) × size class (draft/small/large). The scheduler runs weighted fair queueing across tiers (e.g., pro 60% / plus 30% / free 10% of fleet-seconds, work-conserving so idle share spills over), and within a tier uses **deficit round robin per user** so one user submitting 500 jobs can't starve neighbors. Size classes exist because of bin-packing: draft jobs are the "sand" that fills fragmentation gaps between "boulder" 1080p jobs, which is worth several points of utilization on its own. **Placement.** A final 1080p/20s render needs a full 8×H100 node (tensor-parallel inference); drafts fit on 1–2 GPUs. The scheduler maintains node-level free maps and packs drafts onto partially-used nodes. Model weights (~hundreds of GB with the VAE and text encoder) are pre-loaded on every node and mmap'd — cold-loading per job would destroy latency, so nodes are *specialized by model version*, and rollouts happen by draining and re-imaging cohorts (this is exactly the model-distribution problem from the companion question — see **ML Model Distribution**). **Admission control & honest ETAs.** `est_wait_s` at submit time comes from a live simulation: queue depth per lane × observed service rates. When free-tier ETA exceeds a threshold (say 15 min), degrade *transparently*: cap free jobs at 480p/10s, or show "high demand — join queue?" This is the graceful-degradation NFR made concrete. **Preemption.** Draft passes are preemptible (checkpoint the denoising state — it's just the current latent tensor + step index, a few hundred MB) to make room for pro-tier finals during peaks. Finals are not preempted; killing minute 8 of a 10-minute render wastes more than it saves. > **Rule of thumb to state:** *preempt only jobs whose sunk cost is less than the cost of the wait you're relieving.* **Failure handling.** Workers heartbeat; a job with a dead heartbeat >60s goes back to QUEUED with `attempt_count + 1` (cap 3, then FAILED with automatic credit refund). Because the denoising loop checkpoints latents to local NVMe every ~30 steps, a retried job on the *same* node resumes; cross-node retry restarts the pass. Idempotency: asset registration keys on `job_id`, so a duplicate completion is a no-op. Don't just say "priority queue." Show the money: at ~17K H100s, a 5% utilization improvement is ~850 GPUs — tens of millions of dollars a year. That's why the size-class split (bin-packing), work-conserving WFQ (no idle share), and preemptible drafts (peak absorption) each earn their complexity. Every scheduling mechanism should be justified in GPU-hours saved, not elegance. *Follow-up: if a viral moment doubles submissions for 6 hours, what specifically happens — in order — as the system degrades? Which tier notices first and what do they see?* ### DD2 — Inside a Generation: Making Minutes-Long Inference Schedulable The scheduler can only exploit structure that the inference pipeline exposes. A monolithic "run model, wait 6 minutes" black box cannot be preempted, batched, or checkpointed. Understanding the pipeline's anatomy is what turns generation from an opaque blob into schedulable units. **Pipeline anatomy.** Text encoder → DiT (diffusion transformer) denoising over spacetime patches in latent space → temporal VAE decode → frame post-process → encode. The DiT dominates: cost scales roughly with `duration × pixels × steps`, which is why parameters, not QPS, drive fleet math. **Two-pass generation.** Pass 1 renders a low-step, low-res draft (the FR2 preview). Pass 2 renders finals, optionally conditioned on the draft for consistency. Beyond UX, this creates the small/large job mix the scheduler exploits, and turns early user cancellations (~10–15% after seeing a preview, plausibly) directly into reclaimed fleet capacity. **Parallelism within a job.** Tensor parallelism across the 8 GPUs of one node for finals; for 20s renders, additionally chunk time into overlapping windows denoised in parallel with shared context, then blend — trading a little quality risk at seams for ~2–3× lower wall clock. State the trade; don't hide it. **Batching.** Diffusion steps are lockstep, so same-shape jobs can micro-batch on shared GPUs (drafts especially: batch 4–8 previews per GPU). Finals rarely batch — shapes differ and latency matters — another reason the size-class split earns its complexity. **Checkpointing.** The full job state between steps is (latent tensor, step index, RNG state) — small and serializable. Checkpoint to node NVMe every ~30s of compute; this is what makes preemption and same-node resume nearly free. The checkpointing detail is the one that separates "I've run inference" candidates from hand-wavers: the entire resumable state of a diffusion job is just the current latent tensor, the step index, and the RNG state — a few hundred MB. Once you know that, preemption, retry-with-resume, and draft-pass eviction all become cheap mechanisms instead of scary ones. *Follow-up: your 20s renders blend overlapping time windows denoised in parallel. What visible artifact does this risk, and how would you detect it automatically before the user sees it?* ### DD3 — Safety & Provenance as a Serving-Path Dependency Unlike an upload platform, a generative system creates the risky content itself — moderation-after-publish doesn't work. That puts classifiers inline in the generation SLO, which raises an availability question most candidates never address: what happens when the safety service itself is down? **Three gates:** **The availability problem.** Safety is now in the SLO path: if classifiers are down, do you fail open or closed? Answer explicitly: **fail closed for publish, queue-and-hold for private delivery** — jobs park in MODERATING and drain when the service recovers. Provision the classifier fleet like any tier-1 dependency (it's ~1% of GPU cost; never let it be the bottleneck that idles renders). **Provenance.** Every rendition gets (a) an imperceptible watermark embedded at decode time and (b) a C2PA manifest signed with the platform key, asserting AI generation, model version, and creation time. Watermark detection also runs on *uploaded* conditioning media — remixing our own outputs is fine; laundering third-party content through image-to-video needs the same moderation as a fresh prompt, plus copyright screening. **Feedback loop.** Human-review outcomes flow back as labeled data for the classifiers, and per-user rejection rates feed trust scores that tighten rate limits for repeat abusers — cheaper than fighting each bad job individually. Name the asymmetry: prompt-time moderation is cheap and blocks waste (100ms to save 3 GPU-minutes); frame-time moderation is expensive but unavoidable (the model can generate policy-violating output from an innocent prompt). And commit to a failure posture out loud — "fail closed for publish, queue-and-hold for private" — because "we'd have moderation" without a failure posture is not a design. *Follow-up: a user remixes a public video whose parent was later taken down for policy violation. What happens to the remix, and what should?* ### DD4 — Storage, Lifecycle & Delivery at 55PB/Year 5M videos/day at ~30MB each is 150TB/day of new, undeletable user content — plus latents, previews, and rendition ladders. Storage architecture is not the hard part of this system, but mishandling it (wrong tiering, cache-hostile URLs, HLS overhead on 10s clips) burns real money and latency. **What we store per job.** Final renditions (480/720/1080 ladder, ~50–80MB total), a poster frame, the preview clip, the C2PA manifest, and — decision point — latents. Policy from the FR4 discussion: keep latents hot for public/remixable assets, drop them for private assets after 30 days, re-derive on demand. **Tiering.** **Delivery.** Immutable assets + content-addressed keys (`asset_id/rendition/hash.mp4`) → infinite CDN TTLs, no invalidation logic. Adaptive bitrate via the rendition ladder rather than full HLS packaging (clips are ≤20s; segment overhead isn't worth it — say why). Signed URLs: long-lived for public assets to preserve cache hit rate, 5-minute for private. **Cost sanity check** (interviewers love this closure): The ordering confirms the thesis — every deep-dive minute spent on scheduling and utilization was spent on the right problem. The judgment call worth narrating is the *negative* decision: no HLS packaging. For ≤20s clips, a rendition ladder with range requests delivers ABR-equivalent behavior without per-asset segment/playlist overhead across 5M new assets a day. Knowing when the industry-standard machinery is overkill is a stronger signal than reaching for it. *Follow-up: a video goes viral and gets 50M views in an hour. Walk through exactly which systems notice, and confirm nothing melts.* ## Final Thoughts The shape of a strong answer: recognize early that this is a **GPU-bound async job system wearing a consumer product's clothes**, size the fleet in the estimations, and let that number pull the interview toward scheduling, two-pass generation, and cost — while showing you haven't forgotten that safety gates sit in the critical path and that immutable assets make delivery the easy part. The FR→flow mapping keeps the HLD legible; the four deep dives (scheduling, inference internals, safety, storage) each resolve a tension you named in Core Thinking rather than introducing new machinery for its own sake. **Compute the fleet, then let it drive the interview** The single most important move in this interview is the GPU math: 5M generations/day × 3 GPU-min ÷ 60% utilization ≈ 17K H100s. That number reframes everything — it's why the queue is the product, why utilization beats latency, and why a 5% scheduling improvement is worth tens of millions a year. Candidates who dwell on API QPS never find the real system. **Give a preview, not a progress bar** Users don't need a percentage; they need proof the machine understood them. The draft-pass preview converts a 6-minute opaque wait into a 40-second confirmation — and the ~10–15% of users who cancel after seeing it refund real GPU-minutes to the fleet. It's the rare decision that is simultaneously the best UX choice and the best utilization choice. **Commit to a safety failure posture** "We'd add moderation" is not a design. Say where each gate sits (prompt-time inline, frame-time post-render, publish-time strictest), and commit to the hard case: classifiers down → fail closed for publish, queue-and-hold for private delivery. Safety with an availability budget is engineering; safety without one is a slogan. **Common failure modes to avoid:** treating this as a video *upload* platform (moderation-after-publish doesn't work when you're the one generating the content), hand-waving "autoscale the GPUs" (you can't — procurement lead times are months), and giving progress bars without a preview (users don't need a percentage; they need proof the machine understood them). > These aren't toy problems — they reflect real infrastructure challenges at companies building generative video products. If you want to walk through this design in a mock interview or get feedback on your own approach, book a session with us. We're here to help you sharpen your thinking, build your confidence, and land your next offer.