Every non-trivial application eventually needs to handle large files. In the [File System](/learn/system-design/questions/file-system) question, users sync gigabyte-sized folders. In [Slack](/learn/system-design/questions/slack), people drop images and videos into chat channels. In [Top-K Videos](/learn/system-design/questions/topk), creators upload raw footage that needs to be transcoded before anyone can watch it. The challenge is always the same: how do you move hundreds of megabytes (or more) without crushing your servers? The fundamental insight is simple but powerful: **your application server should coordinate file transfers, not mediate them**. A 500 MB video upload that passes through your API server consumes a worker thread for minutes, eats memory, and starves other requests. The fix is to get the file off your server's back entirely. ## The Problem — Why Direct Upload Fails The naive approach is straightforward: the client sends the file to your API server, which then forwards it to object storage (S3, GCS, Azure Blob). This means the file travels through your server twice — once in, once out. For a 500 MB file, that is a full gigabyte of bandwidth consumed on your server, plus a worker thread blocked for the entire duration. With just a few concurrent uploads, your API server runs out of memory or threads. Meanwhile, the actual work your server needs to do — signing URLs, saving metadata — takes milliseconds. ## Pre-signed URLs — The Core Mechanism A **pre-signed URL** is a temporary, authenticated URL that lets a client upload (or download) directly to object storage without any credentials baked into the client. Your server generates the URL using its own cloud credentials, embeds restrictions (file size limit, content type, expiry time), and hands it to the client. The client then talks to S3 directly. Here is the complete flow: The sequence breaks down into seven lightweight steps: 1. **Client requests an upload URL** from your API, sending the filename, content type, and file size. 2. **Server generates a pre-signed URL** with an expiry (typically 15 minutes), file size cap, and allowed content types. 3. **Server returns the URL** along with an upload ID for tracking. 4. **Client uploads the file directly to S3** using an HTTP PUT to the signed URL. The file never touches your server. 5. **S3 returns a 200 OK** with an ETag (an MD5 hash of the content) that serves as a checksum. 6. **Client confirms completion** to your API, passing the ETag for verification. 7. **Server saves the metadata** (filename, size, owner, storage key, ETag) to the database. Your server's total work: two small HTTP handlers and one database write. The heavy lifting — transferring hundreds of megabytes — happens between the client and S3. **See it in practice:** The [File System](/learn/system-design/questions/file-system) question is the direct application of this pattern — you design upload and download flows with pre-signed URLs and metadata management. In [Slack](/learn/system-design/questions/slack), file attachments and media sharing in chat channels use pre-signed URL uploads for user-generated content. In [ChatGPT](/learn/system-design/questions/chatgpt), file uploads for context documents follow the same pattern. ## Multipart Upload — Handling Very Large Files For files larger than about 100 MB, a single PUT request is fragile. Network hiccups, mobile connections dropping, browser tab closures — any interruption means starting the entire upload from scratch. **Multipart upload** solves this by splitting the file into chunks (typically 5-100 MB each) and uploading them independently. The process works in three phases: 1. **Initiate**: The server calls S3 to start a multipart upload and receives an upload ID. 2. **Upload parts**: The client uploads chunks in parallel, each with its own part number. S3 returns an ETag for every part. 3. **Complete**: The client (or server) sends a "complete" request with all the part numbers and ETags. S3 verifies integrity and assembles the parts into a single object. This gives you four important properties: One operational detail worth knowing: abandoned multipart uploads leave orphaned parts that consume storage. Always configure **lifecycle rules** to auto-delete incomplete multipart uploads after 24 hours. Otherwise, you are paying for storage that nobody can access. **See it in practice:** In [File System](/learn/system-design/questions/file-system), large file syncs use multipart uploads with content-defined chunking so only changed portions re-upload. In [Remote Devbox](/learn/system-design/questions/remote-devbox), pushing large container images or development environments to the cloud uses multipart uploads for reliability. In [CI/CD](/learn/system-design/questions/cicd), build artifacts (Docker images, compiled binaries) are often hundreds of megabytes and need chunked uploads to artifact storage. ## CDN Delivery — Serving Files at Scale Uploading is only half the problem. Once a file is in S3, you need to serve it efficiently to users around the world. Fetching directly from a single S3 region means a user in Tokyo waits 200+ ms for each request to Virginia. Multiply that by every image on a page, and the experience is painful. A **Content Delivery Network** (CDN) caches your files at **edge servers** distributed globally. The first request for a file pulls it from S3 (the "origin"). After that, every subsequent request from the same region is served from the edge — typically in under 20 ms. There is one tricky part: **cache invalidation**. When a user re-uploads a file, stale versions may persist at edge nodes until the TTL expires. The cleanest solution is **content-addressed URLs** — include a hash or version in the URL path (e.g., `/images/abc123-v2.jpg`). New versions get new URLs automatically, so you never need to invalidate anything. The old URLs simply stop being requested and eventually expire from the edge cache. **See it in practice:** In [Yelp](/learn/system-design/questions/yelp), restaurant photos and thumbnails are served entirely from CDN edges so origin servers never handle image traffic. In [Google Maps](/learn/system-design/questions/google-map), pre-rendered map tiles are cached at CDN nodes worldwide for instant map loads. The [Scaling Reads](/learn/system-design/design-patterns/scaling-reads) pattern covers how CDNs fit into the broader layered caching strategy alongside Redis and read replicas. ## Post-Upload Processing — Resize, Transcode, Optimize Raw uploaded files are rarely ready to serve as-is. A 12-megapixel photo from a phone camera is far too large for a thumbnail. A raw video file is not streamable. Users expect your application to handle this transparently. The pattern is an **event-driven processing pipeline**: the upload triggers an event, a worker picks it up, processes the file into multiple variants, and stores the results. For **images**, this typically means generating multiple sizes (thumbnail, medium, large) and converting to efficient formats like WebP or AVIF. A 5 MB JPEG from a phone becomes a 50 KB thumbnail, a 200 KB medium, and a 500 KB WebP — saving bandwidth and load time. For **videos**, the pipeline is more involved: 1. **Splitting**: Divide the video into segments (2-10 seconds each) 2. **Transcoding**: Encode each segment into multiple resolutions (360p, 720p, 1080p) and codecs (H.264, VP9, AV1) 3. **Manifest generation**: Create HLS (.m3u8) or DASH (.mpd) manifests listing available quality levels 4. **CDN distribution**: Push segments and manifests to edge nodes Each step depends on the previous one, forming a **DAG** (Directed Acyclic Graph). Workflow engines like Temporal or AWS Step Functions orchestrate the DAG, handling retries, timeouts, and partial failures. Transcoding is embarrassingly parallel — each segment-times-resolution combination is an independent job. **Adaptive bitrate streaming** (HLS/DASH) lets the video player switch quality mid-stream based on network conditions. The manifest file acts as a menu; the player picks the best quality for each segment. This is why YouTube can start a video instantly at 360p and ramp up to 1080p within seconds. **See it in practice:** In [Top-K Videos](/learn/system-design/questions/topk), uploaded videos pass through a full transcoding pipeline before they can be ranked and served. In [Job Scheduler](/learn/system-design/questions/job-scheduler), video transcoding jobs are a prime example of work that needs reliable scheduling with retries. The [Pub/Sub Messaging](/learn/system-design/design-patterns/pub-sub-messaging) pattern explains how S3 events and message queues decouple the upload from processing. ## State Synchronization — What Can Go Wrong When uploads involve multiple systems (client, API server, object storage, database), things can get out of sync. Here are the failure modes you need to handle: The general principle: **design for partial failure**. Assume that any step can fail independently, and build reconciliation mechanisms to detect and fix inconsistencies. An upload status field that tracks the lifecycle (pending, uploading, processing, ready, failed) makes it straightforward to find and retry stuck items. **See it in practice:** In [Payment](/learn/system-design/questions/payment), idempotency keys solve the same partial-failure problem for financial transactions. In [Webhook](/learn/system-design/questions/webhook), delivery confirmations and retry logic handle the case where the notification succeeds but the acknowledgment fails. The [Scaling Writes](/learn/system-design/design-patterns/scaling-writes) pattern covers write-ahead logging and other techniques for maintaining consistency when multiple systems are involved. ## Content-Defined Chunking — The File Sync Pattern Standard multipart upload uses fixed-size chunks, which works perfectly for one-time uploads. But for **file sync** (like Dropbox or Google Drive), fixed chunks are wasteful. Inserting a single byte at the beginning of a file shifts all chunk boundaries, forcing re-upload of every chunk even though most of the content is identical. **Content-defined chunking** (CDC) solves this by using a **rolling hash** (Rabin fingerprint) to determine chunk boundaries based on file content rather than fixed offsets. A sliding window moves over the data, and when the hash matches a predefined pattern (e.g., lowest 13 bits are zero), a chunk boundary is declared. The key property: inserting or deleting bytes only affects the chunks immediately around the change. All other chunks remain identical, with identical hashes. **Delta sync** takes this further. The client computes chunk hashes locally and sends only the list of hashes to the server. The server compares against stored hashes and requests only new or modified chunks. For a 1 GB file with a small edit, only a few kilobytes might actually transfer over the wire. **See it in practice:** The [File System](/learn/system-design/questions/file-system) question asks you to design exactly this — upload and download flows with content-defined chunking and delta sync. In [Bookstore](/learn/system-design/questions/bookstore), e-book delivery uses CDN caching with content-addressed URLs for efficient distribution. The [Distributed Caching](/learn/system-design/design-patterns/distributed-caching) pattern explains how consistent hashing (a related concept) distributes cache keys across nodes.