Handling Large Blobs

Every non-trivial application eventually needs to handle large files. In the File System question, users sync gigabyte-sized folders. In Slack, people drop images and videos into chat channels. In Top-K Videos, creators upload raw footage that needs to be transcoded before anyone can watch it. The challenge is always the same: how do you move hundreds of megabytes (or more) without crushing your servers?

The fundamental insight is simple but powerful: your application server should coordinate file transfers, not mediate them. A 500 MB video upload that passes through your API server consumes a worker thread for minutes, eats memory, and starves other requests. The fix is to get the file off your server's back entirely.

The Problem — Why Direct Upload Fails#

The naive approach is straightforward: the client sends the file to your API server, which then forwards it to object storage (S3, GCS, Azure Blob). This means the file travels through your server twice — once in, once out. For a 500 MB file, that is a full gigabyte of bandwidth consumed on your server, plus a worker thread blocked for the entire duration.

With just a few concurrent uploads, your API server runs out of memory or threads. Meanwhile, the actual work your server needs to do — signing URLs, saving metadata — takes milliseconds.

Pre-signed URLs — The Core Mechanism#

A pre-signed URL is a temporary, authenticated URL that lets a client upload (or download) directly to object storage without any credentials baked into the client. Your server generates the URL using its own cloud credentials, embeds restrictions (file size limit, content type, expiry time), and hands it to the client. The client then talks to S3 directly.

Here is the complete flow:

The sequence breaks down into seven lightweight steps:

1. Client requests an upload URL from your API, sending the filename, content type, and file size.
2. Server generates a pre-signed URL with an expiry (typically 15 minutes), file size cap, and allowed content types.
3. Server returns the URL along with an upload ID for tracking.
4. Client uploads the file directly to S3 using an HTTP PUT to the signed URL. The file never touches your server.
5. S3 returns a 200 OK with an ETag (an MD5 hash of the content) that serves as a checksum.
6. Client confirms completion to your API, passing the ETag for verification.
7. Server saves the metadata (filename, size, owner, storage key, ETag) to the database.

Your server's total work: two small HTTP handlers and one database write. The heavy lifting — transferring hundreds of megabytes — happens between the client and S3.