ChatGPT Playground

Problem Statement

Design the application backend for a ChatGPT Playground — a lightweight, interactive web tool that lets users experiment with ChatGPT. The system should:

1. Submit prompts and view completions streamed in real-time.
2. Adjust generation parameters (temperature, max tokens, etc.).
3. Save prompts as reusable presets.
4. Search, load, and run existing presets.

Functional Requirements

FR1 – Submit Prompt & View Completion

Users should be able to submit a free-text prompt and view the model response typed out live.

Use case:

A marketing intern types: "Write a tagline for an ice cream shop" and clicks Submit.

The UI shows: "Taste the Joy of Summer at Our Creamery!" like a chat.

FR2 – Adjust Generation Parameters

Users can tweak generation settings like temperature, max tokens, etc., and observe how output changes.

Use case:

A copywriter reruns the same tagline prompt with temperature=0.2 and then 0.9, comparing the creative differences.

FR3 – Save Prompt as a Preset

Users can save a prompt and its parameters as a reusable named preset.

Use case:

After crafting a strong "Brand Tagline Generator," the user taps Save, names it "Taglines – Playful," and stores it.

FR4 – Search, Load & Run Existing Presets

Users can search for and run saved presets using new input text.

Use case:

The user types "summarize" into the preset search bar, selects "Summarize for a 2nd grader" from the filtered list, pastes in a paragraph, and clicks Submit to get a simplified version.

Non-Functional Requirements

NFR1 – Latency: Time-to-First-Token ≤ 300 ms, End-to-End Completion ≤ 2 sec (p95)

Why this matters:

In a system handling 10,000 prompt submissions per second, latency isn't just about speed — it directly affects perceived quality and user engagement. A slow UI feels broken even when the system is technically functional. Instant token streaming gives users a sense of progress and keeps the feedback loop tight — especially critical when users are rapidly iterating prompts.

Deep dive challenge:

How do we prevent slow token starts or long tails from degrading UX?

NFR2 – Availability: 99.9% Success Rate Across Prompt Submission Flow

Why this matters:

At 100M prompts/day, a 0.1% failure rate still means 100,000 broken generations daily. Users expect reliability from AI tools, especially when using saved presets or collaborating across sessions. High availability ensures the Playground remains usable even when parts of the system — like the model — are temporarily degraded, without breaking prompt editing, UI rendering, or preset loading.

Deep dive challenge:

How do we prevent a partial outage from breaking all completions?

NFR3 – Rate Limiting: ≤ 60 Requests per User per Minute, ≤ 2 Concurrent Generations per User

Why this matters:

In a 10M DAU system, even 1% of users misbehaving (intentionally or not) could generate millions of excess requests per minute, potentially spiking model cost and degrading experience for others. Enforcing smart limits and cool-downs protects system stability while keeping honest users unaffected — especially during bursty activity or tab abuse.

Deep dive challenge:

How do we prevent spams from one user choking the system?

NFR4 – Scalability: Support 10k QPS and 100k Concurrent Streaming Sessions

Why this matters:

At scale, usage is never evenly distributed. Peaks, viral usage spikes, and time zone overlaps mean your system must elastically scale while still streaming responses in real time. Supporting 100K concurrent streams without running out of memory, dropping connections, or causing cold-start delays is critical to avoid backlogs and latency cliffs.

Deep dive challenge:

How do we prevent sudden traffic spikes from overwhelming stream infra?

Requirement Summary

High Level Design (Delivery Framework: API → Entity → Workflow → Diagram)

FR1 – Submit Prompt & View Completion

API Signature

Request

{
  "event": "generate",
  "request_id": "uuid-1234",  // Optional idempotency key
  "prompt": "Write a tagline for an ice cream shop",  // User's input prompt
  "temperature": 0.9,    // Controls randomness (0 = deterministic, 1 = creative)
  "max_tokens": 256,     // Limits response length
  "top_p": 1,      // Top-p sampling: limits to top cumulative probability mass
  "frequency_penalty": 0.5,    // Penalizes repeated phrases
  "model": "text-davinci-003", // GPT-3 variant
  "stream": true     // Enables real-time token streaming via WebSocket
}

Response

{
  "request_id": "uuid-1234",
  "completion": "Taste the Joy of Summer at Our Creamery!"
}

(Optional) Stop Request

{
  "event": "stop",
  "request_id": "uuid-1234"
}

Entity

PromptRequest: This is a short-lived object used to validate and relay a single generation request. This object exists in memory for the duration of the request. It is not saved to a database unless explicitly logged.

Workflow: Prompt Submission via Streaming

1. User types a prompt and chooses parameters in the Playground UI. (Before that we assume WebSocket connection is established).
2. Frontend sends a WebSocket message representing the prompt submission, matching the POST /v1/completions schema. This schema aligns with our REST spec for composability, but is transmitted over WebSocket.
3. API Gateway receives the request:
   • It does not authenticate (auth is assumed to be handled upstream, e.g., via login/session cookie or API token).
   • It applies basic rate limits and usage quotas (we will discuss later).
4. Completions Service:
   • Validates parameters (e.g., max tokens, numeric ranges)
   • Checks request_id for idempotency
   • Constructs a model-ready payload and sends it to GPT-3 (black box)
5. GPT-3 begins processing — it takes time to run inference but starts returning tokens one by one as they're ready.
6. Tokens are streamed back to the Completions Service, which:
   • Uses WebSocket to relay tokens to the user's browser in real time
   • Each token is immediately sent as an WebSocket event — the user sees the text "typing out"
7. Frontend renders the streamed tokens linearly until the model signals the end of generation.

Design Diagram

Click to expand

FR2 – Adjust Generation Parameters

API Signature

This is the same WebSocket message structure as FR1, but FR2 emphasizes how we validate and tune the model's behavior using exposed parameters.

{
  "event": "generate",
  "request_id": "uuid-5678",
  "prompt": "Give me 3 startup ideas",
  "temperature": 0.3,           // Lower = more focused, deterministic
  "max_tokens": 150,            // Shorter completion
  "top_p": 0.8,                 // Narrower sampling pool
  "frequency_penalty": 0.7,     // Avoid repeated ideas
  "model": "text-davinci-003",  // GPT-3 variant
  "stream": true                // Stream tokens via WebSocket
}

Entity

GenerationParameters - This is a structured sub-object within the prompt request. It is not stored, but is validated and passed to GPT-3. This is Optional.

Workflow: Parameterized Prompt Submission

1. User adjusts generation settings in the Playground UI via sliders, dropdowns, or advanced options.
2. Frontend sends a WebSocket message to backend, embedding the adjusted parameters along with the prompt.
3. API Gateway receives the message and applies user-level rate limits and quotas.
4. Completions Service:
   • Validates parameter values (e.g. temperature ∈ [0,1], max_tokens ≤ 2048)
   • Fills in sane defaults if optional parameters are missing
   • (Optional) Logs the full GenerationParameters object for A/B testing or observability
   • Assembles a model-ready payload and forwards to GPT-3 API
5. Model API uses these parameters to shape generation behavior.
6. Streaming back of tokens proceeds via WebSocket, same as FR1:
   • Each token is sent incrementally
   • Client can hit "Stop" at any time to cancel
   • Optional: log or visualize how different settings affect outputs

Design Diagram

Click to expand