Sign in to keep reading
“CI/CD System” is free with an account. Your progress and bookmarks follow you across devices.
Sign in to continueDesign a CI/CD System - This system design covers workflow orchestration, parallel execution, artifact management, and real-time visibility at scale.
Written by a Staff+ Engineer and an Engineering Manager
10+ years of experience in build systems and CI/CD pipelines. Tech lead for distributed workflow orchestration at scale. Over 12 years of industry experience with 6 years in engineering leadership, building and scaling tech infrastructure teams that deliver end-to-end large-scale distributed systems.
“CI/CD System” is free with an account. Your progress and bookmarks follow you across devices.
Sign in to continueThink Beyond the Happy Path
Could you answer these today? Tick what you already know.
Design a distributed CI/CD workflow system similar to GitHub Actions that automatically triggers and manages predefined workflows in response to repository events. The system should:
CI/CD isn't just another tech buzzword - it's the backbone of modern software development that keeps your team shipping fast without breaking things.
CI (Continuous Integration) - Your Code's Quality Guardian
Think of CI as that super diligent colleague who checks everyone's work before it gets merged. Every time someone pushes code:
CD (Continuous Delivery/Deployment) - Your Release Automation Hero
FR1 – Trigger workflows on each Git push using the repo-defined workflow file
This covers event ingestion, authentication, deduplication, workflow parsing, and DAG construction from the repository's workflow configuration.
FR2 – Run workflows in isolated sandboxes with scoped secrets
Jobs must execute in containerized or VM-based isolation with per-job resource allocation, secret injection, and artifact sharing between tasks.
FR3 – Provide real-time visibility into execution (status, step progress, logs)
Users should see live log streaming, step-by-step progress, and status updates as workflows execute across distributed runners.
Let's assume the system handles ~10 pushes per second on average, and each workflow takes about 15 minutes to run. That translates to roughly 10,000 concurrent workflows under normal load. During peak traffic — such as coordinated releases or mass test retries — the system may need to handle up to 10x the baseline, i.e., 100,000 concurrent workflows. This framing helps us size the CI/CD platform not just for today's team size but for realistic workload bursts, ensuring fast feedback, minimal queuing, and uninterrupted developer productivity.
NFR1 – High Concurrency
The system shall support concurrent execution of at least 100,000 jobs across the cluster.
NFR2 – Job Isolation
The system shall enforce per-job container isolation to limit blast radius and prevent cross-workflow interference.
NFR3 – Multi-Tenant Fairness
The system shall enforce per-organization/repo quotas to guarantee fairness and prevent noisy-neighbor effects.
NFR4 – Durability & Recovery
The system shall persist workflow state and artifacts across failures to enable safe recovery.
NFR5 – Low Latency Trigger Processing
The system shall process workflow triggers within 10 seconds of receiving a Git event during peak time.
Requirement Summary
| Functional Requirements (FRs) | |
|---|---|
| Name | Description |
| 1. Workflow Triggering | Trigger workflows on Git push events using repo-defined YAML configs with DAG construction. |
| 2. Isolated Execution | Run jobs in containerized sandboxes with scoped secrets and artifact sharing. |
| 3. Real-Time Visibility | Stream logs live, show step progress, and push status updates to users. |
| Non-Functional Requirements (NFRs) | |
|---|---|
| Name | Description |
| 1. High Concurrency | Support 100K+ concurrent jobs across the cluster. |
| 2. Job Isolation | Container-level isolation to prevent cross-workflow interference. |
| 3. Multi-Tenant Fairness | Per-org/repo quotas to prevent noisy-neighbor effects. |
| 4. Durability & Recovery | Persist state and artifacts for crash recovery. |
| 5. Low Latency Triggers | Process triggers within 10s during peak load. |
| Core Entities | |
|---|---|
| Name | Description |
| Repository | A registered source code repository within the CI/CD system. |
Workflow_Run | A versioned specification tied to a commit that defines the pipeline. |
| Task | A single node within the workflow's DAG, executed in isolation. |
Task_Attempt | An individual execution of a task, tracking retries and logs. |
| Artifact | File (build output, logs) produced by a task or task_attempt. |
GitHub Event → WorkflowRun (one per trigger) → Tasks (one per task, DAG edges from prerequisites[]) → TaskAttempts (one per execution try).
How to Deliver High-Level Design in Your Interviews?
Many candidates worry that if they don't mention everything upfront, they'll miss key evaluation points. But rather than frontloading everything, it's better to start from flow, show awareness of interesting decision points, and invite the interviewer to go deeper later.
Here's a structured 5-step approach:
In a modern CI/CD system, every pipeline is triggered by a Git event — such as a push, pull request, or merge. To manage this reliably at scale, we introduce an Ingress Service as the entry point.
Core Responsibilities:
Key Design Principles:
Let's walk through how this works step by step.
Step 1: GitHub Push Event
GitHub Actions is powered internally by GitHub's event system, which is built on webhooks. When something happens (push, pull request, issue comment, release, etc.), GitHub fires an internal webhook-like event.
Step 2: Authenticate GitHub Action Event
The first responsibility is to authenticate the event via API Gateway, since processing an unauthenticated or replayed request could allow attackers to trigger arbitrary builds.
We expose a public endpoint (POST /webhooks/git) where the handler auto-detects the Git provider based on headers. Authentication varies by provider:
| Authentication Methods by Provider | |
|---|---|
| Name | Description |
| GitHub | HMAC-SHA256 via X-Hub-Signature-256 |
| GitLab | |
Step 3: Deduplication
Git providers use at-least-once delivery, meaning the same event may be sent multiple times due to retries, delays, or transient network failures. Without protection, this can cause duplicate workflow runs. We'll cover this in DD1.
Step 4: Fetch & Validate Workflow Definition
Once the event is authenticated and deduplicated, we fetch the workflow definition (e.g., .ci/workflows.yaml) pinned to the exact commit SHA referenced in the event.
Fetching by commit SHA, rather than by branch name (main), ensures that the workflow definition is tied to an immutable snapshot of the codebase. This makes builds reproducible and guards against "moving target" definitions that could change between retries.
Step 5: Parse and Validate the Workflow
After fetching, the workflow YAML is parsed into an internal representation. Validation is multi-layered:
prerequisites[], downstream[] graph, invalid job references, undeclared artifact names.Stricter validation may block valid edge cases, but it reduces runtime failures and strengthens security. Early failure provides clear feedback to developers and saves cluster resources.
Step 6: Construct DAG
Once validated, the workflow definition is transformed into a Directed Acyclic Graph (DAG) of tasks.
In a CI/CD system, you can either model execution as a DAG or as a Step-by-Step flow.
A DAG gives you parallelism, the ability to fan out and fan in jobs, handle matrices, and retry failed nodes independently — which makes pipelines faster and more resilient, though it comes at the cost of added complexity in scheduling and state management.
A Step-by-Step flow is much simpler to reason about: each stage runs after the previous one, which works well for short, strictly linear pipelines, but it limits scalability since failures often require re-running the entire pipeline and there's no room for parallelism.
We suggest defaulting to a DAG for parallelism and selective retries, but collapsing to Step-by-Step when the pipeline is strictly linear.
The execution model is layered:
1. Workflow_run: Provides a reliable top-level object for retries, history, and observability.
2. Tasks: Breaking runs into tasks exposes parallelism, enforces dependencies, and isolates execution.
3. Task_Attempts: Attempts make transitions idempotent and auditable — retries don't corrupt prior state.
Durable Persistence
The DAG is persisted in a durable store, so if the controller crashes, a reconciler can rehydrate state and resume execution. This persistence adds minor latency and storage cost, but in CI/CD systems, durability and correctness far outweigh raw speed.
| Shared token via X-Gitlab-Token |
| Bitbucket | JWT or shared HMAC (if configured) |
All signatures are verified against the raw request body using constant-time comparison to mitigate timing attacks. We also support dual-secret rotation so keys can be rotated without downtime.
HMAC (Hash-based Message Authentication Code) is a cryptographic technique that uses a secret shared key and a hash function to verify both the integrity and authenticity of a message.