You're asked to design a platform that controls a very large fleet of smart devices — EV chargers, HVAC systems, refrigerators, industrial equipment, even whole buildings. A grid operator creates a campaign such as *"Northeast HVAC: reduce power by 20% for two hours, non-critical devices first, observe telemetry, adjust if the reduction is insufficient."* The hard part is that everything happens over the public internet. Devices may be offline. Commands may be delayed or duplicated. Owners may refuse. An ACK does not prove that the physical action happened. Telemetry can arrive late, duplicated, or out of order. After an outage, a million devices may reconnect and upload hours of buffered data at once. The candidate must design a system that: - Lets a user create a campaign targeting a device population, then fans commands out asynchronously - Delivers commands reliably to online *and* offline devices without ever applying a stale command - Uses telemetry to determine what physically happened, and feeds that into the next control decision - Reconciles late telemetry into the correct event-time window and corrects earlier results — without erasing the evidence that earlier decisions were made on older information Before diving in, ask yourself: - What does an ACK actually prove — and what does it not prove? - If a device reconnects after a newer command has already superseded the one it never received, what stops the old one from executing? - What happens to your live control loop when a million devices replay two hours of buffered telemetry at the same moment? - When late data proves you escalated unnecessarily, do you rewrite history — or version it? You don't need to answer all of these right away. A **Senior/Staff+ Engineer** doesn't stop at basic functionality — they **anticipate edge cases**, **design for resilience**, and **push for production-grade reliability**. But **great systems start with great questions.** What would *you* ask next? ## Requirements ### Functional Requirements We'll scope to four FRs. Each one maps to exactly one flow in the High Level Design. **FR1 — Campaign planning** A user creates a campaign that defines the target devices, requested action, desired reduction, effective time, and expiry. The platform persists the campaign first, resolves eligible devices, and then fans the command out asynchronously. **FR2 — Send commands to devices** The platform reliably sends commands to targeted devices even when the network is unreliable. Online devices receive commands quickly; offline devices receive *still-valid* commands when they reconnect. Retries must be safe, stale commands must not take effect, and devices must be able to ACK or refuse. **FR3 — Feedback loop** Devices continuously report telemetry such as power usage. The platform uses this data to determine what actually happened after a command, aggregate the achieved reduction for the campaign, and decide whether the next action should be to hold, escalate, relax, or restore. **FR4 — Reconciliation** Telemetry may arrive substantially late after devices reconnect. The platform must assign late samples to the correct event-time window, deduplicate retries, recompute affected historical results, and publish corrected campaign results without erasing the evidence that earlier decisions were made using older information. > **Out of scope (say this out loud):** grid physics, demand forecasting, settlement/billing, device firmware internals, and detecting maliciously falsified telemetry. Each of these is its own interview. **The key principle behind FR3 and FR4** ``` Command delivered != command executed ACK != physical verification No telemetry = UNKNOWN ``` Most of the design exists to keep those three distinctions honest. ### Non-Functional Requirements **NFR1 — Reliability.** No accepted campaign is lost. Commands use at-least-once delivery with idempotent device behavior. Telemetry is ACKed to the device only after durable acceptance. We are controlling physical equipment over an unreliable network; "probably delivered" is not a contract. **NFR2 — Latency.** P95 command delivery under 10 seconds for online devices; a provisional campaign result within a few minutes. The feedback loop has to react fast enough to be operationally useful during a grid event. **NFR3 — Scalability and burst headroom.** ~10M registered devices; ~700K telemetry samples/sec steady state; several-times burst headroom. Reconnect storms create catch-up traffic that is orders of magnitude above steady state — and they arrive exactly when the operator most needs fresh data. **NFR4 — Correctness under uncertainty.** Deduplicate retries, preserve event time, model UNKNOWN explicitly, and revise historical results when late data arrives. A wrong conclusion about missing or stale data makes the *next* control action wrong. ### Scale & Back-of-Envelope Estimations **The number that identifies the real system:** ```jsx Steady state = 700K samples/sec Reconnect burst = 720M samples arriving within minutes ≈ 17 minutes of steady-state traffic, compressed into a spike → and it lands while the operator needs *current* data most ``` The exact number matters less than the architectural conclusion: steady-state ingestion is already large, and reconnect storms replay hours of buffered data on top of it. That strongly motivates asynchronous processing and a partitioned durable log between the device edge and downstream computation — and, more subtly, isolating live telemetry from catch-up telemetry so the control loop never goes blind during recovery. > **Interviewer signal:** candidates who name the reconnect storm and the live-vs-catch-up tension have found the real system; candidates who size a time-series database and stop have designed a dashboard. ### Follow-up Questions to Expect These four questions give the rest of the design its purpose. Introduce them *before* drawing the final architecture — every component below exists because one of these forces it. - **[DD1] Stale commands:** an offline device reconnects after a newer command has already been issued. What stops the old `REDUCE 20%` from executing after the `RESTORE`? - **[DD2] Reconnect storm:** a million devices reconnect at the same moment and upload two hours of buffered telemetry. How do you absorb the backlog without starving command delivery or live telemetry? - **[DD3] Late data changes history:** missing telemetry caused you to escalate; the data arrives ten minutes later and proves the devices had complied. How do you correct the result without pretending the escalation never happened? - **[DD4] ACK without effect:** the device ACKs the command but telemetry shows no reduction. How does the system decide whether the command actually worked? ## Core Thinking & Design Pattern Before drawing boxes, orient around three tensions that shape everything: ### 1. This is a closed-loop control system, not a monitoring pipeline ``` User plans a campaign → platform sends commands → devices produce telemetry → platform evaluates the result and adjusts the plan → late telemetry may revise an earlier conclusion ``` The design is not just about command delivery and not just about telemetry ingestion. The interesting part is the **relationship between the two**: telemetry is the evidence that closes the loop, and a later plan must be able to say exactly which evidence it was based on. ### 2. Transport state and physical outcome are different dimensions `ACKED` answers "what happened to the message?" `VERIFIED` answers "what did the device physically do?" They come from different channels (command transport vs. telemetry), they change at different times, and `ACKED + NOT_VERIFIED` is a perfectly valid state. Collapsing them into one `SUCCESS` state is the single most common mistake in this interview. ### 3. Correctness must survive incomplete information The platform will act before all the relevant telemetry has arrived — that is the nature of a grid event. So the design has to (a) make uncertainty explicit (`UNKNOWN` is a first-class result, not a failure), (b) process by *event time* rather than arrival time, and (c) version its conclusions so that a decision made on partial data remains auditable after the data completes. **The core recognition** This is a **distributed closed-loop control system over an unreliable edge.** Commands need at-least-once transport made safe by idempotency, per-device ordering, and expiry. Telemetry needs a durable, partitioned log that absorbs reconnect storms and feeds an event-time feedback processor. And the loop's outputs must be versioned, because late data will change the past while the next decision moves the future. ## Core Entities & API ### Core Entities > Keep the model intentionally small — only the objects needed to explain the four FRs. `User` is the starting actor, but identity belongs to the auth system; the campaign keeps only `created_by_user_id` for audit. Presence lives in a TTL cache, not a domain table. #### Device Enough to identify the device and resolve campaign targets. #### Campaign The user's durable intent. Persisted *before* any device is contacted. `parent_campaign_id` and `based_on_rollup_ver` are the two columns that make reconciliation auditable. A follow-up campaign records both the campaign it is reacting to *and* the exact evidence version that triggered it. #### DeviceCommand One row per (campaign, device). The composite key is the **materialized target set**: if `(C1, D7)` exists, D7 was part of C1 at fan-out time. **Transport state machine — deliberately small:** ``` PENDING -> DELIVERED -> ACKED \-> REFUSED PENDING -> EXPIRED DELIVERED -> EXPIRED ``` ``` PENDING -> DELIVERED platform confirms the command reached the device session DELIVERED -> ACKED device explicitly acknowledges DELIVERED -> REFUSED device explicitly refuses (e.g. owner override) * -> EXPIRED validity window ended before the command could be applied ``` > There is **no** `VERIFIED` in this state machine. Verification comes from telemetry and answers a different question: what did the device physically do? (DD4) #### TelemetrySample #### CampaignRollup The versioned, materialized result of a campaign. Late telemetry produces a new version; old versions are never overwritten. ``` Campaign C1, rollup v3 -> 30 MW confirmed Campaign C1, rollup v4 -> 37 MW confirmed, after late telemetry ``` We keep both versions so a later reader can still understand why a follow-up campaign was created from v3. ### API Design The API surface stays small because most of the complexity is asynchronous. #### Create a campaign (FR1) ```bash POST /v1/campaigns Idempotency-Key: northeast-hvac-2026-09-09-01 { "target_selector": { "region_id": "northeast", "device_class": "hvac", "min_priority_tier": 2 }, "action": "reduce_percent", "parameters": { "percent": 20 }, "target_reduction_w": 40000000, "effective_from": "2026-09-09T14:00:00Z", "expires_at": "2026-09-09T16:00:00Z" } Response 202: { "campaign_id": "cmp_01J9...", "status": "active" } ``` > Returns 202 as soon as the campaign is durable. This call does **not** wait for 500,000 devices to be contacted — fan-out is asynchronous (FR1 flow). #### Read the latest campaign result (FR3 / FR4) ```bash GET /v1/campaigns/{campaign_id}/rollup?version=latest Response 200: { "campaign_id": "cmp_01J9...", "version": 4, "watermark": "2026-09-09T14:06:00Z", "targeted_count": 412000, "verified_count": 296000, "refused_count": 18000, "unknown_count": 67000, "achieved_reduction_w": 37000000, "unknown_reduction_w": 1000000 } ``` > The watermark and `unknown_*` fields are part of the contract. A rollup that hides its own staleness is more dangerous than one that says "data before 14:06 is substantially complete; the rest is provisional." #### Catch-up telemetry upload (FR3 / FR4) ```bash POST /v1/devices/{device_id}/telemetry/batch Content-Encoding: gzip { "samples": [ { "sample_seq": 90311, "event_time": "2026-09-09T14:00:10Z", "power_w": 2400, "applied_campaign_id": "cmp_01J9..." }, ... ] } Response 202 (only after durable append to the telemetry log) ``` > The platform ACKs the batch only after it has been durably accepted into the ingestion path. If the response is lost, the device retries the same sequence numbers and the platform deduplicates on `(device_id, sample_seq)`. ## High Level Design > The best way to present this design is **not** to draw the final architecture immediately. Start with the business loop — `plan → command → observe → adjust` — and add one component only when a real failure case forces it. The result is still one closed loop: a **control plane** (Campaign API, Campaign DB, Device Registry, fan-out worker, durable command log, gateway/MQTT broker with an offline outbox) and a **telemetry/feedback plane** (live gateway and catch-up batch API converging on a durable telemetry log, a raw store, an event-time feedback processor, and the versioned CampaignRollup that feeds the next decision). Walk the interviewer through the same picture using the four FRs: ``` FR1 User creates Campaign -> persist -> resolve Devices -> create DeviceCommand rows FR2 DeviceCommand -> durable command log -> gateway/outbox -> Device FR3 Device -> telemetry -> telemetry log -> feedback processor -> CampaignRollup -> next plan FR4 Late telemetry -> original event-time window -> recompute -> CampaignRollup N+1 ``` ### FR1 — Campaign planning: separating intent from fan-out ``` User → POST /v1/campaigns → Campaign API 1. Validate; check Idempotency-Key 2. Persist Campaign (status=active) [durable intent — source of truth] 3. Return 202 { campaign_id } Fan-out Worker (asynchronous): 4. Resolve target_selector against Device Registry (region, device_class, priority_tier, opt_out = false) 5. Insert one DeviceCommand row per device, PK (campaign_id, device_id) 6. Publish each command to the durable command log, keyed by device_id ``` Two design points worth narrating: **First, persist before fan-out.** The naive design resolves 500K devices and contacts them inside the request. If the service crashes halfway, the user doesn't know whether the campaign exists. Persisting the `Campaign` row first turns the API into a fast, idempotent write and moves the expensive work to a worker that can resume after a crash. **Second, `DeviceCommand` is the materialized target set.** The composite key `(campaign_id, device_id)` prevents duplicate target rows when the fan-out worker restarts, and it freezes the answer to "who was in this campaign?" at fan-out time — which the rollup needs later to compute `targeted_count` and `unknown_count` honestly. ### FR2 — Command delivery: at-least-once transport, effectively-once effect ``` DeviceCommand → Durable Command Log (key = device_id, so per-device order holds) → Device Gateway / MQTT Broker online → deliver now offline → retain in per-device outbox until reconnect or expires_at → Device Command contract: command_id stable identity for deduplication device_seq monotonically increasing per device action REDUCE / CAP / RESTORE ... parameters expires_at after this, the command must not take effect Device rule on receipt: command_id already processed? -> ACK again, do not re-apply device_seq < last_applied_seq? -> stale; reject now > expires_at? -> expired; reject otherwise -> apply or refuse; persist last_applied_seq; ACK/REFUSE ``` The network contract is **at-least-once**, not exactly-once. A device may receive the same command more than once, so the command itself carries enough metadata for the device to decide whether applying it is still valid. Delivery state (`DELIVERED`, `ACKED`, `REFUSED`, `EXPIRED`) flows back into `DeviceCommand`; the database remains the source of truth for what we *intended* to send, and the log and broker are delivery machinery. > `ACKED` and `REFUSED` are different outcomes on purpose: a smart-grid device is not a passive server process, and the owner may have the right to refuse. ### FR3 — Feedback loop: telemetry as the evidence channel ``` Device (local buffer) live samples → Live Telemetry Gateway ─┐ buffered batches → Catch-up Batch API ─┴→ Durable Telemetry Log (partition = hash(device_id)) ├→ Raw Telemetry Store [TelemetrySample, replayable] └→ Feedback Processor [event-time windows] → CampaignRollup vN → user / policy → next campaign Per-device assessment (intentionally simple): baseline_w = representative power before the campaign expected_w = baseline_w × (1 − requested_reduction) observed_w = average telemetry in the assessment window coverage = received samples / expected samples explicit refusal -> REFUSED insufficient coverage -> UNKNOWN observed <= expected + tolerance -> VERIFIED otherwise -> NOT_VERIFIED ``` Command delivery tells us what the transport did; telemetry tells us what the physical device did. The feedback processor evaluates by **event time**, aggregates per-device assessments into the campaign rollup, and exposes `achieved_reduction_w` *and* `unknown_reduction_w` so the policy layer can decide how conservatively to react. The exact threshold is replaceable business logic; the separation of concerns is not. ### FR4 — Reconciliation: late data revises the past, not the decision ``` Late TelemetrySample (event_time 10:02, ingest_time 10:10) | dedupe by (device_id, sample_seq) v Telemetry Log | find campaign + ORIGINAL event-time window (10:00–10:05) v Recompute affected campaign evidence | publish corrected aggregate v CampaignRollup version N+1 ├→ user / dashboard reads latest └→ next campaign decision reads latest C2.parent_campaign_id = C1 C2.based_on_rollup_ver = 3 <- unchanged: C2 was decided on v3 ``` Every sample carries `event_time` (when reality happened) and `ingest_time` (when the platform learned about it). A sample measured at 10:02 and received at 10:10 belongs to the **10:02** window. The processor's watermark makes completeness visible — "wall clock 10:12, watermark 10:06" means data before 10:06 is believed substantially complete and everything after is provisional. When late data reopens a window, the system publishes a **new** rollup version rather than updating the old row, so the fact that an earlier decision was reasonable on the information available at the time is never erased. ## Deep Dives The deep dives are the four challenge questions from the requirements section. Each starts from the failure, shows why the simple design breaks, and derives the mechanism that fixes it — rather than jumping to a list of technologies. ### DD1 — What if an offline device reconnects after a newer command has already been issued? Offline delivery by itself is easy: keep the command somewhere durable and send it on reconnect. The dangerous case is **stale command execution**. A device drops offline at 13:55; while it is gone the platform issues `REDUCE 20%` at 14:00 and `RESTORE` at 14:30; the device returns at 15:00. A naive retry queue may deliver both, more than once, in an order the sender never intended. If the device executes whatever arrives, an obsolete reduction is applied after the grid event is already over. ``` 14:00 REDUCE 20% device_seq = 101 expires_at = 14:30 14:30 RESTORE device_seq = 102 expires_at = 16:00 15:00 device reconnects ``` **Correctness cannot depend on the transport.** The network is unreliable, the device can disappear between receive and acknowledge, and either side can crash at an awkward moment. Consider: ``` Platform -> sends command 101 Device -> receives and applies 101 Device -> ACK is lost Platform -> times out, retries 101 ``` The platform cannot know whether the command was lost or only the ACK. Retrying is necessary for reliability — which gives us **at-least-once delivery** whether we like it or not. So the command itself must carry enough metadata for the device to decide: ``` command_id -> duplicates (already processed? ACK again, don't re-apply) device_seq -> superseded commands (older than last_applied_seq? reject as stale) expires_at -> no-longer-safe (past expiry? reject as expired) ``` For the example above, `REDUCE 101` is already expired at 15:00 and `RESTORE 102` is the newest valid command. Even if the broker redelivers both, the physical outcome is deterministic. ``` at-least-once transport + idempotent command identity + per-device ordering + command expiry = retry-safe physical behavior ``` > The phrase to say out loud: we do not need exactly-once **message delivery** if we can achieve effectively-once **physical effect**. **Where should offline commands live?** Two reasonable options: MQTT-style brokers retain pending messages for a device session. Attractive because offline delivery is already part of the edge protocol; the trade-off is how much delivery state you entrust to the broker. The platform stores pending `DeviceCommand` rows and, when presence indicates a reconnect, a delivery worker publishes the still-valid command. More control, more code. Either works. What matters is that the **durability boundary is explicit**: once the API tells the user the campaign was accepted, the command intent must survive service crashes. The DB is the source of truth for intent; the log and broker are delivery machinery. **Expiry must travel with the command.** A TTL only in the broker is not enough — the device itself should refuse a stale action even if the transport misbehaves. And for time-bounded control actions the contract should be *self-expiring*: `apply REDUCE 20%, valid until 16:00; if no newer valid command exists at 16:00, return to normal`. That turns "the platform must be available to undo state" into "the device fails safe on local command validity" — which also answers the platform-outage question before the interviewer asks it. Don't spend five minutes enumerating command states. State the failure — **offline delivery creates stale-command risk** — then explain why `command_id`, `device_seq`, and `expires_at` solve three *different* problems (duplicates, reordering, no-longer-safe). Close with self-expiring commands as the fail-safe when the platform itself is gone. That distinction is worth more than a large state machine. *Follow-up: some commands really are durable desired state ("set logging level to DEBUG until changed"). How would you model both semantics in one command contract without letting a demand-reduction command outlive its grid event?* ### DD2 — What if a million devices reconnect at the same time? A regional outage disconnects one million devices for two hours. They keep measuring locally. When the network returns, every device wants to establish a session, receive pending commands, resume live telemetry, *and* upload 720 buffered samples — 720M historical samples in total — at exactly the moment the operator most needs current telemetry to know whether the grid has recovered. **Why a single ingestion path is dangerous.** In the naive `Device → API → Processor → DB` pipeline, current telemetry and historical catch-up compete for the same connections, CPU, queue capacity, and processor threads. The result is ironic: the system receives more data than ever, and the operator's view of the *current* grid gets less fresh. So distinguish two traffic classes: ``` Live telemetry -> latency-sensitive, needed by the current feedback loop Catch-up telemetry -> correctness-sensitive, can drain over minutes instead of seconds ``` Give them separate edge paths (live gateway vs. catch-up batch API) on separate capacity. They may converge on the same durable log — the log is built to absorb bursts — but catch-up work must never consume every edge connection or processing slot the live path needs. **Why the durable log matters.** Without a durable buffer, a backend that falls behind has two bad choices: reject data (devices retain longer and retry harder, amplifying the storm) or accept into memory (lose it on process failure). A replicated, Kafka-style log changes the problem: the ingestion tier only authenticates, validates, deduplicates cheaply, and appends; heavy processing happens later. A reconnect storm becomes **consumer lag**, which is measurable and recoverable — not data loss. > "Kafka scales" is not the answer. The answer is that a durable log **decouples bursty arrival rate from sustainable processing rate**. **The device protocol must cooperate.** The platform alone cannot solve a synchronized reconnect: ``` randomized reconnect jitter exponential backoff bounded batch size compression server Retry-After / load hints recent data before old data on reconnect: 1. establish current session 2. receive pending valid commands 3. resume live telemetry 4. drain historical backlog in parallel, at a lower quota ``` Should a device upload two hours of history *before* its current reading? No — the control loop cares far more about "what is happening now." **Partitioning.** Partition the telemetry log by `hash(device_id) % N`: traffic spreads evenly even when one region is hot, and one device's samples stay ordered within a partition. Do *not* partition by region unless strict locality is required — a regional outage is precisely the case where region-based partitioning creates a hot partition. The feedback processor scales by partition; if lag grows, add consumers. **When the system still cannot keep up**, degrade in this order: ``` 1. protect command delivery 2. protect live telemetry ingestion 3. durably accept catch-up telemetry 4. allow historical processing to lag ``` — and expose the lag. The rollup's watermark tells the user whether a result is current or provisional. A system that silently shows stale data is more dangerous than one that says "latest fully processed event time is 10:06." Don't answer "autoscaling" and stop — autoscaling doesn't erase a flash crowd that is orders of magnitude above steady state. Walk the chain: *synchronized burst → live and catch-up compete → isolate traffic classes → append quickly to a durable log → process backlog asynchronously → expose lag as a watermark.* That one story demonstrates scalability, reliability, and operational judgment together. *Follow-up: your catch-up batch API is returning 503 with Retry-After to a million devices. What is the exact device-side behavior you want, and how do you keep a million retries from re-forming the same spike 30 seconds later?* ### DD3 — What if missing telemetry caused us to escalate, and late data later proves the devices had complied? This is the most important correctness deep dive, because the platform has to decide before it has complete information. Campaign C1 asks for 40 MW. At 10:05 the loop has 30 MW confirmed, 8 MW from devices whose telemetry hasn't arrived, and 2 MW known non-compliant. It cannot call the 8 MW successful or failed — the correct state is **unknown**. A conservative policy escalates with campaign C2. At 10:10, thousands of devices reconnect and upload telemetry measured during 10:00–10:05 proving another 7 MW *had* been reduced. Now what? ``` 10:05 C1 Rollup v3 confirmed_reduction = 30 MW unknown_reduction = 8 MW | control decision v C2 parent_campaign_id = C1 based_on_rollup_ver = 3 10:10 late TelemetrySample device_id = D123 event_time = 10:02:10 ingest_time = 10:10:04 power_w = 1920 ``` **Which window should the late sample affect?** 10:02 — because that is when the physical measurement happened. If we aggregated by `ingest_time`, the 10:10 window would show phantom activity and the 10:02 window would stay wrong forever. So the model keeps both timestamps, and the processor reopens the 10:00–10:05 assessment window and recomputes: ``` C1 Rollup v3 confirmed = 30 MW unknown = 8 MW C1 Rollup v4 confirmed = 37 MW unknown = 1 MW ``` **Why not simply update the old row?** ```sql UPDATE campaign_rollup SET confirmed_reduction = 37MW WHERE campaign_id = 'C1'; ``` That keeps the dashboard current but destroys an important fact: C2 was created when the platform only knew about 30 MW. An engineer investigating later asks *"if we already knew C1 achieved 37 MW, why did the system escalate?"* — and the system can no longer answer. Versioning preserves two truths at once: ``` Latest event truth: C1 Rollup v4 says 37 MW actually happened Decision-time knowledge: C2 was created from C1 Rollup v3, which knew about 30 MW ``` **Where does the watermark fit?** It answers a different question — *how complete do we believe the event-time data is?* Wall clock 10:15, watermark 10:09 means results after 10:09 are still expected to change frequently. The watermark is not a guarantee that no late sample will ever arrive; it is an operational boundary: "under our lateness policy, data before this point is sufficiently complete for normal processing." **How much late data should we accept?** Any late record reopens its historical window forever. Maximizes correctness, but old partitions stay permanently mutable and expensive. Automatically correct anything up to, say, 24 hours late; older data flows through a periodic backfill path. The exact window is a product decision — what matters is that **lateness policy is explicit**. Previously published reports are never rewritten; corrections are separate adjustment records. Right for financial or regulatory systems; heavier than this operational loop needs. **Reconciliation changes belief about the past; the next control action changes the future.** It does not travel back in time and undo C2. The loop stays forward-moving: ``` 10:05 C2 created on incomplete evidence 10:10 late data improves C1 from 30 MW to 37 MW (rollup v4) 10:15 next planning cycle reads the corrected rollup and may HOLD, RELAX, or RESTORE ``` **If reconciliation processing crashes**, nothing is lost: raw telemetry is durably stored and the log is replayable, so the rollup is a materialized view, not the only copy of truth. If a bug is found in the aggregation logic, replay raw samples and rebuild rollups. Connect four concepts instead of naming them independently: *late telemetry → use event_time → the old window changes → publish a new rollup version → preserve which old version drove each decision.* That is far stronger than "use watermarks for late data," because the interviewer sees exactly which business correctness the watermark and versioning protect. *Follow-up: a late sample arrives for a window that a follow-up campaign has already been decided on, and the correction would have flipped that decision. Who gets notified, and does anything automatic happen?* ### DD4 — What if the device ACKs, but telemetry shows no physical reduction? The platform sends `REDUCE 20%`; the device replies `ACKED`. We know the message reached the device and the device claims to have accepted it. We do **not** know whether electrical load dropped. A device drawing ~2,500 W should land near 2,000 W; instead telemetry reads 2480, 2510, 2490 W. Transport succeeded; physical behavior did not match. One big state machine — `PENDING → SENT → ACKED → SUCCESS` — hides this, because `SUCCESS` fuses two different facts. Keep transport and assessment as **separate dimensions**: ``` Transport (from command delivery): PENDING -> DELIVERED -> ACKED | REFUSED ; * -> EXPIRED answers: "what happened to the command message?" Assessment (from telemetry): VERIFIED | NOT_VERIFIED | UNKNOWN answers: "what does telemetry say about the physical result?" Legal combinations: command_state = ACKED, telemetry_result = NOT_VERIFIED (ACK'd, no effect) command_state = ACKED, telemetry_result = UNKNOWN (ACK'd, then went silent) ``` **How the platform verifies.** No advanced forecasting model is needed to explain the distributed-system behavior: ``` baseline = 2500 W representative power before the command request = reduce 20% expected = 2000 W baseline × (1 − 0.20) observed = 1980 W average over the assessment window -> VERIFIED observed <= expected × 1.05 -> VERIFIED (tolerance band: devices are noisy) observed ≈ baseline -> NOT_VERIFIED too few samples -> UNKNOWN ``` **Why `UNKNOWN` is essential.** A device ACKs at 14:00 and its network disappears; there are no samples from 14:00 to 14:05. Two tempting assumptions are both wrong: ``` "ACK means it probably worked" -> overstates confirmed reduction "No telemetry means it failed" -> overstates shortfall ``` Neither is supported by evidence. The rollup therefore carries uncertainty explicitly — `achieved_reduction_w` *and* `unknown_reduction_w`, `verified_count` *and* `unknown_count` — and the **policy layer** decides what to do with it: a safety-critical grid may treat unknown as zero; a less conservative policy may weight it by historical compliance. The distributed system exposes uncertainty; the policy consumes it. **`applied_campaign_id` in telemetry** is useful supporting evidence — it tells us which command the device *believes* it is following — but not proof. If the device reports `applied_campaign_id = C1` while power is unchanged, the rollup reflects the observed result. **Noisy baselines** (recent median, same-time-of-day history, device-class model, device-reported set point) each have trade-offs, but baseline computation is replaceable business logic. What matters architecturally is that assessment is computed from event-time telemetry and can be *recomputed* when late samples arrive — which is exactly how DD4 connects back to DD3: ``` late telemetry -> observed_w changes -> device assessment may change -> campaign rollup changes -> publish new version ``` The memorable sentence: **ACK proves communication; telemetry provides evidence of physical behavior.** Then show the two state dimensions and say that `ACKED + NOT_VERIFIED` is a valid state. That single example prevents the most common design mistake in this interview and explains why the feedback loop exists at all. *Follow-up: an HVAC unit legitimately takes several minutes for measured load to fall after a set-point change. How do you size the assessment window per device class without turning every slow device into a false NOT_VERIFIED?* ## Common Pitfalls Each of these reveals a different misunderstanding of the problem. Avoiding them is not about memorizing the "correct architecture" — it is about showing where distributed-system guarantees stop and application-level reasoning must begin. Seeing "millions of devices + telemetry" and drawing `Devices → Kafka → Stream Processor → TSDB → Dashboard` is reasonable for observability but misses the center of this problem. The platform is **trying to change fleet behavior**, observe the result, and decide what to do next. A monitoring-only system has no `Campaign`, `DeviceCommand`, command expiry, refusal state, target snapshot, or link between a later plan and the evidence that caused it. Make the control loop visible early — even `User → Campaign → Device → Telemetry → Feedback → User / Policy` in one line signals you understand the problem. The opposite failure: drawing API gateway, command DB, target resolver, Kafka, MQTT, presence store, outbox, telemetry API, catch-up API, a second Kafka, stream processor, object store, OLAP DB, watermark manager, reconciliation worker, and policy engine in the first five minutes. Every box may be defensible, but the interviewer has to reverse-engineer why each exists. Start with the business loop and let a concrete failure force each component in: *"API crashes after accepting a campaign?"* → persist first. *"Device offline?"* → durable command + outbox. *"Telemetry bursts?"* → durable log. *"Data ten minutes late?"* → event time + versioned rollup. Say explicitly: "I'll start with the logical flow, then add reliability and scale as we hit failure cases." Depending on the protocol, an ACK may mean *message received*, *command parsed*, *command accepted*, or *command applied locally*. None proves the physical effect. A refrigerator can accept a command and be unable to reduce power because of a local safety constraint; an EV charger can ACK and be overridden by its owner; an HVAC controller can apply a new set point while measured load stays high for minutes. Trusting device-reported execution state (Option A) is simpler and lower latency; using telemetry as a second evidence channel (Option B) adds delay and an assessment model but distinguishes "delivered" from "effect observed." For this prompt, Option B is the right default — and the interviewer should hear you say `ACKED + NOT_VERIFIED` is a valid state. 100,000 devices go silent right after a command. They may be offline and compliant, offline and non-compliant, behind a degraded telemetry path, online with a crashed local agent, or buffering samples that will arrive later. Mapping silence to `NOT_VERIFIED` over-corrects and sheds unnecessary load; mapping it to success under-corrects. Model `UNKNOWN` explicitly, report confirmed and uncertain capacity separately, and let policy decide how conservative to be. Presence (from a TTL cache) can distinguish "offline" from "online but silent" for diagnosis — but it never changes the rule that missing telemetry does not prove outcome. A sample measured at 10:02 and uploaded at 10:10 must count in the 10:02 window. Grouping by server arrival time leaves 10:02 wrong forever and pollutes 10:10 with a measurement that didn't happen then. Use `event_time` for metrics and compliance; keep `ingest_time` for lag measurement, debugging, and lateness decisions. Device clocks drift — clock sync, monotonic sample age, sequence comparison, or falling back to server time for flagged devices are all options — but you don't need to solve clock synchronization to recognize that arrival time and event time have different semantics. Show one concrete example with both timestamps; it makes watermarks and reconciliation obvious instead of abstract. "Reliable delivery" is not "never stop retrying." A demand-response event ends at 16:00; a device offline all afternoon reconnects at midnight; faithfully delivering `REDUCE 20%` at midnight is a messaging success and a product failure. Three mechanisms, three jobs: *retry* handles temporary loss, *ordering* rejects superseded commands, *expiry* prevents commands taking effect outside their valid window — and expiry must travel with the command, not just live in the broker. Some commands are durable desired state ("set logging level to DEBUG until changed"); a load-shedding command is not. Ask whether executing late is still correct; if not, expiry is part of the contract. Updating the rollup row in place looks attractive until a later action was taken on the earlier value. Rollup v3 says 30 MW at 10:05; the system escalates at 10:06; late data shows 37 MW at 10:10. Overwrite v3 and the record is internally inconsistent — the system appears to have escalated even though the stored evidence says it nearly met target. Version materialized results and record `based_on_rollup_ver` on the follow-up. Connect it to a real action: "we must preserve v3 because a real follow-up campaign was issued from v3" is concrete; "we keep versions for audit" is generic. Persisting only the latest campaign totals works until the aggregation code has a bug, the definition of compliance changes, or a new reporting dimension is needed. Keep raw telemetry as the immutable, replayable source (object storage plus the replayable log) and treat `CampaignRollup` as a derived materialized view for fast decisions. Keep high-resolution raw data for a bounded retention period and downsampled history longer. If the interviewer asks "what if your aggregation logic is wrong?", the answer is replay and backfill from raw — not "we hope our aggregate is correct." Deciding exactly when to escalate, how much more to shed, or which devices to target next are real product and control-policy questions — and a distraction from the distributed-system design. The infrastructure provides trustworthy inputs (confirmed reduction, unknown reduction, refusals, freshness/watermark, campaign version); policy chooses a strategy (conservative: unknown = 0; probabilistic: weight by historical compliance; human-in-the-loop: show uncertainty, require approval). Say the boundary out loud: "I'll design the platform so the policy engine has reliable, versioned evidence. The exact escalation formula is configurable business logic." Most of the conversation is about devices being offline, but the control plane can fail too. If a device needs a later `RESTORE` message to return to normal, a prolonged platform outage leaves physical equipment constrained indefinitely — recovery depends on the very system that is down. For time-bounded control actions, make the device contract self-expiring: `apply REDUCE 20%, valid until 16:00; if no newer valid command exists at 16:00, return to normal`. Reliability is not just "never lose a message"; it is reaching a safe state when communication is unavailable. This is an excellent point to surface right after command expiry. ## Final Thoughts The shape of a strong answer: recognize early that this is a **distributed closed-loop control system**, not a telemetry pipeline, and surface the four failure cases before drawing the final architecture. Then let each failure force exactly one mechanism into the design — persist-before-fan-out, at-least-once transport made safe by identity/ordering/expiry, a durable log that turns reconnect storms into lag, event-time processing with a visible watermark, and versioned rollups that keep both latest truth and decision-time truth. The FR→flow mapping keeps the HLD legible; the four deep dives are the same four questions you raised at the start, now answered. **ACK proves communication; telemetry proves behavior** Keep transport state and physical assessment as separate dimensions. `ACKED + NOT_VERIFIED` and `ACKED + UNKNOWN` are valid states, and the rollup must carry `unknown_reduction_w` explicitly. A single `SUCCESS` state fuses two facts and makes the next control action wrong. **Effectively-once effect, not exactly-once delivery** The network gives you at-least-once, so put the safety in the command contract: `command_id` for duplicates, `device_seq` for superseded commands, `expires_at` for commands that are no longer safe — and make grid actions self-expiring so devices fail safe when the platform is gone. **Version the past; move the future forward** Late telemetry belongs to the event-time window where reality happened, and it produces rollup version N+1 rather than an `UPDATE`. The follow-up campaign records `based_on_rollup_ver`, so the system can always explain why it escalated on the information it had — while the next planning cycle reads the corrected result. **Common failure modes to avoid:** designing a monitoring dashboard when the prompt is a control loop, drawing the full architecture before naming a single failure case, and collapsing uncertainty too early — either by trusting ACKs as outcomes or by treating silence as failure. > These aren't toy problems — they reflect real infrastructure challenges at companies operating large device fleets over the public internet. If you want to walk through this design in a mock interview or get feedback on your own approach, book a session with us. We're here to help you sharpen your thinking, build your confidence, and land your next offer.