Smart Grid Device Control

Video Walkthrough — Part I

53:54

Video Walkthrough — Part II

58:00

Requirements#

Functional Requirements#

We'll scope to four FRs. Each one maps to exactly one flow in the High Level Design.

FR1 — Campaign planning

A user creates a campaign that defines the target devices, requested action, desired reduction, effective time, and expiry. The platform persists the campaign first, resolves eligible devices, and then fans the command out asynchronously.

FR2 — Send commands to devices

The platform reliably sends commands to targeted devices even when the network is unreliable. Online devices receive commands quickly; offline devices receive still-valid commands when they reconnect. Retries must be safe, stale commands must not take effect, and devices must be able to ACK or refuse.

FR3 — Feedback loop

Devices continuously report telemetry such as power usage. The platform uses this data to determine what actually happened after a command, aggregate the achieved reduction for the campaign, and decide whether the next action should be to hold, escalate, relax, or restore.

FR4 — Reconciliation

Telemetry may arrive substantially late after devices reconnect. The platform must assign late samples to the correct event-time window, deduplicate retries, recompute affected historical results, and publish corrected campaign results without erasing the evidence that earlier decisions were made using older information.

Out of scope (say this out loud): grid physics, demand forecasting, settlement/billing, device firmware internals, and detecting maliciously falsified telemetry. Each of these is its own interview.

Command delivered  !=  command executed
ACK                !=  physical verification
No telemetry        =  UNKNOWN

Non-Functional Requirements#

NFR1 — Reliability. No accepted campaign is lost. Commands use at-least-once delivery with idempotent device behavior. Telemetry is ACKed to the device only after durable acceptance. We are controlling physical equipment over an unreliable network; "probably delivered" is not a contract.

NFR2 — Latency. P95 command delivery under 10 seconds for online devices; a provisional campaign result within a few minutes. The feedback loop has to react fast enough to be operationally useful during a grid event.

NFR3 — Scalability and burst headroom. ~10M registered devices; ~700K telemetry samples/sec steady state; several-times burst headroom. Reconnect storms create catch-up traffic that is orders of magnitude above steady state — and they arrive exactly when the operator most needs fresh data.

NFR4 — Correctness under uncertainty. Deduplicate retries, preserve event time, model UNKNOWN explicitly, and revise historical results when late data arrives. A wrong conclusion about missing or stale data makes the next control action wrong.

Scale & Back-of-Envelope Estimations#

The number that identifies the real system:

Steady state   = 700K samples/sec
Reconnect burst = 720M samples arriving within minutes
               ≈ 17 minutes of steady-state traffic, compressed into a spike
               → and it lands while the operator needs *current* data most

The exact number matters less than the architectural conclusion: steady-state ingestion is already large, and reconnect storms replay hours of buffered data on top of it. That strongly motivates asynchronous processing and a partitioned durable log between the device edge and downstream computation — and, more subtly, isolating live telemetry from catch-up telemetry so the control loop never goes blind during recovery.

Interviewer signal: candidates who name the reconnect storm and the live-vs-catch-up tension have found the real system; candidates who size a time-series database and stop have designed a dashboard.

Follow-up Questions to Expect#

These four questions give the rest of the design its purpose. Introduce them before drawing the final architecture — every component below exists because one of these forces it.

• [DD1] Stale commands: an offline device reconnects after a newer command has already been issued. What stops the old REDUCE 20% from executing after the RESTORE?
• [DD2] Reconnect storm: a million devices reconnect at the same moment and upload two hours of buffered telemetry. How do you absorb the backlog without starving command delivery or live telemetry?
• [DD3] Late data changes history: missing telemetry caused you to escalate; the data arrives ten minutes later and proves the devices had complied. How do you correct the result without pretending the escalation never happened?
• [DD4] ACK without effect: the device ACKs the command but telemetry shows no reduction. How does the system decide whether the command actually worked?

Core Thinking & Design Pattern#

Before drawing boxes, orient around three tensions that shape everything:

1. This is a closed-loop control system, not a monitoring pipeline#

User plans a campaign
    → platform sends commands
    → devices produce telemetry
    → platform evaluates the result and adjusts the plan
    → late telemetry may revise an earlier conclusion

The design is not just about command delivery and not just about telemetry ingestion. The interesting part is the relationship between the two: telemetry is the evidence that closes the loop, and a later plan must be able to say exactly which evidence it was based on.

2. Transport state and physical outcome are different dimensions#

ACKED answers "what happened to the message?" VERIFIED answers "what did the device physically do?" They come from different channels (command transport vs. telemetry), they change at different times, and ACKED + NOT_VERIFIED is a perfectly valid state. Collapsing them into one SUCCESS state is the single most common mistake in this interview.

3. Correctness must survive incomplete information#

The platform will act before all the relevant telemetry has arrived — that is the nature of a grid event. So the design has to (a) make uncertainty explicit (UNKNOWN is a first-class result, not a failure), (b) process by event time rather than arrival time, and (c) version its conclusions so that a decision made on partial data remains auditable after the data completes.

Core Entities & API#

Core Entities#

Keep the model intentionally small — only the objects needed to explain the four FRs. User is the starting actor, but identity belongs to the auth system; the campaign keeps only created_by_user_id for audit. Presence lives in a TTL cache, not a domain table.

Device

Enough to identify the device and resolve campaign targets.

Campaign

The user's durable intent. Persisted before any device is contacted.

DeviceCommand

One row per (campaign, device). The composite key is the materialized target set: if (C1, D7) exists, D7 was part of C1 at fan-out time.

Transport state machine — deliberately small:

PENDING -> DELIVERED -> ACKED
                    \-> REFUSED

PENDING   -> EXPIRED
DELIVERED -> EXPIRED

PENDING -> DELIVERED     platform confirms the command reached the device session
DELIVERED -> ACKED       device explicitly acknowledges
DELIVERED -> REFUSED     device explicitly refuses (e.g. owner override)
* -> EXPIRED             validity window ended before the command could be applied

There is no VERIFIED in this state machine. Verification comes from telemetry and answers a different question: what did the device physically do? (DD4)

TelemetrySample

CampaignRollup

The versioned, materialized result of a campaign. Late telemetry produces a new version; old versions are never overwritten.

Campaign C1, rollup v3 -> 30 MW confirmed
Campaign C1, rollup v4 -> 37 MW confirmed, after late telemetry

We keep both versions so a later reader can still understand why a follow-up campaign was created from v3.