Shahzad Bhatti Welcome to my ramblings and rants!

Introduction

Every major cloud now supports Serverless FAAS capabilities like AWS Lambda/Step Functions, Azure Durable Functions, GCP Cloud Functions and Cloudflare Durable Objects where you write a function or a small stateful actor. This allows you to scale it, pay only for what runs but there is a catch, you build on a proprietary runtime, and the runtime’s storage model, invocation model, and IAM rules become part of your application whether you meant them to or not. You cannot easily rewrite it or run it somewhere else. I saw a recent post (Why we’re moving Wire off Cloudflare Durable Objects) from Wire, which ran every container on Cloudflare Durable Objects since day one. They wrote why they rebuilt their own data plane instead of staying, which included extra network hops on the hottest path, drift of state, separation of compute from data, rigid placement policies and lack of self-hosting. None of these are reliability complaints, instead they’re architectural ceilings baked into a runtime you don’t own. And the pattern generalizes past Cloudflare:

• AWS Lambda: SAM and LocalStack approximate the runtime locally, but diverge on execution environment, IAM, and VPC behavior.
• Azure Durable Functions: Azurite emulates the storage layer, but the replay-based orchestration engine behaves differently under real concurrent load than it does in the emulator.
• GCP Cloud Functions: the Functions Framework runs locally, but Eventarc, Pub/Sub push, and Cloud Run triggers all need live GCP resources.
• Cloudflare Workers/DO/Agents: wrangler dev simulates KV with SQLite and alarms with in-process timers, but never replicates the distributed routing that decides which data center actually holds your object.

Every one of these runtimes gives you a great abstraction and takes your operational sovereignty in exchange. This post shows how to keep the abstraction like stateful actors, durable storage, alarms, WASM sandboxing, LLM calls, observability, an event bus, webhooks while running it on infrastructure you control with an open-source framework called PlexSpaces.

PlexSpaces

PlexSpaces is an open-source, polyglot actor framework that gives every actor durable KV storage and alarms, routes messages between actors on one node or across a gRPC mesh, and exposes a host API to Go, TypeScript, Python, and Rust. You write an actor once; the same WASM module deploys to your laptop, a Docker container, Kubernetes, bare metal, or several clouds at once, unchanged. The mental model sits close enough to Cloudflare Durable Objects that migrating existing DO code is mostly mechanical. The key difference: there’s no simulated version to diverge from production, because the production runtime is the development runtime.

Part I: Durable Objects

Cloudflare publishes a short list of rules for writing correct Durable Objects, which are easily mapped to a PlexSpaces constraint, and for most of them the mapping is tighter:

The one meaningful difference: Cloudflare guarantees globally unique placement. PlexSpaces virtual actors are unique per node or per cluster when pinned with @* (any node) or @nodeId (specific node). PlexSpaces provides an object-registry for managing cross-cluster deduplication.

Durable KV Storage

Cloudflare’s ctx.storage gives you a transactionally consistent store scoped to one object:

// Cloudflare DO — JavaScript
const count = await this.ctx.storage.get("count") ?? 0;
await this.ctx.storage.put("count", count + 1);

// Batch operations (storage.get([keys]) / storage.put(map))
const [history, meta] = await this.ctx.storage.get(["room:history", "room:meta"]);
await this.ctx.storage.put(new Map([["room:history", data], ["room:meta", index]]));

PlexSpaces gives you the same guarantee through host.kv. Because each actor processes one message at a time, a plain read-modify-write is safe without extra locking. The migrating_cloudflare_workers TypeScript example restores room history on onInit() using batch KV similar to blockConcurrencyWhile:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
protected override onInit(config: Record<string, unknown>): void {
    this.state.room_id = String(config.actor_id ?? "");

    // blockConcurrencyWhile() equivalent — batch-fetch persisted state before first message
    const keys = [
        "room:" + this.state.room_id + ":history",
        "room:" + this.state.room_id + ":meta",
    ];
    const values = host.kv.multiGet(keys);   // like DO storage.get(["k1","k2"])
    const [historyRaw] = values;
    if (historyRaw) {
        const msgs = JSON.parse(historyRaw);
        if (Array.isArray(msgs)) {
            this.state.messages = msgs;
            this.state.message_seq = msgs[msgs.length - 1]?.seq ?? 0;
        }
    }
}

private persistHistory(): void {
    // Batch write — like DO storage.put(new Map([["k1",v1],["k2",v2]]))
    host.kv.multiPut({
        ["room:" + this.state.room_id + ":history"]: JSON.stringify(this.state.messages),
        ["room:" + this.state.room_id + ":meta"]: JSON.stringify({
            message_seq: this.state.message_seq,
            last_updated: host.nowMs(),
        }),
    });
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
def _load_history(self) -> None:
    room_id = self._room_id()
    index_raw = host.kv.get(f"room:{room_id}:seq_index")
    if index_raw:
        seqs = json.loads(index_raw)
        keys = [f"room:{room_id}:msg:{seq}" for seq in seqs]
        values = host.kv.multi_get(keys)   # DO storage.get([k1,k2,...]) equivalent
        self.messages = [json.loads(v) for v in values if v]
        if self.messages:
            self.msg_seq = self.messages[-1]["seq"]

def _persist_history(self) -> None:
    room_id = self._room_id()
    entries = {}
    seqs = []
    for msg in self.messages:
        entries[f"room:{room_id}:msg:{msg['seq']}"] = json.dumps(msg)
        seqs.append(msg["seq"])
    entries[f"room:{room_id}:seq_index"] = json.dumps(seqs)
    host.kv.multi_put(entries)   # DO storage.put({k:v,...}) equivalent

PlexSpaces also ships atomic operations that Cloudflare leaves you to build yourself with a coordinator object:

// PlexSpaces TypeScript — from RateLimiterActor in guild_chat_actor.ts
// Atomic distributed counter — equivalent to Cloudflare KV atomic increment
const distributedCount = host.kv.increment(windowKey, 1);

// Compare-and-swap — equivalent to DO transactional storage read-modify-write
const applied = await host.kvCas("lock_key", expectedValue, newValue);

// KV with TTL — equivalent to DO storage.put with metadata expiration
await host.kvPutWithTtl("session_token", token, 3600);

The Python RateLimiterActor shows both in context:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("check")
def check(self, user_id: str = "") -> dict:
    # Atomic increment — survives actor restarts, equivalent to Cloudflare KV atomic
    host.kv.increment(f"rate:{user_id}:total", 1)

    # CAS — idempotent token slot reservation, like DO transactional put
    cas_key = f"rate:{user_id}:window"
    current_val = host.kv.get(cas_key) or ""
    host.kv.cas(cas_key, current_val, str(host.now_ms()))

    # Token bucket logic operates on in-memory state (safe: single-threaded actor)
    allowed = self.buckets[user_id]["tokens"] > 0
    if allowed:
        self.buckets[user_id]["tokens"] -= 1
    return {"allowed": allowed, "remaining": self.buckets[user_id]["tokens"]}

Durable Alarms

A DO schedules one future callback that survives node restarts:

// Cloudflare DO
await this.ctx.storage.setAlarm(Date.now() + 10_000);

async alarm() {
    const count = await this.ctx.storage.get("count");
    await this.ctx.storage.delete("count");
    console.log(`Processing ${count} batched requests`);
}

PlexSpaces maps this directly through ReminderFacet, which persists the alarm to durable storage and re-queues it automatically after a restart. The AlarmDemoActor in the guild-chat example demonstrates the full lifecycle like set, query, fire, cancel:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
onEnqueue(_payload: Record<string, unknown>): Record<string, unknown> {
    this.state.queued++;
    if (this.state.queued === 1) {
        // First item — equivalent to: await this.state.storage.setAlarm(Date.now() + 30_000)
        host.alarm.set(host.nowMs() + 30_000);
    }
    return { status: "ok", queued: this.state.queued };
}

// Fires when the scheduled timestamp is reached
// Equivalent to Cloudflare DO: async alarm() { ... }
on__alarm__(_payload: Record<string, unknown>): Record<string, unknown> {
    const processed = this.state.queued;
    this.state.processed += processed;
    this.state.queued = 0;
    this.state.total_alarm_fires++;
    return { status: "ok", processed };
}

onStatus(_payload: Record<string, unknown>): Record<string, unknown> {
    const alarmAt = host.alarm.get();   // DO: await this.state.storage.getAlarm()
    return { ...this.state, alarm_at: alarmAt, alarm_set: alarmAt > 0 };
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("start")
def start(self, delay_ms: int = 30000) -> dict:
    # Equivalent to: this.state.storage.setAlarm(Date.now() + delay_ms)
    host.alarm.set(host.now_ms() + delay_ms)
    return {"status": "ok", "fire_at_ms": host.now_ms() + delay_ms}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Equivalent to Cloudflare DO: async alarm() { ... }
    processed = len(self.pending_requests)
    self.total_processed += processed
    self.pending_requests = []
    return {"status": "ok", "processed": processed}

@handler("cancel")
def cancel(self) -> dict:
    host.alarm.delete()   # DO: this.state.storage.deleteAlarm()
    return {"status": "ok", "action": "alarm_cancelled"}

Get-or-Create (Virtual Actors)

Cloudflare’s core abstraction is the globally unique object that appears on first access, at the cost of a binding declared in wrangler.toml:

// Cloudflare DO — needs a binding in wrangler.toml
const id = env.CHAT_ROOM.idFromName(roomId);
const room = env.CHAT_ROOM.get(id);
await room.fetch("/send", { method: "POST", body: JSON.stringify(msg) });

PlexSpaces virtual actors give you the same behavior with no binding file. The actor ID ({name}//{actorType}::{namespace}@*) encodes both the shard key and the actor class, and the @* suffix lets the runtime place it on whichever node is best; pin it with @node-id for data locality:

// PlexSpaces TypeScript
import { getActorRef } from "@plexspaces/sdk";
const room = getActorRef("ChatRoomActor", roomId, "default");
const reply = await room.ask("send_message", { user_id: userId, content: msg });

# PlexSpaces Python
from plexspaces.host import get_actor_ref
room = get_actor_ref("ChatRoomActor", room_id, "default")
reply = room.ask("send_message", {"user_id": user_id, "content": msg}, timeout_ms=5000)

The actor spins up on its first message, whether or not it existed before.

Listing Actors by Namespace

Cloudflare exposes a Durable Objects namespace list API for management and observability. PlexSpaces has a direct equivalent defined in its proto-first design. The ListActors RPC in actor_runtime.proto accepts namespace, actor_type, state, and node_id filters and returns paginated results:

// proto/plexspaces/v1/actors/actor_runtime.proto
message ListActorsRequest {
    string actor_type = 3;
    ActorState state  = 4;
    string node_id    = 5;
    // Namespace for tenant isolation — only actors in this namespace are returned
    string namespace  = 6;
    PageRequest page_request = 2;
}

message ListActorsResponse {
    repeated Actor actors        = 2;
    PageResponse page_response   = 3;
}

From a PlexSpaces client, listing all active ChatRoomActor instances in the default namespace:

# HTTP API (equivalent to Cloudflare's list-objects endpoint)
curl "http://localhost:8080/api/v1/actors/default/ChatRoomActor?state=active"

// Go SDK
actors, err := client.ListActors(ctx, &ListActorsRequest{
    ActorType: "ChatRoomActor",
    Namespace: "default",
    State:     ActorStateActive,
})

Cloudflare limits listing to metadata (ID, location, storage size). PlexSpaces returns full Actor records including state, facets, node assignment, resource usage, and tenant/namespace tags.

WebSocket Handling

Cloudflare’s Model

Cloudflare gives the Durable Object two WebSocket modes. In the standard model, the object holds the socket directly:

// Cloudflare DO — standard WebSocket
async fetch(request) {
    const [client, server] = Object.values(new WebSocketPair());
    this.ctx.acceptWebSocket(server);
    return new Response(null, { status: 101, webSocket: client });
}

async webSocketMessage(ws, message) {
    for (const peer of this.ctx.getWebSockets()) {
        peer.send(`broadcast: ${message}`);
    }
}

async webSocketClose(ws, code) {
    ws.close(code, "connection closed");
}

The WebSocket Hibernation API (state.acceptWebSocket / getWebSockets()) is Cloudflare’s optimization for objects that hold many sockets but are mostly idle: the object is evicted when no message is being processed, and WebSocket state is restored from durable storage on the next message.

PlexSpaces: A Cleaner Split

PlexSpaces takes a different approach: room state and connection state are in separate actors. A ChatRoomActor holds only membership and message history; each browser connection is a thin-node client registered under its own actor ID. When the room fans out, host.send(actorId, "chat_message", event) routes each delivery through the WsActorTransportClient to the right WebSocket session like the room never holds a socket handle.

This is the same split Discord uses internally in its Elixir stack, where session processes are separate from guild processes. Here’s the real onSend handler from examples/typescript/apps/ws_chat_room/:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onSend(payload: SendPayload): unknown {
    const senderUsername = this.state.members[payload.sender_actor_id] ?? payload.sender_actor_id;
    const ts = host.nowMs();
    this.state.history.push({
        senderActorId: payload.sender_actor_id,
        sender: senderUsername,
        text: payload.text,
        ts,
    });
    if (this.state.history.length > MAX_HISTORY) {
        this.state.history = this.state.history.slice(-MAX_HISTORY);
    }

    const event = {
        sender: payload.sender_actor_id,
        sender_username: senderUsername,
        text: payload.text,
        room_id: this.state.roomId,
        ts,
    };
    // Fan out to all members including sender (delivery confirmation)
    // host.send() routes each tell through WsActorTransportClient ? WsRegistry ? thin-node WS session
    const memberIds = Object.keys(this.state.members);
    for (const actorId of memberIds) {
        host.send(actorId, "chat_message", event);
    }
    return { success: true, members_notified: memberIds.length };
}

A companion PresenceActor in the same file tracks online/offline state using a durable reminder (host.sendAfter) to mark a user offline after 55 seconds of silence without external cron job involved:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onOnline(_payload: OnlinePayload): unknown {
    this.state.online = true;
    this.state.last_seen = host.nowMs();
    host.kv.putJson(`presence:${this.state.userId}`, { online: true, last_seen: this.state.last_seen });
    host.sendAfter(60_000, "timeout_check", {});   // durable reminder, survives restart
    return { success: true, online: true };
}

onTimeout_check(): unknown {
    const idleSince = host.nowMs() - this.state.last_seen;
    if (idleSince > 55_000) {
        this.state.online = false;
        host.kv.putJson(`presence:${this.state.userId}`, {
            online: false, last_seen: this.state.last_seen,
        });
    }
    return { checked: true, idle_ms: idleSince };
}

Gap Analysis: WebSocket Hibernation

The PlexSpaces model costs more code upfront (session actor + room actor) and pays you back with independent scaling: fan-out scales with the number of receivers, not with the room actor’s memory; a crashed session doesn’t lock the room; reconnect logic is client-side only.

Fan-Out: DO’s Missing host.send() Primitive

In Cloudflare, cross-object fan-out means individual fetch() calls or Queues, which are not lean as a fire-and-forget tell. PlexSpaces’s host.send() is a fire-and-forget message routed through the actor mesh, no HTTP overhead, with in-process delivery for co-located actors. The Python guild-chat send_message handler shows this clearly:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("send_message")
def send_message(self, user_id: str = "", content: str = "") -> dict:
    msg = self._add_message(user_id, content, host.now_ms())

    # Fan-out: fire-and-forget to each member actor
    # Mirrors Discord's Manifold pattern for distributed fan-out
    # In Cloudflare DO, this would be individual fetch() calls — expensive
    fan_out_count = 0
    for member_id in list(self.members.keys()):
        if member_id != user_id:
            host.send(member_id, "receive_message", {
                "room_id": self._room_id(),
                "seq": msg["seq"],
                "from": user_id,
                "content": content,
            })
            fan_out_count += 1

    self._persist_history()   # batch multiPut — one KV call for the whole room
    return {"status": "ok", "seq": msg["seq"], "fan_out": fan_out_count}

Part II: Cloudflare Agents SDK

Cloudflare’s Agents SDK builds stateful AI agents on top of Durable Objects. PlexSpaces covers the same patterns; some are direct translations and a few require a different shape.

Conversation State and Memory

Cloudflare stores conversation history in the DO’s storage. PlexSpaces does the same through host.kv, with the same durability guarantee: the ChatAgentActor in examples/python/apps/chat_agent/ stores history under a well-known key and restores it across activations:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
@handler("chat")
def chat(self, message: str = "") -> dict:
    # Load history — equivalent to: await this.storage.get('history')
    history = host.kv.get_json("history") or []

    history.append({"role": "user", "content": message, "timestamp": host.now_ms()})

    assistant_reply = self._call_llm(history)

    history.append({"role": "assistant", "content": assistant_reply, "timestamp": host.now_ms()})

    # Persist — equivalent to: await this.storage.put('history', history)
    host.kv.put_json("history", history)
    self.total_messages += 1

    # Schedule summarization alarm once history is long enough
    if len(history) > _ALARM_THRESHOLD and host.alarm.get() == 0:
        host.alarm.set(host.now_ms() + _ALARM_DELAY_MS)

    return {"status": "ok", "reply": assistant_reply, "history_length": len(history)}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm()
    history = host.kv.get_json("history") or []
    summary = self._call_llm([{
        "role": "user",
        "content": f"Summarize this conversation (2-3 sentences): {json.dumps(history)}"
    }])
    host.kv.put("summary", summary)
    host.kv.delete("history")   # clear after summarizing
    return {"status": "ok", "action": "summarized", "messages_summarized": len(history)}

For long-term cross-session memory, store summaries under a user-scoped key (memory:{user_id}) and inject them into the next conversation’s system prompt similar to Cloudflare’s getMemory/setMemory implementation.

Calling LLMs

Cloudflare’s Agents SDK routes every call through env.AI, Cloudflare’s own inference gateway, locked to providers they support:

// Cloudflare Agents SDK — locked to Cloudflare's AI gateway
const response = await this.env.AI.run("@cf/meta/llama-3-8b-instruct", { messages });

PlexSpaces actors call any provider through a named HTTP service link resolved at deploy time, so the actor code never mentions a specific vendor:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
def _call_llm(self, messages):
    http = ServiceHttpClient("llm-link")
    body = {
        "model": "claude-3-5-haiku-20241022",
        "max_tokens": 1024,
        "messages": [{"role": m["role"], "content": m["content"]} for m in messages],
    }
    resp = http.post("/v1/messages", body)
    # Parse Anthropic response
    if isinstance(resp, dict):
        content = resp.get("content", [])
        if content and isinstance(content, list):
            return content[0].get("text", "")
    return "[LLM unavailable]"

app-config.toml points the link at Ollama locally, Anthropic or OpenAI in production, or an internal AI gateway in a regulated environment:

# Local development — Ollama
[[service_links]]
name = "llm-link"
url  = "http://localhost:11434"

# Production — swap without touching actor code
[[service_links]]
name = "llm-link"
url  = "https://api.anthropic.com"
headers = { "x-api-key" = "${ANTHROPIC_API_KEY}" }

TypeScript actors use the same pattern:

// PlexSpaces TypeScript — examples/typescript/apps/chat_agent/
const resp = host.httpClient("llm-link").post("/v1/messages", {
    model: "claude-3-5-haiku-20241022",
    max_tokens: 1024,
    messages,
});

Durable Workflows

Cloudflare Agents SDK ships workflow primitives that checkpoint multi-step sequences. PlexSpaces WorkflowActor gives you the same like Run, Signal, and Query RPCs map to start, inject external events, and inspect state. For example, the payment workflow in examples/go/apps/migrating_cadence/payment_workflow.go shows the shape:

// PlexSpaces Go — examples/go/apps/migrating_cadence/payment_workflow.go
// PaymentWorkflow implements WorkflowActor for idempotent payment processing.
// Steps: validate ? authorize (with retry) ? capture ? settle.
// Signals: refund, cancel.  Queries: status, payment_id.
type PaymentWorkflow struct {
    plexspaces.BaseActor
    PaymentID       string        `json:"payment_id"`
    Status          string        `json:"status"` // pending ? validated ? authorized ? captured ? settled
    Steps           []PaymentStep `json:"steps"`
    RefundRequested bool          `json:"refund_requested"`
}

func (p *PaymentWorkflow) Run(payloadJSON string) string {
    // Each step checkpoints via getState/setState before proceeding.
    // If the node crashes mid-run, the workflow resumes from the last checkpoint.
    p.Status = "validating"
    if err := p.validatePayment(); err != nil {
        p.Status = "failed"
        return marshal(map[string]any{"error": err.Error()})
    }
    p.addStep("validate")

    // Authorize with retries (idempotency key prevents double-charge)
    p.Status = "authorizing"
    for attempt := 0; attempt < 3; attempt++ {
        if err := p.authorizePayment(); err == nil {
            break
        }
    }
    p.addStep("authorize")
    // ... capture, settle
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

func (p *PaymentWorkflow) Signal(name, _ string) {
    switch name {
    case "refund":
        p.RefundRequested = true
    case "cancel":
        p.Status = "cancelled"
    }
}

func (p *PaymentWorkflow) Query(name, _ string) string {
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

For multi-agent orchestration, OrchestratorActor in examples/go/apps/miniclaw/ decomposes a task into sub-tasks, delegates each to a worker agent discovered via process group, and aggregates results through TupleSpace:

// PlexSpaces Go — examples/go/apps/miniclaw/orchestrator.go
func (o *OrchestratorActor) Run(payloadJSON string) string {
    task := stringVal(parsePayload(payloadJSON), "task", "")
    taskID := fmt.Sprintf("orch-%d", host.NowMs())

    o.Status = "running"
    o.TaskID = taskID

    // Discover available agents via process group membership
    agentID, err := pgFirst("svc:agent")
    if err != nil {
        return marshal(map[string]any{"error": "no agents in svc:agent process group"})
    }

    // Decompose and delegate sub-tasks
    subTasks := decompose(task)
    for i, subTask := range subTasks {
        o.Progress = (i + 1) * 100 / len(subTasks)
        result, err := host.Ask(agentID, "chat", map[string]any{
            "message":    subTask,
            "session_id": fmt.Sprintf("orch-%s-%d", taskID, i),
        }, 30000)
        if err != nil {
            return marshal(map[string]any{"error": "sub-task failed: " + err.Error()})
        }
        // Store result in TupleSpace for aggregation
        host.TS().Write([]any{"orch_result", taskID, i, result})
    }

    o.Status = "completed"
    return marshal(map[string]any{"task_id": taskID, "status": "completed", "sub_tasks": len(subTasks)})
}

Human-in-the-Loop

Cloudflare’s human-in-the-loop pattern pauses an agent on a high-stakes action and waits for external approval before resuming. PlexSpaces implements this natively through a GenFSM actor, e.g., ApprovalGateActor in examples/python/apps/minipi/approval_gate.py:

# PlexSpaces Python — examples/python/apps/minipi/approval_gate.py
@fsm_actor(states=["idle", "awaiting_approval", "approved", "rejected"], initial="idle")
class ApprovalGateActor:
    """
    FSM states: idle ? awaiting_approval ? approved / rejected ? idle

    Key insight: the agent can wait for days. DurabilityFacet preserves all state
    durably — no polling, no timeouts burning tokens.
    """
    fsm_state: str = state(default="idle")
    pending_request: dict = state(default_factory=dict)
    pending_agent_id: str = state(default="")

    @handler("request_approval")
    def request_approval(self, agent_id: str = "", action: str = "", context: dict = None) -> dict:
        """An agent requests human approval for a high-stakes action."""
        if self.fsm_state != "idle":
            return {"status": "busy", "current_agent": self.pending_agent_id}

        self.fsm_state = "awaiting_approval"
        self.pending_agent_id = agent_id
        self.pending_request = {"action": action, "context": context or {}, "requested_at_ms": host.now_ms()}

        # Store request for external review (dashboard, Slack notification, etc.)
        host.kv.put(f"approval_request:{self.actor_id}", json.dumps(self.pending_request))

        return {"status": "pending", "gate_id": self.actor_id}

    @handler("approve")
    def approve(self, approver: str = "", comment: str = "") -> dict:
        """Human approves — signals the suspended agent to resume."""
        agent_id = self.pending_agent_id
        self.fsm_state = "approved"
        self.decision_history.append({
            "action": self.pending_request.get("action"),
            "decision": "approved",
            "approver": approver,
            "decided_at_ms": host.now_ms(),
        })

        # Signal the waiting agent to resume with the decision
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "approved",
            "approver": approver,
            "comment": comment,
        })

        self.fsm_state = "idle"
        self.pending_agent_id = ""
        return {"status": "approved", "agent_id": agent_id}

    @handler("reject")
    def reject(self, approver: str = "", reason: str = "") -> dict:
        """Human rejects — signals the agent with the rejection."""
        agent_id = self.pending_agent_id
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "rejected",
            "approver": approver,
            "reason": reason,
        })
        self.fsm_state = "idle"
        return {"status": "rejected", "agent_id": agent_id}

The agent on the other side calls host.ask("approval_gate", "request_approval", {...}) then processes the workflow_signal:resume message when it arrives. Because state is checkpointed durably, the agent can wait hours or days with no polling loop and no timeout burning tokens.

Long-Running Agents

Cloudflare’s long-running agent pattern uses alarms to wake a dormant agent on a schedule. PlexSpaces handles this identically, e.g., any actor with the ReminderFacet can schedule work arbitrarily far in the future. The ChatAgentActor summarization alarm is one example; for a true long-running loop:

@actor
class ChatAgentActor:
    """Minimal chat agent: conversation in KV, LLM via service link, alarm for summarization."""

    actor_id: str = state(default="")
    total_messages: int = state(default=0)
    total_summarizations: int = state(default=0)

    @init_handler
    def on_init(self, config: dict) -> None:
        self.actor_id = config.get("actor_id", "")
        host.info(f"ChatAgentActor init actor_id={self.actor_id}")


    @handler("__alarm__")
    def on_alarm(self) -> dict:
        """Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm().

        Summarizes conversation history and stores a summary KV key,
        then clears history.
        """
        host.info("ChatAgentActor: alarm fired — summarizing history")

        history = host.kv.get_json("history") or []
        if not history:
            return {"status": "ok", "action": "no_history_to_summarize"}

        # Summarize via LLM
        summary_prompt = (
            f"Summarize this conversation concisely (2-3 sentences): "
            f"{json.dumps([{'role': m['role'], 'content': m['content']} for m in history])}"
        )
        summary = self._call_llm([{"role": "user", "content": summary_prompt}])

        # Persist summary, clear history — equivalent to: storage.put('summary', s); storage.delete('history')
        host.kv.put("summary", summary)
        host.kv.delete("history")

        self.total_summarizations += 1

        host.info(f"ChatAgentActor: summarized {len(history)} messages")
        return {
            "status": "ok",
            "action": "summarized",
            "messages_summarized": len(history),
        }

What PlexSpaces adds beyond Cloudflare’s model: the actor can also be signalled externally at any time via host.send(actorId, "wake_early", {...}) — you’re not limited to the alarm cadence.

Agents Feature Map

Part III: Distributed Computation

Cloudflare Workers and Lambda optimize for millisecond, latency-sensitive request handling. PlexSpaces handles a second problem class: large-scale computation across resources that come and go.

ShardGroups: Scatter-Gather Without the Plumbing

For data-parallel and ML-style workloads, PlexSpaces exposes MPI-style collectives directly through the host API:

// Create a pool of 20 workers, hash-partitioned
let pool_id = client.create_worker_pool(
    "worker-pool-1", "worker", 20,
    PartitionStrategy::Hash, HashMap::new(),
).await?;

// Bulk update: 10,000 messages routed to the right shard by key
client.parallel_update(&pool_id, updates, ConsistencyLevel::Eventual, false).await?;

// Parallel map: query every shard simultaneously
let results = client.parallel_map(&pool_id, json!({ "action": "get_total_count" })).await?;

// Parallel reduce: aggregate stats across all shards
let stats = client.parallel_reduce(
    &pool_id, json!({ "action": "stats" }),
    ShardGroupAggregationStrategy::Concat, 20,
).await?;

Idle Browsers as Compute Nodes

The mersenne_prime TypeScript example runs a GIMPS-style distributed primality search: browser tabs connect as thin WebSocket clients, receive worker JavaScript from a CodeServerActor, and run Lucas-Lehmer tests inside Web Workers. A CoordinatorActor running as WASM on the server assigns exponents, tracks per-worker CPU cores, and dispatches the next candidate immediately on each result:

// PlexSpaces TypeScript — examples/typescript/apps/mersenne_prime/mersenne_actor.ts
onResult(payload: ResultPayload): unknown {
    const item = this.state.work[String(payload?.p)];
    item.status = 'done';
    item.is_prime = Boolean(payload.is_prime);
    item.duration_ms = payload.duration_ms ?? 0;

    // Record every completed candidate in TupleSpace and bump a Prometheus counter
    host.ts.write(['result', String(payload.p), item.is_prime ? 'true' : 'false',
        String(item.duration_ms), payload.actor_id ?? 'unknown']);
    host.incrCounter('ts-mersenne-prime', item.is_prime ? 'primes_found' : 'composites_found');

    // Immediately hand the same worker the next pending candidate
    const next = this._nextPending(this.state.workers[payload.actor_id!]?.cpu_cores ?? 1);
    if (next) {
        next.status = 'assigned';
        host.send(payload.actor_id!, 'assign_work', { p: next.p, done: false });
    }
    return { ok: true };
}

Open the same URL in ten browser tabs and you have ten compute shards, coordinated by one WASM actor, with zero additional infrastructure.

Comprehensive Feature Map

A couple of things need real rework, not a find-and-replace:

• WebSocket architecture. If your DO holds sockets directly today, plan for a thin node that hosts actors.
• Bindings vs children. Cloudflare bindings live in wrangler.toml and show up as env properties. PlexSpaces declares the same relationships as supervision children in app-config.toml.

Running Everywhere

The whole point is that development and production run the identical binary:

# Local development — exact production behavior
plexspaces-node start --config app-config.toml

# Docker — same binary, same config
docker run -v $(pwd):/app plexspaces/node start --config /app/app-config.toml

# Kubernetes — same config via Helm
helm install my-app plexspaces/app-chart \
  --set config.path=app-config.toml \
  --set persistence.storage=postgres

# Multi-cloud — nodes in GCP + AWS joined over a gRPC mesh; actors route transparently

An alarm that fires in production runs through the exact code path you tested on your laptop. There’s no gap to debug between wrangler dev and prod, because there’s only one runtime.

Working Examples

Every example referenced above is in the repository (github.com/bhatti/PlexSpaces):

• Guild chat (DO migration pattern): examples/{go,typescript,python,rust}/apps/migrating_cloudflare_workers/: ChatRoomActor with member fan-out, a token-bucket RateLimiterActor backed by host.kv.increment and host.kv.cas, and an AlarmDemoActor mirroring DO’s full alarm lifecycle
• WebSocket chat room: examples/{typescript,python,go,rust}/apps/ws_chat_room/: ChatRoomActor plus a PresenceActor that uses a durable reminder to detect idle disconnects
• AI chat agent (Cloudflare Agents SDK pattern): examples/{go,typescript,python,rust}/apps/chat_agent/: conversation history in KV, LLM calls through service link, durable summarization alarm
• Human-in-the-loop approval gate: examples/python/apps/minipi/approval_gate.py: FSM-based approval workflow, workflow_signal:resume handoff, state durable across multi-day waits
• Multi-agent orchestration: examples/go/apps/miniclaw/: OrchestratorActor decomposing tasks, delegating via process groups, aggregating through TupleSpace
• Durable payment workflow: examples/go/apps/migrating_cadence/payment_workflow.go: WorkflowActor with Run/Signal/Query, idempotent retry, refund and cancel signals
• Mersenne prime search (browser compute): examples/typescript/apps/mersenne_prime/: browser tabs as Lucas-Lehmer worker shards, coordinated by a WASM CoordinatorActor
• wasmCloud migration: examples/python/apps/migrating_wasmcloud/session_store.py: capability-based session store showing host.kv.list, inter-actor ask, and timer-driven cleanup
• Every abstraction in one place: examples/{go,typescript,python,rust}/apps/abstractions/: durable virtual-actor reactivation, workflow run/signal/query, process-group event delivery, timers, reminders, KV, tuple space, and blob storage

Conclusion

The Cloudflare model is the right model: stateful actors, durable storage, alarms, WebSocket session management, LLM calls baked in. Wire’s post makes the same point from the other direction as they’re not leaving because the model is wrong, they’re leaving because specific architectural ceilings came due at their scale. PlexSpaces keeps the model and removes the ceiling: you write the same actors, get the same alarms and durable storage and observability, and the binary that runs on your laptop is the exact binary that runs in production, on any cloud, on-prem, or across all of them at once. The migration from Cloudflare DO code is mostly mechanical. What you get back is the ability to run anywhere, own your infrastructure and never again debug a divergence between wrangler dev and prod.

GitHub: github.com/bhatti/PlexSpaces

Previous posts in this series:

• Building PlexSpaces: Decades of Distributed Systems Distilled
• Polyglot WebAssembly: Four Languages, One Runtime
• 20+ Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
• Building Mini OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
• Building a Self-Improving AI Agent with Durable Actors: MiniHermes
• Building an Agent Harness and Eval Pipeline with Durable Actors

I have seen some systems never crash, they start cleanly, swallow every error, and keep running no matter what goes wrong. In my experience, they are also the hardest systems to debug, the most dangerous to operate, and the most expensive to maintain. I worked on a similar legacy system for distributed data platform that routed events between hundreds of thousands of nodes. It had zero unhandled exceptions in production. It also had silent authentication failures, invisible data loss, and configuration divergence that took days to diagnose. This is a follow-up to my earlier posts on building an observability platform in Rust, why DRY becomes a liability, and making bad state impossible with ADTs. Those posts were about the type system but this one is about a habit of mind that no type system fixes on its own: the instinct to catch every error with some fallback or default behavior.

The culprit in that system was never a lack of error handling. It was error handling, applied in the wrong places, for the wrong reasons. I call it defensive programming as a religion: every function protects itself against every possible invalid input by inventing a fallback. Missing config? Use a default. Secret unavailable? Generate a random one. Database write failed? Log it and move on. The result is a system that looks healthy on every dashboard while quietly corrupting its own state underneath. In this post, I will walk through the patterns I found, why each one causes more damage than the crash, and what an alternative looks like instead. The whole argument rests on one idea: Every fallback creates a new source of truth, and two sources of truth always drift apart.

I. Why a Fallback Is Worse Than It Looks

It’s tempting to think of a fallback as just “hiding an error.” It’s worse than that. When a function invents a value because the real one is missing, that invented value doesn’t stay hidden, instead it becomes a fact in the system. From that moment on, the system is carrying two truths: the one that should exist but doesn’t, and the one that got made up and does. These two truths never stay in sync and when they drift apart, the failure almost never shows up where the fallback happened. Instead, it shows up somewhere else entirely, in a component with no obvious connection to the code that invented the value. For example, you’ll spend hours in the authentication layer before realizing the signing key was randomly generated at startup by a config migration function three layers away. To be clear about what I mean by “fallback,” I am not talking about:

• Validating input at system boundaries: checking what a user typed, sanitizing data from outside. Correct and necessary.
• Graceful degradation with an explicit signal: returning a typed Degraded state that the caller can see and react to.
• Retrying transient I/O failures with backoff. Standard practice.

I’m talking about code that silently invents state when the real state is missing, and then carries on as if nothing happened. Three things make a fallback harmful:

1. It hides the root cause. The missing value was the bug. The fallback makes it disappear.
2. It persists the invented value. Once it’s written to disk or sent over the wire, every future operation has to succeed against a value that was never correct in the first place.
3. It relocates the symptom. The failure surfaces hours later, in a different component, in a different log file, with no visible thread connecting it back to the startup code that invented the wrong value.

I am not advocating “crash on every error.” It just means: a function that requires X must fail when X is absent and it must never invent X.

II. Postel’s Law

There’s a reason smart engineers build these fallback-heavy systems: they’re following a respected principle: “be conservative in what you send, be liberal in what you accept.” that Jon Postel wrote as guidance for TCP implementations. That advice made a lot of sense in its original context. But this principle leaked out of the protocol layer and turned into a general design philosophy. Engineers started applying “be liberal in what you accept” to function signatures, config loading, and communication between services inside a system they fully control. A function that takes string | undefined and silently substitutes a random value gets called “being robust.” A startup sequence that swallows errors and keeps going gets called “being tolerant.”

Inside your own system, that assumption doesn’t hold. For example, you can fix the sender because you own the caller. But when you own the migration script that’s supposed to write the auth token and you “liberally accept” a missing auth token by inventing a random one instead, you’re not enabling interoperability with an outside party instead you’re hiding a bug in code you wrote. This misapplication creates a ratchet effect. Every “liberal” receiver makes it harder to notice problems at the source. If every function tolerates missing input, the function that’s supposed to supply that input has no pressure to get it right. People also tend to forget that Postel’s Law has a second half: “be conservative in what you send.”

The corrected version for internal systems is this: be strict with components you control, and liberal only at the boundaries where you genuinely can’t fix the sender like external APIs, user input, third-party integrations. Inside your own codebase, strictness isn’t fragility. A function that rejects invalid input tells you exactly where the bug lives.

III. Inventing Values: The Most Dangerous Pattern

This is the category that caused the most damage and I have seen countless bugs due to this anti-pattern. For example, the code generates a random value, assigns it to a security-critical field, and moves on as if that field were properly populated. The invented value becomes a durable fact somewhere and it’s always wrong.

The Archetype: Inventing a Secret

A function runs at startup and writes a signing secret into every worker group’s configuration. Workers and the coordinator use this secret to authenticate each other. If two groups end up with different values, every authentication attempt between them fails silently, showing up as delivery failures instead of auth failures.

// The bug: if authToken is absent, invent a UUID and persist it
const authToken = settings.distributed?.master?.authToken;
const plaintext = authToken != null && authToken.length > 0
  ? authToken
  : randomUUID();  // <-- this line breaks authentication across the cluster

When authToken is missing from the merged settings, which is a perfectly valid state on a fresh install but this function generates a randomUUID() and writes it as the signing secret for every group it touches. Each call produces a different UUID. Meanwhile the token store writes yet another value through a completely different code path. The function reports success for every group it writes to. The symptom 401 errors between nodes shows up minutes or hours later, in worker logs, pointing investigators toward the wrong layer entirely. This is the archetype of the whole problem: if the required input is missing, invent something plausible-looking and keep going. The fix is three lines:

const authToken = settings.distributed?.master?.authToken;
if (authToken == null || authToken.length === 0) {
  logger.warn('authToken absent from settings; cannot write signing secrets');
  return;  // do not proceed; do not invent
}

The “Disabled” Sentinel That Looks Just Like a Real Key

A secret provider has a three-source fallback chain: encrypted store, config file, random bytes. That last fallback is supposed to act as a “poison key” that never validates:

async getSecret(): Promise<string> {
  // Source 1: encrypted store
  const fromStore = await secretsMgr.get(KEY_ID).catch(() => null);
  if (fromStore) return fromStore;

  // Source 2: config file (which may itself contain a well-known default!)
  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return fromFile;

  // Source 3: "disable" by returning random bytes — looks perfectly valid to the caller
  return random(16);
}

The caller gets back a plain string in all three cases. It has no way to tell “a real secret from the store” apart from “a well-known default from the config file” apart from “random bytes that will never work.” It signs a token with whatever string it received and sends it off. This is a type-system failure because the return type Promise<string> squashes three semantically different outcomes into one shape.

The explicit contract fixes this at the type level:

type SecretResult =
  | { kind: 'valid'; value: string; source: 'store' | 'file' }
  | { kind: 'unavailable'; reason: string };

async getSecret(): Promise<SecretResult> {
  const fromStore = await secretsMgr.get(KEY_ID).catch((err) => {
    logger.error('Secret store unavailable', { error: err });
    return null;
  });
  if (fromStore) return { kind: 'valid', value: fromStore, source: 'store' };

  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return { kind: 'valid', value: fromFile, source: 'file' };

  return { kind: 'unavailable', reason: 'no secret in store or config' };
}

Now a caller that receives unavailable marks itself as degraded. It doesn’t sign tokens and surfaces a health check failure instead.

The Token Renewal That Signs With Random Bytes

The token authenticator calls getSecret(), and if that fails, it falls back to random bytes anyway:

let keyStr = await this.secretProvider.getSecret().catch(() => undefined);
if (keyStr == null) {
  this.logger?.debug('Unable to generate a valid token. Disabling.');
  keyStr = random(16);  // this "signed" token will never be accepted by any peer
}
const token = jwt.sign(payload, keyStr);
this.cachedToken = token;  // cached and reused for every future request

The log message says “Disabling,” but nothing gets disabled. The code signs a token with a random key, caches it, and hands it out to every caller for the rest of the process’s life. Workers receive the token, fail to verify it, and log a 401, with nothing to suggest the root cause of the issue.

The explicit contract: if the secret is unavailable, don’t sign anything. Set this.cachedToken = null. Let callers check for null and surface a real health degradation.

The User ID That Regenerates on Every Call

const userId = existing?.username ?? crypto.randomUUID();

When existing is unexpectedly missing, every call generates a brand-new UUID. The user becomes a ghost: every request creates a fresh identity, invisible to deduplication, audit trails, and rate limiters.

The Config Placeholder That Silently Materializes

if (authToken?.token === 'REPLACE_ME') {
  authToken.token = uuidv4();  // silent replacement, no log of the value generated
}

This runs at startup and inside a database migration. If the startup write fails, the generated token is gone.

IV. Swallowed Errors

In one legacy codebase I worked had 656 instances of .catch(NOOP), an empty function attached to a promise rejection that turns any error into undefined and lets execution continue. Many of these were on cleanup paths, which is harmless. But a large number sat on critical data paths like durability, metrics transport, authentication state.

The Durability Guarantee That Wasn’t

A persistent queue exists for exactly one reason: to guarantee that events survive a destination outage. That’s the system’s durability promise.

// PQ flush — the durability guarantee
await this.flushBuffer().catch(NOOP);  // silently swallows disk I/O errors
await this.commit().catch(NOOP);        // silently swallows commit failures

If the flush fails like disk full, I/O error, permission denied, the buffered events are silently gone. The one component whose entire job is preventing data loss is itself a source of silent data loss.

The explicit contract: treat PQ flush errors as fatal to the ingest path. For example, if the flush fails, apply backpressure and pause ingest. Log at error level with the event count at risk and emit a pq_flush_failure metric. Now the operator gets to choose: fix the disk, add capacity, or consciously accept the loss.

Metrics That Vanish

void saasMetrics.sendPacket(packet).catch(NOOP);

Every metrics-send failure is silently swallowed.Dashboards go blank, and nobody knows why.

The Config Load That Treats Corruption as “Empty”

const groups = await conf.loadSystem('internal-groups').catch(() => ({}));

One line here quietly conflates two very different situations:

• “The file doesn’t exist yet”, which is normal on first boot –> return {}
• “The file is corrupt, or a parse error, or permission was denied”, which is a real configuration bug –> also return {}

Either way, the loop over groups never runs. The operator has no way to tell “healthy, no groups configured yet” apart from “broken, groups exist but couldn’t be read.”

Startup That Succeeds Despite Total Failure

export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  try {
    // ... write secrets to all groups ...
  } catch (err) {
    logger.error('failed to sync secrets', { reason: err });
    // swallowed — startup continues, caller receives no signal
  }
}

The function catches everything at the top and resolves successfully no matter what. The caller awaits it and gets no signal that anything went wrong. If the sync fails for every group, the coordinator still starts, workers still connect, and authentication fails across the board but startup “succeeded.”

V. Defensive Defaults

This next category is more subtle. Instead of inventing a random value, the system injects a well-known one like a default credential, a default address, which makes it impossible for downstream code to tell that anything is missing at all.

The Well-Known Default Credential

The shared authentication token has a configuration setting, and by default, the settings loader injects a well-known string whenever nothing is configured:

const { injectDefaultAuthToken = true } = options ?? {};
if (injectDefaultAuthToken && conf.distributed?.master?.authToken == null) {
  conf.distributed.master.authToken = DEFAULT_AUTH_TOKEN;  // 'default-secret'
}

Any caller of getSettings() receives a truthy string for authToken. Code that checks if (authToken) proceeds as if a real token exists. The check passes. Authentication moves forward, using a credential that’s sitting right there in the source cod and every deployment that forgot to override it. The default here is opt-out, not opt-in. Every new call site has to remember to disable the injection.

Workers That Connect to localhost on Config Failure

const server = this.conf.distributed.master
  || { host: 'localhost', port: 5555, authToken: DEFAULT_AUTH_TOKEN };

When a worker fails to load its coordinator address, it silently falls back to localhost:5555 with the default token. On a single-node dev box, this might accidentally work. On a production multi-node deployment, the worker ends up connecting to itself. The symptom looks like a connection timeout, not a config-load failure.

The explicit contract: if conf.distributed.master is missing, throw. A worker cannot function without a coordinator address, and there is no valid default for it. Failing here with a clear message like “coordinator address not configured; set MASTER_URL or add distributed.master to instance.yml“.

VI. Multiple Sources of Truth

This is the pattern that ties everything else together. Every fallback chain creates more than one source of truth and any source of truth that isn’t explicitly designated as the source will eventually drift from the others. In a distributed system, that drift shows up as the hardest class of bug like intermittent or state-dependent failures.

Two Functions, One Secret, Two Different Sources

The bug that inspired this post exists because two functions write the same signing secret to group configs, but read the plaintext from two different physical sources:

A migration populates the token store before syncAllGroupSecrets runs at boot, so the store is meant to be authoritative. But syncAllGroupSecrets predates the store’s existence and still reads straight from the config file. These two sources drift apart after a token rotation or some race condition.

The explicit contract: one source of truth. Both functions read from the store. If the store is unavailable, both fail instead of silent fallback to a secondary source.

The Three-Source Chain Is Three Sources of Truth

Source 1: Encrypted store (authoritative)
  ? (unavailable ? silently falls through)
Source 2: Config file (may hold a stale or default value)
  ? (absent ? silently falls through)
Source 3: Random bytes (structurally valid, semantically useless)

Each fallback quietly downgrades the security posture, and the caller gets back a plain string with no idea which source it came from.

Three branches go into the same sign() call. Only one of them should ever be allowed to reach it, which is the whole argument for making unavailable its own explicit type instead of letting all three collapse into a plain string.

The Cache Nobody Fully Trusts

The settings system keeps a cache for high-availability mode. Some callers pass skipCache: true to bypass it; others don’t, and there’s no documented rule for which is which. This gives the system two sources of truth for the same data: the cache and the disk. If the config file changes between startup and an API call, the API may serve stale data. The skipCache escape hatch is a symptom, not a fix. It means someone stopped trusting the cache’s invalidation and punched a hole through it instead of repairing the underlying mechanism.

The explicit contract: the cache invalidates on every write. Callers never need to know or care whether they’re reading from cache or disk. Remove skipCache as a public option entirely.

VII. Redundant Guards

When a function doesn’t trust its caller’s preconditions, it adds its own guard on top:

// Caller (server.ts):
if (isLeader && featureFlags.check('AUTH_TOKEN_MGMT')) {
  await syncGroupSecrets(conf);
}

// Callee (syncGroupSecrets):
export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  if (!isFreeTier() && !isRunningInSaaS()) return;  // guard 1
  if (!Product.isLeader(settings.distributed?.mode)) return;  // guard 2 (redundant!)
  if (!featureFlags.check('AUTH_TOKEN_MGMT')) return;  // guard 3 (redundant!)
  // ... actual work ...
}

This function lives in a directory named leader/ and is only ever called from the leader startup path, yet it re-checks isLeader internally anyway. The feature flag gets checked at the call site and again inside the function. Three layers of defense against calling this function in the wrong context.

The checks themselves aren’t the problem. It’s what happens when they trip: nothing. The function returns quietly, and the caller gets no signal either way.

The explicit contract: a function either does its job or throws. Preconditions get asserted, not silently absorbed. If the caller already guarantees the precondition, drop the internal check. If the function really can be called from multiple contexts and some of them are invalid, throw on the invalid ones instead of quietly returning.

VIII. “Best-Effort” Writes to State That Isn’t Optional

The most seductive justification for swallowing an error is: “the primary operation already succeeded so we don’t want a secondary failure to undo it.” That reasoning is correct in isolation and catastrophic in aggregate.

The Store Upsert That Swallows Its Own Failure

export async function mirrorTokenToStore(plaintext: string): Promise<void> {
  try {
    await store.upsert({ id: LEGACY_TOKEN_ID, token: encrypt(plaintext) });
  } catch (err) {
    logger.error('failed to mirror token to store', { reason: err });
    // swallowed — the caller sees success
  }
}

The comment above this function explains the intent: it’s “best-effort” so that a store-side failure doesn’t roll back a config file write that already succeeded. That reasoning holds up on its own but config file is now permanently out of sync.

The explicit contract: add reconciliation. On startup, compare the store’s token to the config file’s. If they differ, update the store. Emit a token_store_divergence counter and surface it in a health check similar to reconciliation loops in Kubernetes.

IX. Partial Operations Without a Way Back

The Batch Write That Discards on Failure

const batchTxn = db.transaction((ops) => { ops.forEach(fn => fn()); });
try {
  batchTxn(this.mutationCache.splice(0, maxSize));  // splice removes BEFORE success
} catch (err) {
  logger.error('Batch write failed', err);
  // operations already removed from cache — permanently lost
}

The comment in this code says: operations get removed from the cache regardless of whether the transaction succeeds. But the fix for “one bad operation might stall the queue” ends up being “discard the entire batch, including the good operations.” That isn’t a tradeoff instead it’s silent data loss.

The explicit contract: only remove items from the cache after a confirmed commit. Isolate the failing operation into a dead-letter queue and retry the rest of the batch without it.

The Config Reload Nobody Acknowledges

await conf.triggerReload().catch(NOOP);  // worker continues with old config

After receiving a new config bundle from the coordinator, a worker triggers a reload. If that reload fails, the worker just keeps running the old config. The two sides now disagree about what the worker is actually running.

The explicit contract: report a reload failure back to the coordinator on the next heartbeat. The coordinator marks that worker as “stale config” and can retry or alert. This is how Kubernetes rolling updates work, the controller notices and either retries or halts the rollout.

The Package Install That Saves Despite Partial Failure

for (const op of ops) {
  try {
    switch (op.type) {
      case 'install': await installPackage(op); break;
      case 'uninstall': await uninstallPackage(op); break;
    }
    status.applied.push(op);
  } catch (error) {
    errors.push(error);
  }
}
await this.save(packageManifest);  // saves regardless of how many failed

If three out of five packages install and two fail, the manifest still gets saved with those three. Next startup tries again from this partial state but the failed packages may have left behind lock files or half-written artifacts causing conflicts.

The explicit contract: validate that every operation can succeed before running any of them (a dry-run pass). Execute atomically (ACID transactions).

X. Silent Truncation With No Backpressure

The Metrics Buffer That Silently Drops

Workers piggyback metrics onto heartbeat messages sent to the coordinator, with a hard cap of 100,000 packets. Past that cap, excess metrics are silently dropped without any counter, logs or metrics. This is a nasty failure mode specifically because absence of data is itself meaningful data and silent truncation destroys that signal’s reliability.

The explicit contract: when the buffer nears capacity, reduce granularity instead of dropping outright. When truncation does happen, include metrics_truncated: N in the heartbeat so the coordinator knows its picture is incomplete. Better, instead of piggyback metrics on heartbeats at all, give them their own transport.

The TCP Sender That Zeroes Buffers on Disconnect

// On disconnect: all in-transit events silently lost
this.inTransitBufs = [];
this.bufOffset = 0;
this.bufferEventCount = 0;
this.dropBytes += len;  // only evidence: a counter increment

On a TCP disconnect, every in-transit event gets zeroed out. The only trace left behind is a dropBytes counter buried in internal metrics. Compare that to Kafka’s producer, where unacknowledged messages stay in the producer’s buffer and get retried on reconnect.

The Unbounded Queue That Becomes an OOM

protected queueBatch(): void {
  this.queuedBatches.push({ eventCount, eventsSize, events: this.currentBatch });
  // NO CHECK on length, size, or memory pressure
}

When the output destination is unreachable, failed batches get re-queued, and the queue grows without any bound. Memory climbs until the OOM killer steps in and terminates the process. An unbounded in-memory queue isn’t really a data structure, instead it’s a deferred OOM crash.

The explicit contract: bound the queue. Once it’s full, either apply backpressure to ingest, spill overflow to the persistent queue, or trip a circuit breaker that rejects new events with a typed error. Let the pipeline decide from there: drop, buffer to disk, or pause the source.

XI. Non-Atomic Writes

The Lease File That Can Split-Brain

await writeFile(this.leaseFile, stringify(content));

The failover lease file, the mechanism that’s supposed to guarantee only one coordinator is ever active is written directly with writeFile. On NFS, which is where this system runs in HA mode, writes aren’t atomic. A power failure mid-write leaves behind a truncated or corrupt file. The standby coordinator reads that corrupt lease, fails to parse it, and ends up in an undefined state.

The explicit contract: write to a temp file, fsync, then atomically rename it into place, with a checksum so readers can detect corruption. Every real database does this like SQLite’s WAL.

The Multi-File Config Deploy Without a Journal

Config deployment writes several YAML files in sequence like inputs.yml, outputs.yml, pipelines.yml,, etc. A crash midway through leaves the worker with a partial config and a worker that restarts after a partial deploy loads that inconsistent config.

The explicit contract: write every file to a staging directory first, verify that everything references correctly, then swap atomically like rename the directory, or flip a symlink. This is the same idea behind Docker image layers, Kubernetes ConfigMaps, and Nix store paths.

XII. Six Principles That Cover All of This

Every pattern above breaks one of six well-established principles. They’re standard practice in any system that prioritizes correctness over the appearance of uptime.

1. Fail fast at trust boundaries (Erlang’s “let it crash.”): When a precondition is violated, fail immediately and loudly. Erlang runs telecom infrastructure at 99.9999999% uptime on a philosophy of letting individual processes crash and having a supervisor restart them into known-good state.

2. Make invalid states unrepresentable: If getSecret() can return random(16) as a plain string then every caller is stuck defensively guessing whether it’s “real.” If it returns Secret | Disabled as a discriminated union instead then the type system forces every caller to handle both cases at compile time. I wrote about this pattern in “Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects.”

3. Classify errors (transient vs. fatal): Without classification, every catch block faces an impossible choice: rethrow and break “resilience,” or swallow and hide a real bug. For example, gRPC solves this with status codes like UNAVAILABLE means retry, INVALID_ARGUMENT means don’t, INTERNAL means there’s a bug.

4. Define delivery semantics for critical state: “Fire-and-forget” is fine for debug logs. It’s not fine for persistent queue flushes, token store upserts, or config reloads. If an operation mutates state that downstream code assumes succeeded, it needs at-least-once semantics.

5. One source of truth without fallback chains for critical state: For any given piece of state, there should be exactly one authoritative source. A fallback chain isn’t graceful degradation instead it’s an implicit decision that secondary sources are acceptable substitutes for the truth. If that’s genuinely acceptable, make it explicit with TTLs, version vectors, or consistency levels.

6. Atomic state transitions: State changes should be all-or-nothing: temp file, fsync, atomic rename for files; transactions with rollback for databases; staging plus swap for multi-file deployments.

XIII. Cognitive Load

All this conditional logic and fallback behavior creates the cognitive load, e.g., when any function might silently invent a value, you can’t trust a function’s output without reading its implementation. When errors are swallowed, a successful await no longer means the operation actually succeeded. When defaults get injected into config reads, a non-null value stops meaning “configured.”

Debugging a production incident in a system like this means reading every function in the call chain, understanding every fallback along the way, and reconstructing which code path actually ran. crash tells you exactly where and when an invariant broke. New engineers ask why workers sometimes fail to authenticate after a token rotation, and the honest answer involves reading six functions across four files, understanding a three-source fallback chain. Compare that to: “the store upsert threw on failure, the rotation API returned a 500, the operator re-ran it, it worked.

XIV. Conclusion

Defensive programming isn’t inherently wrong, and neither is Postel’s Law like at the boundary it was designed for. Validating input at system boundaries, handling I/O errors gracefully, protecting against malformed external data are correct applications of defensive thinking. The problem shows up when the same techniques get applied to internal code, inside a system where you control both ends of every interface. The alternative, in short:

• Throw Error when a required state that’s missing: The function refuses to proceed and the caller finds out immediately.
• Explicit return when an optional state that’s missing: null, undefined, Option<T>, a discriminated union.
• Transient failures = retry with backoff: Never .catch(NOOP).
• One source of truth per piece of state: Not a fallback chain that quietly degrades. Not a cache with no real invalidation.
• Bounded queues with backpressure: Not unbounded buffers waiting to OOM.
• Atomic state transitions: Not multi-step operations that can half-finish.
• Reconciliation loops for distributed state: Not one-shot “best-effort” writes that quietly drift apart.

This mud didn’t accumulate overnight, and it won’t disappear overnight either. But every fallback you remove, every error you refuse to swallow will makes the next incident roughly ten times faster to diagnose.

Related Blogs

1. From Big Ball of Mud to Functional Pipeline
2. The Reusability Trap: When DRY Becomes a Liability
3. Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects

Observability is a key part of any infrastructure but I’ve watched teams repeat the same mistakes around measuring availability. For example, they track uptime and watch average latency. They run a TCP health check on port 80 and call it good. Then support learns about the availability issues from customers but the health dashboard shows everything is green. This post covers how to measure availability correctly: what signals to collect, how monitoring tools compute the rolling statistics you see, why percentiles beat averages and what happens to tail latency at scale in microservices.

1. What Availability Actually Means

The textbook definition of availability is uptime, e.g., the fraction of time a service is running. This splits into two independent questions:

Availability = P(request succeeds) AND P(request completes within SLA)

A service can answer every request successfully but take 30 seconds per response then that’s functionally unavailable. Conversely, a service can respond in 5ms but return errors to 50% of requests is also functionally unavailable.

2. User Errors vs Server Errors — Why the Distinction Matters

This is the most commonly conflated measurement in production monitoring. HTTP status codes carry clear semantic meaning that should drive entirely different alert responses:

4xx errors are client/user errors like 400/Bad Request, 401/Unauthorized. 5xx errors means service is failing like 500/Internal Server, 503/Service Unavailable. There is one gray area: client timeouts. If your client times out after 5s waiting for your 10s response, the client sees a 408 or a network error, which look like a 4xx but the root cause is server-side latency. This is why tracking latency separately from error codes is essential.

from prometheus_client import Counter, Histogram

# Track errors with full status code granularity
request_counter = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status_code', 'status_class']
)

latency_histogram = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

def record_request(method: str, endpoint: str, status: int, duration_s: float):
    status_class = f"{status // 100}xx"
    request_counter.labels(
        method=method,
        endpoint=endpoint,
        status_code=str(status),
        status_class=status_class
    ).inc()
    latency_histogram.labels(method=method, endpoint=endpoint).observe(duration_s)


# --- Prometheus queries that actually measure availability ---

# Server error rate (5xx only — excludes client errors)
SERVER_ERROR_RATE = """
sum(rate(http_requests_total{status_class="5xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Availability (only penalize server errors)
AVAILABILITY = """
1 - (
  sum(rate(http_requests_total{status_class="5xx"}[5m]))
  /
  sum(rate(http_requests_total[5m]))
)
"""

# Client error rate (useful to watch, but not availability)
CLIENT_ERROR_RATE = """
sum(rate(http_requests_total{status_class="4xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Latency SLA compliance — fraction of requests completing within 500ms
LATENCY_SLA_COMPLIANCE = """
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
"""

A spike in 4xx that isn’t paired with a 5xx spike is almost certainly a misbehaving client, not your service. Alert on them differently: 5xx pages your on-call, 4xx goes to a ticket queue for review.

3. SLAs, SLOs, and Error Budgets

These three terms are used interchangeably in many organizations and they shouldn’t be.

• SLA (Service Level Agreement) is a contractual commitment to external customers. Violating it has legal or financial consequences. Example: “We guarantee 99.9% availability per calendar month. If we breach this, we issue service credits.”
• SLO (Service Level Objective) is an internal engineering target, usually tighter than the SLA. Example: “We target 99.95% availability.” The gap between SLO and SLA is your buffer.
• Error Budget is what you get to spend before you breach your SLO. For a 99.9% SLO over 30 days:

Total minutes in 30 days = 30 × 24 × 60 = 43,200 minutes
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes

The error budget is your 43.2 minutes. Every minute of downtime spends from it. This reframes the conversation from “is the service up?” to “how fast are we burning through our budget?”

from datetime import datetime, timedelta

class ErrorBudget:
    """
    Track error budget consumption in real time.
    
    Example: 99.9% SLO over 30 days = 43.2 minutes of allowed downtime.
    """
    def __init__(self, slo_target: float, window_days: int = 30):
        self.slo_target = slo_target          # e.g., 0.999 for 99.9%
        self.window_minutes = window_days * 24 * 60
        self.allowed_downtime_minutes = self.window_minutes * (1 - slo_target)
        self.downtime_minutes_spent = 0.0
        self.start_time = datetime.now()

    def record_downtime(self, minutes: float):
        self.downtime_minutes_spent += minutes

    def budget_remaining_minutes(self) -> float:
        return max(0, self.allowed_downtime_minutes - self.downtime_minutes_spent)

    def budget_remaining_pct(self) -> float:
        return (self.budget_remaining_minutes() / self.allowed_downtime_minutes) * 100

    def burn_rate(self) -> float:
        """How fast are we burning budget vs. expected rate? 1.0 = on track, >1.0 = burning fast."""
        elapsed = (datetime.now() - self.start_time).total_seconds() / 60
        expected_spent = (elapsed / self.window_minutes) * self.allowed_downtime_minutes
        if expected_spent == 0:
            return 0.0
        return self.downtime_minutes_spent / expected_spent

    def summary(self) -> str:
        return (
            f"SLO: {self.slo_target*100:.2f}%  |  "
            f"Budget: {self.allowed_downtime_minutes:.1f} min  |  "
            f"Spent: {self.downtime_minutes_spent:.1f} min  |  "
            f"Remaining: {self.budget_remaining_pct():.1f}%  |  "
            f"Burn rate: {self.burn_rate():.2f}x"
        )

# Usage
budget = ErrorBudget(slo_target=0.999, window_days=30)
budget.record_downtime(minutes=12.5)   # incident on day 3
budget.record_downtime(minutes=8.0)    # incident on day 11
print(budget.summary())
# SLO: 99.90%  |  Budget: 43.2 min  |  Spent: 20.5 min  |  Remaining: 52.5%  |  Burn rate: ...

A burn rate above 1.0 means you’ll exceed your error budget before the window closes. Burn rate above 14.4x means you’ll exhaust it within 48 hours, which is a PagerDuty alert.

4. The Health Check Anti-Pattern

I need to address something I’ve seen sink production deployments before we even get to metrics: health checks that only verify the process is listening on a port. A port check tells you the process hasn’t crashed. It tells you nothing about whether the process can serve traffic. I’ve seen this exact scenario: database connection pool was exhausted, port was open, load balancer marked the instance healthy, every request returned a 500. The monitoring was dark green the whole time.

A real health check must exercise the actual request path: connect to dependencies, perform a lightweight but genuine operation, return structured status. In Kubernetes this means a readiness probe hitting a /health endpoint that checks dependency connectivity. Critically, readiness and liveness are different probes:

• Liveness: Is the process deadlocked? If not, keep it alive. If yes, kill and restart it.
• Readiness: Can it serve traffic right now? If not, remove it from the load balancer pool, but don’t kill it.

A process that is alive but not ready (warming up a cache, waiting for a dependency) should fail readiness but pass liveness. Confusing these two causes cascading restarts during startup under load is a failure mode I’ve seen multiple times in prod. See my Zero-Downtime Services on Kubernetes and Istio post for the full treatment.

5. Why Average Latency Lies

Here’s a production story I’ve seen more than once. The team does an efficiency push: optimizes the hot path, ships a 30% improvement in p50 latency. Dashboards celebrate but three weeks later, the p99 is back to where it started. The answer is queuing theory. Consider a server with a queue in front of it. Define utilization P as:

P = arrival rate / service rate

The average number of items in the system in queue plus being served is:

E[N] = P / (1 - P)

This is not a linear relationship. It’s an asymptote that goes vertical as you approach full utilization:

When you make the code faster (higher service-rate), P drops, and you slide left on this curve, i.e., fewer items queuing with lower tail latency. But then traffic grows or you reduce servers to “realize the savings.” P climbs back to where it was, and latency returns with it. The key lesson is that the average latency reflects the fast path but high-percentile latency (p99, p99.9) is extremely sensitive to queue depth. High percentile latency is a leading indicator that you’re approaching overload.

There’s a counterintuitive implication from this: p99 is a terrible way to measure whether your efficiency work succeeded. It’s so sensitive to the queuing nonlinearity that changes in utilization will swamp the signal from your actual code changes. For measuring efficiency, mean latency is actually better because it tracks the true cost of processing one request without queue effects. Use percentiles for alerting and use mean for efficiency measurement.

6. Percentiles From First Principles

Let’s go over percentiles from scratch, because monitoring tools throw around “p50”, “p99”, “p99.9” without ever explaining what they actually represent, and misunderstanding them leads to misreading dashboards. Given a set of N latency measurements, sort them from fastest to slowest. The Nth percentile is the value at position N% in that sorted list.

Latencies (ms): [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
Sorted:          [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
                  ^              ^              ^
                  p10           p50            p90

p50 = 10ms  (50% of requests were at or below this speed)
p90 = 13ms  (90% of requests were at or below this speed)
p99 = 400ms (99% of requests were at or below this speed)

What p99 tells you is: at most 1% of your requests see latency worse than this number. Equivalently, 999 out of every 1000 requests complete faster than p99. The catch is that p99 is a single value and it summarizes nothing about the shape of the distribution between p90 and p99. Latency can get dramatically worse for customers in that range without your p99 alarm firing.

import numpy as np

def explain_percentile(latencies_ms: list[float]):
    """Show what percentiles mean in plain English."""
    arr = np.array(sorted(latencies_ms))
    n = len(arr)
    
    stats = {
        "mean":  np.mean(arr),
        "p50":   np.percentile(arr, 50),
        "p90":   np.percentile(arr, 90),
        "p95":   np.percentile(arr, 95),
        "p99":   np.percentile(arr, 99),
        "p99.9": np.percentile(arr, 99.9),
        "max":   np.max(arr),
    }
    
    print(f"{'Statistic':<10} {'Value':>10}   Plain English")
    print("-" * 65)
    print(f"{'mean':<10} {stats['mean']:>10.1f}ms  Average — hides bimodal distributions")
    print(f"{'p50':<10} {stats['p50']:>10.1f}ms  Half of requests faster than this")
    print(f"{'p90':<10} {stats['p90']:>10.1f}ms  90% of requests faster than this")
    print(f"{'p95':<10} {stats['p95']:>10.1f}ms  95% of requests faster than this")
    print(f"{'p99':<10} {stats['p99']:>10.1f}ms  99% of requests faster than this")
    print(f"{'p99.9':<10} {stats['p99.9']:>10.1f}ms  999/1000 requests faster than this")
    print(f"{'max':<10} {stats['max']:>10.1f}ms  Worst single request (very noisy)")

# Simulate a bimodal latency distribution
# 95% fast requests (cache hit), 5% slow (cache miss + DB query)
import random
random.seed(42)
latencies = [
    random.gauss(10, 2) if random.random() > 0.05 else random.gauss(300, 40)
    for _ in range(1000)
]
explain_percentile(latencies)

Statistic       Value   Plain English
-----------------------------------------------------------------
mean             24.8ms  Average — hides bimodal distributions
p50              10.4ms  Half of requests faster than this
p90              12.1ms  90% of requests faster than this
p95              17.9ms  95% of requests faster than this
p99             302.1ms  99% of requests faster than this
p99.9           375.8ms  999/1000 requests faster than this
max             392.4ms  Worst single request (very noisy)

7. Moving Averages and Rolling Percentiles

When Grafana shows you a p99 or Datadog shows you an error rate, it’s not summing up all-time data. It’s computing over a rolling time window.

Simple Moving Average vs EWMA

A Simple Moving Average (SMA) gives equal weight to every sample in the window:

from collections import deque
import statistics

class SMA:
    """Simple Moving Average — every sample in the window weighted equally."""
    def __init__(self, window: int):
        self.buf = deque(maxlen=window)
    
    def add(self, v: float) -> float:
        self.buf.append(v)
        return statistics.mean(self.buf)

An Exponentially Weighted Moving Average (EWMA) gives more weight to recent samples, fading older ones smoothly:

class EWMA:
    """
    Exponentially Weighted Moving Average.
    alpha: 0 < alpha < 1
    - High alpha (e.g. 0.3): reacts fast, noisier
    - Low alpha  (e.g. 0.05): smoother, slower to detect changes
    
    StatsD uses EWMA for gauge values. Prometheus uses time-window sums.
    """
    def __init__(self, alpha: float = 0.1):
        self.alpha = alpha
        self.value: float | None = None

    def add(self, sample: float) -> float:
        if self.value is None:
            self.value = sample
        else:
            self.value = self.alpha * sample + (1 - self.alpha) * self.value
        return self.value

# Demonstrate: same spike, different alphas
spike_data = [10, 10, 10, 10, 250, 10, 10, 10, 10, 10]
slow_ewma = EWMA(alpha=0.05)
fast_ewma = EWMA(alpha=0.30)

print(f"{'Sample':>8} {'Value':>8} {'alpha=0.05':>10} {'alpha=0.30':>10}")
for i, v in enumerate(spike_data):
    print(f"{i:>8} {v:>8.0f} {slow_ewma.add(v):>10.1f} {fast_ewma.add(v):>10.1f}")

  Sample    Value     alpha=0.05   alpha=0.30
       0       10       10.0       10.0
       1       10       10.0       10.0
       4      250       21.9       82.0   --> fast alpha sees the spike much louder
       5       10       21.3       58.4   --> slow alpha recovers faster
       9       10       18.5       17.2

Rolling Percentile

Computing exact percentiles over a moving window requires keeping raw samples and re-sorting. For production scale, the T-Digest algorithm computes approximate percentiles with bounded memory. Here’s the conceptual version first:

import numpy as np
from collections import deque

class RollingPercentile:
    """
    Rolling percentile over a fixed window of recent samples.
    
    Production note: At high throughput, use T-Digest or DDSketch instead.
    Prometheus uses pre-defined histogram buckets + linear interpolation.
    """
    def __init__(self, window: int, pctile: float):
        self.buf = deque(maxlen=window)
        self.pctile = pctile

    def add(self, v: float) -> float | None:
        self.buf.append(v)
        if len(self.buf) < 2:
            return None
        return float(np.percentile(list(self.buf), self.pctile))

# Show how window size affects sensitivity
import random
random.seed(7)

data = [random.gauss(10, 2) for _ in range(90)] + \
       [random.gauss(200, 20) for _ in range(10)]  # degradation at t=90

p99_small  = RollingPercentile(window=20,  pctile=99)
p99_medium = RollingPercentile(window=100, pctile=99)

print("How window size affects p99 detection of a latency spike:")
print(f"{'t':>4} {'value':>8} {'p99 w=20':>12} {'p99 w=100':>12}")
for t, v in enumerate(data[80:]):   # show the transition region
    small  = p99_small.add(v)
    medium = p99_medium.add(v)
    marker = " --> spike starts" if t == 10 else ""
    if small and medium:
        print(f"{t+80:>4} {v:>8.1f} {small:>12.1f} {medium:>12.1f}{marker}")

Prometheus histogram vs. summary: Prometheus offers two ways to track latency. A Summary computes quantiles client-side over a rolling window but you can’t aggregate across instances. A Histogram records counts in pre-defined buckets and approximates quantiles server-side, which is slightly less accurate, but fully aggregatable. For microservices with multiple replicas, always use Histogram.

8. Trimmed Mean: More Signal, Real Tradeoffs

Here’s the core difference between a percentile and a trimmed mean, using the product review analogy:

100 latency measurements, sorted by speed:

p99  = the single worst measurement in the best 99%
       (the 99th measurement out of 100, sorted fastest-to-slowest)

tm99 = the average of all 99 measurements in the best 99%
       (discard the 1 slowest, average the remaining 99)

tm99 summarizes 99 times more data than p99. That makes it more stable (less spiky under low traffic), harder to game (a gradual degradation can hide between percentile checkpoints, but tm99 will catch it), and more representative of typical customer experience.

• tm99 tracks the average experience of your bulk of customers
• TM(99%:) tracks the average of your slowest 1%; ensures outlier experience doesn’t silently worsen

Together these two numbers cover 100% of your requests with just two metrics.

import numpy as np

def compute_tm_stats(samples: list[float]) -> dict:
    """
    Compute a full suite of trimmed mean statistics.

    Syntax mirrors CloudWatch / AWS Embedded Metrics Format:
      tm99        = TM(0%:99%)  = average of fastest 99%
      TM(99%:)    = TM(99%:100%) = average of slowest 1%  
      TM(1%:99%)  = drop both extremes (handles unbounded latency)
      IQM         = TM(25%:75%) = Interquartile Mean
    """
    arr = np.sort(np.array(samples))
    n = len(arr)

    def tm(lower_pct: float, upper_pct: float) -> float:
        lo = np.percentile(arr, lower_pct)
        hi = np.percentile(arr, upper_pct)
        trimmed = arr[(arr >= lo) & (arr <= hi)]
        return float(np.mean(trimmed)) if len(trimmed) else float('nan')

    return {
        "mean":       float(np.mean(arr)),
        "p50":        float(np.percentile(arr, 50)),
        "p99":        float(np.percentile(arr, 99)),
        "tm99":       tm(0, 99),      # avg of fastest 99%
        "TM(99%:)":   tm(99, 100),    # avg of slowest 1%  --> watch your outliers here
        "TM(1%:99%)": tm(1, 99),      # drop both extremes (use for unbounded latency)
        "IQM":        tm(25, 75),     # interquartile mean
    }

# Scenario: a cache-miss spike where 2% of requests are slow
rng = np.random.default_rng(42)
fast = rng.normal(10, 1.5, 980)
slow = rng.normal(350, 30, 20)
samples = np.concatenate([fast, slow]).tolist()

stats = compute_tm_stats(samples)
print(f"{'Metric':<14} {'Value':>10}   Notes")
print("-" * 65)
for k, v in stats.items():
    notes = {
        "mean":       "Pulled up by slow tail — misleading",
        "p50":        "Median — fine but ignores tail",
        "p99":        "Single value at 99th position",
        "tm99":       "Average of 98% of customers --> primary SLO metric",
        "TM(99%:)":   "Average of slowest 2% --> outlier watchdog",
        "TM(1%:99%)": "Drops both extremes — good for browser metrics",
        "IQM":        "Middle 50% average — robust to both extremes",
    }.get(k, "")
    print(f"{k:<14} {v:>10.1f}ms  {notes}")

Metric              Value   Notes
-----------------------------------------------------------------
mean                16.8ms  Pulled up by slow tail — misleading
p50                  9.9ms  Median — fine but ignores tail
p99                335.2ms  Single value at 99th position
tm99                10.1ms  Average of 98% of customers --> primary SLO metric
TM(99%:)           351.4ms  Average of slowest 2% --> outlier watchdog
TM(1%:99%)          10.1ms  Drops both extremes — good for browser metrics
IQM                  9.8ms  Middle 50% average — robust to both extremes

Bounded vs. unbounded latency:

• Bounded latency (server-side, with request timeouts): use tm99 + TM(99%:). Since latency is capped by your timeout, even the worst measurements are meaningful.
• Unbounded latency (client-side browser metrics, user-perceived time): use TM(1%:99%). A user who closes their laptop mid-request and reopens it days later may log a latency of 230,400 seconds. These shouldn’t contaminate your outlier statistics. Drop the top and bottom extremes.

I have seen in a real-life production services where teams work towards improving p50/median but everything else gets worse. You only find this out when you examine tm95 because latency was consistently worse for a growing number of customers. The key lesson is that percentiles create blind spots “between the checkpoints.” A degradation that affects the 40th–60th percentile range will move neither p25 nor p75 much. Trimmed mean, because it averages across the entire range, catches these shifts. However, trimmed mean has its own blind spot. It deliberately removes the part of the distribution that dominates user experience in fan-out architectures. The right answer is not to choose between percentiles and trimmed mean but use both.

10. Winsorized Mean, Percentile Rank, and IQM

These statistics show up in CloudWatch and modern observability platforms, and they each solve a specific problem.

Winsorized Mean (WM)

Like trimmed mean, but instead of discarding outliers, it replaces them with the boundary value. For wm99:

• Find the value at the 99th percentile (= p99)
• Treat all 1% outliers as if they had exactly that p99 value
• Average all 100% of samples

def winsorized_mean(samples: list[float], lower_pct: float = 0, upper_pct: float = 99) -> float:
    arr = np.array(samples, dtype=float)
    lo = np.percentile(arr, lower_pct)
    hi = np.percentile(arr, upper_pct)
    # Clip: anything below lo becomes lo, anything above hi becomes hi
    winsorized = np.clip(arr, lo, hi)
    return float(np.mean(winsorized))

Winsorized mean gives some weight to outliers without letting extreme values skew the average. The difference between tm99 and wm99 is subtle at high percentages and wm99 will be slightly higher because it includes the outliers rather than dropping them.

Percentile Rank PR()

Percentile rank answers the inverse question from percentile. Percentile says: “What latency value marks the Nth percent?” Percentile rank says: “What percent of requests are below a given latency value?”

If you have an SLA of “respond within 500ms to 99% of users,” you’d normally monitor p99 and check it’s <= 500ms. With Percentile Rank, you instead plot PR(:500ms, i.e., the percentage of requests completing within 500ms and drive that number toward 99% or higher. This is more directly action-oriented: you always know exactly how far below your SLA you are.

def percentile_rank(samples: list[float], threshold: float) -> float:
    """What fraction of samples are at or below threshold?"""
    arr = np.array(samples)
    return float(np.mean(arr <= threshold) * 100)

# Example: SLA is p99 < 500ms
samples_ms = [10, 12, 9, 11, 450, 10, 13, 600, 11, 10]  # small sample
pr_500 = percentile_rank(samples_ms, 500)
print(f"PR(:500ms) = {pr_500:.1f}%  (SLA requires 99%)")
# PR(:500ms) = 90.0%  (SLA requires 99%) — you're 9 percentage points short

IQM (Interquartile Mean)

IQM is simply TM(25%:75%), the average of the middle 50% of samples, discarding the top and bottom 25%. It’s extremely robust to outliers in both directions, useful when you expect noise from both ends of the distribution (e.g., some requests are trivially fast cache hits, others are pathologically slow).

11. The Inspection Paradox: Your Users Experience Worse Than Your Metrics Show

As Marc Brooker’s explained in his blog, this is the most underappreciated gap in distributed systems reliability. For example, say your service has outages with very different durations: some resolve in 30 seconds, but occasionally one runs for 3 hours. Your MTTR (Mean Time to Recovery) might calculate to 5 minutes. But when a user hits your service during an outage, they’re more likely to land in a long outage than a short one because long outages have more time-slots for users to arrive in.

Customer-experienced mean recovery = (1/2) × (MTTR + Variance/MTTR)

The second term is what kills you. If your outage duration has high variance, e.g., fast recovery most of the time, but occasional 3-hour events then that variance term dominates. Your customers experience something dramatically worse than your MTTR.

import random
import math
import statistics

def inspection_paradox_demo(
    median_recovery_min: float,
    p99_recovery_min: float,
    arrivals_per_min: float = 100,
    n_outages: int = 2000
) -> dict:
    """
    Simulate the gap between operator MTTR and customer-experienced recovery.
    
    Key insight: customers are t-weighted samplers of your outage distribution.
    A 10-minute outage gets sampled by ~10x as many clients as a 1-minute outage.
    """
    # Fit lognormal to median and p99
    mu = math.log(median_recovery_min)
    sigma = (math.log(p99_recovery_min) - mu) / 2.326

    server_durations = []
    client_wait_times = []

    for _ in range(n_outages):
        duration = random.lognormvariate(mu, sigma)
        server_durations.append(duration)

        # Clients arrive as a Poisson process during the outage
        t = 0.0
        while True:
            gap = random.expovariate(arrivals_per_min)
            if t + gap > duration:
                break
            # This client arrived at time t, waits until outage ends
            client_wait_times.append(duration - t)
            t += gap

    return {
        "operator_mttr":        statistics.mean(server_durations),
        "operator_p99":         sorted(server_durations)[int(len(server_durations) * 0.99)],
        "customer_mean_wait":   statistics.mean(client_wait_times) if client_wait_times else 0,
        "customer_p99_wait":    sorted(client_wait_times)[int(len(client_wait_times) * 0.99)] if client_wait_times else 0,
        "experience_gap_ratio": (statistics.mean(client_wait_times) / statistics.mean(server_durations)) if client_wait_times else 0,
    }

result = inspection_paradox_demo(
    median_recovery_min=1,    # median outage resolves in 1 minute
    p99_recovery_min=60,      # but 1% of outages take an hour
)

print("Scenario: 1-minute median recovery, 60-minute p99 recovery")
print()
print("What your on-call dashboard shows:")
print(f"  MTTR:              {result['operator_mttr']:.1f} minutes")
print(f"  p99 recovery:      {result['operator_p99']:.1f} minutes")
print()
print("What your customers actually experience:")
print(f"  Mean recovery:     {result['customer_mean_wait']:.1f} minutes")
print(f"  p99 recovery:      {result['customer_p99_wait']:.1f} minutes")
print(f"  Experience gap:    {result['experience_gap_ratio']:.1f}x worse than MTTR")

Scenario: 1-minute median recovery, 60-minute p99 recovery

What your on-call dashboard shows:
  MTTR:              4.9 minutes
  p99 recovery:      56.6 minutes

What your customers actually experience:
  Mean recovery:     60.0 minutes
  p99 recovery:      797.3 minutes
  Experience gap:    12.1x worse than MTTR

This is why tail recovery time matters more than averages suggest. Timeout-and-retry can hide individual request latency, but it cannot hide recovery time. Once a client gets stuck in an outage, retries don’t shorten the outage, they just add load to an already struggling service. The right takeaway: minimize variance in recovery time, not just its mean. Bounded, predictable recovery is far better for customers than fast-average-but-occasional-disaster.

12. Tail Latency Amplifies in Microservices

Modern architectures decompose user requests into many service calls. This creates two topologies, and both amplify tail latency:

Fan-out math: If each service has a 1% probability of a slow response, the probability that at least one is slow when calling N services in parallel is:

P(at least one slow) = 1 - (1 - 0.01)^N

What was a rare 1% tail now affects the majority of user interactions. And here’s the pernicious part: your per-service p99 metric looks perfectly fine. The damage is invisible at the service level, only visible at the user-experience level.

import numpy as np, random

def simulate_fanout(n_backends: int, tail_prob: float = 0.01, n_reqs: int = 20_000):
    """
    Simulate client experience when calling n_backends in parallel.
    Each backend: (1-tail_prob) chance of fast, tail_prob chance of slow.
    """
    results = []
    slow_count = 0
    for _ in range(n_reqs):
        latencies = []
        for _ in range(n_backends):
            if random.random() < tail_prob:
                latencies.append(random.gauss(250, 25))
                slow_count += 1
            else:
                latencies.append(random.gauss(10, 2))
        results.append(max(latencies))  # fan-out: wait for slowest
    
    arr = np.array(results)
    return {
        "p50":  np.percentile(arr, 50),
        "p99":  np.percentile(arr, 99),
        "mean": np.mean(arr),
        "pct_slow_user_requests": np.mean(arr > 50) * 100,
    }

print(f"{'N':>4} {'p50 (ms)':>10} {'p99 (ms)':>10} {'mean (ms)':>10} {'% users hit slow':>18}")
for n in [1, 5, 10, 25, 50, 100]:
    r = simulate_fanout(n)
    print(f"{n:>4} {r['p50']:>10.1f} {r['p99']:>10.1f} {r['mean']:>10.1f} {r['pct_slow_user_requests']:>18.1f}%")

The trimmed mean blind spot revisited. At N=50, nearly 40% of user requests are slow. But your per-service tm99 (averaging the best 99% of individual service calls) still looks great because it’s averaging the fast cluster. This is exactly the case where trimmed mean gives you false comfort. You need explicit end-to-end latency tracking at the user-request level, not just per-service tail tracking.

13. The Pooling Dividend: Why Redundancy Is Non-Linear

Adding servers doesn’t just increase capacity linearly but it also improves latency through pooling. This comes from the Erlang C model in queuing theory. For example, two designs, both handling the same total load:

• Design A: 1 server at 80% utilization
• Design B: 10 servers sharing load, each at 80% utilization

Design A has roughly a 13% chance of any incoming request finding the server busy and joining a queue. Design B has roughly a 3.6% chance. Double the fleet to 20 servers at the same 80% per-server utilization, and the queueing probability drops toward 1%. You’re getting better latency and better tail behavior at the same per-server cost.

import math
from functools import lru_cache

def erlang_c(c: int, rho: float) -> float:
    """
    Erlang C formula: probability an arriving request must queue
    (rather than being served immediately) in an M/M/c system.
    
    c: number of servers
    rho: per-server utilization (0 < rho < 1)
    """
    a = c * rho  # total offered load
    
    @lru_cache(maxsize=None)
    def factorial(n: int) -> int:
        return 1 if n <= 1 else n * factorial(n - 1)
    
    # Sum term for the denominator
    sum_term = sum(a**k / factorial(k) for k in range(c))
    last_term = (a**c / factorial(c)) * (1 / (1 - rho))
    
    ec = last_term / (sum_term + last_term)
    return ec

print("Probability a request must queue before being served:")
print(f"{'Servers':>8} {'Utilization':>12} {'Queue prob':>12}   {'Queue %':>8}")
for c in [1, 2, 5, 10, 20, 50]:
    ec = erlang_c(c=c, rho=0.8)
    print(f"{c:>8} {'80%':>12} {ec:>12.4f}   {ec*100:>7.1f}%")

Probability a request must queue before being served:
 Servers  Utilization   Queue prob    Queue %
       1          80%       0.8000      80.0%
       2          80%       0.7111      71.1%
       5          80%       0.5541      55.4%
      10          80%       0.4092      40.9%
      20          80%       0.2561      25.6%
      50          80%       0.0870       8.7%

Most of the benefit materializes at modest fleet sizes. You don’t need to be at hyperscale to get pooling gains. A fleet of 5-10 servers sharing load through a proper load balancer will have dramatically better tail latency behavior than the same compute running as independent instances.

14. Retries, Circuit Breakers, and the Amplification Trap

Retries protect against transient failures like a GC pause, a brief network glitch, a thundering herd. In past production deployment, I use up to 3 retries with exponential backoff for idempotent read operations. The protection against false positives is real and worthwhile. But retries have a catastrophic failure mode: retry amplification.

A single user request can generate 3 × 3 × 3 = 27 actual requests to a struggling downstream service. This turns a partial overload into a total collapse. I’ve watched this happen in production, e.g., a service that was at 60% capacity receives a burst of retries from a misbehaving upstream and immediately spikes to 200% load, failing every request, causing more retries, a feedback loop.

The mitigations:

import time
import threading
from collections import deque

class RetryBudget:
    """
    Limit total retry rate as a fraction of total traffic.
    If retries exceed the budget, fail fast instead of retrying.
    
    Classic mitigation for retry amplification.
    """
    def __init__(self, budget_fraction: float = 0.10, window_seconds: int = 60):
        self.budget_fraction = budget_fraction
        self.window = window_seconds
        self.total_requests: deque = deque()
        self.retry_requests: deque = deque()
        self._lock = threading.Lock()

    def _prune(self):
        cutoff = time.monotonic() - self.window
        while self.total_requests and self.total_requests[0] < cutoff:
            self.total_requests.popleft()
        while self.retry_requests and self.retry_requests[0] < cutoff:
            self.retry_requests.popleft()

    def record_request(self):
        with self._lock:
            self.total_requests.append(time.monotonic())

    def should_retry(self) -> bool:
        """Returns True if we have retry budget remaining."""
        with self._lock:
            self._prune()
            total = len(self.total_requests)
            retries = len(self.retry_requests)
            if total == 0:
                return True
            current_rate = retries / total
            if current_rate < self.budget_fraction:
                self.retry_requests.append(time.monotonic())
                return True
            return False  # budget exhausted — fail fast, don't amplify


class CircuitBreaker:
    """
    Stop sending requests to a failing downstream.
    Transitions: CLOSED -> OPEN -> HALF_OPEN -> CLOSED
    """
    CLOSED, OPEN, HALF_OPEN = "CLOSED", "OPEN", "HALF_OPEN"

    def __init__(self, failure_threshold: float = 0.5, cooldown_seconds: float = 30):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown_seconds
        self.state = self.CLOSED
        self.failures = 0
        self.total = 0
        self.opened_at: float | None = None

    def call_allowed(self) -> bool:
        if self.state == self.CLOSED:
            return True
        if self.state == self.OPEN:
            if time.monotonic() - self.opened_at > self.cooldown:
                self.state = self.HALF_OPEN
                return True  # let one probe through
            return False  # fail fast
        return True  # HALF_OPEN: let one probe through

    def record_success(self):
        self.failures = 0
        self.total = 0
        self.state = self.CLOSED

    def record_failure(self):
        self.failures += 1
        self.total += 1
        if self.total >= 10 and self.failures / self.total >= self.failure_threshold:
            self.state = self.OPEN
            self.opened_at = time.monotonic()

Hedge requests are often better than retries for latency problems. Instead of waiting for a timeout and retrying, fire a second request after a short delay (say, the p90 latency). Accept whichever responds first, cancel the other. This cuts your tail exposure without amplifying load as aggressively, because typically one of the two requests will succeed quickly.

15. Synthetic Canaries in Production

Error rates and latency percentiles tell you what’s happening to real traffic but only after users are affected. Synthetic canaries fill the gap: background processes that continuously exercise your API end-to-end, giving you availability signal even at 3am when real traffic is low.

Key design decisions from production experience:

• Test the full workflow, not just the health endpoint. A canary for a data API should create, read, update, and delete a record. One for an auth service should issue a token, validate it, and revoke it. Shallow canaries that only call GET /health will miss the exact failures that health check anti-patterns also miss.
• Track first-attempt and final success separately. If your canary succeeds on retry 2 90% of the time, the final success rate looks fine but something is quietly broken. First-attempt success rate catches this.
• Keep canary observability separate from production. Mixing them has two failure modes: canary failures inflate your production error rate, and canary successes can mask production degradation if canaries hit warm caches or a separate code path.
• Account for canary bias. Canaries hit warm caches and have predictable access patterns. Their p99 is almost always better than real user p99. Use canary latency to detect regressions relative to a baseline, not to claim absolute performance numbers.
• Use retries in canaries, but with a limit. Up to 3 retries prevents false positives from transient network blips. But record the retry count per run, e..g, a canary that regularly needs 2+ retries is a signal worth investigating even if it eventually succeeds.

16. Putting It All Together: A Layered Monitoring Strategy

After decades of building and operating distributed systems, here’s the monitoring architecture I’d deploy for any production service from day one:

Closing: The Number That Matters Most

After all of this, the insight that has most changed how I think about availability is this: your users don’t experience your MTTR. They experience a version of it weighted by how long outages last, which skews dramatically toward your worst events. A service with a 1-minute median recovery but occasional 2-hour outages will have customers experiencing something closer to hours, not minutes. The variance in your tail events matters more than the central tendency. This is why the tail cannot be trimmed away from your visibility. Build observability that shows you the tail. Use redundancy and retries but understand how they amplify under pressure. Run canaries that exercise the whole path. Track user errors and server errors separately. Keep SLO burn rate visible so you always know how much budget you’ve spent. And when your customers say the service is slow and your dashboard says everything is green then believe the customers.

A self-guided path for junior and mid-level engineers whose core skills are quietly eroding

I have observed a contrast productivity gap watching a senior engineer use an agentic coding tool and watching a junior engineer use the same tool. The senior engineer moves faster, catches more problems, and produces better outcomes, while the junior engineer often ships code that looks finished but quietly breaks at the seams. The reason is not the tool, it is what each engineer brings to the tool.

Senior engineers are more effective with agentic AI because they have already built the skills that make AI useful: writing precise specifications, designing systems that hold together under real load, spotting code smells in a diff, understanding trade-offs between correctness and performance, maintaining the conceptual integrity of a codebase across hundreds of changes. These skills aren’t separate from coding experience, they are products of it, built up over years of writing code, breaking things, debugging production incidents, and internalizing the consequences of design decisions.

Junior and mid-level engineers haven’t built those skills yet, which used to be fine because the path to building them was clear: you wrote code, made mistakes, got reviewed by someone who caught what you missed, and learned. Repeat for several years. The trouble is that agentic coding short-circuits exactly that path. When an agent generates the code, a junior engineer faces a key problem: they cannot reliably distinguish between code that is actually correct and code that is plausibly correct. Agentic code looks right in the narrow context where it was generated, the function compiles, the tests pass, the logic seems sound. But a system is not a collection of locally correct functions. It is a web of interacting decisions, constraints, and invariants that only hold together if someone understands the whole. Senior engineers have built that whole-system mental model through years of implementation.

Some companies have responded to this by stopping junior engineer hiring entirely, reasoning that agents can now fill entry-level roles. This is a serious mistake, and a slow-moving disaster. It optimizes for short-term output while eliminating the pipeline through which every senior and principal engineer is eventually produced. Today’s junior engineers are tomorrow’s architects. When companies stop hiring and developing them, they are consuming the seed corn and they will feel it in three to five years when there are no experienced engineers left to review what the agents produce.

The risk doesn’t stop at junior engineers. Senior engineers face a subtler version of the same problem. When you stop writing code regularly, the skills built through writing it, the intuition for design, the eye for code smells, the ability to hold a large system in your head begin to decay. Specification writing becomes abstract rather than grounded. Architecture decisions lose their connection to implementation reality. Code review gets shallower because you’re no longer maintaining the mental model of how things fit together. The most important thing being lost is not any individual skill but shared understanding, what Fred Brooks called conceptual integrity, the coherence of design philosophy across an entire system that only exists when the people building it have deeply internalized how it works.

This post is about what to do about all of it. How to deliberately build the skills that agentic coding doesn’t hand you. How to maintain the skills you’ve built, as the nature of the work changes. And how to grow from junior to senior to principal in an era when the traditional feedback loop between design and implementation has been broken.

How Engineers Used to Grow

For decades, the career path followed a recognizable arc. You joined at entry level, wrote code, broke things, got your code torn apart in review, made better mistakes, and gradually developed what researchers call tacit understanding, the ability to look at a system and feel what is wrong before you can fully articulate why.

The Dreyfus model of skill acquisition describes this progression across five stages:

The Japanese martial arts concept of Shu Ha Ri mirrors this exactly. First you follow the form faithfully (Shu learn the rules). Then you find the exceptions and break with tradition (Ha question the rules). Then form dissolves into natural action (Ri transcend the rules). You cannot skip stages. The competent engineer who writes their first distributed system will make mistakes the expert would never make because they haven’t yet built the mental model that only comes from doing the work and suffering the consequences.

What made this work was consequence. Writing code gave you direct feedback. A missing lock caused a race condition. A clever abstraction became unmaintainable by the third person to touch it. A shared base class six levels deep broke four products when a parent changed. These lessons were visceral, and they stuck. The Dreyfus model would say you accumulated the situational exposure that moves you from rule-following to intuition.

Books codified what masters had learned. The Pragmatic Programmer showed how to develop craft. Code Complete provided the vocabulary for code quality. A Philosophy of Software Design showed what makes modules deep or shallow. The Mythical Man-Month showed why adding engineers to a late project makes it later, coordination cost, not coding hours, drives timelines. Research quoted there put coding at roughly 14% of total project effort. Requirements, design, testing, debugging, coordination, documentation, and operations consumed the rest. Agentic coding has compressed that 14% toward zero. The other 86% remains entirely human.

The Disruption: Design and Build Are No Longer Learned Together

In traditional software development, design and build were not two separate activities. They were one activity experienced from two angles. When you wrote the code yourself, you felt every consequence of your design decisions in real time. A bad abstraction made your own implementation painful. A missing transaction boundary caused a bug you personally had to trace to its source at midnight. You didn’t just observe these consequences. You lived them, and that is what made the lessons stick.

Agentic coding severs this connection. The engineer writes a specification, the agent produces code, and the engineer reviews the result. This workflow feels productive. It is often highly productive, for engineers who already have the judgment to specify well and review rigorously. But for engineers who are still building that judgment, it removes the primary mechanism that builds engineering judgment.

The specific failure mode for junior and mid-level engineers is the plausibility trap. Agent-generated code looks correct in the narrow local context: functions are clean, tests pass, the logic holds for the cases the spec described. What the code often lacks is correctness at the system level, the consistency guarantees that span service boundaries, the failure modes that only appear under concurrent load, the invariants that hold only if you understand the domain well enough to define them. A senior engineer reviewing that code has a whole-system mental model built through years of implementation experience. They feel when something is off even before they can articulate why. A junior engineer doesn’t have that model yet, and reviewing agent-generated code without it is like trying to spot a structural flaw in a building you’ve never seen the blueprints for.

Bertrand Meyer, in his analysis in Communications of the ACM makes this point precisely: AI-generated code creates a dangerous psychological bias because it is significantly harder to spot a subtle logical flaw in well-structured generated code than in the messy human-written code reviewers are used to. Cleanliness produces false confidence. Agents write plausible code. Plausible is not the same as correct, and the gap between the two is exactly where junior engineers without deep mental models get stuck.

For senior engineers, the risk is different but equally real. Specification, design, architecture, code review, and debugging are not static skills you acquire once and keep forever. They are maintained through practice through the practice of building systems, not just reviewing them. When senior engineers stop writing code regularly, their design intuitions gradually lose contact with implementation reality. Architectural decisions start floating free from the constraints that make them achievable. Specifications become abstract rather than grounded in how things actually work. Code review gets shallower because the reviewer is no longer holding a live mental model of how the system fits together under pressure. The decay is slow and invisible until it isn’t. I had previously seen this decay when principal/staff stopped writing code and became pure architects but agentic coding is making it more prevalent.

In AI Writes Code. You Own the Design. Here’s How to Keep It That Way, I described how AI agents resemble offshore teams more than co-located colleagues: they have a narrow context window, they lack shared understanding of your codebase, they produce locally correct work that misses the bigger picture, and they have no memory between sessions. Every session starts from zero. Amazon AWS teams learned this the hard way, AI-generated code that looked right, passed review, and then caused production incidents. Their response was to significantly tighten review policies. When a production incident costs customers millions or exposes a security breach, you cannot file a bug against Cursor or Claude Code. The engineer who approved the change is accountable.

What’s at stake, underneath all of this, is shared understanding. Fred Brooks called it conceptual integrity, the coherence of design philosophy that runs through an entire system. Conceptual integrity doesn’t live in a document. It lives in the heads of engineers who have thought deeply about the system, implemented parts of it themselves, debugged its failures, and built up a shared mental model of how the pieces fit. That shared model is what gets lost when design and implementation are permanently separated. It is also the most important and hardest-to-recover thing a team can lose. Code can be rewritten. Conceptual integrity, once gone, takes years to rebuild. Instead, the system accumulates exactly what Brooks warned against: many locally reasonable decisions that don’t cohere into a coherent whole.

We cannot give up on junior and mid-level engineers developing real skills, even as agents handle more of the typing. Code reviews by senior engineers help but we mostly learn by doing, not by watching. Engineers still need to understand how code works, how it fits into the larger system, and what the trade-offs mean under real conditions. What we build and what we understand are not separate. Pulling them apart entirely is a quality risk the whole team will pay for, slowly at first and then all at once.

The Two Skill Trees

Engineering growth has always required two parallel tracks. Agentic coding affects each differently.

Hard skills are the technical capabilities: designing systems, understanding trade-offs, debugging complex failures, recognizing code smells, reasoning about correctness under concurrency, and mastering both functional and non-functional requirements. The traditional path built these incrementally through writing code, breaking things, and fixing them. Agentic coding removes that feedback loop without replacing it.

Soft skills are the interpersonal and organizational capabilities: writing clearly, building consensus, managing ambiguity, estimating honestly, communicating with non-technical stakeholders, mentoring others, and owning outcomes across an entire project lifecycle. These have always mattered for growth from mid-level to senior and from senior to principal. Agentic coding hasn’t reduced their importance, it has raised the bar, because the differentiating value of a senior engineer shifts away from code production and toward judgment, communication, and design thinking.

Both tracks require deliberate practice. The diagram below shows the full skill landscape across career levels, mapped to the hard/soft split.

The Career Levels in Detail

Before getting to specific advice, it helps to have a clear picture of what each level actually requires.

Junior: You own software components and work on well-defined problems. You produce high-quality code under guidance, learn from review feedback. You collaborate across the full development lifecycle including code, tests, deployment, documentation but you rely on peers and managers for guidance on design. You are expected to deliver reliably within a clear scope, and you actively seek to learn.

Mid-level: You are an autonomous contributor, owning features, not just components. You design software solutions for difficult problems, though you still seek guidance on architectural strategy. You coach junior engineers. You make priority trade-offs between feature work and operational work. You participate meaningfully in code reviews not just catching bugs, but providing direction.

Senior: You lead multi-engineer projects and own team-level architecture. You work on complex problems with multiple conflicting constraints. You write for both technical and non-technical audiences. You solve problems that don’t yet have a defined technology strategy. You balance short-term delivery against long-term architectural health. You become a force multiplier and your presence should make everyone meaningfully better.

Staff / Principal: You lead across an organization, not just a team. You define technical strategy and roadmaps that span multiple teams. You take on intrinsically hard problems like major bottlenecks and undefined high-impact opportunities. You align teams toward cohesive technical visions. You earn influence through credibility and results, not title.

Junior engineers are largely tactical. Seniors span tactical and operational. Staffs/Principals are primarily operational and strategic.

Hard Skills: What to Build Deliberately

1. Learn to Write Specifications

The most immediately practical hard skill in the agentic era is writing precise specifications. Agents produce what you specify. Vague specifications produce code that fills gaps with training-data assumptions, which may or may not match your domain. This is a learnable craft.

I wrote you-got-skills framework to demonstrate how to build specification skills with use of RFC 2119 discipline. Write a one-page spec before prompting an agent, every time. For significant features, use the full design document structure I previously shared: problem statement, proposal with trade-offs, alternatives considered, non-functional requirements, and rollout plan. The act of writing this forces you to make decisions you were previously leaving implicit.

2. Build a Design Sense

One of the clearest failure patterns in junior and mid-level engineers using agentic coding is an underdeveloped design sense. They can describe what they want. They struggle to explain why one design is better than another, or to recognize when generated code silently violates conceptual integrity, which is identified in The Mythical Man-Month as the single most important property of a well-designed system.

Build this sense deliberately. Read A Philosophy of Software Design and practice the deletion test: if you deleted this module, where would the complexity go? Deep modules with small interfaces earn their place. Shallow pass-throughs add indirection without value. AI defaults to shallow modules, lots of small classes, each delegating to the next. Learning to recognize this pattern and push back on it is a concrete skill you can develop right now.

In Applying Domain-Driven Design and Clean/Hexagonal Architecture to MicroServices, I shared how Domain-Driven Design can employed for an application architecture. When AI generates code for your domain, it has no idea what your domain means. Practice making invalid states unrepresentable. Sum types that enumerate valid states, state machines that encode valid transitions, parse-don’t-validate at boundaries, These design patterns matter more in the agentic era because the compiler becomes your code reviewer when humans can’t catch everything. When AI generates code within a well-typed system, category errors that would slip through casual review become compile errors.

Study the Stable Dependencies Principle: depend in the direction of stability. As illustrated in the reusability trap analysis, the most expensive bugs often don’t come from duplicated code, they come from code shared prematurely. Recognizing when DRY has become a liability is a senior-engineer skill that requires real practice to develop.

3. Develop a Nose for Code Smells and Code Review

Reviewing AI-generated code is not casual reading. Agents write clean, plausible code. The bugs that slip through are not obvious, they’re missing idempotency tokens, race conditions that appear only under concurrent load, enum values that propagate without being handled by all consumers.

Build a structured review practice. Apply two explicit passes. The first pass looks for correctness and security: logic errors (off-by-one, null handling, TOCTOU races), security holes (injection, missing auth checks, hardcoded secrets), data loss risks, and error swallowing. The second pass looks for design: are modules deep or shallow? Are invalid states representable in the type system? Does this code separate commands from queries? Is the complexity justified by the actual problem, or has the agent added abstractions for a feature used by twelve people?

Practice this on every pull request you review, whether AI-generated or human-written. The structured passes build the intuition that experienced engineers call a “nose for code smells”.

4. Master Non-Functional Requirements

Most junior engineers understand functional requirements. Senior engineers understand non-functional requirements , how reliably, under what conditions, and with what failure behavior. This is arguably the most important distinction on the path from mid-level to senior.

When you read a feature request, train yourself to immediately ask: what is the latency budget, and at what percentile? What is the consistency model between these two data stores? What happens if this operation half-succeeds? What’s the blast radius if this component fails completely? What happens at 10x current load? These questions are what agents cannot answer from a vague prompt. In Failures in MicroService Architecture, I shared a number of production issues that I experienced with distributed systems. You can apply the outbox pattern, circuit breaker, retry with jitter, bulkheads and other patterns to remedy common production issues.

5. Keep Your Hands in the Code

Agentic coding creates pressure to delegate implementation entirely. Resist it. You do not build judgment about systems you have never built yourself. Write the spike yourself before committing to full design. Implement the critical path at least once, even if an agent later handles the boilerplate. Trace the execution of generated code in a debugger until you understand what it actually does before approving it for production.

This matters most for debugging. When something fails in production, the mental model you’ve built through implementation is what lets you form hypotheses quickly. Engineers who have only reviewed AI-generated code without deeply understanding it will struggle to diagnose the failures that code produces. The CACM analysis shows that AI-generated code introduces logical and concurrency bugs in clean-looking code that humans find harder to spot than equivalent bugs in messy human-written code.

Furthermore, as agentic coding produces more and more code we don’t fully own mentally, understanding decay sets in. Storey calls this cognitive debt. A team can have low technical debt while sitting on a mountain of cognitive debt where no one can confidently predict the impact of a change. Over time, no single engineer holds the complete picture of how the system works. This makes production incidents progressively harder to diagnose. Keeping your hands in the code, owning critical-path implementations, using agents to explain generated code you don’t immediately understand.

6. Learn Formal Methods Basics

One underappreciated direction for junior/mid-level engineers is to begin learning specification and verification techniques, not as academic exercises but as practical tools for the agentic era. I shared my experience applying TLA+ for specifications in Beyond Vibe Coding: Using TLA+ and Executable Specifications with Claude. But, you can start with property-based testing: instead of writing examples, write invariants that your system must maintain regardless of input. Start with static analysis tools and learn to interpret what they find. Write explicit pre-conditions and post-conditions for complex functions, even as comments. These habits build the specification discipline that makes your agent prompts more precise and your reviews more effective.

Soft Skills: What Separates Mid-Level from Senior from Principal

7. Write with Precision and Clarity

Writing is the highest-leverage soft skill for any engineer who wants to grow. Design documents, post-mortems, and stakeholder communications all require the same underlying capability: translating technical thinking into prose that creates shared understanding.

Practice this deliberately. Write a design document for every significant thing you build, using the full structure described in How Not to Write a Design Document: problem statement, proposal, trade-offs, alternatives considered, non-functional requirements, rollout plan. Show it to a senior engineer. Ask what questions it fails to answer. Design documents are also how you develop design skills. A bad design doc does exactly what a bad design does: it makes the solution sound inevitable, skips trade-offs, and pushes hard questions into implementation. That feels fast until production starts collecting interest on every shortcut.

8. Bring Clarity to Ambiguity

The most important skill a senior engineer develops is the ability to look at a fuzzy problem and make it concrete. This is the single most valued contribution that humans still provide in the agentic era: not the code, but the thinking that makes the code correct.

Ambiguity reduction works in both directions. On the problem side: understand the actual customer need before finalizing a solution, push back on specs that describe a solution rather than a problem, ask what the real constraint is. On the solution side: identify which design decisions are reversible versus which are one-way doors. Practice this in every design review, every planning discussion, every incident retrospective.

9. Build Alignment and Consensus

The transition from proficient to expert in any domain requires operating at the social level: building consensus among people with competing interests, aligning a technical direction through an organization that has other priorities, and navigating disagreements constructively. The trust equation from Maister et al. shows that trust has four components: credibility, reliability, intimacy (safety), and self-orientation (does it serve the system or you?). Engineers who lose influence at the senior and principal level almost always fail on the fourth element. Proposals that come across as serving “my architecture” rather than “our actual problem” collapse trust fast.

Build alignment by listening before proposing. Spend time understanding what actually hurts the team before advocating for a technical direction. Frame proposals in terms of reduced toil, reduced uncertainty instead of architectural purity. Find a long-standing pain and solve it visibly. The Aikido principle from Jerry Weinberg applies here: center, enter, turn. First be aware of yourself and what you want to accomplish. Then enter the world of the other person. Then together turn the energy in a more effective direction.

10. Communicate Upward in Business Terms

Translating technical decisions into business impact separates senior engineers from principals. The ability to tell a VP concisely, what the risk is, and what it costs to address it. Learn the metrics that matter to leadership: revenue impact, customer retention, incident cost, deployment frequency, engineer productivity. Practice expressing technical proposals in those terms. This is the failure mode highlighted in How Senior Engineers Lose Trust: communicating technical complexity without translating it into business impact, focusing on engineering outputs.

11. Estimate Honestly and Decompose Work Well

Engineers who consistently underestimate erode trust. Engineers who consistently overestimate become known as blockers. Honest estimation with explicit uncertainty ranges, clear assumptions, and candid identification of the biggest risks is a key skill.

Three practices make estimation better. First, decompose into vertical slices, not horizontal layers. A vertical slice cuts through all layers and produces something independently demoable. Horizontal slicing delays feedback as you don’t know if the feature works until the last layer is complete. Second, use three-point estimation for commitments: (Best + 4×MostLikely + Worst) / 6, and present ranges rather than single numbers. Capacity is never 100%. Budget explicitly for KTLO like operational work, incident response, and technical debt.

12. Own Outcomes Beyond Your Code

The clearest signal of an engineer ready for senior responsibility is willingness to own the work nobody wants to do: the failing test that has been skipped for months, the runbook that was never written, the technical debt accumulating in the corner nobody touches, the onboarding documentation that every new hire struggles with. This is what some call being the janitor, taking responsibility for team health and code health. It builds organizational trust faster than any individual feature. Own incidents that aren’t yours. When a production problem occurs on your team, treat it as your problem regardless of who wrote the code. In Writing Post Mortems That Actually Make You Better: A Practitioner’s Guide, I explained how to use the Five Whys and the Swiss Cheese model for documenting incident post-mortems.

13. Become a Go-To Person

Focused expertise builds the kind of reputation that earns you higher-impact work. The path to being a go-to person has three branches: project ownership, technology expertise, and domain expertise. Pick one to start. Host a learning session on something you know well. Write about it internally. Help others who are stuck on it.

14. Mentor Others

Teaching is one of the fastest ways to consolidate your own understanding. When you explain a design decision to a junior engineer, you discover exactly what you do and don’t understand. When you give code review feedback that helps someone see a flaw they missed, you sharpen your own eye.

In the agentic era, junior engineers need mentorship more than ever because the traditional mechanism of learning through building and breaking code is less available. Senior engineers who help juniors understand why AI-generated code works the way it does, how to critique it structurally, and how to reason about trade-offs are providing something genuinely important. The psychological safety research from Google’s Project Aristotle applies here: teams where members feel safe raising concerns, asking questions, and challenging designs outperform teams where they don’t. You build that culture one mentoring conversation at a time.

The T-Shape and Broken Comb Model

The most useful framework for thinking about hard skill investment is the T-shape: one area of genuine depth combined with broad familiarity across adjacent areas (the horizontal bar). As engineers progress toward principal level, the shape often becomes what practitioners call a broken comb, multiple verticals of depth across different domains, connected by broad horizontal understanding. A principal engineer might go deep in distributed systems, in observability, and in the security model of their specific domain, while maintaining enough breadth to lead design conversations across the full stack.

A Concrete Self-Guided Growth Plan

Here is a practical, time-bounded path for engineers at each stage.

If you are a junior engineer (0–3 years):

• Write a one-page spec before prompting an agent. Compare what the agent produced to what you specified.
• Ask to implement at least one non-trivial feature entirely yourself, even if it takes longer.
• Read The Pragmatic Programmer and Code Complete.
• Request structured feedback on every code review you submit.
• Use agents to explain generated code you don’t understand.

If you are a mid-level engineer (3–6 years):

• Write a full design document for the next significant feature you build. Share it with a senior engineer and ask specifically what questions it fails to answer.
• Own one domain on your team completely: its documentation, its monitoring, its failure modes, its onboarding.
• Start hosting one internal learning session per quarter on something you know well. Write it up afterward.
• Apply a structured two-pass review to every pull request you review. Track what you catch over a month.
• Read A Philosophy of Software Design. Apply the deletion test and bounded context thinking to your current codebase.
• Write one post-mortem per incident using the Five Whys structure.

If you are a senior engineer aiming for staff/principal:

• Lead one project that coordinates work across multiple engineers. Own the design and run the design review. Drive the post-project retrospective.
• Translate one technical proposal into business impact language: metrics, incident cost, customer effect.
• Mentor junior engineers specifically in how to critically evaluate AI-generated code.
• Identify the most painful systemic problem on your team, the thing everyone complains about and nobody fixes. Fix it, document it, and share what you learned.

Ongoing, at every level:

Keep your hands in the code. The fraction of code that engineers write themselves will keep shrinking, but understanding what the code does, how it fits the larger system, and what its failure modes are requires someone who can read it critically, reason about it deeply, and debug it under pressure.

What We Cannot Give Up

There is real pressure in many organizations to reduce engineering involvement in requirements, design, and review to automate the entire lifecycle. This deserves serious assessment. Agents accelerate delivery but they do not absorb accountability. When code fails in production, the customer doesn’t care whether the bug was introduced by a human or a model. The engineer who approved it is responsible. Code review, even partially automated still requires human engineers who understand the system well enough to know what they’re reviewing. Junior engineers who bypass the developmental stages that build that understanding will produce reviews that miss what matters. Organizations that accept this trade-off in exchange for short-term velocity will eventually pay compounding interest.

The goal is not to resist agentic coding. The productivity gains are real and the trend is irreversible. The goal is to keep all three in check: technical debt in the code, cognitive debt in the team’s shared understanding, and intent debt in the artifacts. Agentic coding, used carelessly, accelerates all three simultaneously.

Further Reading

• AI Writes Code. You Own the Design.
• How Not to Write a Design Document
• Writing Post Mortems That Actually Make You Better: A Practitioner’s Guide
• The Reusability Trap: When DRY Becomes a Liability
• Key Takeaways from Leading Effective Engineering Teams
• Insights from Become a Great Engineering Leader
• Highlights from The Engineering Executive’s Primer
• you-got-skills structured SDLC skill framework for agentic development

I have worked for both large tech companies and startups. Two patterns kept showing up across every company I worked at startup and large company alike that both punish the engineers doing the right thing.

At startups, the pressure is entirely on shipping features. Engineers who move fast and ship constantly get rewarded. Security, observability, scalability become “future problems.” The engineers who slow down to build things properly, who push back on cutting corners, get treated as obstacles. The corners get cut anyway. When the system eventually breaks under load or gets breached, nobody connects it back to the decisions made two years earlier. The engineers who raised concerns are long gone or drowned out.

At large companies, a different trap. Ship something clean with simple design, solid implementation, few follow-up bugs and people move on. Nobody notices the problems that didn’t happen. Nobody gets promoted for the outages that never occurred. But ship something overengineered, watch it fall apart in production, spend months firefighting and suddenly you’re a hero. The tech lead who pushed patches at 2am gets noticed. Management reads the complexity as evidence of a hard problem solved. The tech lead gets promoted and moves to the next team. The engineers left behind inherit the mess.

Same outcome, different path. In both cases, the engineers who built things well are invisible. The ones who created the problems or thrived on them get ahead.

Essential vs. Accidental Complexity

In The Mythical Man-Month, Fred Brooks defined two kinds of complexity. Essential complexity is the irreducible difficulty built into the problem domain itself. Accidental complexity is the difficulty we add through poor abstractions, unnecessary coupling, and artificial layers. Larry Tesler’s Law of Conservation of Complexity says essential complexity can’t be eliminated, only moved. Push it out of the user interface and it lands in your middleware.

What most companies reward the accidental kind. Many moving parts, multiple failure modes, a fleet of services with their own deployment pipelines as these look like a hard problem solved by smart engineers. A system that just works, simply and reliably, signals nothing. The people who built it must have been working on something easy. I saw this repeatedly at larger companies. Senior engineers with years of incremental, principled improvements couldn’t get promoted because their work wasn’t considered “complex enough.” The implicit rule was clear: elegance doesn’t get you promoted.

War Stories

The database migration that became a platform. At a large tech company, we needed a simple migration from one database to another but it turned into a real-time data synchronization system. Suddenly there were shadow testing components, reconciliation pipelines, anti-entropy jobs for fixing discrepancies, and runbooks for each failure mode. The project stretched from months into years. The original problem, move data from A to B, never required any of it. But the complexity generated headcount, resources, and career advancement that a clean migration would never have produced.

The microservices migration that never finished. A monolith-to-microservices transition ran so long the team ended up maintaining both systems simultaneously. The migration date kept slipping. Nobody could tell you which services were fully cut over. The codebase became a graveyard of abandoned halfway points. Years of engineering time consumed, several promotions justified. The engineers who eventually inherited it had no idea what was intentional and what was just never cleaned up.

The Erlang rewrite. At a FinTech company, senior executive decided to rewrite an order management system from Java to Erlang, not for a specific technical reason, but because Erlang was interesting. Brooks called this the second-system effect: when engineers rewrite something they think they now understand, they pile in everything they held back the first time. The effort was far larger than anyone expected. Management abandoned it partway through. The team was left with two halves of the same system in two different languages, domain knowledge split across both.

The Go rewrite. The same executive years later decided to rewrite a Java financial system in Go because Go was what the industry was talking about. Years passed, the migration stalled. Some parts in Go, most still in Java. The team gave up. Meanwhile the actual urgent problems like data consistency, observability, performance at scale went unaddressed because everyone’s attention was on the rewrite. Nobody owned the full picture of dependencies or understood the consistency guarantees. Meanwhile, sales sold the system as a low-latency and four nine availability but in practice it was based on false illusion due to poor observability.

The postscript at that second company: when AI became the new shiny thing, the pattern played out again. Engineers who built flashy demos got promoted. The people fixing real infrastructure problems had nothing visible to show.

Conceptual Integrity Breaks Down as Organizations Grow

In the original Mythical Man-Month, Brooks argued that the most important property a system can have is conceptual integrity, one coherent design philosophy, with someone who holds the whole system in mind and says no to things that don’t fit. His prescription was a chief architect with real authority over what goes in and what stays out. That works when one person can still comprehend the system. As organizations grow and systems get divided among teams, nobody has that view anymore. Each team makes locally reasonable decisions. Accidental complexity accumulates not from individual mistakes but from the disconnect between groups who can’t see each other’s work.

Cross-cutting concerns like security, authentication, observability are where this gets dangerous fastest. I saw one system where authentication behaved differently depending on whether you were on-premises or in the cloud, and whether you were hitting the control plane or data plane. Secrets in some places, JWTs in others, config files in some environments, environment variables in others, a wall of conditional logic tying it together. No single person understood the whole thing. That mess led to a significant security breach and customer churn. Nobody designed it. It grew, one locally reasonable decision at a time.

Two Different Failure Modes

Startups and large companies both get this wrong, but for opposite reasons.

Startups are under pressure to ship customer-facing features. Security, observability, performance, operational burden become “future problems.” Sometimes that’s the right call. A startup that dies building the perfect architecture ships nothing. But the technical debt from ignored non-functionals doesn’t disappear. It accumulates, and it usually arrives all at once right when the company is trying to scale. That’s the worst possible time to deal with it.

Large companies have the opposite problem. The incentive structure rewards visible complexity. Tech leads propose ambitious architectures, staff up around them, ship something complicated, and move to the next team before the consequences mature. The engineers who inherit the system didn’t choose the design, can’t fully explain it, and can’t safely simplify it because they don’t understand what each piece is actually doing.

In both cases, the people who make the architectural decisions aren’t around to live with them. That gap between decision and consequence is the core of the problem.

The Goldilocks Principle

The approach that actually works is simpler than it sounds: start with the least complex architecture that handles the real requirements. Add complexity only when something forces you to.

Not simple for its own sake, e.g., if the domain genuinely requires distributed coordination, the design should say so. But the default should be: prove the complexity is necessary before building it. “This is how I’ve seen it done at bigger companies” and “this technology is interesting” are not justifications. Neither is designing for scale you don’t have. I’ve watched teams build for ten million users when they had ten thousand, then spend two years maintaining infrastructure that served no real requirement.

Vertical slices enforce this discipline. When you ship thin, end-to-end cuts of real functionality that a user can actually touch then you find out fast whether your design is right. The feedback loop is short. A wrong assumption costs a week, not six months. You can correct before the mistake becomes load-bearing.

AI Accelerates This Problem

With tools like Claude Code and Cursor, the implementation bottleneck is largely gone. A team using AI assistants can build a distributed system with five services in the time it used to take to build one. That’s progress if the design is right. If the incentive structure still rewards accidental complexity, AI just produces it faster.

In When Copying Kills Innovation: My Journey Through Software’s Cargo Cult Problem, I shared the cargo-cult behavior like adding components because they look sophisticated happens at higher velocity now. An AI agent given a vague prompt and no design constraints defaults to patterns common in its training data. That means microservices when a monolith would do, event buses when a direct call would do, five abstractions where two would do.

As I wrote in AI Writes Code. You Own the Design., the thinking parts like the what and why can’t be delegated to an agent. AI handles the how. Engineers who can identify essential complexity, strip the accidental kind, and hold a design together are more valuable now than before. But only if the organization’s reward structure reflects that.

How Do You Fix the Reward Structure?

I don’t have a clean answer. But here’s where the levers are.

• Reward outcomes, not artifacts. Most promotion processes credit visible artifacts: the design doc for a complex system, the heroic incident response, the fleet of services owned. The outcomes that actually matter, a system that stayed up for two years, a migration that finished in six weeks, a design that five new engineers understood on day one are harder to see and usually go uncredited. Engineering leaders have to explicitly define what good engineering looks like and measure it over time horizons long enough to see consequences.
• Make accountability follow decisions. Connect tech leads to the consequences of their architectural choices twelve to eighteen months later. Not as punishment as designs fail for unforeseeable reasons. But an engineer who never sees what their decisions cost never updates their model. Right now the feedback loop doesn’t exist for most people who make these calls.
• Credit the “no.” The engineers who prevent bad architectures from being built are the hardest to recognize. The bad system was never built, so there’s nothing to point to. If you want more of this behavior, name it explicitly and credit it explicitly. Otherwise the rational move for any ambitious engineer is to propose the complex thing and let someone else clean it up.
• Add a simplicity lens to design reviews. Most design reviews ask: will this work? Fewer ask: is this more complex than it needs to be? Formally asking “what would we remove without losing essential functionality?” changes the conversation. The burden of proof shifts to adding a component, not removing one.

The Conversation Worth Having

Brooks wrote that conceptual integrity is the most important consideration in system design. What the book doesn’t address is that most organizations are structured to undermine it like rewarding the engineers who add complexity and moving them on before they face the consequences. The engineers who hold the line against unnecessary moving parts, who ship systems that work quietly for years, who say “we don’t need this” and mean it are doing some of the hardest work in software. In most companies, they’re not the ones getting promoted.

With AI accelerating the implementation layer, the judgment required to distinguish essential from accidental complexity matters more than it ever has. If the reward structure doesn’t change to reflect that, we’ll just build the wrong things faster.

Related reading:

• How Abstraction Is Killing Software.
• When Copying Kills Innovation.
• AI Writes Code. You Own the Design.