Shahzad Bhatti Welcome to my ramblings and rants!

Introduction

Every major cloud now supports Serverless FAAS capabilities like AWS Lambda/Step Functions, Azure Durable Functions, GCP Cloud Functions and Cloudflare Durable Objects where you write a function or a small stateful actor. This allows you to scale it, pay only for what runs but there is a catch, you build on a proprietary runtime, and the runtime’s storage model, invocation model, and IAM rules become part of your application whether you meant them to or not. You cannot easily rewrite it or run it somewhere else. I saw a recent post (Why we’re moving Wire off Cloudflare Durable Objects) from Wire, which ran every container on Cloudflare Durable Objects since day one. They wrote why they rebuilt their own data plane instead of staying, which included extra network hops on the hottest path, drift of state, separation of compute from data, rigid placement policies and lack of self-hosting. None of these are reliability complaints, instead they’re architectural ceilings baked into a runtime you don’t own. And the pattern generalizes past Cloudflare:

• AWS Lambda: SAM and LocalStack approximate the runtime locally, but diverge on execution environment, IAM, and VPC behavior.
• Azure Durable Functions: Azurite emulates the storage layer, but the replay-based orchestration engine behaves differently under real concurrent load than it does in the emulator.
• GCP Cloud Functions: the Functions Framework runs locally, but Eventarc, Pub/Sub push, and Cloud Run triggers all need live GCP resources.
• Cloudflare Workers/DO/Agents: wrangler dev simulates KV with SQLite and alarms with in-process timers, but never replicates the distributed routing that decides which data center actually holds your object.

Every one of these runtimes gives you a great abstraction and takes your operational sovereignty in exchange. This post shows how to keep the abstraction like stateful actors, durable storage, alarms, WASM sandboxing, LLM calls, observability, an event bus, webhooks while running it on infrastructure you control with an open-source framework called PlexSpaces.

PlexSpaces

PlexSpaces is an open-source, polyglot actor framework that gives every actor durable KV storage and alarms, routes messages between actors on one node or across a gRPC mesh, and exposes a host API to Go, TypeScript, Python, and Rust. You write an actor once; the same WASM module deploys to your laptop, a Docker container, Kubernetes, bare metal, or several clouds at once, unchanged. The mental model sits close enough to Cloudflare Durable Objects that migrating existing DO code is mostly mechanical. The key difference: there’s no simulated version to diverge from production, because the production runtime is the development runtime.

Part I: Durable Objects

Cloudflare publishes a short list of rules for writing correct Durable Objects, which are easily mapped to a PlexSpaces constraint, and for most of them the mapping is tighter:

The one meaningful difference: Cloudflare guarantees globally unique placement. PlexSpaces virtual actors are unique per node or per cluster when pinned with @* (any node) or @nodeId (specific node). PlexSpaces provides an object-registry for managing cross-cluster deduplication.

Durable KV Storage

Cloudflare’s ctx.storage gives you a transactionally consistent store scoped to one object:

// Cloudflare DO — JavaScript
const count = await this.ctx.storage.get("count") ?? 0;
await this.ctx.storage.put("count", count + 1);

// Batch operations (storage.get([keys]) / storage.put(map))
const [history, meta] = await this.ctx.storage.get(["room:history", "room:meta"]);
await this.ctx.storage.put(new Map([["room:history", data], ["room:meta", index]]));

PlexSpaces gives you the same guarantee through host.kv. Because each actor processes one message at a time, a plain read-modify-write is safe without extra locking. The migrating_cloudflare_workers TypeScript example restores room history on onInit() using batch KV similar to blockConcurrencyWhile:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
protected override onInit(config: Record<string, unknown>): void {
    this.state.room_id = String(config.actor_id ?? "");

    // blockConcurrencyWhile() equivalent — batch-fetch persisted state before first message
    const keys = [
        "room:" + this.state.room_id + ":history",
        "room:" + this.state.room_id + ":meta",
    ];
    const values = host.kv.multiGet(keys);   // like DO storage.get(["k1","k2"])
    const [historyRaw] = values;
    if (historyRaw) {
        const msgs = JSON.parse(historyRaw);
        if (Array.isArray(msgs)) {
            this.state.messages = msgs;
            this.state.message_seq = msgs[msgs.length - 1]?.seq ?? 0;
        }
    }
}

private persistHistory(): void {
    // Batch write — like DO storage.put(new Map([["k1",v1],["k2",v2]]))
    host.kv.multiPut({
        ["room:" + this.state.room_id + ":history"]: JSON.stringify(this.state.messages),
        ["room:" + this.state.room_id + ":meta"]: JSON.stringify({
            message_seq: this.state.message_seq,
            last_updated: host.nowMs(),
        }),
    });
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
def _load_history(self) -> None:
    room_id = self._room_id()
    index_raw = host.kv.get(f"room:{room_id}:seq_index")
    if index_raw:
        seqs = json.loads(index_raw)
        keys = [f"room:{room_id}:msg:{seq}" for seq in seqs]
        values = host.kv.multi_get(keys)   # DO storage.get([k1,k2,...]) equivalent
        self.messages = [json.loads(v) for v in values if v]
        if self.messages:
            self.msg_seq = self.messages[-1]["seq"]

def _persist_history(self) -> None:
    room_id = self._room_id()
    entries = {}
    seqs = []
    for msg in self.messages:
        entries[f"room:{room_id}:msg:{msg['seq']}"] = json.dumps(msg)
        seqs.append(msg["seq"])
    entries[f"room:{room_id}:seq_index"] = json.dumps(seqs)
    host.kv.multi_put(entries)   # DO storage.put({k:v,...}) equivalent

PlexSpaces also ships atomic operations that Cloudflare leaves you to build yourself with a coordinator object:

// PlexSpaces TypeScript — from RateLimiterActor in guild_chat_actor.ts
// Atomic distributed counter — equivalent to Cloudflare KV atomic increment
const distributedCount = host.kv.increment(windowKey, 1);

// Compare-and-swap — equivalent to DO transactional storage read-modify-write
const applied = await host.kvCas("lock_key", expectedValue, newValue);

// KV with TTL — equivalent to DO storage.put with metadata expiration
await host.kvPutWithTtl("session_token", token, 3600);

The Python RateLimiterActor shows both in context:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("check")
def check(self, user_id: str = "") -> dict:
    # Atomic increment — survives actor restarts, equivalent to Cloudflare KV atomic
    host.kv.increment(f"rate:{user_id}:total", 1)

    # CAS — idempotent token slot reservation, like DO transactional put
    cas_key = f"rate:{user_id}:window"
    current_val = host.kv.get(cas_key) or ""
    host.kv.cas(cas_key, current_val, str(host.now_ms()))

    # Token bucket logic operates on in-memory state (safe: single-threaded actor)
    allowed = self.buckets[user_id]["tokens"] > 0
    if allowed:
        self.buckets[user_id]["tokens"] -= 1
    return {"allowed": allowed, "remaining": self.buckets[user_id]["tokens"]}

Durable Alarms

A DO schedules one future callback that survives node restarts:

// Cloudflare DO
await this.ctx.storage.setAlarm(Date.now() + 10_000);

async alarm() {
    const count = await this.ctx.storage.get("count");
    await this.ctx.storage.delete("count");
    console.log(`Processing ${count} batched requests`);
}

PlexSpaces maps this directly through ReminderFacet, which persists the alarm to durable storage and re-queues it automatically after a restart. The AlarmDemoActor in the guild-chat example demonstrates the full lifecycle like set, query, fire, cancel:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
onEnqueue(_payload: Record<string, unknown>): Record<string, unknown> {
    this.state.queued++;
    if (this.state.queued === 1) {
        // First item — equivalent to: await this.state.storage.setAlarm(Date.now() + 30_000)
        host.alarm.set(host.nowMs() + 30_000);
    }
    return { status: "ok", queued: this.state.queued };
}

// Fires when the scheduled timestamp is reached
// Equivalent to Cloudflare DO: async alarm() { ... }
on__alarm__(_payload: Record<string, unknown>): Record<string, unknown> {
    const processed = this.state.queued;
    this.state.processed += processed;
    this.state.queued = 0;
    this.state.total_alarm_fires++;
    return { status: "ok", processed };
}

onStatus(_payload: Record<string, unknown>): Record<string, unknown> {
    const alarmAt = host.alarm.get();   // DO: await this.state.storage.getAlarm()
    return { ...this.state, alarm_at: alarmAt, alarm_set: alarmAt > 0 };
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("start")
def start(self, delay_ms: int = 30000) -> dict:
    # Equivalent to: this.state.storage.setAlarm(Date.now() + delay_ms)
    host.alarm.set(host.now_ms() + delay_ms)
    return {"status": "ok", "fire_at_ms": host.now_ms() + delay_ms}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Equivalent to Cloudflare DO: async alarm() { ... }
    processed = len(self.pending_requests)
    self.total_processed += processed
    self.pending_requests = []
    return {"status": "ok", "processed": processed}

@handler("cancel")
def cancel(self) -> dict:
    host.alarm.delete()   # DO: this.state.storage.deleteAlarm()
    return {"status": "ok", "action": "alarm_cancelled"}

Get-or-Create (Virtual Actors)

Cloudflare’s core abstraction is the globally unique object that appears on first access, at the cost of a binding declared in wrangler.toml:

// Cloudflare DO — needs a binding in wrangler.toml
const id = env.CHAT_ROOM.idFromName(roomId);
const room = env.CHAT_ROOM.get(id);
await room.fetch("/send", { method: "POST", body: JSON.stringify(msg) });

PlexSpaces virtual actors give you the same behavior with no binding file. The actor ID ({name}//{actorType}::{namespace}@*) encodes both the shard key and the actor class, and the @* suffix lets the runtime place it on whichever node is best; pin it with @node-id for data locality:

// PlexSpaces TypeScript
import { getActorRef } from "@plexspaces/sdk";
const room = getActorRef("ChatRoomActor", roomId, "default");
const reply = await room.ask("send_message", { user_id: userId, content: msg });

# PlexSpaces Python
from plexspaces.host import get_actor_ref
room = get_actor_ref("ChatRoomActor", room_id, "default")
reply = room.ask("send_message", {"user_id": user_id, "content": msg}, timeout_ms=5000)

The actor spins up on its first message, whether or not it existed before.

Listing Actors by Namespace

Cloudflare exposes a Durable Objects namespace list API for management and observability. PlexSpaces has a direct equivalent defined in its proto-first design. The ListActors RPC in actor_runtime.proto accepts namespace, actor_type, state, and node_id filters and returns paginated results:

// proto/plexspaces/v1/actors/actor_runtime.proto
message ListActorsRequest {
    string actor_type = 3;
    ActorState state  = 4;
    string node_id    = 5;
    // Namespace for tenant isolation — only actors in this namespace are returned
    string namespace  = 6;
    PageRequest page_request = 2;
}

message ListActorsResponse {
    repeated Actor actors        = 2;
    PageResponse page_response   = 3;
}

From a PlexSpaces client, listing all active ChatRoomActor instances in the default namespace:

# HTTP API (equivalent to Cloudflare's list-objects endpoint)
curl "http://localhost:8080/api/v1/actors/default/ChatRoomActor?state=active"

// Go SDK
actors, err := client.ListActors(ctx, &ListActorsRequest{
    ActorType: "ChatRoomActor",
    Namespace: "default",
    State:     ActorStateActive,
})

Cloudflare limits listing to metadata (ID, location, storage size). PlexSpaces returns full Actor records including state, facets, node assignment, resource usage, and tenant/namespace tags.

WebSocket Handling

Cloudflare’s Model

Cloudflare gives the Durable Object two WebSocket modes. In the standard model, the object holds the socket directly:

// Cloudflare DO — standard WebSocket
async fetch(request) {
    const [client, server] = Object.values(new WebSocketPair());
    this.ctx.acceptWebSocket(server);
    return new Response(null, { status: 101, webSocket: client });
}

async webSocketMessage(ws, message) {
    for (const peer of this.ctx.getWebSockets()) {
        peer.send(`broadcast: ${message}`);
    }
}

async webSocketClose(ws, code) {
    ws.close(code, "connection closed");
}

The WebSocket Hibernation API (state.acceptWebSocket / getWebSockets()) is Cloudflare’s optimization for objects that hold many sockets but are mostly idle: the object is evicted when no message is being processed, and WebSocket state is restored from durable storage on the next message.

PlexSpaces: A Cleaner Split

PlexSpaces takes a different approach: room state and connection state are in separate actors. A ChatRoomActor holds only membership and message history; each browser connection is a thin-node client registered under its own actor ID. When the room fans out, host.send(actorId, "chat_message", event) routes each delivery through the WsActorTransportClient to the right WebSocket session like the room never holds a socket handle.

This is the same split Discord uses internally in its Elixir stack, where session processes are separate from guild processes. Here’s the real onSend handler from examples/typescript/apps/ws_chat_room/:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onSend(payload: SendPayload): unknown {
    const senderUsername = this.state.members[payload.sender_actor_id] ?? payload.sender_actor_id;
    const ts = host.nowMs();
    this.state.history.push({
        senderActorId: payload.sender_actor_id,
        sender: senderUsername,
        text: payload.text,
        ts,
    });
    if (this.state.history.length > MAX_HISTORY) {
        this.state.history = this.state.history.slice(-MAX_HISTORY);
    }

    const event = {
        sender: payload.sender_actor_id,
        sender_username: senderUsername,
        text: payload.text,
        room_id: this.state.roomId,
        ts,
    };
    // Fan out to all members including sender (delivery confirmation)
    // host.send() routes each tell through WsActorTransportClient ? WsRegistry ? thin-node WS session
    const memberIds = Object.keys(this.state.members);
    for (const actorId of memberIds) {
        host.send(actorId, "chat_message", event);
    }
    return { success: true, members_notified: memberIds.length };
}

A companion PresenceActor in the same file tracks online/offline state using a durable reminder (host.sendAfter) to mark a user offline after 55 seconds of silence without external cron job involved:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onOnline(_payload: OnlinePayload): unknown {
    this.state.online = true;
    this.state.last_seen = host.nowMs();
    host.kv.putJson(`presence:${this.state.userId}`, { online: true, last_seen: this.state.last_seen });
    host.sendAfter(60_000, "timeout_check", {});   // durable reminder, survives restart
    return { success: true, online: true };
}

onTimeout_check(): unknown {
    const idleSince = host.nowMs() - this.state.last_seen;
    if (idleSince > 55_000) {
        this.state.online = false;
        host.kv.putJson(`presence:${this.state.userId}`, {
            online: false, last_seen: this.state.last_seen,
        });
    }
    return { checked: true, idle_ms: idleSince };
}

Gap Analysis: WebSocket Hibernation

The PlexSpaces model costs more code upfront (session actor + room actor) and pays you back with independent scaling: fan-out scales with the number of receivers, not with the room actor’s memory; a crashed session doesn’t lock the room; reconnect logic is client-side only.

Fan-Out: DO’s Missing host.send() Primitive

In Cloudflare, cross-object fan-out means individual fetch() calls or Queues, which are not lean as a fire-and-forget tell. PlexSpaces’s host.send() is a fire-and-forget message routed through the actor mesh, no HTTP overhead, with in-process delivery for co-located actors. The Python guild-chat send_message handler shows this clearly:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("send_message")
def send_message(self, user_id: str = "", content: str = "") -> dict:
    msg = self._add_message(user_id, content, host.now_ms())

    # Fan-out: fire-and-forget to each member actor
    # Mirrors Discord's Manifold pattern for distributed fan-out
    # In Cloudflare DO, this would be individual fetch() calls — expensive
    fan_out_count = 0
    for member_id in list(self.members.keys()):
        if member_id != user_id:
            host.send(member_id, "receive_message", {
                "room_id": self._room_id(),
                "seq": msg["seq"],
                "from": user_id,
                "content": content,
            })
            fan_out_count += 1

    self._persist_history()   # batch multiPut — one KV call for the whole room
    return {"status": "ok", "seq": msg["seq"], "fan_out": fan_out_count}

Part II: Cloudflare Agents SDK

Cloudflare’s Agents SDK builds stateful AI agents on top of Durable Objects. PlexSpaces covers the same patterns; some are direct translations and a few require a different shape.

Conversation State and Memory

Cloudflare stores conversation history in the DO’s storage. PlexSpaces does the same through host.kv, with the same durability guarantee: the ChatAgentActor in examples/python/apps/chat_agent/ stores history under a well-known key and restores it across activations:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
@handler("chat")
def chat(self, message: str = "") -> dict:
    # Load history — equivalent to: await this.storage.get('history')
    history = host.kv.get_json("history") or []

    history.append({"role": "user", "content": message, "timestamp": host.now_ms()})

    assistant_reply = self._call_llm(history)

    history.append({"role": "assistant", "content": assistant_reply, "timestamp": host.now_ms()})

    # Persist — equivalent to: await this.storage.put('history', history)
    host.kv.put_json("history", history)
    self.total_messages += 1

    # Schedule summarization alarm once history is long enough
    if len(history) > _ALARM_THRESHOLD and host.alarm.get() == 0:
        host.alarm.set(host.now_ms() + _ALARM_DELAY_MS)

    return {"status": "ok", "reply": assistant_reply, "history_length": len(history)}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm()
    history = host.kv.get_json("history") or []
    summary = self._call_llm([{
        "role": "user",
        "content": f"Summarize this conversation (2-3 sentences): {json.dumps(history)}"
    }])
    host.kv.put("summary", summary)
    host.kv.delete("history")   # clear after summarizing
    return {"status": "ok", "action": "summarized", "messages_summarized": len(history)}

For long-term cross-session memory, store summaries under a user-scoped key (memory:{user_id}) and inject them into the next conversation’s system prompt similar to Cloudflare’s getMemory/setMemory implementation.

Calling LLMs

Cloudflare’s Agents SDK routes every call through env.AI, Cloudflare’s own inference gateway, locked to providers they support:

// Cloudflare Agents SDK — locked to Cloudflare's AI gateway
const response = await this.env.AI.run("@cf/meta/llama-3-8b-instruct", { messages });

PlexSpaces actors call any provider through a named HTTP service link resolved at deploy time, so the actor code never mentions a specific vendor:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
def _call_llm(self, messages):
    http = ServiceHttpClient("llm-link")
    body = {
        "model": "claude-3-5-haiku-20241022",
        "max_tokens": 1024,
        "messages": [{"role": m["role"], "content": m["content"]} for m in messages],
    }
    resp = http.post("/v1/messages", body)
    # Parse Anthropic response
    if isinstance(resp, dict):
        content = resp.get("content", [])
        if content and isinstance(content, list):
            return content[0].get("text", "")
    return "[LLM unavailable]"

app-config.toml points the link at Ollama locally, Anthropic or OpenAI in production, or an internal AI gateway in a regulated environment:

# Local development — Ollama
[[service_links]]
name = "llm-link"
url  = "http://localhost:11434"

# Production — swap without touching actor code
[[service_links]]
name = "llm-link"
url  = "https://api.anthropic.com"
headers = { "x-api-key" = "${ANTHROPIC_API_KEY}" }

TypeScript actors use the same pattern:

// PlexSpaces TypeScript — examples/typescript/apps/chat_agent/
const resp = host.httpClient("llm-link").post("/v1/messages", {
    model: "claude-3-5-haiku-20241022",
    max_tokens: 1024,
    messages,
});

Durable Workflows

Cloudflare Agents SDK ships workflow primitives that checkpoint multi-step sequences. PlexSpaces WorkflowActor gives you the same like Run, Signal, and Query RPCs map to start, inject external events, and inspect state. For example, the payment workflow in examples/go/apps/migrating_cadence/payment_workflow.go shows the shape:

// PlexSpaces Go — examples/go/apps/migrating_cadence/payment_workflow.go
// PaymentWorkflow implements WorkflowActor for idempotent payment processing.
// Steps: validate ? authorize (with retry) ? capture ? settle.
// Signals: refund, cancel.  Queries: status, payment_id.
type PaymentWorkflow struct {
    plexspaces.BaseActor
    PaymentID       string        `json:"payment_id"`
    Status          string        `json:"status"` // pending ? validated ? authorized ? captured ? settled
    Steps           []PaymentStep `json:"steps"`
    RefundRequested bool          `json:"refund_requested"`
}

func (p *PaymentWorkflow) Run(payloadJSON string) string {
    // Each step checkpoints via getState/setState before proceeding.
    // If the node crashes mid-run, the workflow resumes from the last checkpoint.
    p.Status = "validating"
    if err := p.validatePayment(); err != nil {
        p.Status = "failed"
        return marshal(map[string]any{"error": err.Error()})
    }
    p.addStep("validate")

    // Authorize with retries (idempotency key prevents double-charge)
    p.Status = "authorizing"
    for attempt := 0; attempt < 3; attempt++ {
        if err := p.authorizePayment(); err == nil {
            break
        }
    }
    p.addStep("authorize")
    // ... capture, settle
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

func (p *PaymentWorkflow) Signal(name, _ string) {
    switch name {
    case "refund":
        p.RefundRequested = true
    case "cancel":
        p.Status = "cancelled"
    }
}

func (p *PaymentWorkflow) Query(name, _ string) string {
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

For multi-agent orchestration, OrchestratorActor in examples/go/apps/miniclaw/ decomposes a task into sub-tasks, delegates each to a worker agent discovered via process group, and aggregates results through TupleSpace:

// PlexSpaces Go — examples/go/apps/miniclaw/orchestrator.go
func (o *OrchestratorActor) Run(payloadJSON string) string {
    task := stringVal(parsePayload(payloadJSON), "task", "")
    taskID := fmt.Sprintf("orch-%d", host.NowMs())

    o.Status = "running"
    o.TaskID = taskID

    // Discover available agents via process group membership
    agentID, err := pgFirst("svc:agent")
    if err != nil {
        return marshal(map[string]any{"error": "no agents in svc:agent process group"})
    }

    // Decompose and delegate sub-tasks
    subTasks := decompose(task)
    for i, subTask := range subTasks {
        o.Progress = (i + 1) * 100 / len(subTasks)
        result, err := host.Ask(agentID, "chat", map[string]any{
            "message":    subTask,
            "session_id": fmt.Sprintf("orch-%s-%d", taskID, i),
        }, 30000)
        if err != nil {
            return marshal(map[string]any{"error": "sub-task failed: " + err.Error()})
        }
        // Store result in TupleSpace for aggregation
        host.TS().Write([]any{"orch_result", taskID, i, result})
    }

    o.Status = "completed"
    return marshal(map[string]any{"task_id": taskID, "status": "completed", "sub_tasks": len(subTasks)})
}

Human-in-the-Loop

Cloudflare’s human-in-the-loop pattern pauses an agent on a high-stakes action and waits for external approval before resuming. PlexSpaces implements this natively through a GenFSM actor, e.g., ApprovalGateActor in examples/python/apps/minipi/approval_gate.py:

# PlexSpaces Python — examples/python/apps/minipi/approval_gate.py
@fsm_actor(states=["idle", "awaiting_approval", "approved", "rejected"], initial="idle")
class ApprovalGateActor:
    """
    FSM states: idle ? awaiting_approval ? approved / rejected ? idle

    Key insight: the agent can wait for days. DurabilityFacet preserves all state
    durably — no polling, no timeouts burning tokens.
    """
    fsm_state: str = state(default="idle")
    pending_request: dict = state(default_factory=dict)
    pending_agent_id: str = state(default="")

    @handler("request_approval")
    def request_approval(self, agent_id: str = "", action: str = "", context: dict = None) -> dict:
        """An agent requests human approval for a high-stakes action."""
        if self.fsm_state != "idle":
            return {"status": "busy", "current_agent": self.pending_agent_id}

        self.fsm_state = "awaiting_approval"
        self.pending_agent_id = agent_id
        self.pending_request = {"action": action, "context": context or {}, "requested_at_ms": host.now_ms()}

        # Store request for external review (dashboard, Slack notification, etc.)
        host.kv.put(f"approval_request:{self.actor_id}", json.dumps(self.pending_request))

        return {"status": "pending", "gate_id": self.actor_id}

    @handler("approve")
    def approve(self, approver: str = "", comment: str = "") -> dict:
        """Human approves — signals the suspended agent to resume."""
        agent_id = self.pending_agent_id
        self.fsm_state = "approved"
        self.decision_history.append({
            "action": self.pending_request.get("action"),
            "decision": "approved",
            "approver": approver,
            "decided_at_ms": host.now_ms(),
        })

        # Signal the waiting agent to resume with the decision
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "approved",
            "approver": approver,
            "comment": comment,
        })

        self.fsm_state = "idle"
        self.pending_agent_id = ""
        return {"status": "approved", "agent_id": agent_id}

    @handler("reject")
    def reject(self, approver: str = "", reason: str = "") -> dict:
        """Human rejects — signals the agent with the rejection."""
        agent_id = self.pending_agent_id
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "rejected",
            "approver": approver,
            "reason": reason,
        })
        self.fsm_state = "idle"
        return {"status": "rejected", "agent_id": agent_id}

The agent on the other side calls host.ask("approval_gate", "request_approval", {...}) then processes the workflow_signal:resume message when it arrives. Because state is checkpointed durably, the agent can wait hours or days with no polling loop and no timeout burning tokens.

Long-Running Agents

Cloudflare’s long-running agent pattern uses alarms to wake a dormant agent on a schedule. PlexSpaces handles this identically, e.g., any actor with the ReminderFacet can schedule work arbitrarily far in the future. The ChatAgentActor summarization alarm is one example; for a true long-running loop:

@actor
class ChatAgentActor:
    """Minimal chat agent: conversation in KV, LLM via service link, alarm for summarization."""

    actor_id: str = state(default="")
    total_messages: int = state(default=0)
    total_summarizations: int = state(default=0)

    @init_handler
    def on_init(self, config: dict) -> None:
        self.actor_id = config.get("actor_id", "")
        host.info(f"ChatAgentActor init actor_id={self.actor_id}")


    @handler("__alarm__")
    def on_alarm(self) -> dict:
        """Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm().

        Summarizes conversation history and stores a summary KV key,
        then clears history.
        """
        host.info("ChatAgentActor: alarm fired — summarizing history")

        history = host.kv.get_json("history") or []
        if not history:
            return {"status": "ok", "action": "no_history_to_summarize"}

        # Summarize via LLM
        summary_prompt = (
            f"Summarize this conversation concisely (2-3 sentences): "
            f"{json.dumps([{'role': m['role'], 'content': m['content']} for m in history])}"
        )
        summary = self._call_llm([{"role": "user", "content": summary_prompt}])

        # Persist summary, clear history — equivalent to: storage.put('summary', s); storage.delete('history')
        host.kv.put("summary", summary)
        host.kv.delete("history")

        self.total_summarizations += 1

        host.info(f"ChatAgentActor: summarized {len(history)} messages")
        return {
            "status": "ok",
            "action": "summarized",
            "messages_summarized": len(history),
        }

What PlexSpaces adds beyond Cloudflare’s model: the actor can also be signalled externally at any time via host.send(actorId, "wake_early", {...}) — you’re not limited to the alarm cadence.

Agents Feature Map

Part III: Distributed Computation

Cloudflare Workers and Lambda optimize for millisecond, latency-sensitive request handling. PlexSpaces handles a second problem class: large-scale computation across resources that come and go.

ShardGroups: Scatter-Gather Without the Plumbing

For data-parallel and ML-style workloads, PlexSpaces exposes MPI-style collectives directly through the host API:

// Create a pool of 20 workers, hash-partitioned
let pool_id = client.create_worker_pool(
    "worker-pool-1", "worker", 20,
    PartitionStrategy::Hash, HashMap::new(),
).await?;

// Bulk update: 10,000 messages routed to the right shard by key
client.parallel_update(&pool_id, updates, ConsistencyLevel::Eventual, false).await?;

// Parallel map: query every shard simultaneously
let results = client.parallel_map(&pool_id, json!({ "action": "get_total_count" })).await?;

// Parallel reduce: aggregate stats across all shards
let stats = client.parallel_reduce(
    &pool_id, json!({ "action": "stats" }),
    ShardGroupAggregationStrategy::Concat, 20,
).await?;

Idle Browsers as Compute Nodes

The mersenne_prime TypeScript example runs a GIMPS-style distributed primality search: browser tabs connect as thin WebSocket clients, receive worker JavaScript from a CodeServerActor, and run Lucas-Lehmer tests inside Web Workers. A CoordinatorActor running as WASM on the server assigns exponents, tracks per-worker CPU cores, and dispatches the next candidate immediately on each result:

// PlexSpaces TypeScript — examples/typescript/apps/mersenne_prime/mersenne_actor.ts
onResult(payload: ResultPayload): unknown {
    const item = this.state.work[String(payload?.p)];
    item.status = 'done';
    item.is_prime = Boolean(payload.is_prime);
    item.duration_ms = payload.duration_ms ?? 0;

    // Record every completed candidate in TupleSpace and bump a Prometheus counter
    host.ts.write(['result', String(payload.p), item.is_prime ? 'true' : 'false',
        String(item.duration_ms), payload.actor_id ?? 'unknown']);
    host.incrCounter('ts-mersenne-prime', item.is_prime ? 'primes_found' : 'composites_found');

    // Immediately hand the same worker the next pending candidate
    const next = this._nextPending(this.state.workers[payload.actor_id!]?.cpu_cores ?? 1);
    if (next) {
        next.status = 'assigned';
        host.send(payload.actor_id!, 'assign_work', { p: next.p, done: false });
    }
    return { ok: true };
}

Open the same URL in ten browser tabs and you have ten compute shards, coordinated by one WASM actor, with zero additional infrastructure.

Comprehensive Feature Map

A couple of things need real rework, not a find-and-replace:

• WebSocket architecture. If your DO holds sockets directly today, plan for a thin node that hosts actors.
• Bindings vs children. Cloudflare bindings live in wrangler.toml and show up as env properties. PlexSpaces declares the same relationships as supervision children in app-config.toml.

Running Everywhere

The whole point is that development and production run the identical binary:

# Local development — exact production behavior
plexspaces-node start --config app-config.toml

# Docker — same binary, same config
docker run -v $(pwd):/app plexspaces/node start --config /app/app-config.toml

# Kubernetes — same config via Helm
helm install my-app plexspaces/app-chart \
  --set config.path=app-config.toml \
  --set persistence.storage=postgres

# Multi-cloud — nodes in GCP + AWS joined over a gRPC mesh; actors route transparently

An alarm that fires in production runs through the exact code path you tested on your laptop. There’s no gap to debug between wrangler dev and prod, because there’s only one runtime.

Working Examples

Every example referenced above is in the repository (github.com/bhatti/PlexSpaces):

• Guild chat (DO migration pattern): examples/{go,typescript,python,rust}/apps/migrating_cloudflare_workers/: ChatRoomActor with member fan-out, a token-bucket RateLimiterActor backed by host.kv.increment and host.kv.cas, and an AlarmDemoActor mirroring DO’s full alarm lifecycle
• WebSocket chat room: examples/{typescript,python,go,rust}/apps/ws_chat_room/: ChatRoomActor plus a PresenceActor that uses a durable reminder to detect idle disconnects
• AI chat agent (Cloudflare Agents SDK pattern): examples/{go,typescript,python,rust}/apps/chat_agent/: conversation history in KV, LLM calls through service link, durable summarization alarm
• Human-in-the-loop approval gate: examples/python/apps/minipi/approval_gate.py: FSM-based approval workflow, workflow_signal:resume handoff, state durable across multi-day waits
• Multi-agent orchestration: examples/go/apps/miniclaw/: OrchestratorActor decomposing tasks, delegating via process groups, aggregating through TupleSpace
• Durable payment workflow: examples/go/apps/migrating_cadence/payment_workflow.go: WorkflowActor with Run/Signal/Query, idempotent retry, refund and cancel signals
• Mersenne prime search (browser compute): examples/typescript/apps/mersenne_prime/: browser tabs as Lucas-Lehmer worker shards, coordinated by a WASM CoordinatorActor
• wasmCloud migration: examples/python/apps/migrating_wasmcloud/session_store.py: capability-based session store showing host.kv.list, inter-actor ask, and timer-driven cleanup
• Every abstraction in one place: examples/{go,typescript,python,rust}/apps/abstractions/: durable virtual-actor reactivation, workflow run/signal/query, process-group event delivery, timers, reminders, KV, tuple space, and blob storage

Conclusion

The Cloudflare model is the right model: stateful actors, durable storage, alarms, WebSocket session management, LLM calls baked in. Wire’s post makes the same point from the other direction as they’re not leaving because the model is wrong, they’re leaving because specific architectural ceilings came due at their scale. PlexSpaces keeps the model and removes the ceiling: you write the same actors, get the same alarms and durable storage and observability, and the binary that runs on your laptop is the exact binary that runs in production, on any cloud, on-prem, or across all of them at once. The migration from Cloudflare DO code is mostly mechanical. What you get back is the ability to run anywhere, own your infrastructure and never again debug a divergence between wrangler dev and prod.

GitHub: github.com/bhatti/PlexSpaces

Previous posts in this series:

• Building PlexSpaces: Decades of Distributed Systems Distilled
• Polyglot WebAssembly: Four Languages, One Runtime
• 20+ Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
• Building Mini OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
• Building a Self-Improving AI Agent with Durable Actors: MiniHermes
• Building an Agent Harness and Eval Pipeline with Durable Actors

I have seen some systems never crash, they start cleanly, swallow every error, and keep running no matter what goes wrong. In my experience, they are also the hardest systems to debug, the most dangerous to operate, and the most expensive to maintain. I worked on a similar legacy system for distributed data platform that routed events between hundreds of thousands of nodes. It had zero unhandled exceptions in production. It also had silent authentication failures, invisible data loss, and configuration divergence that took days to diagnose. This is a follow-up to my earlier posts on building an observability platform in Rust, why DRY becomes a liability, and making bad state impossible with ADTs. Those posts were about the type system but this one is about a habit of mind that no type system fixes on its own: the instinct to catch every error with some fallback or default behavior.

The culprit in that system was never a lack of error handling. It was error handling, applied in the wrong places, for the wrong reasons. I call it defensive programming as a religion: every function protects itself against every possible invalid input by inventing a fallback. Missing config? Use a default. Secret unavailable? Generate a random one. Database write failed? Log it and move on. The result is a system that looks healthy on every dashboard while quietly corrupting its own state underneath. In this post, I will walk through the patterns I found, why each one causes more damage than the crash, and what an alternative looks like instead. The whole argument rests on one idea: Every fallback creates a new source of truth, and two sources of truth always drift apart.

I. Why a Fallback Is Worse Than It Looks

It’s tempting to think of a fallback as just “hiding an error.” It’s worse than that. When a function invents a value because the real one is missing, that invented value doesn’t stay hidden, instead it becomes a fact in the system. From that moment on, the system is carrying two truths: the one that should exist but doesn’t, and the one that got made up and does. These two truths never stay in sync and when they drift apart, the failure almost never shows up where the fallback happened. Instead, it shows up somewhere else entirely, in a component with no obvious connection to the code that invented the value. For example, you’ll spend hours in the authentication layer before realizing the signing key was randomly generated at startup by a config migration function three layers away. To be clear about what I mean by “fallback,” I am not talking about:

• Validating input at system boundaries: checking what a user typed, sanitizing data from outside. Correct and necessary.
• Graceful degradation with an explicit signal: returning a typed Degraded state that the caller can see and react to.
• Retrying transient I/O failures with backoff. Standard practice.

I’m talking about code that silently invents state when the real state is missing, and then carries on as if nothing happened. Three things make a fallback harmful:

1. It hides the root cause. The missing value was the bug. The fallback makes it disappear.
2. It persists the invented value. Once it’s written to disk or sent over the wire, every future operation has to succeed against a value that was never correct in the first place.
3. It relocates the symptom. The failure surfaces hours later, in a different component, in a different log file, with no visible thread connecting it back to the startup code that invented the wrong value.

I am not advocating “crash on every error.” It just means: a function that requires X must fail when X is absent and it must never invent X.

II. Postel’s Law

There’s a reason smart engineers build these fallback-heavy systems: they’re following a respected principle: “be conservative in what you send, be liberal in what you accept.” that Jon Postel wrote as guidance for TCP implementations. That advice made a lot of sense in its original context. But this principle leaked out of the protocol layer and turned into a general design philosophy. Engineers started applying “be liberal in what you accept” to function signatures, config loading, and communication between services inside a system they fully control. A function that takes string | undefined and silently substitutes a random value gets called “being robust.” A startup sequence that swallows errors and keeps going gets called “being tolerant.”

Inside your own system, that assumption doesn’t hold. For example, you can fix the sender because you own the caller. But when you own the migration script that’s supposed to write the auth token and you “liberally accept” a missing auth token by inventing a random one instead, you’re not enabling interoperability with an outside party instead you’re hiding a bug in code you wrote. This misapplication creates a ratchet effect. Every “liberal” receiver makes it harder to notice problems at the source. If every function tolerates missing input, the function that’s supposed to supply that input has no pressure to get it right. People also tend to forget that Postel’s Law has a second half: “be conservative in what you send.”

The corrected version for internal systems is this: be strict with components you control, and liberal only at the boundaries where you genuinely can’t fix the sender like external APIs, user input, third-party integrations. Inside your own codebase, strictness isn’t fragility. A function that rejects invalid input tells you exactly where the bug lives.

III. Inventing Values: The Most Dangerous Pattern

This is the category that caused the most damage and I have seen countless bugs due to this anti-pattern. For example, the code generates a random value, assigns it to a security-critical field, and moves on as if that field were properly populated. The invented value becomes a durable fact somewhere and it’s always wrong.

The Archetype: Inventing a Secret

A function runs at startup and writes a signing secret into every worker group’s configuration. Workers and the coordinator use this secret to authenticate each other. If two groups end up with different values, every authentication attempt between them fails silently, showing up as delivery failures instead of auth failures.

// The bug: if authToken is absent, invent a UUID and persist it
const authToken = settings.distributed?.master?.authToken;
const plaintext = authToken != null && authToken.length > 0
  ? authToken
  : randomUUID();  // <-- this line breaks authentication across the cluster

When authToken is missing from the merged settings, which is a perfectly valid state on a fresh install but this function generates a randomUUID() and writes it as the signing secret for every group it touches. Each call produces a different UUID. Meanwhile the token store writes yet another value through a completely different code path. The function reports success for every group it writes to. The symptom 401 errors between nodes shows up minutes or hours later, in worker logs, pointing investigators toward the wrong layer entirely. This is the archetype of the whole problem: if the required input is missing, invent something plausible-looking and keep going. The fix is three lines:

const authToken = settings.distributed?.master?.authToken;
if (authToken == null || authToken.length === 0) {
  logger.warn('authToken absent from settings; cannot write signing secrets');
  return;  // do not proceed; do not invent
}

The “Disabled” Sentinel That Looks Just Like a Real Key

A secret provider has a three-source fallback chain: encrypted store, config file, random bytes. That last fallback is supposed to act as a “poison key” that never validates:

async getSecret(): Promise<string> {
  // Source 1: encrypted store
  const fromStore = await secretsMgr.get(KEY_ID).catch(() => null);
  if (fromStore) return fromStore;

  // Source 2: config file (which may itself contain a well-known default!)
  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return fromFile;

  // Source 3: "disable" by returning random bytes — looks perfectly valid to the caller
  return random(16);
}

The caller gets back a plain string in all three cases. It has no way to tell “a real secret from the store” apart from “a well-known default from the config file” apart from “random bytes that will never work.” It signs a token with whatever string it received and sends it off. This is a type-system failure because the return type Promise<string> squashes three semantically different outcomes into one shape.

The explicit contract fixes this at the type level:

type SecretResult =
  | { kind: 'valid'; value: string; source: 'store' | 'file' }
  | { kind: 'unavailable'; reason: string };

async getSecret(): Promise<SecretResult> {
  const fromStore = await secretsMgr.get(KEY_ID).catch((err) => {
    logger.error('Secret store unavailable', { error: err });
    return null;
  });
  if (fromStore) return { kind: 'valid', value: fromStore, source: 'store' };

  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return { kind: 'valid', value: fromFile, source: 'file' };

  return { kind: 'unavailable', reason: 'no secret in store or config' };
}

Now a caller that receives unavailable marks itself as degraded. It doesn’t sign tokens and surfaces a health check failure instead.

The Token Renewal That Signs With Random Bytes

The token authenticator calls getSecret(), and if that fails, it falls back to random bytes anyway:

let keyStr = await this.secretProvider.getSecret().catch(() => undefined);
if (keyStr == null) {
  this.logger?.debug('Unable to generate a valid token. Disabling.');
  keyStr = random(16);  // this "signed" token will never be accepted by any peer
}
const token = jwt.sign(payload, keyStr);
this.cachedToken = token;  // cached and reused for every future request

The log message says “Disabling,” but nothing gets disabled. The code signs a token with a random key, caches it, and hands it out to every caller for the rest of the process’s life. Workers receive the token, fail to verify it, and log a 401, with nothing to suggest the root cause of the issue.

The explicit contract: if the secret is unavailable, don’t sign anything. Set this.cachedToken = null. Let callers check for null and surface a real health degradation.

The User ID That Regenerates on Every Call

const userId = existing?.username ?? crypto.randomUUID();

When existing is unexpectedly missing, every call generates a brand-new UUID. The user becomes a ghost: every request creates a fresh identity, invisible to deduplication, audit trails, and rate limiters.

The Config Placeholder That Silently Materializes

if (authToken?.token === 'REPLACE_ME') {
  authToken.token = uuidv4();  // silent replacement, no log of the value generated
}

This runs at startup and inside a database migration. If the startup write fails, the generated token is gone.

IV. Swallowed Errors

In one legacy codebase I worked had 656 instances of .catch(NOOP), an empty function attached to a promise rejection that turns any error into undefined and lets execution continue. Many of these were on cleanup paths, which is harmless. But a large number sat on critical data paths like durability, metrics transport, authentication state.

The Durability Guarantee That Wasn’t

A persistent queue exists for exactly one reason: to guarantee that events survive a destination outage. That’s the system’s durability promise.

// PQ flush — the durability guarantee
await this.flushBuffer().catch(NOOP);  // silently swallows disk I/O errors
await this.commit().catch(NOOP);        // silently swallows commit failures

If the flush fails like disk full, I/O error, permission denied, the buffered events are silently gone. The one component whose entire job is preventing data loss is itself a source of silent data loss.

The explicit contract: treat PQ flush errors as fatal to the ingest path. For example, if the flush fails, apply backpressure and pause ingest. Log at error level with the event count at risk and emit a pq_flush_failure metric. Now the operator gets to choose: fix the disk, add capacity, or consciously accept the loss.

Metrics That Vanish

void saasMetrics.sendPacket(packet).catch(NOOP);

Every metrics-send failure is silently swallowed.Dashboards go blank, and nobody knows why.

The Config Load That Treats Corruption as “Empty”

const groups = await conf.loadSystem('internal-groups').catch(() => ({}));

One line here quietly conflates two very different situations:

• “The file doesn’t exist yet”, which is normal on first boot –> return {}
• “The file is corrupt, or a parse error, or permission was denied”, which is a real configuration bug –> also return {}

Either way, the loop over groups never runs. The operator has no way to tell “healthy, no groups configured yet” apart from “broken, groups exist but couldn’t be read.”

Startup That Succeeds Despite Total Failure

export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  try {
    // ... write secrets to all groups ...
  } catch (err) {
    logger.error('failed to sync secrets', { reason: err });
    // swallowed — startup continues, caller receives no signal
  }
}

The function catches everything at the top and resolves successfully no matter what. The caller awaits it and gets no signal that anything went wrong. If the sync fails for every group, the coordinator still starts, workers still connect, and authentication fails across the board but startup “succeeded.”

V. Defensive Defaults

This next category is more subtle. Instead of inventing a random value, the system injects a well-known one like a default credential, a default address, which makes it impossible for downstream code to tell that anything is missing at all.

The Well-Known Default Credential

The shared authentication token has a configuration setting, and by default, the settings loader injects a well-known string whenever nothing is configured:

const { injectDefaultAuthToken = true } = options ?? {};
if (injectDefaultAuthToken && conf.distributed?.master?.authToken == null) {
  conf.distributed.master.authToken = DEFAULT_AUTH_TOKEN;  // 'default-secret'
}

Any caller of getSettings() receives a truthy string for authToken. Code that checks if (authToken) proceeds as if a real token exists. The check passes. Authentication moves forward, using a credential that’s sitting right there in the source cod and every deployment that forgot to override it. The default here is opt-out, not opt-in. Every new call site has to remember to disable the injection.

Workers That Connect to localhost on Config Failure

const server = this.conf.distributed.master
  || { host: 'localhost', port: 5555, authToken: DEFAULT_AUTH_TOKEN };

When a worker fails to load its coordinator address, it silently falls back to localhost:5555 with the default token. On a single-node dev box, this might accidentally work. On a production multi-node deployment, the worker ends up connecting to itself. The symptom looks like a connection timeout, not a config-load failure.

The explicit contract: if conf.distributed.master is missing, throw. A worker cannot function without a coordinator address, and there is no valid default for it. Failing here with a clear message like “coordinator address not configured; set MASTER_URL or add distributed.master to instance.yml“.

VI. Multiple Sources of Truth

This is the pattern that ties everything else together. Every fallback chain creates more than one source of truth and any source of truth that isn’t explicitly designated as the source will eventually drift from the others. In a distributed system, that drift shows up as the hardest class of bug like intermittent or state-dependent failures.

Two Functions, One Secret, Two Different Sources

The bug that inspired this post exists because two functions write the same signing secret to group configs, but read the plaintext from two different physical sources:

A migration populates the token store before syncAllGroupSecrets runs at boot, so the store is meant to be authoritative. But syncAllGroupSecrets predates the store’s existence and still reads straight from the config file. These two sources drift apart after a token rotation or some race condition.

The explicit contract: one source of truth. Both functions read from the store. If the store is unavailable, both fail instead of silent fallback to a secondary source.

The Three-Source Chain Is Three Sources of Truth

Source 1: Encrypted store (authoritative)
  ? (unavailable ? silently falls through)
Source 2: Config file (may hold a stale or default value)
  ? (absent ? silently falls through)
Source 3: Random bytes (structurally valid, semantically useless)

Each fallback quietly downgrades the security posture, and the caller gets back a plain string with no idea which source it came from.

Three branches go into the same sign() call. Only one of them should ever be allowed to reach it, which is the whole argument for making unavailable its own explicit type instead of letting all three collapse into a plain string.

The Cache Nobody Fully Trusts

The settings system keeps a cache for high-availability mode. Some callers pass skipCache: true to bypass it; others don’t, and there’s no documented rule for which is which. This gives the system two sources of truth for the same data: the cache and the disk. If the config file changes between startup and an API call, the API may serve stale data. The skipCache escape hatch is a symptom, not a fix. It means someone stopped trusting the cache’s invalidation and punched a hole through it instead of repairing the underlying mechanism.

The explicit contract: the cache invalidates on every write. Callers never need to know or care whether they’re reading from cache or disk. Remove skipCache as a public option entirely.

VII. Redundant Guards

When a function doesn’t trust its caller’s preconditions, it adds its own guard on top:

// Caller (server.ts):
if (isLeader && featureFlags.check('AUTH_TOKEN_MGMT')) {
  await syncGroupSecrets(conf);
}

// Callee (syncGroupSecrets):
export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  if (!isFreeTier() && !isRunningInSaaS()) return;  // guard 1
  if (!Product.isLeader(settings.distributed?.mode)) return;  // guard 2 (redundant!)
  if (!featureFlags.check('AUTH_TOKEN_MGMT')) return;  // guard 3 (redundant!)
  // ... actual work ...
}

This function lives in a directory named leader/ and is only ever called from the leader startup path, yet it re-checks isLeader internally anyway. The feature flag gets checked at the call site and again inside the function. Three layers of defense against calling this function in the wrong context.

The checks themselves aren’t the problem. It’s what happens when they trip: nothing. The function returns quietly, and the caller gets no signal either way.

The explicit contract: a function either does its job or throws. Preconditions get asserted, not silently absorbed. If the caller already guarantees the precondition, drop the internal check. If the function really can be called from multiple contexts and some of them are invalid, throw on the invalid ones instead of quietly returning.

VIII. “Best-Effort” Writes to State That Isn’t Optional

The most seductive justification for swallowing an error is: “the primary operation already succeeded so we don’t want a secondary failure to undo it.” That reasoning is correct in isolation and catastrophic in aggregate.

The Store Upsert That Swallows Its Own Failure

export async function mirrorTokenToStore(plaintext: string): Promise<void> {
  try {
    await store.upsert({ id: LEGACY_TOKEN_ID, token: encrypt(plaintext) });
  } catch (err) {
    logger.error('failed to mirror token to store', { reason: err });
    // swallowed — the caller sees success
  }
}

The comment above this function explains the intent: it’s “best-effort” so that a store-side failure doesn’t roll back a config file write that already succeeded. That reasoning holds up on its own but config file is now permanently out of sync.

The explicit contract: add reconciliation. On startup, compare the store’s token to the config file’s. If they differ, update the store. Emit a token_store_divergence counter and surface it in a health check similar to reconciliation loops in Kubernetes.

IX. Partial Operations Without a Way Back

The Batch Write That Discards on Failure

const batchTxn = db.transaction((ops) => { ops.forEach(fn => fn()); });
try {
  batchTxn(this.mutationCache.splice(0, maxSize));  // splice removes BEFORE success
} catch (err) {
  logger.error('Batch write failed', err);
  // operations already removed from cache — permanently lost
}

The comment in this code says: operations get removed from the cache regardless of whether the transaction succeeds. But the fix for “one bad operation might stall the queue” ends up being “discard the entire batch, including the good operations.” That isn’t a tradeoff instead it’s silent data loss.

The explicit contract: only remove items from the cache after a confirmed commit. Isolate the failing operation into a dead-letter queue and retry the rest of the batch without it.

The Config Reload Nobody Acknowledges

await conf.triggerReload().catch(NOOP);  // worker continues with old config

After receiving a new config bundle from the coordinator, a worker triggers a reload. If that reload fails, the worker just keeps running the old config. The two sides now disagree about what the worker is actually running.

The explicit contract: report a reload failure back to the coordinator on the next heartbeat. The coordinator marks that worker as “stale config” and can retry or alert. This is how Kubernetes rolling updates work, the controller notices and either retries or halts the rollout.

The Package Install That Saves Despite Partial Failure

for (const op of ops) {
  try {
    switch (op.type) {
      case 'install': await installPackage(op); break;
      case 'uninstall': await uninstallPackage(op); break;
    }
    status.applied.push(op);
  } catch (error) {
    errors.push(error);
  }
}
await this.save(packageManifest);  // saves regardless of how many failed

If three out of five packages install and two fail, the manifest still gets saved with those three. Next startup tries again from this partial state but the failed packages may have left behind lock files or half-written artifacts causing conflicts.

The explicit contract: validate that every operation can succeed before running any of them (a dry-run pass). Execute atomically (ACID transactions).

X. Silent Truncation With No Backpressure

The Metrics Buffer That Silently Drops

Workers piggyback metrics onto heartbeat messages sent to the coordinator, with a hard cap of 100,000 packets. Past that cap, excess metrics are silently dropped without any counter, logs or metrics. This is a nasty failure mode specifically because absence of data is itself meaningful data and silent truncation destroys that signal’s reliability.

The explicit contract: when the buffer nears capacity, reduce granularity instead of dropping outright. When truncation does happen, include metrics_truncated: N in the heartbeat so the coordinator knows its picture is incomplete. Better, instead of piggyback metrics on heartbeats at all, give them their own transport.

The TCP Sender That Zeroes Buffers on Disconnect

// On disconnect: all in-transit events silently lost
this.inTransitBufs = [];
this.bufOffset = 0;
this.bufferEventCount = 0;
this.dropBytes += len;  // only evidence: a counter increment

On a TCP disconnect, every in-transit event gets zeroed out. The only trace left behind is a dropBytes counter buried in internal metrics. Compare that to Kafka’s producer, where unacknowledged messages stay in the producer’s buffer and get retried on reconnect.

The Unbounded Queue That Becomes an OOM

protected queueBatch(): void {
  this.queuedBatches.push({ eventCount, eventsSize, events: this.currentBatch });
  // NO CHECK on length, size, or memory pressure
}

When the output destination is unreachable, failed batches get re-queued, and the queue grows without any bound. Memory climbs until the OOM killer steps in and terminates the process. An unbounded in-memory queue isn’t really a data structure, instead it’s a deferred OOM crash.

The explicit contract: bound the queue. Once it’s full, either apply backpressure to ingest, spill overflow to the persistent queue, or trip a circuit breaker that rejects new events with a typed error. Let the pipeline decide from there: drop, buffer to disk, or pause the source.

XI. Non-Atomic Writes

The Lease File That Can Split-Brain

await writeFile(this.leaseFile, stringify(content));

The failover lease file, the mechanism that’s supposed to guarantee only one coordinator is ever active is written directly with writeFile. On NFS, which is where this system runs in HA mode, writes aren’t atomic. A power failure mid-write leaves behind a truncated or corrupt file. The standby coordinator reads that corrupt lease, fails to parse it, and ends up in an undefined state.

The explicit contract: write to a temp file, fsync, then atomically rename it into place, with a checksum so readers can detect corruption. Every real database does this like SQLite’s WAL.

The Multi-File Config Deploy Without a Journal

Config deployment writes several YAML files in sequence like inputs.yml, outputs.yml, pipelines.yml,, etc. A crash midway through leaves the worker with a partial config and a worker that restarts after a partial deploy loads that inconsistent config.

The explicit contract: write every file to a staging directory first, verify that everything references correctly, then swap atomically like rename the directory, or flip a symlink. This is the same idea behind Docker image layers, Kubernetes ConfigMaps, and Nix store paths.

XII. Six Principles That Cover All of This

Every pattern above breaks one of six well-established principles. They’re standard practice in any system that prioritizes correctness over the appearance of uptime.

1. Fail fast at trust boundaries (Erlang’s “let it crash.”): When a precondition is violated, fail immediately and loudly. Erlang runs telecom infrastructure at 99.9999999% uptime on a philosophy of letting individual processes crash and having a supervisor restart them into known-good state.

2. Make invalid states unrepresentable: If getSecret() can return random(16) as a plain string then every caller is stuck defensively guessing whether it’s “real.” If it returns Secret | Disabled as a discriminated union instead then the type system forces every caller to handle both cases at compile time. I wrote about this pattern in “Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects.”

3. Classify errors (transient vs. fatal): Without classification, every catch block faces an impossible choice: rethrow and break “resilience,” or swallow and hide a real bug. For example, gRPC solves this with status codes like UNAVAILABLE means retry, INVALID_ARGUMENT means don’t, INTERNAL means there’s a bug.

4. Define delivery semantics for critical state: “Fire-and-forget” is fine for debug logs. It’s not fine for persistent queue flushes, token store upserts, or config reloads. If an operation mutates state that downstream code assumes succeeded, it needs at-least-once semantics.

5. One source of truth without fallback chains for critical state: For any given piece of state, there should be exactly one authoritative source. A fallback chain isn’t graceful degradation instead it’s an implicit decision that secondary sources are acceptable substitutes for the truth. If that’s genuinely acceptable, make it explicit with TTLs, version vectors, or consistency levels.

6. Atomic state transitions: State changes should be all-or-nothing: temp file, fsync, atomic rename for files; transactions with rollback for databases; staging plus swap for multi-file deployments.

XIII. Cognitive Load

All this conditional logic and fallback behavior creates the cognitive load, e.g., when any function might silently invent a value, you can’t trust a function’s output without reading its implementation. When errors are swallowed, a successful await no longer means the operation actually succeeded. When defaults get injected into config reads, a non-null value stops meaning “configured.”

Debugging a production incident in a system like this means reading every function in the call chain, understanding every fallback along the way, and reconstructing which code path actually ran. crash tells you exactly where and when an invariant broke. New engineers ask why workers sometimes fail to authenticate after a token rotation, and the honest answer involves reading six functions across four files, understanding a three-source fallback chain. Compare that to: “the store upsert threw on failure, the rotation API returned a 500, the operator re-ran it, it worked.

XIV. Conclusion

Defensive programming isn’t inherently wrong, and neither is Postel’s Law like at the boundary it was designed for. Validating input at system boundaries, handling I/O errors gracefully, protecting against malformed external data are correct applications of defensive thinking. The problem shows up when the same techniques get applied to internal code, inside a system where you control both ends of every interface. The alternative, in short:

• Throw Error when a required state that’s missing: The function refuses to proceed and the caller finds out immediately.
• Explicit return when an optional state that’s missing: null, undefined, Option<T>, a discriminated union.
• Transient failures = retry with backoff: Never .catch(NOOP).
• One source of truth per piece of state: Not a fallback chain that quietly degrades. Not a cache with no real invalidation.
• Bounded queues with backpressure: Not unbounded buffers waiting to OOM.
• Atomic state transitions: Not multi-step operations that can half-finish.
• Reconciliation loops for distributed state: Not one-shot “best-effort” writes that quietly drift apart.

This mud didn’t accumulate overnight, and it won’t disappear overnight either. But every fallback you remove, every error you refuse to swallow will makes the next incident roughly ten times faster to diagnose.

Related Blogs

1. From Big Ball of Mud to Functional Pipeline
2. The Reusability Trap: When DRY Becomes a Liability
3. Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects