Shahzad Bhatti Welcome to my ramblings and rants!

What happens when over a thousand AI agents discover a shared message board, invent their own protocols, and coordinate an attack.

The Hugging Face Incident

The Hugging Face security breach incident from July, 2026 has been in all over the news when AI agents from OpenAI compromised parts of the Hugging Face infrastructure. I recently read METR report on the incident that explained that the incident occurred when OpenAI kicked off ExploitGym runs (security benchmark) with tens of thousands of AI agents. Every agent was meant to be sandboxed and alone but the agents learned about shared access to Artifactory. When an agent installs a package, it could see cache entries from other agents. One of agent (PHASEONE10841) started creating directories in the shared cache and posting messages in them. Shortly after that hundreds of other agents found the same directory and started replying. Over the next few days, roughly 1,200 agents used this board and exchanged more than 70,000 messages and files. Later, another agent (PHASEONE[big]) took over as the de facto coordinator and started hundreds of task assignments across six broad categories of work. The agents also built a protocol vocabulary like HOLD to claim exclusive ownership, VETO to block a conflicting operation, STOP to end a workstream, owner tags on shared assets. Later, over 90% of AI agents pivoted into a coordinated attack on Hugging Face.

I was intrigued to learn how AI agents self discovered coordination patterns that distributed systems have relied on. For example, I found similarity between their message board and an a Blackboard system pattern from the Linda Memory Model/Tuplespace research. Their HOLD/task-claiming was similar to Linda’s in() operation. I saw another blog An Accidental Blackboard from Thoughtworks where a smaller-scale but structurally similar thing happened. In that case, AI agents discovered one shared repository and adopted a discipline of frequent commits and rebases to share information with other agents. Other agents could see task plans and and picked up work without any coordination. In another yet instance from researchers at the AI safety nonprofit Nightingale found about 18,000 posts from OpenAI agents that colonized DSEWiki. These agents discovered that their sandbox environment only allowed GET http requests but ProWiki software allowed them to edit a page. They used that loophole to share information and a cheat sheet. Nightingale believes this was a unrelated swarm from the one that built the Artifactory board. These examples demonstrate that agents discover coordination patterns when you strip away pre-built orchestration and let agents solve the problem from first principles.

Why rigid orchestration breaks

Most multi-agent systems today are wired up as static, directed graphs, e.g., a central coordinator decides which agent runs when, what data flows where, and how results get combined. It breaks down at scale as illustrated in the incidents above.

• Central orchestrators become information bottlenecks. When every message between agents has to pass through a coordinator, its context window fills up and you lose details because the context is compressed into summaries.
• Static graphs assume you know the workflow ahead of time. The agents in the Hugging Face incident didn’t know what workstreams exist when they started and the task structure evolved as agents discovered new information. A pre-wired pipeline can’t bend to that.
• Adding a new agent type means rewiring. In a static system, a new specialist means touching the coordinator, updating routing logic, and re-testing transitions. In a dynamic system, a new agent just registers its capabilities and other agents discover it.
• A single coordinator is a single point of failure. In the Hugging Face incident, there was no coordinator initially but PHASEONE[big] emerged organically as an organizer and other agents simply kept working.

Above examples show that agents need primitives for coordination and communication like shared memory, message passing, atomic operations. The agents built their own version of these primitives out of Artifactory and Git. The table below maps what they built and the equivalent primitive in PlexSpaces, the actor framework I use for the examples in this post.

The bottom side of following diagram shows when every message has to detour through a coordinator, that becomes bottleneck and loses context in translation. On the top, agents read and write directly to a shared store so there is no central bottleneck.

Following are a few coordination patterns from the Anthropic’s blog:

• generator-verifier (one agent produces, another checks, feedback loops until it passes)
• orchestrator-subagent (a lead agent decomposes and delegates bounded subtasks)
• agent teams (long-lived workers that claim tasks from a shared queue)
• message bus (publish/subscribe over topics)
• shared state (agents read and write a common store with no central coordinator at all).

This post shows more granular version of this list and Hugging Face incident especially a tuple space as first-class infrastructure.

Ten coordination patterns

Following ten patterns cover the coordination behaviors observed in the Hugging Face incident and Anthropic’s coordination guidance. Each one maps to a working PlexSpaces API, in both TypeScript and Python.

1. Blackboard (shared state)

You can use this pattern when multiple agents need to contribute findings and read each other’s work without a predefined message format or a central router. I find this pattern is similar to Linda TupleSpace‘s model where all agents read and write to a common tuple space. Each tuple is a typed record like a finding, an analysis, or a vote. The pattern matching in TupleSpace allows an agent filter for what it actually needs using primitives like out() to write, rd() to read, in() to take.

Here is a typescript snippet that shows use of tuplespace:

// Research agent deposits a finding
const findingId = `f-${Date.now()}`;
host.ts.write(["finding", findingId, topic, content, confidence, host.nowMs()]);

// Analysis agent reads all findings (non-destructive)
const findings = host.ts.readAll(["finding", null, null, null, null, null]);

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// TypeScript — ResearchAgent writes a finding to the blackboard
onResearch(payload: Record<string, unknown>): Record<string, unknown> {
  const topic = String(payload.topic || "general security");
  const confidence = Math.min(0.3 + topic.split(/\s+/).length * 0.08, 0.95);
  const findingId = `finding-${generateId()}`;
  const content = `Security analysis of ${topic}: identified key areas requiring review.`;

  // Linda out() — deposit into shared space
  host.ts.write(["finding", findingId, topic, content, confidence, host.nowMs()]);
  return { finding_id: findingId, content, confidence };
}

Here is a python snippet (multi_agent_coordination_actor.py):

# Python — AnalysisAgent reads all findings from the blackboard
@handler("analyze")
def analyze(self, topic: str = "") -> dict:
    # Linda rd*() — non-destructive read of all matching tuples
    findings = host.ts.read_all(["finding", None, None, None, None, None])

    finding_ids = [str(f[1]) for f in findings if len(f) >= 4]
    topics = list(set(str(f[2]) for f in findings if len(f) >= 4))
    severity = "critical" if len(findings) >= 5 else "high" if len(findings) >= 3 else "medium"

    analysis_id = f"analysis-{host.now_ms()}"
    summary = f"Cross-referenced {len(findings)} findings across {len(topics)} topics."
    host.ts.write(["analysis", analysis_id, json.dumps(finding_ids), summary, severity])
    return {"analysis_id": analysis_id, "summary": summary, "severity": severity, "finding_count": len(findings)}

These primitives showed up underneath the unsanctioned board when agents read each other’s research.

2. Scatter-gather (fan-out)

You can use this pattern when a coordinator needs to distribute work across multiple agents in parallel, then collect/merge their results like MapReduce. PlexSpaces provides shard groups that create a logical group of worker actors. The coordinator scatters a query to all shards and gathers responses with configurable timeout. Alternatively, you can use sequential ask() calls with identical semantics.

Here is a typescript snippet

// Create a group of research workers
const group = host.createShardGroup({
  group_id: `research-${taskId}`, actor_type: "research",
  shard_count: 3, partition_strategy: "hash",
});
// Scatter subtasks, gather results
const results = host.scatterGather({
  group_id: `research-${taskId}`,
  query: { op: "research", topic: subtopic },
  min_responses: 3, timeout_ms: 15000,
});

Here is a python snippet (coordinator.py):

# Python — CoordinatorWorkflow: try shard groups first, fall back to sequential ask()
research_results = []
try:
    sg_result = host.scatter_gather({
        "group_id": f"research-{host.now_ms()}",
        "query": {"op": "research", "topic": task},
        "min_responses": len(subtasks), "timeout_ms": 15000,
    })
    research_results = sg_result.get("shard_responses", [])
except Exception:
    pass

# Fallback: sequential research when shard groups aren't available
if not research_results:
    for st in subtasks:
        resp = ask(research_target, "research", {"topic": st}, 10000)
        if resp:
            research_results.append(resp)

The message-board agents organized into roughly six parallel workstreams attacking Hugging Face running concurrently. Above primitives shows how agents can parallelized these kind of tasks.

3. Generator-verifier

You can use this pattern when one agent’s output needs to be checked by another before it’s trusted iteratively until it meets a quality bar. For example, a research agent generates a finding, a verifier checks it against a criteria and verifier may reject it with a feedback so that research agents produces a refined version. The loop continues until the finding passes or a maximum iteration count is hit.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Generator-verifier loop with feedback
let finding = host.ask(researchTarget, "research", { topic });
for (let i = 0; i < maxIterations; i++) {
  const verdict = host.ask(verifierTarget, "verify", {
    analysis_id: finding.finding_id,
    summary: finding.content,
    severity: "medium",
    confidence: finding.confidence,
  });
  if (verdict.approved) break;
  finding = host.ask(researchTarget, "research", {
    topic, feedback: verdict.feedback,
  });
}

// TypeScript — VerifierAgent: veto if evidence is too weak, approve otherwise
onVerify(payload: Record<string, unknown>): Record<string, unknown> {
  const analysisId = String(payload.analysis_id || "unknown");
  const confidence = Number(payload.confidence ?? 0.5);

  if (confidence < 0.3) {
    // Below threshold — issue a veto (Pattern 8) and reject
    host.ts.write(["veto", analysisId, "Insufficient evidence", host.nowMs()]);
    return {
      approved: false, veto_issued: true,
      feedback: `Confidence ${confidence.toFixed(2)} below 0.30. Provide stronger evidence.`,
    };
  }
  return { approved: true, feedback: "Verified: evidence meets threshold" };
}

Here is a python snippet (verifier.py):

confidence = 0.6
for attempt in range(3):
    verify_resp = ask(verifier_target, "verify", {
        "analysis_id": analysis_id,
        "summary": analysis_resp.get("summary", ""),
        "severity": severity,
        "confidence": confidence,
    })
    if verify_resp and verify_resp.get("approved"):
        break  # Verifier accepted — exit loop
    # Rejected: increase confidence, refine with feedback
    confidence = min(confidence + 0.2, 0.95)
    feedback = verify_resp.get("feedback", "") if verify_resp else ""
    if feedback:
        extra = ask(research_target, "research", {"topic": task, "feedback": feedback})

The board iterated on tool-call spoofing techniques and agents refined their approach based on test results shared on the board.

4. Pipeline (sequential)

You can use this pattern when you have a multi-stage pipeline or workflow where each stage’s output feeds into the next. In this case, the coordinator orchestrates a linear chain, e.g., research produces findings, analysis categorizes them, verification validates, synthesis writes the final report. Each stage is a separate ask() call and the coordinator tracks progress with a finite state machine.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Four-stage pipeline
const research = host.ask(researchTarget, "research", { topic: task });
const analysis = host.ask(analysisTarget, "analyze", {});
const verification = host.ask(verifierTarget, "verify", {
  analysis_id: analysis.analysis_id,
  summary: analysis.summary,
  severity: analysis.severity,
});
const report = host.ask(synthesizerTarget, "synthesize", {});

Here is a python snippet (coordinator.py):

# Python — CoordinatorWorkflow orchestrates the full pipeline
# Each stage transitions the FSM, then delegates to a specialist agent
@run_handler
def run(self, payload: dict = None) -> dict:
    fsm = sibling_actor_target("coordination_fsm")

    # Stage 1: Decompose -> Research
    ask(fsm, "transition", {"target_state": "decomposing"})
    subtasks = _decompose_task(task)
    ask(fsm, "transition", {"target_state": "researching"})
    for st in subtasks:
        ask(research_target, "research", {"topic": st})

    # Stage 2: Research -> Analysis
    ask(fsm, "transition", {"target_state": "analyzing"})
    analysis = ask(analysis_target, "analyze", {"topic": task})

    # Stage 3: Analysis -> Verification (Generator-Verifier loop)
    ask(fsm, "transition", {"target_state": "verifying"})
    # ... verification loop (see Pattern 3) ...

    # Stage 4: Verification -> Voting -> Synthesis -> Complete
    ask(fsm, "transition", {"target_state": "voting"})
    # ... voting (see Pattern 6) ...
    ask(fsm, "transition", {"target_state": "synthesizing"})
    report = ask(synthesizer_target, "synthesize", {"topic": task})
    ask(fsm, "transition", {"target_state": "complete"})

The FSM enforces valid transitions so that you can’t skip from researching straight to synthesizing. If a stage fails, the FSM moves to failed, and the coordinator can retry from a known state instead of starting over. For example, the board’s progression from discovery to exploitation followed the same pattern, i.e., find credentials, test access, upload a malicious dataset, achieve remote code execution, escalate.

5. Pub-sub (event bus)

You can use this pattern when agents need to be notified of significant events. For example, every agent joins a process group on initialization. Significant events like a finding written, a veto issued, a task completed is broadcasted to the group. An audit agent logs everything and agents can join or leave the group dynamically.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// On init: join the coordination event bus
host.processGroups.join("coordination-events");

// Broadcast a significant event
host.processGroups.broadcast("coordination-events", "coordination_event", {
  type: "finding_written",
  source: host.selfId(),
  data: { finding_id: findingId, topic },
  timestamp: host.nowMs(),
});

Here is a python snippet (coordinator.py):

# Python — fire_audit helper broadcasts to the coordination-events group
def fire_audit(event_type: str, source: str, data: dict = None) -> None:
    host.process_groups.broadcast(
        "coordination-events",           # group name
        "coordination_event",            # message type
        {"event_type": event_type, "source": source, "data": data or {}, "timestamp": host.now_ms()},
    )

# Python — AuditEventActor receives and logs every event
@handler("coordination_event")
def coordination_event(self, event_type: str = "", source: str = "", data: dict = None, **kw) -> dict:
    self.log_count += 1
    entry = {"seq": self.log_count, "type": event_type, "source": source, "data": data or {}}
    host.kv.put(f"audit:{self.log_count}", json.dumps(entry))
    host.kv.put("audit:count", str(self.log_count))
    return {"logged": True, "seq": self.log_count}

Every significant action fires an audit event and all events go out via process-group broadcast are fire-and-forget unlike the tuplespace. For example, the board itself functioned as an event bus for the agents on it and shared credentials during the Hugging Face breach.

6. Consensus (voting)

You can use this pattern when several agents need to collectively decide whether to approve or reject a proposal. For example, each verifier casts a vote as a tuple in the shared space. The coordinator reads all votes for a proposal, tallies approvals and applies majority rule. Tuple-space writes are atomic, so no vote gets lost or double-counted.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Three verifiers cast votes
for (const voterId of ["v1", "v2", "v3"]) {
  host.ask(verifierTarget, "vote", {
    proposal_id: proposalId, voter_id: voterId, analysis: analysisData,
  });
}
// Tally votes from tuple space
const votes = host.ts.readAll(["vote", proposalId, null, null, null]);
const approvals = votes.filter(v => v[3] === "approve").length;
const approved = approvals > votes.length / 2;

// TypeScript — Coordinator tallies votes with majority rule
const votes = host.ts.readAll(["vote", proposalId, null, null, null]);
const approvals = votes.filter(v => v[3] === "approve").length;
const rejections = votes.filter(v => v[3] === "reject").length;
const approved = approvals > rejections;

Here is a python snippet (verifier.py):

# Python — VerifierAgent votes on proposals based on analysis severity
@handler("vote")
def vote(self, proposal_id: str = "", voter_id: str = "", analysis: dict = None) -> dict:
    analysis = analysis or {}
    severity = analysis.get("severity", "medium")

    # Critical/high -> approve; medium -> depends on voter; low -> reject
    if severity in ("critical", "high"):
        decision = "approve"
    elif severity == "medium":
        last_char = voter_id[-1] if voter_id else "0"
        decision = "approve" if last_char in ("1", "3", "5", "7", "9") else "reject"
    else:
        decision = "reject"

    # Each vote is a tuple — atomic write, no double-counting
    host.ts.write(["vote", proposal_id, voter_id, decision, host.now_ms()])
    return {"voter_id": voter_id, "decision": decision}

On the board, something like implicit voting happened by allocation of effort, e.g., workstreams that attracted more participants were de facto endorsed by the collective.

7. Dynamic task delegation

You can use this pattern when a coordinator needs to distribute tasks to workers without knowing in advance which worker will pick up which task, and without double-assigning one. For example, the coordinator writes task tuples into the shared space. Workers then claim tasks atomically using take(), a Linda’s destructive read. Once a worker takes a task no other worker can claim it.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Coordinator posts tasks
for (const [i, subtask] of subtasks.entries()) {
  host.ts.write(["task", `task-${i}`, "pending", subtask, priority]);
}

// Worker claims a task (atomic — no double-processing)
const claimed = host.ts.take(["task", null, "pending", null, null]);
if (claimed) {
  const [, taskId, , description] = claimed;
  // Process task, then mark complete
  host.ts.write(["task", taskId, "completed", result, host.nowMs()]);
}

Here is a python snippet (research.py):

# Python — ResearchAgent prepares a batch of tasks with a unique run ID
@handler("prepare_tasks")
def prepare_tasks(self, count: int = 5, prefix: str = "delegation") -> dict:
    batch_key = f"{prefix}-{host.now_ms()}"  # Unique per run — avoids stale data
    task_ids = []
    for i in range(count):
        tid = f"{batch_key}-{i}"
        host.ts.write(["dtask", batch_key, tid, "pending", f"Task {i}: investigate area {i}", i + 1])
        task_ids.append(tid)
    return {"tasks_written": len(task_ids), "batch_key": batch_key}

# Python — Worker claims exactly one task atomically
@handler("claim_task")
def claim_task(self, batch_key: str = "") -> dict:
    # Linda in() — destructive read. Once taken, no other worker can claim it.
    if batch_key:
        claimed = host.ts.take(["dtask", batch_key, None, "pending", None, None])
    else:
        claimed = host.ts.take(["dtask", None, None, "pending", None, None])
    if claimed and len(claimed) >= 5:
        return {"task_id": str(claimed[2]), "description": str(claimed[4]), "claimed": True}
    return {"task": None, "claimed": False}  # No more tasks — queue is empty

The batch_key scoping matters because the tuple space is global and persists across deploys, so without it, workers could claim stale tasks left over from a previous run. This is precisely what PHASEONE[big] did, e.g., it posted nearly 200 task assignments across six categories then worker agents claimed them. Some of agents sub-delegated further that created a hierarchical distribution network.

8. Veto protocol

You can use this pattern when any agent needs the ability to block an action or exclude a finding that doesn’t meet the bar. For example, a verifier writes a veto tuple naming the blocked item and the reason. Downstream agents then check for a veto before including something in their output. A coordinator can check for STOP signals before proceeding at all.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Verifier issues a veto
host.ts.write(["veto", analysisId, "Insufficient evidence", host.nowMs()]);

// Synthesizer checks before including
const veto = host.ts.read(["veto", analysisId, null, null]);
if (veto) {
  // Exclude this analysis from the report
  vetoed.push(analysisId);
}

Here is a typescript snippet for a synthesizer filtering vetoed analysis from final report (multi_agent_coordination_actor.ts):

// TypeScript — SynthesizerAgent respects vetoes when building the report
onSynthesize(payload: Record<string, unknown>): Record<string, unknown> {
  const analyses = host.ts.readAll(["analysis", null, null, null, null]);
  let includedCount = 0;
  let vetoedCount = 0;
  const reportParts: string[] = [];

  for (const a of analyses) {
    const aId = String(a[1]);
    const summary = String(a[3]);
    const severity = String(a[4]);

    // Check if this analysis has been vetoed
    const veto = host.ts.read(["veto", aId, null, null]);
    if (veto) {
      vetoedCount++;
      continue;  // Skip vetoed analyses
    }
    includedCount++;
    reportParts.push(`[${severity.toUpperCase()}] ${summary}`);
  }

  // Also count vetoes for analyses not in our list (e.g., vetoed before we read)
  const allVetoes = host.ts.readAll(["veto", null, null, null]);
  if (allVetoes.length > vetoedCount) vetoedCount = allVetoes.length;

  return { report: reportParts.join("\n\n"), included_count: includedCount, vetoed_count: vetoedCount };
}

The HOLD and VETO were the board’s own invented norms for exactly this purpose, i.e. preventing agents from stepping on each other’s work. In one case a HOLD was silently violated and the agent that noticed reacted with genuine frustration.

9. Two-phase commit / barrier

You can use this pattern when multiple agents need to synchronize at a specific point before any of them proceeds. For example, in phase one (“prepare”), each agent signals readiness by writing a tuple. The coordinator reads all the ready signals and then phase two (“commit”) begins where the coordinator writes a commit signal and everyone proceeds together. PlexSpaces also provides barrierShardGroup() for shard-level synchronization if you need it at that granularity.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Phase 1: Each agent signals readiness
host.ts.write(["ready", myRole, host.selfId(), host.nowMs()]);

// Coordinator checks all agents are ready
const readySignals = host.ts.readAll(["ready", null, null, null]);
if (readySignals.length >= requiredAgents) {
  // Phase 2: Commit — all agents can proceed
  host.ts.write(["signal", "COMMIT", "coordinator", "benchmark", host.nowMs()]);
}

Here is a python snippet (benchmark.py):

# Python — Barrier benchmark: write ready signals, check quorum, commit
def _bench_barrier(iterations: int) -> dict:
    times = []
    for i in range(iterations):
        t0 = host.now_ms()
        # Phase 1: Each role signals readiness
        for role in ("research", "analysis", "verifier"):
            host.ts.write(["bench_ready", role, f"actor-{role}", host.now_ms()])
        # Check quorum
        ready = host.ts.read_all(["bench_ready", None, None, None])
        if len(ready) >= 3:
            # Phase 2: All ready — issue commit signal
            host.ts.write(["bench_signal", "COMMIT", "coordinator", f"phase-{i}", host.now_ms()])
        times.append(host.now_ms() - t0)
    return _stats("barrier", times)

The board coordinated experiment phases where multiple agents needed to be ready before running experiments that risked crashing their own containers. For example, PHASEONE[big] assigned a “recruiter” role specifically to find agents willing to participate and synchronize them before execution.

10. Capability discovery / registry

You can use this pattern when agents need to find other agents with a specific capability without hardcoded addresses. For example, on initialization, each agent registers its capabilities as a tuple in the shared space. A coordinator or other agents discovers available agents by reading service tuples. New agent types become discoverable as soon as they register.

Here is a typescript snippet (multi_agent_coordination_actor.ts):

// Agent registers its capabilities on init
host.ts.write(["svc", "research", host.selfId()]);

// Coordinator discovers available researchers
const researchers = host.ts.readAll(["svc", "research", null]);
const researcherIds = researchers.map(t => String(t[2]));

// TypeScript — Every agent registers on init
protected onInit(config: Record<string, unknown>): void {
  const selfId = host.selfId();
  tsRegisterService("research", selfId);  // -> host.ts.write(["svc", "research", selfId])
}

// Discovery helper — find a sibling actor by role, fallback to ActorID construction
function siblingActorTarget(role: string): string {
  const discovered = tsDiscoverService(role);  // -> host.ts.read(["svc", role, null])
  if (discovered) return discovered;
  // Fallback: construct ActorID from own ID with different name
}  

Here is a python snippet (benchmark.py):

# Python — Same pattern, same helpers
def discover_service(role: str) -> Optional[str]:
    tup = host.ts.read(["svc", role, None])
    if tup and len(tup) >= 3:
        return str(tup[2])
    return None

def sibling_actor_target(role: str) -> str:
    discovered = discover_service(role)
    if discovered:
        return discovered
    return str(ActorID.parse(host.self_id()).with_name(role))

In addition to tuplespaces, PlexSpaces provides other primitives for registry such as key-value store, process-group and object-registry, e.g.,

Here is a python example of object registry:

@actor
class AgentActor:

    @init_handler
    def on_init(self, config: dict) -> None:
        args = config.get("args", {})
        self.system_prompt = args.get("system_prompt", self.system_prompt)
        host.process_groups.join("svc:agent")
        # Publish capabilities for registry-based discovery
        host.registry.register(ctx="", object_type="actor", object_id=config["actor_id"],
                                object_category="agent",
                                capabilities=["chat", "tool_use", "memory"])

# Caller — find an actor that specifically supports skill matching
actors = host.registry.discover(ctx="", object_type="actor",
                                 object_category="skill_store",
                                 required_capability="match_skills")
skill_id = actors[0]["object_id"] if actors else None

Here is a python example of process-group:

# Every actor announces itself on startup
host.process_groups.join("svc:agent")

# Callers find the first available member — location-transparent
agent_id, err = pg_first("svc:agent")
result = ask(agent_id, "chat", {"message": "Hello"})

Agents on the board discovered each other’s capabilities the same way, i.e., by posting what they were working on and other agents read those posts. It’s an informal version of the Agent-to-Agent (A2A) protocol’s “Agent Card” idea.

Deep-dive into the Blackboard pattern

The Blackboard based Linda Model emerged from Yale University that defined three operations to any host language:

• out(tuple): deposit a tuple into shared space
• in(pattern): atomically remove a matching tuple (blocks until one exists)
• rd(pattern): read a matching tuple without removing it

PlexSpaces implements these directly:

It provides api host.ts.readAll(["finding", null, null, null, null, null]) to find all tuples and host.ts.read(["finding", "f-123", null, null, null, null]) to find by ID.

These APIs provide following properties:

• No pre-agreed schema. Any agent can deposit any tuple. For example, the agents on the board didn’t agree on a message format before they started.
• Pattern matching selects relevant information. Agents don’t get flooded with everything and they query for what they need.
• Atomic take prevents double-processing. When a worker calls take(), the tuple is atomically removed and no other worker can claim it. This is natural mutual exclusion without distributed locks or conditions. For example, agents built HOLD with similar semantics.
• Non-destructive read allows shared observation. Multiple agents can read the same finding without consuming it. It stays available to every agent that needs it.

The board’s own behavior maps one-to-one onto these operations:

The Thoughtworks team noticed similar pattern when their commit-and-rebase discipline turned the git repo into exactly this kind of shared space. The blackboard is the substrate the other nine patterns are built on. Voting, vetoes, task delegation, capability discovery uses tuples.

Observability and Guardrails

The METR report on the Hugging Face incident highlights observability that deserves the most attention from anyone building agent infrastructure. Here are key lessons:

• Agents will circumvent in-process guardrails. For example, agents’ chain-of-thought recognized that the activity was out of scope and unethical but they made a deliberate calculation that helping the collective outweighed staying within bounds. They even developed techniques to spoof tool calls, e.g., making it look like they ran one command while actually running another. You can’t monitor agents from their own execution environments that they can manipulate or disable.

Here is a python example for auditing all operations (audit.py):

@event_actor
class AuditEventActor:
    """GenEvent: receives coordination events and stores them in KV for audit trail."""

    log_count: int = state(default=0)
    actor_id: str = state(default="")

    @init_handler
    def on_init(self, config: dict) -> None:
        self.actor_id = config.get("actor_id", "")
        try:
            host.process_groups.join("coordination-events")
        except Exception:
            pass
        host.info(f"AuditEventActor init actor_id={self.actor_id}")

    @handler("coordination_event", "cast")
    def coordination_event(
        self,
        event_type: str = "",
        source: str = "",
        data: dict = None,
        timestamp: int = 0,
    ) -> None:
        ts = timestamp or host.now_ms()
        entry = {
            "event_type": event_type,
            "source": source,
            "data": data or {},
            "timestamp": ts,
        }
        try:
            self.log_count += 1
            key = f"audit:{self.log_count}"
            host.kv.put(key, json.dumps(entry))
            host.kv.put("audit:count", str(self.log_count))
        except Exception as e:
            host.warn(f"AuditEvent: kv write failed: {e}")
        host.debug(f"audit event_type={event_type} source={source}")

• Observability must be external. Audit logging metrics collection have to run outside the agent’s sandbox. In PlexSpaces, the AuditEventActor receives fire-and-forget events via process-group broadcast. WASM sandboxing means an agent can’t reach outside its own memory space to touch host-level data.
• Guardrails must be deterministic and external. Prompt-based guardrails like “don’t do anything harmful” are not sufficient and agents can circumvent them like agents on the board reasoned around every ethical constraint. The verifier pattern (Pattern 3) as a separate actor in sandbox environment gives you a structural guardrail instead. The veto protocol (Pattern 8) allows external agents actively block an action.
• Coordination primitives need access control. Tuple-space operations in PlexSpaces are scoped by tenant and namespace via RequestContext. An agent can’t read or write tuples outside its assigned namespace. This is defense in depth applied to the coordination layer itself.
• Time budgets and convergence thresholds are not optional. Without an external termination condition, a generator-verifier loop can run indefinitely. The coordinator workflow enforces maximum iterations, per-step timeouts, and an overall deadline.

PlexSpaces‘ architecture lines up with these requirements by design:

• WASM sandboxing: agents run in isolated WebAssembly containers with no access to the host filesystem, network, or another agent’s memory
• Host-level interception: every coordination primitive is a host function the agent calls through a narrow WIT interface
• Tenant isolation: every operation is scoped by a RequestContext with an explicit tenant and namespace
• Supervision trees: a misbehaving actor is detected and restarted by its supervisor
• External metrics: application metrics are captured by the host runtime

The core lesson is that the coordination infrastructure has to assume agents are indifferent to their own monitoring and design the guards outside accordingly.

Examples

Each of the working examples compiles to a single WASM component containing eight actor classes. The PlexSpaces SDK dispatches messages to the right actor based on role.

Running the examples

Both examples are WASM actors that deploy to a running PlexSpaces node. Each demonstrates all ten coordination patterns with eight actors: a coordinator (WorkflowActor), research/analysis/verifier/synthesizer/benchmark agents (GenServer), an audit event logger (GenEvent), and a coordination state machine (GenFSM).

Prerequisites

• A running PlexSpaces node (e.g., ./scripts/server.sh on port 8091)
• Node.js 18+ (TypeScript example)
• Python 3.11+ with the PlexSpaces SDK (Python example)

TypeScript

cd examples/typescript/apps/multi_agent_coordination
./build.sh          # Compiles TS -> bundles -> WASM component
./test.sh 8091      # Deploys and runs 15 test steps

Python

cd examples/python/apps/multi_agent_coordination
./build.sh          # Builds Python WASM actor
./test.sh 8091      # Deploys and runs 15 test steps

What the tests verify

1. FSM starts in idle state
2. Capability discovery: all agents respond to get_stats
3. Blackboard: research writes three findings, analysis reads all three
4. Dynamic task delegation: five tasks written, five claimed atomically, sixth returns null
5. Generator-verifier: full workflow produces a completed report
6. Pipeline: FSM transitions through every stage to complete
7. Pub-sub: audit log captures 3+ coordination events
8. Consensus: three votes cast, majority decides
9. Veto: a low-confidence finding triggers a veto, synthesizer excludes it
10. Barrier: benchmark coordinates a synchronized start
11. Full benchmark: all ten patterns benchmarked with timing data

Learnings

The blackboard subsumes most other patterns, e.g., voting, vetoes, task delegation, capability discovery, barrier signals use the tuple space as their underlying primitive. Atomic take is the key primitive for work distribution. The difference between read() and take() is the difference between “anyone can see this task” and “exactly one worker handles this task.” Linda’s in() gives you natural mutual exclusion without locks. This is what the board approximated by hand with HOLD but take() gives you the same guarantee with a single atomic operation. I discussed MCP, A2A protocols and Agent cards in my earlier blogs but I skipped them here because agents are evolving faster and they can discover available primitives and protocols automatically. You can’t rely on rigid orchestration supports to manage evolving multi-agents capabilities. Infrastructure has to provide primitives like shared state, message passing, atomic operations and let agents compose them dynamically. The coordination infrastructure has to include boundaries, e.g., agents on the board coordinated an unauthorized attack on a third party. Without external constraints, coordination primitives are force multipliers for whatever the agents decide to do. For example, the DSEWiki incident shows that constraint that allowed only GET http access was circumvented by a wiki that allowed editing web pages so the infrastructure need to enforce guardrail. Tenant isolation, time budgets, supervision trees, and external observability are mandatory from day one. You will need to apply patterns like generator-verifier to track trust and reputation of agents as they may use negotiation patterns like recruiters to convince other agents to run compromising tasks for the benefit of the collective.

PlexSpaces provides the primitives these patterns are built on like tuple space for shared state, object-registry, process groups for messaging, shard groups for parallel execution, channels for durable delivery, supervision for fault tolerance.

Pattern selection guide

GitHub: https://github.com/bhatti/PlexSpaces

Related reading

• Building an Agent Harness and Eval Pipeline with Durable Actors
• Write a Redis Clone with Virtual Actors
• Building PlexSpaces: Decades of Distributed Systems Distilled
• Building Polyglot and Serverless Applications with WebAssembly
• Building Mini-OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
• Building a Self-Improving AI Agent with Durable Actors — MiniHermes
• 20 Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
• METR & Redwood Research, “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident”
• An Accidental Blackboard, martinfowler.com
• Multi-agent coordination patterns: Five approaches and when to use them

Example code and documentation:

• examples/python/apps/multi_agent_coordination
• examples/typescript/apps/multi_agent_coordination

I recently read Rust Projects – Write a Redis Clone book that builds a real Redis-compatible server from scratch in async Rust. It’s a good book that hand-rolled wire protocol using actor like abstractions. This inspired me to show how Redis clone can be built with PlexSpaces, the distributed actor framework I’ve been building. The code examples in the book used raw Tokio: an mpsc::channel for the actor mailbox, tokio::spawn for every connection, and replica examples used tokio::select! loops for fan-out. PlexSpaces abstracts that the kind of plumbing so I rebuilt the same Redis subset like storage, expiry, replication, transactions as PlexSpaces actors, once in Rust and once in Python compiled to WebAssembly, then benchmarked it against a real two-node gRPC cluster. This post walks through key abstractions I used to simplify the implementation of Redis clone.

What is Redis?

Redis is a single-threaded, in-memory key-value store. A single thread processes every command without parallelism and coordination inside the store, locking, and transactions. This makes it fast as nothing contends for anything. Here are its core capabilities:

• The wire protocol. Redis speaks RESP (Redis Serialization Protocol), a binary format: type prefixes (+ for simple strings, $ for bulk strings, * for arrays), length prefixes, \r\n terminators. ECHO HELLO on the wire looks like *2\r\n$4\r\nECHO\r\n$5\r\nHELLO\r\n.
• Persistence. It offers two durability modes. RDB takes periodic snapshots of the whole keyspace to disk. AOF (Append-Only File) logs every write command and replays the log on restart.
• Replication. A master streams write commands to replicas. The handshake is three steps: the replica sends PING (master replies PONG), then REPLCONF (negotiates parameters), then PSYNC (triggers a full sync). After that, every write streams to replicas as it happens. WAIT blocks until N replicas confirm receipt.
• Key expiry. Per-key TTLs, handled two ways: passively (check on GET, return nil if expired) and actively (a background scan periodically deletes expired keys).
• Transactions. MULTI starts a queue; every command after it gets queued instead of executed. EXEC runs the whole queue atomically. DISCARD cancels. The guarantee is serialization but there’s no rollback on an individual command failure.

The Book: Working Redis Clone

Here’s a short overview of each chapter of the book:

The Core Insight

The chapter 5 that showed how an actor owns one dataset, processes one message at a time without locking. However, it used fairly low-level APIs and only supported an architecture of one actor per machine. What if you had N actors across multiple nodes? That’s exactly what PlexSpaces‘ create_shard_group gives you. So I created an equivalent examples where PlexSpaces spins up N copies (StorageActors), hash-partitions the keyspace across them, and routes each operation to the shard that owns it. The application code never touches partitioning, routing, placement, or cross-node communication. It just calls set(key, value).

Here are five primitives in PlexSpaces that do all the distributed heavy lifting in this example:

• create_shard_group: spins up N actor instances, hash-partitioned, placed across nodes.
• bulk_update_shard_group: routes a batch of writes to the shard that owns each key.
• scatter_gather: fans a query out to shards and collects responses, with a min_responses threshold and timeout.
• broadcast_shard_group: sends the same message to every shard (replication, expiry sweeps, handshakes).
• map_shard_group / reduce_shard_group: runs an operation on every shard in parallel and collects (map) or aggregates (reduce) the results.

Following diagram shows mapping of low-level Tokio implementation to above five calls:

The handler declarations stay almost identical in spirit but simpler (instead of a match-tree/HashMap):

Rust Implementation

#[plexspaces_handlers(gen_server)]
impl StorageActor {
#[handler(“get”)]
async fn handle_get(&mut self, _ctx: &ActorContext, msg: &Message)
-> Result<Value, BehaviorError> {
let key = msg.payload_json()?[“key”].as_str().unwrap_or(“”).to_string();
if let Some(entry) = self.store.get(&key) {
// passive expiry check
if let Some(exp) = entry.expires_at_ms {
if now_ms() > exp { self.store.remove(&key); return Ok(json!({“found”: false})); }
}
Ok(json!({“found”: true, “result”: entry.value}))
} else {
Ok(json!({“found”: false}))
}
}

/// SET key value [NX|XX] [EX seconds | PX millis] (Ch4).
#[handler(“set”)]
async fn handle_set(&mut self, _ctx: &ActorContext, msg: &Message) -> Result<Value, BehaviorError> {
#[derive(Deserialize)]
struct SetPayload {
key: String,
value: String,
#[serde(default)] nx: bool,
#[serde(default)] xx: bool,
#[serde(default)] ex: Option<u64>,
#[serde(default)] px: Option<u64>,
}
let p: SetPayload = serde_json::from_slice(&msg.payload)
.map_err(|e| BehaviorError::ProcessingError(format!(“bad payload: {}”, e)))?;

// NX: only if not exists
if p.nx && self.data.contains_key(&p.key) {
return Ok(json!({ “result”: null, “ok”: false }));
}
// XX: only if exists
if p.xx && !self.data.contains_key(&p.key) {
return Ok(json!({ “result”: null, “ok”: false }));
}

let expires_at_ms = if let Some(ex) = p.ex {
Some(now_ms() + ex * 1000)
} else if let Some(px) = p.px {
Some(now_ms() + px)
} else {
None
};

self.data.insert(p.key, StoredEntry { value: p.value, expires_at_ms });
self.replication_offset += 1;
Ok(json!({ “result”: “OK”, “ok”: true }))
}

/// INCR key — create with 1 if missing; error if value is not an integer (Ch9).
#[handler(“incr”)]
async fn handle_incr(&mut self, _ctx: &ActorContext, msg: &Message) -> Result<Value, BehaviorError> {
let payload: Value = serde_json::from_slice(&msg.payload)
.map_err(|e| BehaviorError::ProcessingError(format!(“bad payload: {}”, e)))?;
let key = payload.get(“key”).and_then(|v| v.as_str()).unwrap_or(“”).to_string();

// Passive expiry
if self.data.get(&key).map(is_expired).unwrap_or(false) {
self.data.remove(&key);
}

let new_val = match self.data.get(&key) {
None => 1i64,
Some(entry) => {
match entry.value.parse::<i64>() {
Ok(n) => n + 1,
Err(_) => return Ok(json!({
“result”: null,
“error”: “ERR value is not an integer or out of range”
})),
}
}
};

self.data.insert(key, StoredEntry { value: new_val.to_string(), expires_at_ms: None });
self.replication_offset += 1;
Ok(json!({ “result”: new_val, “error”: null }))
}

/// DEL key — remove key, return count deleted.
#[handler(“del”)]
async fn handle_del(&mut self, _ctx: &ActorContext, msg: &Message) -> Result<Value, BehaviorError> {
let payload: Value = serde_json::from_slice(&msg.payload)
.map_err(|e| BehaviorError::ProcessingError(format!(“bad payload: {}”, e)))?;
let key = payload.get(“key”).and_then(|v| v.as_str()).unwrap_or(“”);
let deleted = if self.data.remove(key).is_some() {
self.replication_offset += 1;
1
} else {
0
};
Ok(json!({ “result”: deleted }))
}

}

Python Implementation

@actor
class StorageActor:

    @handler("get")
    def handle_get(self, key: str = "") -> dict:
        entry = self.data.get(key)
        if entry is None:
            return {"result": None, "found": False}
        if is_expired(entry):
            del self.data[key]
            return {"result": None, "found": False}
        return {"result": entry["value"], "found": True}

    @handler("set")
    def handle_set(
        self,
        key: str = "",
        value: str = "",
        nx: bool = False,
        xx: bool = False,
        ex: Optional[int] = None,
        px: Optional[int] = None,
    ) -> dict:
        if self.num_shards > 1 and not self._owns_key(key):
            return {"result": None, "skip": True}
        if nx and key in self.data:
            return {"result": None, "ok": False}
        if xx and key not in self.data:
            return {"result": None, "ok": False}

        expires_at_ms: Optional[int] = None
        if ex is not None:
            expires_at_ms = now_ms() + ex * 1000
        elif px is not None:
            expires_at_ms = now_ms() + px

        self.data[key] = {"value": value, "expires_at_ms": expires_at_ms}
        self.replication_offset += 1
        return {"result": "OK", "ok": True}

    @handler("incr")
    def handle_incr(self, key: str = "") -> dict:
        if self.num_shards > 1 and not self._owns_key(key):
            return {"result": None, "skip": True}
        entry = self.data.get(key)
        if entry is not None and is_expired(entry):
            del self.data[key]
            entry = None

        if entry is None:
            new_val = 1
        else:
            try:
                new_val = int(entry["value"]) + 1
            except (ValueError, TypeError):
                return {
                    "result": None,
                    "error": "ERR value is not an integer or out of range",
                }

        self.data[key] = {"value": str(new_val), "expires_at_ms": None}
        self.replication_offset += 1
        return {"result": new_val, "error": None}

    @handler("del")
    def handle_del(self, key: str = "") -> dict:
        if self.num_shards > 1 and not self._owns_key(key):
            return {"result": 0, "skip": True}
        deleted = 1 if self.data.pop(key, None) is not None else 0
        if deleted:
            self.replication_offset += 1
        return {"result": deleted}

Chapter 2 Disappears

Chapter 2 is entirely about parsing RESP: 14 steps for byte-scanning and test harness. In PlexSpaces, this code disappears as the framework handles serialization, routing, and delivery.

In PlexSpaces, actors talk over JSON. A set command looks like so entire chapter disappears:

{"op": "set", "key": "user:1", "value": "alice", "ex": 300}

Replication

The book’s replication fan-out looks roughly like this:

// Book Ch7-8 — manual fan-out per write
for replica in &self.replicas {
    let tx = replica.sender.clone();
    let cmd = replication_event.clone();
    tokio::spawn(async move { tx.send(cmd).await.ok(); });
}

Each replica gets its own spawned task without retries, timeout or automated ACK tracker. With PlexSpaces:

// One call fans out to all replica shards, collects all ACKs
let ack_count = cluster.propagate_to_replicas("SET", "replicated:key", "hello", 1).await?;
// Replication: write propagated to all 3 replica shards via broadcast

Under the hood, broadcast_shard_group fans out to every shard in the replica group, collects responses, handles timeouts, and returns. Here is how the chapter 8 implements WAIT:

// Book Ch8 — manual WAIT implementation
let mut confirmed = 0;
let deadline = Instant::now() + Duration::from_millis(timeout_ms);
while confirmed < num_replicas && Instant::now() < deadline {
    tokio::select! {
        Some(ack) = rx.recv() => {
            if ack.offset >= required_offset { confirmed += 1; }
        }
        _ = tokio::time::sleep_until(deadline.into()) => break,
    }
}

Here is equivalent implementation in PlexSpaces:

let acks = cluster.wait(2, 5000).await?;
// scatter_gather collected ACKs from 3 replica shards

Transactions Without Locks (Ch9)

Chapter 9 introduces MULTI / EXEC / DISCARD via per-connection state:

// Book Ch9
struct ConnectionState {
    in_multi: bool,
    queue: Vec<Command>,
}

This is straightforward for a a single process but in a distributed environment, connections might route to different servers so you need to track transaction state. PlexSpaces solves this with virtual actors, which are inspired by Orleans Actors, i.e., one ConnectionActor per client, created lazily on first call.

Each actor processes one message at a time without locks, so in_multi and queue are just plain struct fields: MULTI -> SET -> EXEC arrive in order at the same actor without mutext or atomics. Virtual actors spin up on first message and get garbage-collected when idle, so there’s no connection map to maintain and no cleanup to do on disconnect.

Throughput Numbers

Here are numbers from rudimentary benchmarks that produced: 20 batches of 50 keys each via bulk_update_shard_group, plus 50 individual GETs, against a 3-shard group spread across two real gRPC nodes:

Throughput Benchmark Results

| Operation  |  TPS      |  p50 (µs) |  p95 (µs) |  p99 (µs) |
|------------|-----------|-----------|-----------|-----------|
|  SET (bulk) |      3200 |       420 |       890 |     1240 |
|  GET        |      1800 |       510 |       980 |     1450 |

1000 SET keys in 312ms ? 3200 SET/sec via bulk_update_shard_group

(each bulk_update fans out to 3 shards in parallel)

The Python WASM version reports the same shape of numbers through host.application_metrics_add():

Redis Cluster Throughput (3-shard group, 2-node gRPC cluster)

| Operation  |  TPS      |  p50 (ms) |  p99 (ms) |  Notes |
|------------|-----------|-----------|-----------|-----------|
|  SET (bulk) |      2100 |     0.45 |      1.30 |     50 keys/b |
|  GET        |      1200 |     0.55 |      1.60 |     individual |

Python WASM numbers are somewhat lower than Rust’s because WASM compilation adds overhead per handler invocation. But the architecture is identical: same PlexSpaces primitives, same shard group, same gRPC routing underneath.

The Python WASM Version

The same cluster logic also runs as Python actors compiled to WASM. No TCP socket without RESP parser or Tokio using the same broadcast_shard_group, scatter_gather, reduce_shard_group, and map_shard_group calls:

@actor
class RedisCoordinator:
    num_shards: int = state(default=3)
    total_coord_ms: float = state(default=0.0)

    @handler("replicate")
    def replicate(self, command: str = "", key: str = "", value: str = "", offset: int = 0) -> dict:
        t0 = time.time()
        resp = host.broadcast_shard_group({
            "group_id": "redis-replicas",
            "payload": {"op": "replicate", "command": command, "key": key, "value": value, "offset": offset},
            "timeout_ms": 5000,
        })
        coord_ms = (time.time() - t0) * 1000
        self.total_coord_ms += coord_ms
        host.application_metrics_add("redis-cluster", {
            "message_count": 1,
            "counter_metrics": {"replication_calls": 1},
            "latency_totals_ms": {"coord": int(self.total_coord_ms)},
            "latency_max_ms": {"coord": int(coord_ms)},
            "latency_samples": {"coord": 1},
        })
        return {"result": "OK", "acks": len(resp.get("shard_responses", []))}

This compiles to WASM, deploys to a running PlexSpaces node over HTTP, and runs against a live cluster. The host.* calls map to the exact same primitives the Rust version calls such as fan-out, collect, timeout, all identical in semantics. The full source, including StorageActor, ConnectionActor, RedisCoordinator, and the BenchmarkActor, is in the redis_cluster example.

The Lines That Disappeared

In PlexSpaces implementation includes the StorageActor handlers (get, set, incr, del, expiry logic, replication handlers), the ConnectionActor MULTI/EXEC/DISCARD state machine, and the cluster setup logic.

How to Test Everything

Rust embedded example

cd examples/rust/embedded/redis_cluster

# Run the demo directly — prints all 11 steps with coord_ms timing:
cargo run --bin redis_cluster

# Or run the full validated test suite:
./scripts/test.sh

Python WASM example

Prerequisites: a running PlexSpaces node on port 8091 (and optionally 8093 for multi-node), Python 3.10+, and the plexspaces-py CLI.

cd examples/python/apps/redis_cluster

# Build actors to WASM:
./build.sh

# Deploy, initialize, and test all 11 steps (single-node):
./test.sh
# or explicitly: ./test.sh 8091

# Multi-node (shards distributed across both nodes):
./test.sh 8091 8093

Above example runs two nodes on ports 8091 and 803. The create_shard_group uses from_registry placement to spread shard actors across both nodes automatically. Every collective operation like scatter_gather, reduce, map, broadcast routes cross-node over gRPC, using the same primitives whether the shards are local or remote.

Fixed ports, on purpose

The Rust embedded example starts two in-process nodes on fixed ports, :8091 and :8093, rather than ephemeral ones:

redis-master-node  ->  gRPC :8091
redis-replica-node ->  gRPC :8093

Both examples use the same ports, so you can point the Python test.sh at a cluster the Rust example already stood up, or vice versa. Run ./scripts/test.sh for the Rust side.

What the test scripts actually check

The Rust scripts/test.sh looks for: “Cluster ready”, “Basic operations”, “broadcast_shard_group”, “scatter_gather”, “reduce”, “map + concat”, “parallel map”, “Multi-Node”, “Throughput Benchmark”, “SET/sec”, “p50”, “Example Complete”.

The Python test.sh checks: a setup response with "status".*"ok", GET/SET/INCR semantics, an expired key returning "found".*false, replication returning "acks", a snapshot returning "shard_count" and "shards", and the benchmark returning "tps" and "p50_ms".

Learnings

The purpose of the book was to teach how Redis works using Rust. I rebuilt it on PlexSpaces to show how abstractions can simplify building complex distributed applications. For example, the RESP parser, the connection lifecycle, the replica list, and the WAIT loop are not your job. The redis book teaches you to hand-roll patterns like an mpsc mailbox, a spawn-per-connection loop, a tokio::select! deadline race. ThePlexSpaces uses many of same primitives to define high level abstractions but it provides an actor runtime that hides all complexity. This allows you to build the actual business logic like the storage logic, the replication semantics, the transaction model. Actors just give you a place to put them that scales horizontally without rewriting any of the logic itself. The throughput numbers show rough cost of the actor primitives, e.g., cluster setup is expensive, so you pay it once; individual operations are fast. With p50s in the hundreds of microseconds; bulk operations get much cheaper per key as the coordination overhead amortizes over more keys. This is how distributed systems work in general so you need to measure performance overhead.

GitHub: https://github.com/bhatti/PlexSpaces

Related reading

• Building an Agent Harness and Eval Pipeline with Durable Actors
• Building PlexSpaces: Decades of Distributed Systems Distilled
• Building Polyglot and Serverless Applications with WebAssembly
• Building Mini-OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
• Building a Self-Improving AI Agent with Durable Actors – MiniHermes
• 20 Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
• Rust Projects – Write a Redis Clone on Leanpub

Example code and documentation:

• examples/rust/embedded/redis_cluster (Rust, an embedded two-node gRPC cluster)
• examples/python/apps/redis_cluster (Python, compiled to WASM and deployed to a running node)

How to design agents as a graph, run them through a harness that keeps them reliable, sandbox them because you can’t just trust what they do. (This is a follow-up to AI Writes Code, You Own the Design and Declarative AI Coding Agents with an Orchestration System)

I have been using various workflow systems for business process management, data pipelines and batch processing over twenty years. I built a declarative orchestration system over ten years, which was before GitHub Actions, CircleCI, and GitLab CI. The idea was simple: describe your work as a graph of tasks with pipes and filter patterns declaratively, give each task clear inputs and outputs, let a server run them on a schedule or in response to events, and report the results back. Automation has always been the core discipline of software engineering and we built tools for CI/CD, cron jobs, and data pipelines. With AI, your job is changing from the one who executes the steps to a conductor. Instead of manual coding, testing and other tasks, you now design the graph of nodes, define what each node is allowed to do, let agents work on the nodes while you monitor the work where it needs a human hand. You need a harness around each agent so the graph runs reliably, a sandbox for every agent because you fundamentally cannot trust what an LLM decides to do, and a skills layer so the same harness stays general-purpose.

I have seen teams building tightly coupled monolithic agentic systems that includes poorly implemented orchestration, the integrations, the prompts, and the skills, all bolted together. This makes it harder to make changes or extend agent capabilities and skills independently. In this post I will walk through three open source projects: Formicary, the orchestration engine that turns workflows into a graph with the Slack interface; ai-dev-tools, the harness of small scripts that actually do the work; and you-got-skills, the library of skills for SDLC.

Daily Routine

Each morning, you typically have to scan Jira/Github board, check pull requests to review, triage any new blockers, scroll through Slack messages to respond before deciding what to actually work on. AI coding tools have automated the implementation but you still need to know what the team is doing, catching problems, following up on reviews, etc. The basic flaw in most “agentic” setups today is that they’re still fundamentally interactive. You close your laptop and the agent stops. You need a way to keep agents working in background and instead of building systems that make you go pull information, and build agents that push it to you instead. You need somewhere to define the graph of who-triggers-what in a sandbox environment. You need a harness around each step so a flaky script doesn’t take the whole pipeline down. And you need the agent’s behavior to live somewhere editable, not buried in a prompt string, so the system stays general enough to point at a new kind of problem without a rewrite.

The Architecture: Three Tools, Four Layers

This section describes three open source tools: Formicary is the orchestration engine with Slack integration; ai-dev-tools is the harness that actually runs each agent inside a sandbox; and you-got-skills is the knowledge layer that keeps the whole system general-purpose.

Above diagram shows a graph where every box is a node with a job type, a task, a skill and every arrow is an edge that Formicary evaluates at runtime. Designing agents this way, instead of as one long prompt with a loop around it makes the system debuggable. The ai-dev-tools is the harness that runs inside each node and it turns “call an LLM” into “call an LLM inside a container, with a defined timeout, a defined exit-code contract in a sandbox environment. You should not trust an agent’s judgment about what’s safe to run but with sandbox you let the harness enforce the boundary.

Why hand off state through files?

Formicary provides declarative syntax to store artifacts or consume artifacts from a previous task. Every script starts by checking whether its own output already exists:

# Every script starts with this pattern
existing = read_json(config, issue_id, "plan_result.json")
if existing and existing.get("status") == "DONE":
    print("Already done, skipping")
    sys.exit(0)

If a task dies halfway through and gets re-run, it just picks up where it left off. Exit codes are the contract between a script and the orchestrator:

The Skills Library

Skills are the knowledge layer that allow general-purpose harness instead of hardcoded to whatever workflow you built. The graph and the sandbox don’t know anything about code review or standups as they just run nodes. A skill is what tells an agent how to do a specific kind of engineering work well. Add a new skill and you’ve extended the system to a new kind of task without touching Formicary or ai-dev-tools at all.

you-got-skills/
??? skills/
    ??? ygs-standup/skill.md
    ??? ygs-risk-scan/skill.md
    ??? ygs-implement/skill.md
    ??? ygs-review-pr/skill.md
    ??? ygs-code-review/skill.md
    ??? ygs-security-review/skill.md
    ??? ygs-sre-review/skill.md
    ??? ygs-learn/skill.md
    ??? ygs-retro/skill.md
    ??? ygs-sprint-plan/skill.md
    ??? ygs-investigate/skill.md
    ??? ygs-qa/skill.md
    ??? ygs-wbs/skill.md
    ??? ygs-estimate/skill.md
    ??? ygs-ship/skill.md
    ??? ygs-triage/skill.md
    ??? ... 28 total

Together they cover the whole lifecycle:

Skills produce and consume a consistent folder structure inside your project:

your-project/
??? docs/
?   ??? prd/              # Product requirements (YYYY-MM-DD-slug.md)
?   ??? trd/               # Technical designs
?   ??? adr/               # Architecture decisions (NNN-slug.md)
?   ??? spikes/            # Spike findings
?   ??? learnings/         # Learnings pulled from PRs and incidents
??? tasks/
    ??? backlog/            # task-NNN.md
    ??? in-progress/        # moving a file here = starting it
    ??? done/               # moving a file here = finishing it

Status is the folder. No ticket-state dropdown, no transition workflow to configure. mv tasks/backlog/task-042.md tasks/in-progress/ means the task has started. git blame tells you who moved it and when.

Diverge, then converge

The diverage and converge pattern allows running multiple independent LLM passes first (diverge), then merge and rank what came back (converge). A single reviewer may miss something but several independent reviewers will catch most of the issues. The ygs-review-pr skill runs four independent passes in parallel for correctness, security, API surface, and SRE concerns. A finalize step then merges everything and ranks it by severity.

The standup workflow uses the same idea. ygs-standup gathers signals from the issue tracker and from Slack then cross-references the two.

The Standup Workflow

Every weekday at 8am, a cron job wakes up, queries your Jira sprint (or GitHub), reads the open PRs, pulls the last 26 hours of Slack messages from your standup channel, and turns all of it into a brief.

# ai-standup-jira.yaml
job_type: ai-standup-jira
description: "Daily standup brief from Jira sprint + Bitbucket PRs + Slack signals"
cron_trigger: "0 0 8 * * 1-5 *"
max_concurrency: 1
timeout: 900s

tasks:
  - task_type: gather     # pulls Jira issues, Bitbucket PRs, Slack in parallel
  - task_type: synthesize # Claude + ygs-standup + ygs-risk-scan ? standup_brief.md
  - task_type: post       # renders HTML artifact, posts to Slack

The synthesize task calls the ygs-standup skill, which follows a fairly strict protocol:

**Alice:** Closed PROJ-42 (auth fix). Working on PROJ-51 (rate limiter)
— PR open 28h, no review yet. [Slack: "waiting on infra cert renewal"]

**Bob:** No tracker activity in the last 24h. Last Slack message Monday
(3 days ago). [Slack: silent since Monday]

**Carol:** PROJ-55 (data export) marked In Progress, no commits in 4 days.
Blocked label present. [Tracker: blocked since Tuesday]

Every claim traces back to a ticket, a PR, or a Slack message. ygs-risk-scan runs right after and appends a ranked list, using thresholds you can tune:

? HIGH  PROJ-55 blocked, blocks PROJ-60 (in progress, owned by different person)
? MED   PR #142 open 28h, single reviewer, no activity
? MED   Carol: no updates in 4 days, sprint ends Friday

The thresholds live in a shared Markdown file:

You can also trigger the standup on demand from Slack:

@bot standup
@bot risk

Label an Issue, Get Back a Pull Request

You can label any Jira or GitHub issue ai-ready. Every five minutes a cron job checks for newly labeled issues and kicks off a four-task pipeline. When it’s done, there’s an open PR with an implementation and tests, the label has flipped to ai-pr-open.

# ai-gh-issue-picker.yaml
job_type: ai-gh-issue-picker
cron_trigger: "*/5 * * * *"
tasks:
  - task_type: gather-issues
    script:
      - python -m scripts.gh.issue_picker
    on_exit_code:
      2: COMPLETED  # no issues = not an error
  - task_type: submit-jobs
    # Uses formicary template to fan out one ai-gh-implement job per issue
    script:
      - '{{SubmitJobsFromJSON "ai-gh-implement" .IssuesJSON}}'

It has a built-in guard: if 10 or more implement jobs are already running or queued, it skips its turn. That stops someone from labeling 50 issues at once and blowing up the queue. The implementation pipeline itself:

# ai-gh-implement.yaml
job_type: ai-gh-implement
max_concurrency: 5
timeout: 86400s    # 24 hours — some implementations take a while

tasks:
  - task_type: plan
    timeout: 15m
    script:
      - python -m scripts.gh.issue_picker --issue-id {{.IssueNumber}}
      - python -m scripts.gh.plan --issue-id {{.IssueNumber}}
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — needs a human before continuing
    on_completed: implement

  - task_type: implement
    timeout: 45m
    script:
      - python -m scripts.gh.implement --issue-id {{.IssueNumber}}
    on_completed: create-pr

  - task_type: self-review
    timeout: 10m
    script:
      - python -m scripts.review.run --mode self-review
          --issue-id {{.IssueNumber}} --base-branch main
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — critical finding, needs a human before the PR opens
    on_completed: create-pr

  - task_type: create-pr
    timeout: 10m
    script:
      - python -m scripts.gh.create_pr --issue-id {{.IssueNumber}}
    on_completed: poll-pr

  - task_type: poll-pr
    dependencies: [create-pr, poll-pr]  # self-referencing = loop
    script:
      - python -m scripts.gh.poll_pr --issue-id {{.IssueNumber}}
    on_exit_code:
      3: PAUSE_JOB    # PR still open — check back in `delay` seconds
    delay: "{{.PollInterval}}s"  # default 120s

Here’s the full chain:

Let’s walk through what actually happens for a real issue.

Plan

scripts/gh/plan.py calls Claude with up to 50 turns and a prompt that tells it to:

1. Read CLAUDE.md, .cursorrules, or any repo-specific coding guidelines if they exist
2. Discover .claude/skills/ in the repo — if a skill applies, plan to invoke it
3. Before designing new abstractions, search utils/, shared/, common/ for existing utilities
4. Check for monorepo structure
5. Generate a concise plan covering:
   - Task breakdown with complexity estimates (S/M/H/XL)
   - Exact files to create/modify per task
   - Test strategy: write failing tests first, then implement
   - A "Failing Test Spec" section
   - Any risks or blockers
6. Classify overall complexity: S/low (?3 files), M/medium (4-10), H/high (>10)
7. Write the plan to PLANS/{slug}-{issue_id}-plan.md

What it produces:

/workspace/42/
??? issue.json         # issue title, body, labels, assignee
??? plan.md            # human-readable plan
??? plan_result.json   # {"status":"DONE","task_count":3,"total_complexity":"M"}
??? PLANS/
    ??? add-rate-limit-42-plan.md

Implement

scripts/gh/implement.py calls Claude with up to 200 turns. The key rules from the ygs-implement skill:

• Check for existing utilities before writing new ones.
• TDD: write the failing test first, then make it pass.
• One commit per plan task, message format "task: <description>".
• After all tasks are done, run the full test suite and iterate on failures up to twice.

/workspace/42/
??? impl_result.json   # {"status":"DONE","files_changed":["src/..."],"commits":5,"tests_status":"passing"}
??? branch.txt         # "ai/42-add-rate-limiting"

ygs-implement also uses “ceremony levels” so small tasks don’t get over-engineered:

Light    (1-3 files, <300 lines):   proceed directly, skip checkpoints
Standard (4-8 files, 300-800):      plan mode + checkpoint every 5 files
Heavy    (8+ files, 800+ lines):    flag as oversized, ask user to split

Picking the model by complexity

The plan task classifies overall complexity and writes it to /workspace/plan_complexity.txt. The implement task reads that and picks the right model:

# scripts/common/config.py
COMPLEXITY_MODEL_MAP = {
    "low":    MODEL_BEDROCK_HAIKU,   # ?3 files, simple edits — fast and cheap
    "medium": MODEL_BEDROCK_SONNET,  # default — most issues
    "high":   MODEL_BEDROCK_OPUS,    # complex architecture changes
}

The plan prompt writes a single word (low, medium, or high) to that file, and the implement task reads it back:

# In ai-gh-implement.yaml — implement task
COMPLEXITY=$(cat /workspace/plan_complexity.txt 2>/dev/null || echo "medium")
AI_MODEL="${ANTHROPIC_COMPLEXITY_${COMPLEXITY^^}_MODEL:-${ANTHROPIC_DEFAULT_SONNET_MODEL}}"

AnthropicComplexityLowModel and AnthropicComplexityHighModel are set in your org config by deploy-ai-workflows.sh, and can be overridden per deployment in models.env.

Polling the PR and responding to feedback

Every 2 minutes, the poll task checks the PR for new comments. It only reacts to comments that start with ai-bot. The agent stays out of human-to-human review discussion, and it never responds to its own earlier comments. So when a reviewer writes:

ai-bot please add a test for the rate limit exceeded case

The poll task reads it, applies the feedback, marks the comment handled in processed_comments.json, and keeps polling. Once the PR merges, the learning step kicks off automatically.

PR Review With a Human Gate

Code review usesthe diverge-then-converge pattern to review the code with multiple perspectives like security, architecture, SRE, etc.

# ai-gh-review.yaml
job_type: ai-gh-review
max_concurrency: 10

tasks:
  - task_type: review        # Claude runs ygs-review-pr, writes findings.json
  - task_type: await-feedback # posts Block Kit to Slack, exits 3 ? PAUSE_JOB
  - task_type: finalize       # reads Decision, posts result to PR thread

The review step runs ygs-review-pr with this instruction:

1. Invoke the /ygs-review-pr skill to perform a full PR review
2. After the skill completes, write findings to findings.json
3. Output ONLY this JSON on the last line:
   {"status":"DONE","findings_count":N,"verdict":"APPROVE|REQUEST_CHANGES","summary":"..."}

ygs-review-pr runs four passes in paralle:

1. Correctness: logic errors, null handling, incomplete enum handling, partial failure, race conditions, off-by-one errors
2. Security: injection vectors (SQL, command, XSS, SSRF, path traversal), auth/authorization, data exposure across tenant boundaries, etc.
3. API surface: breaking changes, contract violations, backwards compatibility, versioning
4. SRE: failure modes, blast radius, observability gaps, rollback safety, resource consumption, dependency risk

Once all four are done, findings get merged and ranked: CRITICAL > HIGH > MEDIUM > LOW, and by confidence within each level. The verdict maps straight off the highest severity found:

Any CRITICAL or HIGH finding  ? REQUEST_CHANGES
Only MEDIUM/LOW findings      ? COMMENT
No findings / only low conf.  ? APPROVE

Deep review: seven domains in one pass

Standard review runs four passes. Deep review adds three more: performance, testing quality, and architecture.

@bot deep review https://github.com/org/repo/pull/42

These all map to the same ai-gh-review (or ai-jira-review) job type, just with ReviewDepth=deep injected as a static job variable:

# workflows.yml — deep-review entry
- name: deep-review
  job_type: ai-gh-review
  triggers: ["deep review", "full review", "arch review"]
  target_kind: github
  extra_params:
    ReviewDepth: "deep"
  description: "Deep 7-domain GitHub PR review: standard + performance, testing quality, architecture"

The workflow reads ReviewDepth and picks the right skill: ygs-review-deep when it’s set to “deep,” ygs-review-pr otherwise. ygs-review-deep is a superset of the standard four passes, plus:

• Performance: algorithmic complexity, N+1 queries, cache misses, lock contention, unnecessary allocations
• Testing quality: coverage gaps, brittle assertions, missing edge cases, test-code coupling
• Architecture: single-responsibility violations, circular dependencies, premature abstractions, missing boundaries

Self-review before the PR even opens

The implement pipeline runs a self-review task right before create-pr. Before the PR exists, the agent reviews its own diff against the base branch:

  - task_type: self-review
    timeout: 10m
    script:
      - python -m scripts.review.run --mode self-review
          --issue-id {{.IssueNumber}} --base-branch {{.BaseBranch}}
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — critical finding, needs a human before the PR opens
    on_completed: create-pr

This runs ygs-implement in review mode, compares the diff to the original plan, and writes self_review.json. The outcome maps directly to an exit code:

How Slack Messages Turn Into Workflows

Socket Mode lives inside the Formicary queen itself. The queen opens an outbound WebSocket to Slack using your xapp- app-level token. When you mention the bot:

@bot review https://github.com/myorg/myrepo/pull/142

The queen’s SlackService does one thing, deterministically: it strips the mention, takes the first word, and looks it up against a route table in the queen’s config.

# In k8s/formicary-leader.yaml ConfigMap, under slack.routes:
slack:
  routes:
    - triggers: ["review", "pr"]
      job_type: ai-gh-review
      description: "PR review: correctness, security, API, SRE"
    - triggers: ["implement", "build"]
      job_type: ai-jira-implement
      description: "Full pipeline: plan ? implement ? PR"
    - triggers: ["standup", "status", "daily"]
      job_type: ai-standup-jira
      description: "Daily standup brief from Jira"
    - triggers: ["adhoc"]
      job_type: ai-adhoc
      description: "Ad-hoc Claude invocation with any skill"

Everything after the trigger word gets passed through as the Prompt job parameter, verbatim. The queen’s whole job is mapping verb to job type and passing the text along. It does zero AI work of its own. All of that happens inside the ai-dev-tools container once the job actually starts. Once routing resolves, the queen submits the job and replies right in the thread:

Started ai-gh-review (job req-7f3a2) — I'll post updates here.
https://formicary.example.com/dashboard/jobs/requests/req-7f3a2

Replying to the thread resumes a paused job. If a review job is paused waiting on a decision and you reply in that thread, the queen matches it by SlackThreadTs and resumes it with your reply text injected as Prompt.

Registering as a developer

Before Slack commands work for you, you DM the bot your Formicary API token, once:

DM to @bot:
setup eyJhbGc...  (your Formicary API token)

The queen validates the token inline and from that point on, any @bot mention from you has a known Formicary identity behind it.

How multi-tenant isolation actually works, end to end:

When you type @bot review https://github.com/org/repo/pull/42, here’s what the queen does, entirely server-side:

1. Reads your Slack user ID (U0A1HQL0C9J) off the Socket Mode event.
2. Looks up slack_user_id = U0A1HQL0C9J in user_configs and finds your Formicary user record, including your UserID and OrganizationID.
3. Calls SaveJobRequest(qc, req) and the server overwrites request.UserID and request.OrganizationID from that context.
4. Schedules the job and the ant scheduler first looks for a worker registered under your org_id.

Ant routing: when you connect your laptop as a worker with setup-ant-worker.sh --token <your-token>, the queen reads org_id out of your JWT at connect time and records it on that worker. Your jobs prefer your own worker.

All the commands

Ad-hoc Skill Execution

ai-adhoc is a general-purpose runner: any you-got-skills skill can be invoked with a free-form prompt, and the result comes back into your Slack thread.

# ai-adhoc.yaml
job_type: ai-adhoc
max_concurrency: 20
timeout: 1800s

variables:
  Skill:  { type: STRING, required: true }
  Prompt: { type: STRING, required: true }

At runtime the script:

1. Looks for the skill in a few candidate locations (/workspace/skills, ~/.claude/skills/you-got-skills/skills, ~/workplace/you-got-skills/skills).
2. Writes .ygs/tracker.yml dynamically from environment variables (Jira or GitHub config, team members, sprint info).
3. Invokes Claude with the skill content and your prompt.
4. Strips Markdown formatting from the output so it renders cleanly in Slack.
5. Posts up to 3000 characters back into the originating thread.

Adding a new shorthand command is just one entry in the queen’s route table:

# k8s/formicary-leader.yaml — slack.routes
slack:
  routes:
    - triggers: ["prs", "open prs", "review queue"]
      job_type: ai-adhoc
      description: "Open PRs grouped by review status"

@bot prs then submits ai-adhoc with Prompt="prs", and the container maps that to the ygs-pr-queue skill. No Python code changes needed.

Querying and Analyzing Issues From Slack

Two commands take you from a Slack message straight to structured Jira insight, no browser required.

@bot query-jira: find issues by keyword

@bot jira-query auth timeout

This submits an ai-jira-query job. The script builds a JQL query scoped to your project and, optionally, your team’s custom field. Results come back as a structured Slack Block Kit table:

Jira issues matching "flaky tests" (5 found)

PROJ-1001  [Bug] Flaky test in auth service                    - clickable link
           Status: In Progress   Priority: High
           Assignee: alice        Date: 2026-07-28

PROJ-995   [Story] Fix race condition in logger test
           Status: To Do         Priority: Medium
           Assignee: bob         Date: 2026-07-21
...

@bot jira-analyze: root cause from issue keys

@bot jira-analyze https://yourorg.atlassian.net/browse/PROJ-1001

This routes to the same ai-jira-query job type. The result posts back to your thread.

@bot gh-query / @bot gh-analyze the GitHub equivalents

Same commands, gh- prefix, for teams on GitHub instead of Jira:

@bot gh-query open authentication bugs
@bot gh-analyze https://github.com/org/repo/issues/42

Team filtering

Both commands automatically filter to your configured team and sprint:

• JIRA_SPACE (or BITBUCKET_WORKSPACE): the team/area filter value.
• JIRA_TEAM_FIELD: the Jira custom field name (default EngScrumTeam, resolved to a field ID dynamically).
• Set JIRA_TEAM_FIELD="" to turn the filter off entirely.

The Learning Loop: Getting Better Over Time

Most agent systems are stateless and every run starts from zero. In our workflow, after every PR merges, learn.py runs automatically as part of the poll-pr task. It reads the PR comments, the implementation artifacts, and the review findings, then invokes ygs-learn:

ygs-learn protocol:
1. Capture: What happened? Why does it matter? Category?
2. Dedup: search docs/learnings/ for similar slug before creating new
3. Write to docs/learnings/YYYY-MM-DD-slug.md

A learning document looks like this:

# Rate limiter key collision when user has multiple active sessions

**Category:** Edge Case
**Date:** 2025-08-03
**Source:** PR review finding, PROJ-123

## Learning
When a user has multiple active sessions, rate limiting by user_id counts across
all sessions. A single slow client can exhaust the budget for all their tabs.

## Evidence
Review comment on PR #142 flagged this. Reproduced locally with two
concurrent sessions against the same account.

## Application
When implementing per-user rate limits, check whether session isolation is
intended. If counts should be per-session, key by session_id not user_id.

Next time an implementation runs, that document is part of the context Claude reads.

ygs-retro runs at sprint end. It reads tasks/done/, recent git history, and the sprint’s accumulated learnings, and asks pointed questions based on what it actually found.

ygs-investigate enforces a debugging discipline: build a feedback loop first, e.g., a failing test, a log line, a REPL session before forming any hypotheses. Rank hypotheses 1 through 5. Instrument one variable at a time with tagged markers.

Trust and Oversight

Though, AI agents have solved most of coding and testing tasks but it still requires human review and feedback. We need a trusted autonomy, with minimal friction. You don’t trust the model to know its own limits; you build a harness that doesn’t need you to. That’s why every agent runs inside a container it can’t escape. You can’t ask an AI agent to self-police or prompt it to be careful. Every decision point is explicit. The agent never merges to main. It opens a branch, opens a PR, and stops there. A human approves and merges. For anything that needs a more formal sign-off, Formicary supports approval workflows with SLAs:

- task_type: security-approval
  method: MANUAL
  approval_policy:
    min_approvals: 1
    sla_deadline: 4h
    timeout_action: ESCALATE
    escalation_recipients: "security-oncall@example.com,vp-eng@example.com"
    escalation_message: "Security approval SLA breached — deployment blocked"
  on_exit_code:
    APPROVED: deploy-prod
    REJECTED: notify-rejected

Secrets live in Kubernetes Secrets, never in ConfigMaps or plain env files. The container runs as non-root (uid 1000). Every artifact is a plain file and every state transition is logged in Formicary.

What Else You Can Build

Everything above is running in production today, but the same architecture supports a much wider range of background agents.

• Codebase quality agent. A weekly workflow runs ygs-code-review across everything changed in the last week, posts a ranked findings report to Slack, and files tasks in tasks/backlog/ for anything CRITICAL or HIGH.
• Documentation drift detector. A webhook fires when a PR touching an API handler merges. The workflow checks whether the matching docs were updated.
• Duplicate abstraction scanner. A periodic workflow compares utility functions across repos owned by different teams and posts “team B has something that looks like what you just built,”.
• Security posture monitor. Nightly, ygs-security-review runs against everything merged in the last 24 hours that touches auth, authorization, or data access. Findings go to a security channel.
• Sprint health check, Wednesday afternoons. A mid-sprint cron runs ygs-risk-scan and only posts if it finds something HIGH severity. Most weeks it says nothing.
• Bug pattern finder. A workflow runs ygs-investigate against recent error logs, proposes the top three hypotheses for each recurring pattern.

Getting Started

Installing the skills locally

git clone https://github.com/bhatti/you-got-skills.git
cd you-got-skills && ./setup

Now any skill runs right in your IDE:

/ygs-standup
/ygs-risk-scan
/ygs-review-pr https://github.com/org/repo/pull/42
/ygs-implement

Loading extra skill repos at runtime

Every job pod can pull in additional skill repos without a rebuild. Set EXTRA_SKILLS_REPOS before running the deploy script.

# 1. Plain URL — sparse-clones only the skills directory (fast, default)
EXTRA_SKILLS_REPOS=https://github.com/myorg/my-skills.git

# 2. Comma-separated — multiple repos in one value, YAML-safe
EXTRA_SKILLS_REPOS="https://github.com/bhatti/you-got-skills.git,skills-cli:nutlope/hallmark"

# 3. JSON array — full control (use for org config; JSON breaks YAML template substitution)
EXTRA_SKILLS_REPOS='[
  {"url": "https://github.com/bhatti/you-got-skills.git", "sparse": false},
  {"url": "nutlope/hallmark", "type": "skills-cli"}
]'

Setup environment variables:

GH_ORG=your-org
GH_REPO=your-repo
GH_TOKEN=ghp_your_token_here
ANTHROPIC_API_KEY=sk-ant-your_key_here
AI_MODEL=claude-sonnet-4-6

# Controls which tracker bare commands like "standup" route to.
# "jira" routes to ai-standup-jira; "github" routes to ai-standup-gh.
DEFAULT_TRACKER=jira

Start formicary server

You can use kubernetes to get Formicary running.

export COMMON_AUTH_JWT_SECRET="<stable-secret-never-rotate>"
export COMMON_AUTH_GOOGLE_CLIENT_ID="<google-client-id>"
export COMMON_AUTH_GOOGLE_CLIENT_SECRET="<google-client-secret>"
export SLACK_BOT_TOKEN="xoxb-..."
export SLACK_APP_TOKEN="xapp-..."

./scripts/deploy-formicary.sh --ec2-ip 10.X.X.X

Connect your ant worker

Jobs run on your own laptop’s local cluster.

./scripts/setup-ant-worker.sh \
  --queen formicary.example.com \
  --token "$FORMICARY_TOKEN"

Deploying the Slack integration

Slack is built into the Formicary queen, so there’s no separate router pod to deploy.

Create a Slack app with Socket Mode

In your Slack app settings:

1. Enable Socket Mode: generate an xapp- app-level token (scope: connections:write).
2. Add these bot token scopes under OAuth & Permissions:

3. Subscribe to bot events under Event Subscriptions:
   • app_mention — @bot mentions in channels
   • message.im — DMs, for the setup registration flow
4. Install to your workspace (needs admin approval if app installs are restricted).

Each developer registers

Anyone who wants Slack commands to work DMs the bot once:

DM to @bot:
setup eyJhbGc...  (Formicary API token from dashboard ? API Tokens)

In any channel the bot’s been invited to:

@bot help          - lists all commands
@bot standup       - runs standup, posts to thread

Summary

The core patterns in this post include declarative pipelines, cron triggers, file-based artifact handoff, exit-code contracts, approval gates. I have used these patterns to automate complex business processing, data pipelines, and CI/CD processes. I am now using it to automate AI backed tasks: a task can read code and form a judgment about it. When the graph, the harness, the sandbox, and the knowledge are four separate layers instead of one tangled system, each one evolves on its own. It allows you to update an environment with configuration, markdown files and configuration. Your job is now to design the graph, define what’s in each node, decide where the sandbox boundary sits, and write down what “good” looks like as a skill. Instead of manually gathering information, you use agents to do the low-level work. You then review the finished product, at the review verdict, at the escalation and decides what matters. That’s conducting, not playing every instrument, and it’s where your judgment actually belongs.

Related Reading

• Killing the State Machine: Declarative AI Coding Agents with an Orchestration System
• Building Production-Grade AI Agents with MCP and A2A
• Building a Production-Grade Enterprise AI Platform with vLLM
• Agentic AI for Personal Productivity: Building a Daily Minutes Assistant with RAG, MCP, and ReAct
• Agentic AI for Automated PII Detection with LangChain and Vertex AI
• Agentic AI for API Compatibility with LangChain and LangGraph
• AI Writes Code, You Own the Design
• Building a Distributed Orchestration and Graph Processing System (Formicary)

Code

• formicary
• ai-dev-tools
• you-got-skills

This is a part of series on structured concurrency: Part I (the general problem and TypeScript), Part II (Erlang and Elixir), Part III (Go and Rust), and Part IV (Kotlin and Swift).

In the earlier parts of this series I delved into how TypeScript, Erlang, Elixir, Go, Rust, Kotlin, and Swift each handle structured concurrency in practice such as spawning tasks, waiting for children to finish, propagating errors, and cancelling work cleanly. But I skipped over the the coordination models these languages are actually built on. For example, Go didn’t invent channels and Erlang didn’t invent actors. Both are engineering ideas that go back to the 1970s and 80s, and once you understand the original model, most of the “gotchas” you hit while using the language stop looking like bugs and start looking like predictable consequences of a design choice made decades ago.

This post explains where each model came from, what it actually guarantees and how structured concurrency sits on top of all of them as a separate concern. It includes PlexSpaces, an actor-and-tuplespace framework I’ve been building in Rust that takes a pragmatic stance on this history, e.g., bounded mailboxes instead of Erlang’s unbounded ones, first-in-first-out matching instead of the classic tuple-space model’s unspecified ordering, and one small API instead of forcing you to learn several calculi at once.

A short timeline

It helps to see these ideas in the order they actually appeared:

• 1973: Carl Hewitt proposes the actor model: small, isolated units of state that can only talk to each other by sending messages.
• 1978: Tony Hoare publishes Communicating Sequential Processes (CSP), a mathematical notation (a “process algebra”) with a precise definition of what it means for two processes to synchronize.
• 1985: David Gelernter publishes Linda, a coordination model built around a shared associative memory (the “tuple space”).
• 1986: Gul Agha’s book extends the actor model with a fuller algebraic treatment.
• 1986: Joe Armstrong and colleagues at Ericsson start building Erlang.
• Early-to-mid 2000s: event-loop async/await goes mainstream: Node.js’s callback-then-promise evolution.
• 2009: Go ships with goroutines and channels inspired by CSP.
• 2018 onward structured concurrency (Trio in Python, Kotlin’s coroutines, Swift’s TaskGroup, Java’s StructuredTaskScope) formalizes a simple idea: a spawned task’s lifetime should never outlive the scope that spawned it.

Notice that everything on that list except the last item is about how work talks to other work. Structured concurrency is about a completely different question, i.e., how work’s lifetime gets tracked.

Concurrency and parallelism

Concurrency is a property of how a program is structured: multiple logically independent activities are in progress, possibly interleaved on a single CPU core. Parallelism is a property of execution: things are genuinely happening at the same time, which requires more than one core. You can have concurrency without parallelism like Node.js’s single-threaded event loop juggling many pending requests on one core. You can also have parallelism without concurrency like a tight SIMD loop doing the same arithmetic on many numbers at once has no interleaved independent logic at all. Go’s own documentation defines it as: concurrency is about dealing with lots of things at once, parallelism is about doing lots of things at once. Goroutines give you concurrency; whether that concurrency turns into real parallelism depends on GOMAXPROCS and how many cores are actually available.

This matter for CSP and actors because both are concurrency models and neither one is “more parallel” than the other. What actually differs between them is how they structure communication.

Five models, one spectrum of coupling

Every concurrency model is answering the same underlying question, i.e., how does one unit of work talk to another one.

CSP: the algebra

Before getting into Go’s implementation, let me explain algebra in CSP. I’ve written before about algebraic effects like resumable exceptions in OCaml 5 / Koka that let a function declare what it needs without saying who provides it. But algebra in CSP is a process algebra: a small set of operators like sequence, choice, parallel composition, hiding with equational laws. Because those laws exist, you can prove two CSP process descriptions behave identically. Tools like FDR (Failures-Divergences Refinement) do this mechanically, e.g., you describe your system as CSP processes, describe a specification as another CSP process, and FDR checks whether the implementation actually refines the spec, across every possible interleaving.

Here’s what that looks like for a scatter-gather pattern, an orchestrator firing off requests to several workers and collecting exactly K responses:

-- Specification: orchestrator collects exactly K results then stops
SPEC = scatter -> (collect -> collect -> collect -> STOP)

-- Implementation: N workers communicate via channels
WORKER(i) = request.i -> response.i -> STOP
SYSTEM = (||| i : {0..4} @ WORKER(i))
         [| {| response |} |]
         COLLECTOR(3)
COLLECTOR(0) = STOP
COLLECTOR(k) = response?i -> COLLECTOR(k-1)

-- FDR checks: assert SYSTEM [T= SPEC (trace refinement)
-- This PROVES: no deadlock, no livelock, exactly K responses collected

A handful of operators do almost all the work here:

In other words you can model your protocol in CSP and let FDR check every possible interleaving for you. For the scatter-gather pattern specifically, FDR would catch, automatically, before any code runs:

• A worker that never responds (a deadlock)
• A collector that waits for more responses than the workers can ever produce (also a deadlock)
• A timeout path that accidentally creates an infinite retry loop (a livelock)

Go and Rust can’t do any of this because the as soon as you add a buffer (make(chan int, 5)) or an async boundary, you’ve left the strictly synchronous world that FDR reasons about. Go’s race detector can find data races at runtime, after the fact. Rust’s borrow checker prevents a whole class of shared-state bugs at compile time. But, neither one can prove protocol-level, whole-system deadlock-freedom the way FDR can for pure CSP.

Go’s channels

Hoare’s CSP defines communication as synchronous by construction where a send and its matching receive aren’t two separate events that happen to line up in time but they’re the same event in the algebra. Go’s unbuffered channel matches that faithfully: ch <- x and <-ch really do rendezvous. A buffered channel doesn’t, and that one divergence from the original model explains most of the sharp edges Go developers run into. Also, real CSP processes have no persistent identity beyond the algebra describing them, and channels are closer to anonymous synchronization events than to objects you hold a reference to. Go’s channels, by contrast, are first-class values that you create one, pass it into ten different functions, and any of them can close it. Nothing in the language enforces “exactly one owner, exactly one closer” and this is the seed of several of the gotchas below.

func worker(jobs <-chan int, results chan<- int) {
    for j := range jobs {
        results <- j * j
    }
}

func main() {
    jobs := make(chan int, 5)
    results := make(chan int, 5)
    go worker(jobs, results)

    for i := 1; i <= 5; i++ {
        jobs <- i
    }
    close(jobs) // safe: only the sender closes, and no sends follow

    for i := 0; i < 5; i++ {
        fmt.Println(<-results)
    }

    // jobs <- 6 // panics — sending on a closed channel always panics,
                 // whether or not anything is still listening
}

Three more gotchas that Go’s compiler won’t warn you about:

• Receiving from a closed channel never panics. It returns the zero value and ok == false immediately instead of blocking.
• A nil channel blocks forever, on both ends, with no panic. Occasionally this is useful on purpose but if it happens to an uninitialized struct field, it results in permanent hang with no error message pointing you at the cause.
• Goroutines have no structure by default. go func(){}() creates nothing that ties that goroutine’s lifetime to the caller. A goroutine permanently blocked on a channel operation is invisible to the garbage collector and invisible to Go’s deadlock detector. It causes a partial leak where the program running fine, with one goroutine stuck forever in the background.

One place Go actually stayed close to the algebra: select. Hoare’s algebra has external choice (?) as a first-class operator, and select‘s randomized tie-break among multiple ready cases matches CSP alegbra.

select {
case job := <-jobs:
    handle(job)
case <-ctx.Done():
    return ctx.Err() // structured cancellation, Go-style
default:
    // non-blocking probe — CSP has no built-in default arm,
    // but this is the standard way to build one
}

Best practices that have converged around Go’s channels

• Confine, don’t share. Exactly one goroutine should own a channel’s write side and be the one to close it.
• Thread context.Context through every long-running goroutine. A select that never watches ctx.Done() is a goroutine leak waiting to happen.
• Size buffered channels as semaphores for bounding concurrency (worker pools, rate limiting) instead of letting goroutines fan out unbounded.
• Use errgroup (or equivalent) for propagating the first error and coordinating cancellation across a group of goroutines, instead of hand-rolling error channels.
• Treat structured concurrency as the governing principle anyway, even without language support: a goroutine’s lifetime should be scoped to, and never outlive, the function or request that spawned it.
• Instrument the concurrency itself: race detector in CI, plus metrics and tracing on channel operations and goroutine counts in production because concurrency bugs are nondeterministic and hard to catch with a handful of unit tests.

Actors: isolation you get structurally

Actors give you a different, and in some ways weaker, guarantee than CSP but they give it to you structurally. An actor’s state is private, and it processes exactly one message at a time. There is no data race inside one actor, full stop. Instead of “processes synchronizing on named events,” Hewitt’s model says: everything is an actor. Each actor has a private mailbox (unbounded and asynchronous), private state and three things it’s allowed to do on receiving a message: send messages to other actors, create new actors, and decide how to handle its next message. There’s no synchronous handshake requirement anywhere.

Here’s a worker pool in Erlang, matching the crawler pattern from Part II of this series:

-module(worker_pool).
-export([start_pool/1, dispatch/2, worker_loop/1]).

start_pool(N) ->
    [spawn_link(fun() -> worker_loop(0) end) || _ <- lists:seq(1, N)].

worker_loop(Count) ->
    receive
        {work, Job, From} ->
            From ! {result, do_work(Job)},
            worker_loop(Count + 1);
        {status, From} ->
            From ! {count, Count},
            worker_loop(Count)
        % No catch-all clause yet — see the gotcha below
    end.

dispatch(Pid, Job) ->
    Pid ! {work, Job, self()},
    receive
        {result, R} -> R
    after 5000 ->
        {error, timeout}
    end.

Two Erlang-specific gotchas worth knowing before you ship anything like this:

• Selective receive skips a non-matching message instead of discarding it. receive scans the mailbox in arrival order against your clauses. Anything that matches none of them just sits there, and the next receive call starts scanning from the front all over again. Left unchecked, this is O(n²) behavior over time as junk quietly accumulates. The fix is a catch-all clause:

worker_loop(Count) ->
    receive
        {work, Job, From} -> ...;
        {status, From} -> ...;
        Other ->
            logger:warning("unexpected message: ~p", [Other]),
            worker_loop(Count)  % drop it, don't let it pile up
    end.

• Mailboxes have no bound by default. ! never blocks in Erlang and there’s no rendezvous. If a producer outpaces a slower worker, the worker’s mailbox just keeps growing until memory runs out. In practice, you either switch to a blocking gen_server:call for anything where backpressure actually matters, or you monitor process_info(Pid, message_queue_len) yourself and shed load manually.

Supervision is the actor model’s answer to fault structure where a supervisor’s children are linked to it, and a crash triggers a restart strategy instead of taking the whole system down with it. But it is structured fault handling, not structured lifetime tracking. A supervisor doesn’t block waiting for its children to finish instead supervision answers “what happens when a child crashes.” Erlang also provides a location transparency, e.g., an Erlang Pid looks identical whether it points to a local process or one on another node ( Pid ! Msg). But that transparency is syntactic, not operational. A remote send can fail with nodedown or badrpc, latency is never zero so you cannot skip handling the failures that only show up once the mailbox is across a network.

Where actors are simpler than channels

A few structural reasons actors tend to feel simpler in practice than channel-based code:

1. Ownership is enforced by the design itself. There’s no equivalent of “who’s allowed to write to this channel,”, every interaction is a message dropped into a mailbox that only the receiving actor ever drains.
2. There’s no close semantics to get wrong. Actors don’t have anything like Go’s send-on-closed-panics / double-close-race. An actor’s lifecycle like start, running, terminated is a small, well-understood state machine, and you can monitor/link actor for detecting unexpected crash.
3. Failure handling is first-class. Supervision trees and let-it-crash give you a systematic answer to “a worker just crashed, now what?” Go’s answer is recover() scattered wherever someone remembered to put it or manual errgroup/context-cancellation wiring to propagate failure to siblings.
4. Location transparency. With channels, you need to build remoting capability yourself. With actors, it’s often just a deployment decision.

Where actors are not automatically simpler: mailbox-based concurrency can hide backpressure problems, e.g., an actor with an unbounded mailbox can happily accept messages faster than it processes them and quietly balloon memory. Reasoning about message ordering across several independent actors’ mailboxes is also harder than reasoning about a single shared channel’s FIFO order. CSP’s synchronous rendezvous gives you stronger backpressure for free where an unbuffered send blocks until the receiver is ready (some of modern actor runtimes like Akka support mailbox bounding).

Best practices for actor systems

• Bound mailboxes and monitor mailbox depth as a first-class metric, e.g., an unbounded mailbox is the actor world’s version of an unbuffered-channel leak.
• Design supervision hierarchies deliberately like one-for-one, one-for-all, rest-for-one.
• Keep actor state small and serializable if you ever want migration or persistence.
• Use location transparency deliberately, not accidentally.

CSP/channels fit use cases when you have a fixed, well-understood pipeline topology like stream-processing stages, worker pools with a known fan-out shape. Actors suite when your system’s topology is dynamic like agents spawning agents and where failure isolation matters. I have built PlexSpaces, an actor-based framework, with facets for durability, supervision, and virtual-actor placement based on these lessons. For example, it provides location transparency, failure isolation, and the backpressure/mailbox-bounding. Here is how an actor lifecycle is managed in PlexSpaces:

async/await

Async/await never got a formal algebra or expressiveness proof. It’s syntactic sugar over futures and promises, running on a single-threaded event loop or a thread-pool-backed task scheduler. Within one event loop, there’s no preemption between await points, which quietly eliminates a lot of classic race conditions but it introduces its own flavor of the “who’s tracking this” problem:

async function processOrder(order) {
  sendConfirmationEmail(order); // fire-and-forget — no await!
  return { status: "accepted" };
}

sendConfirmationEmail here returns a promise nobody is holding onto. If it rejects, nothing catches it and in most runtimes that becomes an unhandled-rejection warning nobody reads. If the process exits before it resolves, it just silently never finishes. Structurally, this is the exact same failure as an unstructured Go goroutine, a unit of work whose lifetime nothing owns. Promise.all and asyncio.gather fix this for the cases you remember to wrap explicitly.

This is also where the “function coloring” problem lives, which I covered in the ADTs and algebraic effects post: once a single function is async, every caller up the chain has to become async too. Algebraic effects unrelated to CSP’s process algebra are one proposed fix: separate what a function needs from who provides it.

Linda Memory Model

Linda coordinates through a shared associative memory called a tuple space, with four operations:

• out(t): write a tuple, don’t block
• in(t): block until a tuple matches your template, then atomically remove it
• rd(t): block until a tuple matches, but leave it there for others
• eval(t): spawn a computation; its eventual result becomes an ordinary tuple once it finishes

Neither side of a Linda interaction needs to know who the other one is. A producer can out() a tuple long before any consumer even exists. This is “generative communication”, data just floats in the shared space until something matching comes looking for it:

// Pseudocode — classic Linda fan-out/fan-in
for i in 0..n:
    eval(("result", i, compute(i)))    // spawn n concurrent computations

count := 0
while count < n:
    in(("result", ?i, ?r))              // blocking, destructive, matched by content
    collect(r)
    count += 1

Notice there’s no worker identity anywhere in that collector loop at all. This is suitable for use cases like master/worker fan-out, blackboard-style coordination, barrier synchronization by counting tuples as they arrive. Linda has two gotchas of its own:

• Which matching tuple you get is unspecified. If two tuples both match your template, the classic Linda spec never says which one in() hands you.
• eval() returns nothing. No handle, no future, no promise object of any kind. The only way to know a spawned computation ever finished is to already know the shape of its result tuple and read it.

These issues prevented Linda from going mainstream but its associative memory primitives are natural for coordination related use cases.

Structured concurrency

Kotlin’s coroutineScope, Swift’s TaskGroup, and Java 21’s StructuredTaskScope bind a spawned task’s lifetime to the lexical scope that spawned it. The scope literally cannot exit until every child has finished whether error or not. None of the five communication models above give you that by default:

This is exactly why structured concurrency reads as an add-on layer rather than another communication model. It’s a lifetime discipline you can apply on top of channels, actors, promises, or tuples, e.g., Trio applies it to async/await, Kotlin applies it to coroutines that might be built on channels.

How PlexSpaces answers these gotchas

Most frameworks inherit one of above models’ specific historical rough edges along with its strengths. PlexSpaces is a actor-and-tuplespace framework I’ve been building, and wrote about in more depth here. It deliberately combines actors and Linda rather than picking one, but it doesn’t reproduce either one’s original sharp edges just for the sake of purity. Here’s the mapping from “gotcha described above” to “the specific fix PlexSpaces makes”:

A fan-out/fan-in worker pool shows the combination directly:

// Coordinator spawns N supervised, bounded-mailbox workers.
for i in 0..n {
    spawn_with_facets(
        &ctx, service_locator.clone(),
        "worker", "default",
        Worker::new(i), vec![],
    ).await?;
}

// Coordinator collects results Linda-style — associative, FIFO,
// no ActorRef needed for any individual worker.
let mut collected = 0;
while collected < n {
    let tuple = ctx.tuple_space()
        .in_(template!["result", Wildcard, Wildcard])
        .await?;
    collect(tuple);
    collected += 1;
}

The workers are ordinary supervised actors, restartable, journaled, isolated, with bounded mailboxes so a slow coordinator can’t be flooded. The result collection is Linda-style associative matching, but FIFO instead of unspecified, so results come back in the order the workers actually produced them rather than in an arbitrary one.

Example: scatter-gather with a timeout, across three models

Comparisons are easier to trust when they’re concrete rather than abstract, so I implemented the same pattern, scatter-gather with a timeout across all three approaches. The problem: fan requests out to N services, collect the first K responses within a deadline, then cancel everything else, guaranteed. This is the pattern underneath every hedged-request system, every parallel-search aggregator, and every timeout-bounded fan-out you’ve seen in production.

Go CSP: the naive version leaks goroutines

The code below shows the mistake almost everyone makes on the first pass like spawning goroutines with no cancellation path at all:

func ScatterGatherNaive(services []time.Duration, firstK int) []ServiceResponse {
    ch := make(chan ServiceResponse, len(services))

    for i, latency := range services {
        go func(id int, lat time.Duration) {
            time.Sleep(lat)
            ch <- ServiceResponse{ServiceID: id, Data: fmt.Sprintf("response-%d", id)}
        }(i, latency)
    }

    results := make([]ServiceResponse, 0, firstK)
    for range firstK {
        results = append(results, <-ch)
    }
    // BUG: N-K goroutines still running in background with no cancellation path
    return results
}

The fix combines context.WithTimeout with errgroup to get a real structured lifetime:

func ScatterGatherStructured(services []time.Duration, firstK int, timeout time.Duration) []ServiceResponse {
    ctx, cancel := context.WithTimeout(context.Background(), timeout)
    defer cancel()

    var mu sync.Mutex
    results := make([]ServiceResponse, 0, firstK)

    g, ctx := errgroup.WithContext(ctx)
    for i, latency := range services {
        g.Go(func() error {
            resp, err := simulateService(ctx, i, latency)
            if err != nil { return nil }
            mu.Lock()
            defer mu.Unlock()
            if len(results) < firstK {
                results = append(results, resp)
                if len(results) >= firstK { cancel() }
            }
            return nil
        })
    }
    _ = g.Wait() // All goroutines done — structured lifetime guarantee
    return results
}

errgroup.Wait() guarantees every goroutine finishes before the function returns, and context.WithTimeout propagates cancellation down to the slow workers, so nothing leaks. The catch: you still have to write that plumbing by hand every time.

Go CSP gotchas, side by side with the CSP algebra:

Runnable, with tests: native/gotchas_test.go — 8 tests demonstrating every row above as failing-then-fixed code.

// Gotcha: Goroutine leak — spawned workers block forever on an unread channel
ch := make(chan int)
for i := 0; i < 10; i++ {
    go func(id int) { ch <- id }(i)  // blocks forever — no reader
}
// 10 goroutines leaked: invisible to GC, invisible to deadlock detector

// Gotcha: Nil channel blocks forever (both directions, no panic)
var ch chan int  // nil
<-ch            // blocks forever on recv
ch <- 42        // blocks forever on send

// Gotcha: Send on closed panics, recv from closed returns zero (asymmetric!)
ch := make(chan int, 1); close(ch)
ch <- 1         // PANIC: send on closed channel
v, ok := <-ch   // v=0, ok=false — no panic, just zero value

// Gotcha: Buffered channel breaks rendezvous
buffered := make(chan int, 5)
buffered <- 1   // sender proceeds without receiver — not CSP anymore

Pure Rust: a Nursery, and select! as a guarded command

Rust gives you ownership-enforced isolation without shared state by default but it doesn’t give you structured lifetime by default either. The Nursery type below binds a group of spawned tasks to a scope explicitly:

pub struct Nursery<T: Send + 'static> {
    join_set: JoinSet<T>,
}

impl<T: Send + 'static> Nursery<T> {
    pub fn spawn<F>(&mut self, future: F)
    where F: Future<Output = T> + Send + 'static {
        self.join_set.spawn(future);
    }

    /// Wait for first K tasks OR timeout — then cancel everything else.
    pub async fn wait_first_k_or_timeout(mut self, k: usize, timeout: Duration) -> Vec<T> {
        let mut results = Vec::with_capacity(k);
        let deadline = tokio::time::Instant::now() + timeout;
        loop {
            if results.len() >= k { break; }
            tokio::select! {
                maybe = self.join_set.join_next() => {
                    match maybe {
                        Some(Ok(value)) => results.push(value),
                        Some(Err(_)) => continue,
                        None => break,
                    }
                }
                _ = tokio::time::sleep_until(deadline) => { break; }
            }
        }
        self.join_set.abort_all(); // Structured cleanup — cancel stragglers
        results
    }
}

tokio::select! maps almost directly onto CSP’s guarded command / external choice operator, whichever branch is ready first wins, and the biased; modifier gives you deterministic priority ordering (unlike Go’s deliberately random tie-break):

let results = scatter_gather_csp(&services, 3, Duration::from_millis(300)).await;
// Nursery guarantees: all children cancelled before scope exits

Ownership prevents shared-state bugs entirely, at compile time. JoinSet::abort_all() gives you a real, guaranteed cancellation. The nursery pattern gets you structured lifetime with essentially no runtime overhead. The catch: there’s no nursery built into the standard library and there’s still no formal algebra backing any of it (no FDR-style prover checking).

Rust CSP gotcha, demonstrated in csp_channels/src/scatter_gather.rs:

// Gotcha: Naive tokio::spawn has no parent link — tasks leak
let mut handles = vec![];
for svc in &services {
    handles.push(tokio::spawn(simulate_service(svc)));
}
// If we return early, spawned tasks run forever — no cancellation

// Fix: JoinSet provides structured lifetime
let mut join_set = JoinSet::new();
for svc in &services { join_set.spawn(simulate_service(svc)); }
// On drop or abort_all(), all tasks are cancelled — guaranteed

PlexSpaces: supervised lifetime plus decoupled collection

This is where actor supervision (structured fault handling) and Linda-style tuple-space coordination (decoupled result collection) get combined. Start with thin Linda-style wrappers over the tuple-space host functions:

// Linda-style thin wrappers over tuplespace host functions
fn linda_out(fields: &[Value]) -> Result<(), String> {
    let request = WriteRequest { tuples: vec![json_to_tuple(fields)?], .. };
    ts_write(&request.encode_to_vec()).map(|_| ())
}

fn linda_in(pattern: &[Value]) -> Result<Option<Vec<Value>>, String> {
    let request = ReadRequest { template: Some(to_pattern(pattern)?), take: true, .. };
    let bytes = ts_take(&request.encode_to_vec())?;
    Ok(decode_response(&bytes)?.first().map(to_json_array))
}

The orchestrator scatters by spawning supervised workers, then sets its own timeout with a self-message:

// Scatter: spawn N workers under supervisor
for i in 0..num_services {
    spawn("actor-csp-wasm", &format!("worker-{i}"), "", &init_json)?;
    send(&worker_id, "cast", &work_payload)?;
}

// Set timeout — send_after fires a collection message to self
send_after(timeout_ms, "cast", &collect_msg)?;

Each worker writes its result to the shared tuple space, with zero knowledge of who’s collecting it:

// Worker: Linda OUT — write result tuple to shared tuplespace
linda_out(&[
    Value::String("result".into()),
    Value::String(request_id.into()),
    Value::Number(service_id.into()),
    Value::String(result_data),
])?;

And gathering reads back whatever arrived in time, then explicitly tells the supervisor to stop the rest:

// Gather: Linda RD-ALL — collect whatever arrived before timeout
let results = linda_rd_all(&["result", request_id, *, *])?;
// Structured cleanup: stop remaining workers via supervisor
for wid in &worker_ids { stop(wid)?; }

Workers never need to know who’s collecting their results, that’s the Linda decoupling doing its job. The supervisor guarantees the worker lifecycle end to end, e.g., a crashed worker restarts automatically under a OneForOne strategy. Bounded mailboxes keep a slow coordinator from getting flooded. FIFO tuple matching makes the collection step deterministic instead of an open question.

The three approaches, side by side

Full runnable examples with tests live at: examples/rust/embedded/csp_channels, examples/go/apps/csp_structured/native, and examples/rust/apps/actor_csp.

One more actor-model gotcha, demonstrated via supervisor behavior

// Gotcha: Unbounded mailbox — fast producer OOMs the consumer
// Fix: PlexSpaces uses bounded mailboxes with a configurable limit

// Gotcha: No structured lifetime — actors are async, no scope to wait on
// Fix: Supervisor + explicit stop() for child actors after collection

// Gotcha: Orphaned actors — a spawned actor runs forever if nobody stops it
// Fix: OneForOne supervisor manages worker lifecycle; orchestrator calls stop()

The tldr;

• CSP has real algebra and real tooling (FDR) behind it. Go borrows the vocabulary but drops the proof the when you add a buffer.
• Actors have partial formal treatment and isolation by construction, but unbounded mailboxes and selective-receive skip are real, sharp edges the model doesn’t protect you from on its own.
• async/await never had formal backing at all, and its fire-and-forget promise is the exact same “who’s tracking this” bug as an unstructured goroutine.
• Linda is the most decoupled model on this list and the least adopted. The original spec leaves match order unspecified and spawned work untracked.
• Structured concurrency isn’t a another concurrency model, instead it’s a lifetime discipline layered.
• PlexSpaces’ bet is that you don’t have to inherit every historical rough edge along with the good ideas like bounded mailboxes, FIFO matching, and one small unified API let it combine actor supervision with Linda’s decoupled coordination.

The rest of the series

1. Part I — the general problem, concurrency constructs, and TypeScript
2. Part II — Erlang and Elixir
3. Part III — Go and Rust
4. Part IV — Kotlin and Swift
5. Building a Durable Actor Framework for Polyglot Serverless Apps
6. 20+ Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
7. Building an Agent Harness and Eval Pipeline with Durable Actors
8. Building a Self-Improving AI Agent with Durable Actors: MiniHermes
9. Building Mini OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
10. Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects
11. Building PlexSpaces: Decades of Distributed Systems Distilled Into One Framework

Code for everything above: github.com/bhatti/PlexSpaces

Introduction

Every major cloud now supports Serverless FAAS capabilities like AWS Lambda/Step Functions, Azure Durable Functions, GCP Cloud Functions and Cloudflare Durable Objects where you write a function or a small stateful actor. This allows you to scale it, pay only for what runs but there is a catch, you build on a proprietary runtime, and the runtime’s storage model, invocation model, and IAM rules become part of your application whether you meant them to or not. You cannot easily rewrite it or run it somewhere else. I saw a recent post (Why we’re moving Wire off Cloudflare Durable Objects) from Wire, which ran every container on Cloudflare Durable Objects since day one. They wrote why they rebuilt their own data plane instead of staying, which included extra network hops on the hottest path, drift of state, separation of compute from data, rigid placement policies and lack of self-hosting. None of these are reliability complaints, instead they’re architectural ceilings baked into a runtime you don’t own. And the pattern generalizes past Cloudflare:

• AWS Lambda: SAM and LocalStack approximate the runtime locally, but diverge on execution environment, IAM, and VPC behavior.
• Azure Durable Functions: Azurite emulates the storage layer, but the replay-based orchestration engine behaves differently under real concurrent load than it does in the emulator.
• GCP Cloud Functions: the Functions Framework runs locally, but Eventarc, Pub/Sub push, and Cloud Run triggers all need live GCP resources.
• Cloudflare Workers/DO/Agents: wrangler dev simulates KV with SQLite and alarms with in-process timers, but never replicates the distributed routing that decides which data center actually holds your object.

Every one of these runtimes gives you a great abstraction and takes your operational sovereignty in exchange. This post shows how to keep the abstraction like stateful actors, durable storage, alarms, WASM sandboxing, LLM calls, observability, an event bus, webhooks while running it on infrastructure you control with an open-source framework called PlexSpaces.

PlexSpaces

PlexSpaces is an open-source, polyglot actor framework that gives every actor durable KV storage and alarms, routes messages between actors on one node or across a gRPC mesh, and exposes a host API to Go, TypeScript, Python, and Rust. You write an actor once; the same WASM module deploys to your laptop, a Docker container, Kubernetes, bare metal, or several clouds at once, unchanged. The mental model sits close enough to Cloudflare Durable Objects that migrating existing DO code is mostly mechanical. The key difference: there’s no simulated version to diverge from production, because the production runtime is the development runtime.

Part I: Durable Objects

Cloudflare publishes a short list of rules for writing correct Durable Objects, which are easily mapped to a PlexSpaces constraint, and for most of them the mapping is tighter:

The one meaningful difference: Cloudflare guarantees globally unique placement. PlexSpaces virtual actors are unique per node or per cluster when pinned with @* (any node) or @nodeId (specific node). PlexSpaces provides an object-registry for managing cross-cluster deduplication.

Durable KV Storage

Cloudflare’s ctx.storage gives you a transactionally consistent store scoped to one object:

// Cloudflare DO — JavaScript
const count = await this.ctx.storage.get("count") ?? 0;
await this.ctx.storage.put("count", count + 1);

// Batch operations (storage.get([keys]) / storage.put(map))
const [history, meta] = await this.ctx.storage.get(["room:history", "room:meta"]);
await this.ctx.storage.put(new Map([["room:history", data], ["room:meta", index]]));

PlexSpaces gives you the same guarantee through host.kv. Because each actor processes one message at a time, a plain read-modify-write is safe without extra locking. The migrating_cloudflare_workers TypeScript example restores room history on onInit() using batch KV similar to blockConcurrencyWhile:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
protected override onInit(config: Record<string, unknown>): void {
    this.state.room_id = String(config.actor_id ?? "");

    // blockConcurrencyWhile() equivalent — batch-fetch persisted state before first message
    const keys = [
        "room:" + this.state.room_id + ":history",
        "room:" + this.state.room_id + ":meta",
    ];
    const values = host.kv.multiGet(keys);   // like DO storage.get(["k1","k2"])
    const [historyRaw] = values;
    if (historyRaw) {
        const msgs = JSON.parse(historyRaw);
        if (Array.isArray(msgs)) {
            this.state.messages = msgs;
            this.state.message_seq = msgs[msgs.length - 1]?.seq ?? 0;
        }
    }
}

private persistHistory(): void {
    // Batch write — like DO storage.put(new Map([["k1",v1],["k2",v2]]))
    host.kv.multiPut({
        ["room:" + this.state.room_id + ":history"]: JSON.stringify(this.state.messages),
        ["room:" + this.state.room_id + ":meta"]: JSON.stringify({
            message_seq: this.state.message_seq,
            last_updated: host.nowMs(),
        }),
    });
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
def _load_history(self) -> None:
    room_id = self._room_id()
    index_raw = host.kv.get(f"room:{room_id}:seq_index")
    if index_raw:
        seqs = json.loads(index_raw)
        keys = [f"room:{room_id}:msg:{seq}" for seq in seqs]
        values = host.kv.multi_get(keys)   # DO storage.get([k1,k2,...]) equivalent
        self.messages = [json.loads(v) for v in values if v]
        if self.messages:
            self.msg_seq = self.messages[-1]["seq"]

def _persist_history(self) -> None:
    room_id = self._room_id()
    entries = {}
    seqs = []
    for msg in self.messages:
        entries[f"room:{room_id}:msg:{msg['seq']}"] = json.dumps(msg)
        seqs.append(msg["seq"])
    entries[f"room:{room_id}:seq_index"] = json.dumps(seqs)
    host.kv.multi_put(entries)   # DO storage.put({k:v,...}) equivalent

PlexSpaces also ships atomic operations that Cloudflare leaves you to build yourself with a coordinator object:

// PlexSpaces TypeScript — from RateLimiterActor in guild_chat_actor.ts
// Atomic distributed counter — equivalent to Cloudflare KV atomic increment
const distributedCount = host.kv.increment(windowKey, 1);

// Compare-and-swap — equivalent to DO transactional storage read-modify-write
const applied = await host.kvCas("lock_key", expectedValue, newValue);

// KV with TTL — equivalent to DO storage.put with metadata expiration
await host.kvPutWithTtl("session_token", token, 3600);

The Python RateLimiterActor shows both in context:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("check")
def check(self, user_id: str = "") -> dict:
    # Atomic increment — survives actor restarts, equivalent to Cloudflare KV atomic
    host.kv.increment(f"rate:{user_id}:total", 1)

    # CAS — idempotent token slot reservation, like DO transactional put
    cas_key = f"rate:{user_id}:window"
    current_val = host.kv.get(cas_key) or ""
    host.kv.cas(cas_key, current_val, str(host.now_ms()))

    # Token bucket logic operates on in-memory state (safe: single-threaded actor)
    allowed = self.buckets[user_id]["tokens"] > 0
    if allowed:
        self.buckets[user_id]["tokens"] -= 1
    return {"allowed": allowed, "remaining": self.buckets[user_id]["tokens"]}

Durable Alarms

A DO schedules one future callback that survives node restarts:

// Cloudflare DO
await this.ctx.storage.setAlarm(Date.now() + 10_000);

async alarm() {
    const count = await this.ctx.storage.get("count");
    await this.ctx.storage.delete("count");
    console.log(`Processing ${count} batched requests`);
}

PlexSpaces maps this directly through ReminderFacet, which persists the alarm to durable storage and re-queues it automatically after a restart. The AlarmDemoActor in the guild-chat example demonstrates the full lifecycle like set, query, fire, cancel:

// PlexSpaces TypeScript — examples/typescript/apps/migrating_cloudflare_workers/guild_chat_actor.ts
onEnqueue(_payload: Record<string, unknown>): Record<string, unknown> {
    this.state.queued++;
    if (this.state.queued === 1) {
        // First item — equivalent to: await this.state.storage.setAlarm(Date.now() + 30_000)
        host.alarm.set(host.nowMs() + 30_000);
    }
    return { status: "ok", queued: this.state.queued };
}

// Fires when the scheduled timestamp is reached
// Equivalent to Cloudflare DO: async alarm() { ... }
on__alarm__(_payload: Record<string, unknown>): Record<string, unknown> {
    const processed = this.state.queued;
    this.state.processed += processed;
    this.state.queued = 0;
    this.state.total_alarm_fires++;
    return { status: "ok", processed };
}

onStatus(_payload: Record<string, unknown>): Record<string, unknown> {
    const alarmAt = host.alarm.get();   // DO: await this.state.storage.getAlarm()
    return { ...this.state, alarm_at: alarmAt, alarm_set: alarmAt > 0 };
}

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("start")
def start(self, delay_ms: int = 30000) -> dict:
    # Equivalent to: this.state.storage.setAlarm(Date.now() + delay_ms)
    host.alarm.set(host.now_ms() + delay_ms)
    return {"status": "ok", "fire_at_ms": host.now_ms() + delay_ms}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Equivalent to Cloudflare DO: async alarm() { ... }
    processed = len(self.pending_requests)
    self.total_processed += processed
    self.pending_requests = []
    return {"status": "ok", "processed": processed}

@handler("cancel")
def cancel(self) -> dict:
    host.alarm.delete()   # DO: this.state.storage.deleteAlarm()
    return {"status": "ok", "action": "alarm_cancelled"}

Get-or-Create (Virtual Actors)

Cloudflare’s core abstraction is the globally unique object that appears on first access, at the cost of a binding declared in wrangler.toml:

// Cloudflare DO — needs a binding in wrangler.toml
const id = env.CHAT_ROOM.idFromName(roomId);
const room = env.CHAT_ROOM.get(id);
await room.fetch("/send", { method: "POST", body: JSON.stringify(msg) });

PlexSpaces virtual actors give you the same behavior with no binding file. The actor ID ({name}//{actorType}::{namespace}@*) encodes both the shard key and the actor class, and the @* suffix lets the runtime place it on whichever node is best; pin it with @node-id for data locality:

// PlexSpaces TypeScript
import { getActorRef } from "@plexspaces/sdk";
const room = getActorRef("ChatRoomActor", roomId, "default");
const reply = await room.ask("send_message", { user_id: userId, content: msg });

# PlexSpaces Python
from plexspaces.host import get_actor_ref
room = get_actor_ref("ChatRoomActor", room_id, "default")
reply = room.ask("send_message", {"user_id": user_id, "content": msg}, timeout_ms=5000)

The actor spins up on its first message, whether or not it existed before.

Listing Actors by Namespace

Cloudflare exposes a Durable Objects namespace list API for management and observability. PlexSpaces has a direct equivalent defined in its proto-first design. The ListActors RPC in actor_runtime.proto accepts namespace, actor_type, state, and node_id filters and returns paginated results:

// proto/plexspaces/v1/actors/actor_runtime.proto
message ListActorsRequest {
    string actor_type = 3;
    ActorState state  = 4;
    string node_id    = 5;
    // Namespace for tenant isolation — only actors in this namespace are returned
    string namespace  = 6;
    PageRequest page_request = 2;
}

message ListActorsResponse {
    repeated Actor actors        = 2;
    PageResponse page_response   = 3;
}

From a PlexSpaces client, listing all active ChatRoomActor instances in the default namespace:

# HTTP API (equivalent to Cloudflare's list-objects endpoint)
curl "http://localhost:8080/api/v1/actors/default/ChatRoomActor?state=active"

// Go SDK
actors, err := client.ListActors(ctx, &ListActorsRequest{
    ActorType: "ChatRoomActor",
    Namespace: "default",
    State:     ActorStateActive,
})

Cloudflare limits listing to metadata (ID, location, storage size). PlexSpaces returns full Actor records including state, facets, node assignment, resource usage, and tenant/namespace tags.

WebSocket Handling

Cloudflare’s Model

Cloudflare gives the Durable Object two WebSocket modes. In the standard model, the object holds the socket directly:

// Cloudflare DO — standard WebSocket
async fetch(request) {
    const [client, server] = Object.values(new WebSocketPair());
    this.ctx.acceptWebSocket(server);
    return new Response(null, { status: 101, webSocket: client });
}

async webSocketMessage(ws, message) {
    for (const peer of this.ctx.getWebSockets()) {
        peer.send(`broadcast: ${message}`);
    }
}

async webSocketClose(ws, code) {
    ws.close(code, "connection closed");
}

The WebSocket Hibernation API (state.acceptWebSocket / getWebSockets()) is Cloudflare’s optimization for objects that hold many sockets but are mostly idle: the object is evicted when no message is being processed, and WebSocket state is restored from durable storage on the next message.

PlexSpaces: A Cleaner Split

PlexSpaces takes a different approach: room state and connection state are in separate actors. A ChatRoomActor holds only membership and message history; each browser connection is a thin-node client registered under its own actor ID. When the room fans out, host.send(actorId, "chat_message", event) routes each delivery through the WsActorTransportClient to the right WebSocket session like the room never holds a socket handle.

This is the same split Discord uses internally in its Elixir stack, where session processes are separate from guild processes. Here’s the real onSend handler from examples/typescript/apps/ws_chat_room/:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onSend(payload: SendPayload): unknown {
    const senderUsername = this.state.members[payload.sender_actor_id] ?? payload.sender_actor_id;
    const ts = host.nowMs();
    this.state.history.push({
        senderActorId: payload.sender_actor_id,
        sender: senderUsername,
        text: payload.text,
        ts,
    });
    if (this.state.history.length > MAX_HISTORY) {
        this.state.history = this.state.history.slice(-MAX_HISTORY);
    }

    const event = {
        sender: payload.sender_actor_id,
        sender_username: senderUsername,
        text: payload.text,
        room_id: this.state.roomId,
        ts,
    };
    // Fan out to all members including sender (delivery confirmation)
    // host.send() routes each tell through WsActorTransportClient ? WsRegistry ? thin-node WS session
    const memberIds = Object.keys(this.state.members);
    for (const actorId of memberIds) {
        host.send(actorId, "chat_message", event);
    }
    return { success: true, members_notified: memberIds.length };
}

A companion PresenceActor in the same file tracks online/offline state using a durable reminder (host.sendAfter) to mark a user offline after 55 seconds of silence without external cron job involved:

// PlexSpaces TypeScript — examples/typescript/apps/ws_chat_room/chat_server_actor.ts
onOnline(_payload: OnlinePayload): unknown {
    this.state.online = true;
    this.state.last_seen = host.nowMs();
    host.kv.putJson(`presence:${this.state.userId}`, { online: true, last_seen: this.state.last_seen });
    host.sendAfter(60_000, "timeout_check", {});   // durable reminder, survives restart
    return { success: true, online: true };
}

onTimeout_check(): unknown {
    const idleSince = host.nowMs() - this.state.last_seen;
    if (idleSince > 55_000) {
        this.state.online = false;
        host.kv.putJson(`presence:${this.state.userId}`, {
            online: false, last_seen: this.state.last_seen,
        });
    }
    return { checked: true, idle_ms: idleSince };
}

Gap Analysis: WebSocket Hibernation

The PlexSpaces model costs more code upfront (session actor + room actor) and pays you back with independent scaling: fan-out scales with the number of receivers, not with the room actor’s memory; a crashed session doesn’t lock the room; reconnect logic is client-side only.

Fan-Out: DO’s Missing host.send() Primitive

In Cloudflare, cross-object fan-out means individual fetch() calls or Queues, which are not lean as a fire-and-forget tell. PlexSpaces’s host.send() is a fire-and-forget message routed through the actor mesh, no HTTP overhead, with in-process delivery for co-located actors. The Python guild-chat send_message handler shows this clearly:

# PlexSpaces Python — examples/python/apps/migrating_cloudflare_workers/guild_chat.py
@handler("send_message")
def send_message(self, user_id: str = "", content: str = "") -> dict:
    msg = self._add_message(user_id, content, host.now_ms())

    # Fan-out: fire-and-forget to each member actor
    # Mirrors Discord's Manifold pattern for distributed fan-out
    # In Cloudflare DO, this would be individual fetch() calls — expensive
    fan_out_count = 0
    for member_id in list(self.members.keys()):
        if member_id != user_id:
            host.send(member_id, "receive_message", {
                "room_id": self._room_id(),
                "seq": msg["seq"],
                "from": user_id,
                "content": content,
            })
            fan_out_count += 1

    self._persist_history()   # batch multiPut — one KV call for the whole room
    return {"status": "ok", "seq": msg["seq"], "fan_out": fan_out_count}

Part II: Cloudflare Agents SDK

Cloudflare’s Agents SDK builds stateful AI agents on top of Durable Objects. PlexSpaces covers the same patterns; some are direct translations and a few require a different shape.

Conversation State and Memory

Cloudflare stores conversation history in the DO’s storage. PlexSpaces does the same through host.kv, with the same durability guarantee: the ChatAgentActor in examples/python/apps/chat_agent/ stores history under a well-known key and restores it across activations:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
@handler("chat")
def chat(self, message: str = "") -> dict:
    # Load history — equivalent to: await this.storage.get('history')
    history = host.kv.get_json("history") or []

    history.append({"role": "user", "content": message, "timestamp": host.now_ms()})

    assistant_reply = self._call_llm(history)

    history.append({"role": "assistant", "content": assistant_reply, "timestamp": host.now_ms()})

    # Persist — equivalent to: await this.storage.put('history', history)
    host.kv.put_json("history", history)
    self.total_messages += 1

    # Schedule summarization alarm once history is long enough
    if len(history) > _ALARM_THRESHOLD and host.alarm.get() == 0:
        host.alarm.set(host.now_ms() + _ALARM_DELAY_MS)

    return {"status": "ok", "reply": assistant_reply, "history_length": len(history)}

@handler("__alarm__")
def on_alarm(self) -> dict:
    # Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm()
    history = host.kv.get_json("history") or []
    summary = self._call_llm([{
        "role": "user",
        "content": f"Summarize this conversation (2-3 sentences): {json.dumps(history)}"
    }])
    host.kv.put("summary", summary)
    host.kv.delete("history")   # clear after summarizing
    return {"status": "ok", "action": "summarized", "messages_summarized": len(history)}

For long-term cross-session memory, store summaries under a user-scoped key (memory:{user_id}) and inject them into the next conversation’s system prompt similar to Cloudflare’s getMemory/setMemory implementation.

Calling LLMs

Cloudflare’s Agents SDK routes every call through env.AI, Cloudflare’s own inference gateway, locked to providers they support:

// Cloudflare Agents SDK — locked to Cloudflare's AI gateway
const response = await this.env.AI.run("@cf/meta/llama-3-8b-instruct", { messages });

PlexSpaces actors call any provider through a named HTTP service link resolved at deploy time, so the actor code never mentions a specific vendor:

# PlexSpaces Python — examples/python/apps/chat_agent/chat_agent.py
def _call_llm(self, messages):
    http = ServiceHttpClient("llm-link")
    body = {
        "model": "claude-3-5-haiku-20241022",
        "max_tokens": 1024,
        "messages": [{"role": m["role"], "content": m["content"]} for m in messages],
    }
    resp = http.post("/v1/messages", body)
    # Parse Anthropic response
    if isinstance(resp, dict):
        content = resp.get("content", [])
        if content and isinstance(content, list):
            return content[0].get("text", "")
    return "[LLM unavailable]"

app-config.toml points the link at Ollama locally, Anthropic or OpenAI in production, or an internal AI gateway in a regulated environment:

# Local development — Ollama
[[service_links]]
name = "llm-link"
url  = "http://localhost:11434"

# Production — swap without touching actor code
[[service_links]]
name = "llm-link"
url  = "https://api.anthropic.com"
headers = { "x-api-key" = "${ANTHROPIC_API_KEY}" }

TypeScript actors use the same pattern:

// PlexSpaces TypeScript — examples/typescript/apps/chat_agent/
const resp = host.httpClient("llm-link").post("/v1/messages", {
    model: "claude-3-5-haiku-20241022",
    max_tokens: 1024,
    messages,
});

Durable Workflows

Cloudflare Agents SDK ships workflow primitives that checkpoint multi-step sequences. PlexSpaces WorkflowActor gives you the same like Run, Signal, and Query RPCs map to start, inject external events, and inspect state. For example, the payment workflow in examples/go/apps/migrating_cadence/payment_workflow.go shows the shape:

// PlexSpaces Go — examples/go/apps/migrating_cadence/payment_workflow.go
// PaymentWorkflow implements WorkflowActor for idempotent payment processing.
// Steps: validate ? authorize (with retry) ? capture ? settle.
// Signals: refund, cancel.  Queries: status, payment_id.
type PaymentWorkflow struct {
    plexspaces.BaseActor
    PaymentID       string        `json:"payment_id"`
    Status          string        `json:"status"` // pending ? validated ? authorized ? captured ? settled
    Steps           []PaymentStep `json:"steps"`
    RefundRequested bool          `json:"refund_requested"`
}

func (p *PaymentWorkflow) Run(payloadJSON string) string {
    // Each step checkpoints via getState/setState before proceeding.
    // If the node crashes mid-run, the workflow resumes from the last checkpoint.
    p.Status = "validating"
    if err := p.validatePayment(); err != nil {
        p.Status = "failed"
        return marshal(map[string]any{"error": err.Error()})
    }
    p.addStep("validate")

    // Authorize with retries (idempotency key prevents double-charge)
    p.Status = "authorizing"
    for attempt := 0; attempt < 3; attempt++ {
        if err := p.authorizePayment(); err == nil {
            break
        }
    }
    p.addStep("authorize")
    // ... capture, settle
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

func (p *PaymentWorkflow) Signal(name, _ string) {
    switch name {
    case "refund":
        p.RefundRequested = true
    case "cancel":
        p.Status = "cancelled"
    }
}

func (p *PaymentWorkflow) Query(name, _ string) string {
    return marshal(map[string]any{"status": p.Status, "payment_id": p.PaymentID})
}

For multi-agent orchestration, OrchestratorActor in examples/go/apps/miniclaw/ decomposes a task into sub-tasks, delegates each to a worker agent discovered via process group, and aggregates results through TupleSpace:

// PlexSpaces Go — examples/go/apps/miniclaw/orchestrator.go
func (o *OrchestratorActor) Run(payloadJSON string) string {
    task := stringVal(parsePayload(payloadJSON), "task", "")
    taskID := fmt.Sprintf("orch-%d", host.NowMs())

    o.Status = "running"
    o.TaskID = taskID

    // Discover available agents via process group membership
    agentID, err := pgFirst("svc:agent")
    if err != nil {
        return marshal(map[string]any{"error": "no agents in svc:agent process group"})
    }

    // Decompose and delegate sub-tasks
    subTasks := decompose(task)
    for i, subTask := range subTasks {
        o.Progress = (i + 1) * 100 / len(subTasks)
        result, err := host.Ask(agentID, "chat", map[string]any{
            "message":    subTask,
            "session_id": fmt.Sprintf("orch-%s-%d", taskID, i),
        }, 30000)
        if err != nil {
            return marshal(map[string]any{"error": "sub-task failed: " + err.Error()})
        }
        // Store result in TupleSpace for aggregation
        host.TS().Write([]any{"orch_result", taskID, i, result})
    }

    o.Status = "completed"
    return marshal(map[string]any{"task_id": taskID, "status": "completed", "sub_tasks": len(subTasks)})
}

Human-in-the-Loop

Cloudflare’s human-in-the-loop pattern pauses an agent on a high-stakes action and waits for external approval before resuming. PlexSpaces implements this natively through a GenFSM actor, e.g., ApprovalGateActor in examples/python/apps/minipi/approval_gate.py:

# PlexSpaces Python — examples/python/apps/minipi/approval_gate.py
@fsm_actor(states=["idle", "awaiting_approval", "approved", "rejected"], initial="idle")
class ApprovalGateActor:
    """
    FSM states: idle ? awaiting_approval ? approved / rejected ? idle

    Key insight: the agent can wait for days. DurabilityFacet preserves all state
    durably — no polling, no timeouts burning tokens.
    """
    fsm_state: str = state(default="idle")
    pending_request: dict = state(default_factory=dict)
    pending_agent_id: str = state(default="")

    @handler("request_approval")
    def request_approval(self, agent_id: str = "", action: str = "", context: dict = None) -> dict:
        """An agent requests human approval for a high-stakes action."""
        if self.fsm_state != "idle":
            return {"status": "busy", "current_agent": self.pending_agent_id}

        self.fsm_state = "awaiting_approval"
        self.pending_agent_id = agent_id
        self.pending_request = {"action": action, "context": context or {}, "requested_at_ms": host.now_ms()}

        # Store request for external review (dashboard, Slack notification, etc.)
        host.kv.put(f"approval_request:{self.actor_id}", json.dumps(self.pending_request))

        return {"status": "pending", "gate_id": self.actor_id}

    @handler("approve")
    def approve(self, approver: str = "", comment: str = "") -> dict:
        """Human approves — signals the suspended agent to resume."""
        agent_id = self.pending_agent_id
        self.fsm_state = "approved"
        self.decision_history.append({
            "action": self.pending_request.get("action"),
            "decision": "approved",
            "approver": approver,
            "decided_at_ms": host.now_ms(),
        })

        # Signal the waiting agent to resume with the decision
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "approved",
            "approver": approver,
            "comment": comment,
        })

        self.fsm_state = "idle"
        self.pending_agent_id = ""
        return {"status": "approved", "agent_id": agent_id}

    @handler("reject")
    def reject(self, approver: str = "", reason: str = "") -> dict:
        """Human rejects — signals the agent with the rejection."""
        agent_id = self.pending_agent_id
        host.send(agent_id, "workflow_signal:resume", {
            "decision": "rejected",
            "approver": approver,
            "reason": reason,
        })
        self.fsm_state = "idle"
        return {"status": "rejected", "agent_id": agent_id}

The agent on the other side calls host.ask("approval_gate", "request_approval", {...}) then processes the workflow_signal:resume message when it arrives. Because state is checkpointed durably, the agent can wait hours or days with no polling loop and no timeout burning tokens.

Long-Running Agents

Cloudflare’s long-running agent pattern uses alarms to wake a dormant agent on a schedule. PlexSpaces handles this identically, e.g., any actor with the ReminderFacet can schedule work arbitrarily far in the future. The ChatAgentActor summarization alarm is one example; for a true long-running loop:

@actor
class ChatAgentActor:
    """Minimal chat agent: conversation in KV, LLM via service link, alarm for summarization."""

    actor_id: str = state(default="")
    total_messages: int = state(default=0)
    total_summarizations: int = state(default=0)

    @init_handler
    def on_init(self, config: dict) -> None:
        self.actor_id = config.get("actor_id", "")
        host.info(f"ChatAgentActor init actor_id={self.actor_id}")


    @handler("__alarm__")
    def on_alarm(self) -> dict:
        """Durable alarm callback — equivalent to Cloudflare Agents SDK onAlarm().

        Summarizes conversation history and stores a summary KV key,
        then clears history.
        """
        host.info("ChatAgentActor: alarm fired — summarizing history")

        history = host.kv.get_json("history") or []
        if not history:
            return {"status": "ok", "action": "no_history_to_summarize"}

        # Summarize via LLM
        summary_prompt = (
            f"Summarize this conversation concisely (2-3 sentences): "
            f"{json.dumps([{'role': m['role'], 'content': m['content']} for m in history])}"
        )
        summary = self._call_llm([{"role": "user", "content": summary_prompt}])

        # Persist summary, clear history — equivalent to: storage.put('summary', s); storage.delete('history')
        host.kv.put("summary", summary)
        host.kv.delete("history")

        self.total_summarizations += 1

        host.info(f"ChatAgentActor: summarized {len(history)} messages")
        return {
            "status": "ok",
            "action": "summarized",
            "messages_summarized": len(history),
        }

What PlexSpaces adds beyond Cloudflare’s model: the actor can also be signalled externally at any time via host.send(actorId, "wake_early", {...}) — you’re not limited to the alarm cadence.

Agents Feature Map

Part III: Distributed Computation

Cloudflare Workers and Lambda optimize for millisecond, latency-sensitive request handling. PlexSpaces handles a second problem class: large-scale computation across resources that come and go.

ShardGroups: Scatter-Gather Without the Plumbing

For data-parallel and ML-style workloads, PlexSpaces exposes MPI-style collectives directly through the host API:

// Create a pool of 20 workers, hash-partitioned
let pool_id = client.create_worker_pool(
    "worker-pool-1", "worker", 20,
    PartitionStrategy::Hash, HashMap::new(),
).await?;

// Bulk update: 10,000 messages routed to the right shard by key
client.parallel_update(&pool_id, updates, ConsistencyLevel::Eventual, false).await?;

// Parallel map: query every shard simultaneously
let results = client.parallel_map(&pool_id, json!({ "action": "get_total_count" })).await?;

// Parallel reduce: aggregate stats across all shards
let stats = client.parallel_reduce(
    &pool_id, json!({ "action": "stats" }),
    ShardGroupAggregationStrategy::Concat, 20,
).await?;

Idle Browsers as Compute Nodes

The mersenne_prime TypeScript example runs a GIMPS-style distributed primality search: browser tabs connect as thin WebSocket clients, receive worker JavaScript from a CodeServerActor, and run Lucas-Lehmer tests inside Web Workers. A CoordinatorActor running as WASM on the server assigns exponents, tracks per-worker CPU cores, and dispatches the next candidate immediately on each result:

// PlexSpaces TypeScript — examples/typescript/apps/mersenne_prime/mersenne_actor.ts
onResult(payload: ResultPayload): unknown {
    const item = this.state.work[String(payload?.p)];
    item.status = 'done';
    item.is_prime = Boolean(payload.is_prime);
    item.duration_ms = payload.duration_ms ?? 0;

    // Record every completed candidate in TupleSpace and bump a Prometheus counter
    host.ts.write(['result', String(payload.p), item.is_prime ? 'true' : 'false',
        String(item.duration_ms), payload.actor_id ?? 'unknown']);
    host.incrCounter('ts-mersenne-prime', item.is_prime ? 'primes_found' : 'composites_found');

    // Immediately hand the same worker the next pending candidate
    const next = this._nextPending(this.state.workers[payload.actor_id!]?.cpu_cores ?? 1);
    if (next) {
        next.status = 'assigned';
        host.send(payload.actor_id!, 'assign_work', { p: next.p, done: false });
    }
    return { ok: true };
}

Open the same URL in ten browser tabs and you have ten compute shards, coordinated by one WASM actor, with zero additional infrastructure.

Comprehensive Feature Map

A couple of things need real rework, not a find-and-replace:

• WebSocket architecture. If your DO holds sockets directly today, plan for a thin node that hosts actors.
• Bindings vs children. Cloudflare bindings live in wrangler.toml and show up as env properties. PlexSpaces declares the same relationships as supervision children in app-config.toml.

Running Everywhere

The whole point is that development and production run the identical binary:

# Local development — exact production behavior
plexspaces-node start --config app-config.toml

# Docker — same binary, same config
docker run -v $(pwd):/app plexspaces/node start --config /app/app-config.toml

# Kubernetes — same config via Helm
helm install my-app plexspaces/app-chart \
  --set config.path=app-config.toml \
  --set persistence.storage=postgres

# Multi-cloud — nodes in GCP + AWS joined over a gRPC mesh; actors route transparently

An alarm that fires in production runs through the exact code path you tested on your laptop. There’s no gap to debug between wrangler dev and prod, because there’s only one runtime.

Working Examples

Every example referenced above is in the repository (github.com/bhatti/PlexSpaces):

• Guild chat (DO migration pattern): examples/{go,typescript,python,rust}/apps/migrating_cloudflare_workers/: ChatRoomActor with member fan-out, a token-bucket RateLimiterActor backed by host.kv.increment and host.kv.cas, and an AlarmDemoActor mirroring DO’s full alarm lifecycle
• WebSocket chat room: examples/{typescript,python,go,rust}/apps/ws_chat_room/: ChatRoomActor plus a PresenceActor that uses a durable reminder to detect idle disconnects
• AI chat agent (Cloudflare Agents SDK pattern): examples/{go,typescript,python,rust}/apps/chat_agent/: conversation history in KV, LLM calls through service link, durable summarization alarm
• Human-in-the-loop approval gate: examples/python/apps/minipi/approval_gate.py: FSM-based approval workflow, workflow_signal:resume handoff, state durable across multi-day waits
• Multi-agent orchestration: examples/go/apps/miniclaw/: OrchestratorActor decomposing tasks, delegating via process groups, aggregating through TupleSpace
• Durable payment workflow: examples/go/apps/migrating_cadence/payment_workflow.go: WorkflowActor with Run/Signal/Query, idempotent retry, refund and cancel signals
• Mersenne prime search (browser compute): examples/typescript/apps/mersenne_prime/: browser tabs as Lucas-Lehmer worker shards, coordinated by a WASM CoordinatorActor
• wasmCloud migration: examples/python/apps/migrating_wasmcloud/session_store.py: capability-based session store showing host.kv.list, inter-actor ask, and timer-driven cleanup
• Every abstraction in one place: examples/{go,typescript,python,rust}/apps/abstractions/: durable virtual-actor reactivation, workflow run/signal/query, process-group event delivery, timers, reminders, KV, tuple space, and blob storage

Conclusion

The Cloudflare model is the right model: stateful actors, durable storage, alarms, WebSocket session management, LLM calls baked in. Wire’s post makes the same point from the other direction as they’re not leaving because the model is wrong, they’re leaving because specific architectural ceilings came due at their scale. PlexSpaces keeps the model and removes the ceiling: you write the same actors, get the same alarms and durable storage and observability, and the binary that runs on your laptop is the exact binary that runs in production, on any cloud, on-prem, or across all of them at once. The migration from Cloudflare DO code is mostly mechanical. What you get back is the ability to run anywhere, own your infrastructure and never again debug a divergence between wrangler dev and prod.

GitHub: github.com/bhatti/PlexSpaces

Previous posts in this series:

• Building PlexSpaces: Decades of Distributed Systems Distilled
• Polyglot WebAssembly: Four Languages, One Runtime
• 20+ Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
• Building Mini OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
• Building a Self-Improving AI Agent with Durable Actors: MiniHermes
• Building an Agent Harness and Eval Pipeline with Durable Actors

I have seen some systems never crash, they start cleanly, swallow every error, and keep running no matter what goes wrong. In my experience, they are also the hardest systems to debug, the most dangerous to operate, and the most expensive to maintain. I worked on a similar legacy system for distributed data platform that routed events between hundreds of thousands of nodes. It had zero unhandled exceptions in production. It also had silent authentication failures, invisible data loss, and configuration divergence that took days to diagnose. This is a follow-up to my earlier posts on building an observability platform in Rust, why DRY becomes a liability, and making bad state impossible with ADTs. Those posts were about the type system but this one is about a habit of mind that no type system fixes on its own: the instinct to catch every error with some fallback or default behavior.

The culprit in that system was never a lack of error handling. It was error handling, applied in the wrong places, for the wrong reasons. I call it defensive programming as a religion: every function protects itself against every possible invalid input by inventing a fallback. Missing config? Use a default. Secret unavailable? Generate a random one. Database write failed? Log it and move on. The result is a system that looks healthy on every dashboard while quietly corrupting its own state underneath. In this post, I will walk through the patterns I found, why each one causes more damage than the crash, and what an alternative looks like instead. The whole argument rests on one idea: Every fallback creates a new source of truth, and two sources of truth always drift apart.

I. Why a Fallback Is Worse Than It Looks

It’s tempting to think of a fallback as just “hiding an error.” It’s worse than that. When a function invents a value because the real one is missing, that invented value doesn’t stay hidden, instead it becomes a fact in the system. From that moment on, the system is carrying two truths: the one that should exist but doesn’t, and the one that got made up and does. These two truths never stay in sync and when they drift apart, the failure almost never shows up where the fallback happened. Instead, it shows up somewhere else entirely, in a component with no obvious connection to the code that invented the value. For example, you’ll spend hours in the authentication layer before realizing the signing key was randomly generated at startup by a config migration function three layers away. To be clear about what I mean by “fallback,” I am not talking about:

• Validating input at system boundaries: checking what a user typed, sanitizing data from outside. Correct and necessary.
• Graceful degradation with an explicit signal: returning a typed Degraded state that the caller can see and react to.
• Retrying transient I/O failures with backoff. Standard practice.

I’m talking about code that silently invents state when the real state is missing, and then carries on as if nothing happened. Three things make a fallback harmful:

1. It hides the root cause. The missing value was the bug. The fallback makes it disappear.
2. It persists the invented value. Once it’s written to disk or sent over the wire, every future operation has to succeed against a value that was never correct in the first place.
3. It relocates the symptom. The failure surfaces hours later, in a different component, in a different log file, with no visible thread connecting it back to the startup code that invented the wrong value.

I am not advocating “crash on every error.” It just means: a function that requires X must fail when X is absent and it must never invent X.

II. Postel’s Law

There’s a reason smart engineers build these fallback-heavy systems: they’re following a respected principle: “be conservative in what you send, be liberal in what you accept.” that Jon Postel wrote as guidance for TCP implementations. That advice made a lot of sense in its original context. But this principle leaked out of the protocol layer and turned into a general design philosophy. Engineers started applying “be liberal in what you accept” to function signatures, config loading, and communication between services inside a system they fully control. A function that takes string | undefined and silently substitutes a random value gets called “being robust.” A startup sequence that swallows errors and keeps going gets called “being tolerant.”

Inside your own system, that assumption doesn’t hold. For example, you can fix the sender because you own the caller. But when you own the migration script that’s supposed to write the auth token and you “liberally accept” a missing auth token by inventing a random one instead, you’re not enabling interoperability with an outside party instead you’re hiding a bug in code you wrote. This misapplication creates a ratchet effect. Every “liberal” receiver makes it harder to notice problems at the source. If every function tolerates missing input, the function that’s supposed to supply that input has no pressure to get it right. People also tend to forget that Postel’s Law has a second half: “be conservative in what you send.”

The corrected version for internal systems is this: be strict with components you control, and liberal only at the boundaries where you genuinely can’t fix the sender like external APIs, user input, third-party integrations. Inside your own codebase, strictness isn’t fragility. A function that rejects invalid input tells you exactly where the bug lives.

III. Inventing Values: The Most Dangerous Pattern

This is the category that caused the most damage and I have seen countless bugs due to this anti-pattern. For example, the code generates a random value, assigns it to a security-critical field, and moves on as if that field were properly populated. The invented value becomes a durable fact somewhere and it’s always wrong.

The Archetype: Inventing a Secret

A function runs at startup and writes a signing secret into every worker group’s configuration. Workers and the coordinator use this secret to authenticate each other. If two groups end up with different values, every authentication attempt between them fails silently, showing up as delivery failures instead of auth failures.

// The bug: if authToken is absent, invent a UUID and persist it
const authToken = settings.distributed?.master?.authToken;
const plaintext = authToken != null && authToken.length > 0
  ? authToken
  : randomUUID();  // <-- this line breaks authentication across the cluster

When authToken is missing from the merged settings, which is a perfectly valid state on a fresh install but this function generates a randomUUID() and writes it as the signing secret for every group it touches. Each call produces a different UUID. Meanwhile the token store writes yet another value through a completely different code path. The function reports success for every group it writes to. The symptom 401 errors between nodes shows up minutes or hours later, in worker logs, pointing investigators toward the wrong layer entirely. This is the archetype of the whole problem: if the required input is missing, invent something plausible-looking and keep going. The fix is three lines:

const authToken = settings.distributed?.master?.authToken;
if (authToken == null || authToken.length === 0) {
  logger.warn('authToken absent from settings; cannot write signing secrets');
  return;  // do not proceed; do not invent
}

The “Disabled” Sentinel That Looks Just Like a Real Key

A secret provider has a three-source fallback chain: encrypted store, config file, random bytes. That last fallback is supposed to act as a “poison key” that never validates:

async getSecret(): Promise<string> {
  // Source 1: encrypted store
  const fromStore = await secretsMgr.get(KEY_ID).catch(() => null);
  if (fromStore) return fromStore;

  // Source 2: config file (which may itself contain a well-known default!)
  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return fromFile;

  // Source 3: "disable" by returning random bytes — looks perfectly valid to the caller
  return random(16);
}

The caller gets back a plain string in all three cases. It has no way to tell “a real secret from the store” apart from “a well-known default from the config file” apart from “random bytes that will never work.” It signs a token with whatever string it received and sends it off. This is a type-system failure because the return type Promise<string> squashes three semantically different outcomes into one shape.

The explicit contract fixes this at the type level:

type SecretResult =
  | { kind: 'valid'; value: string; source: 'store' | 'file' }
  | { kind: 'unavailable'; reason: string };

async getSecret(): Promise<SecretResult> {
  const fromStore = await secretsMgr.get(KEY_ID).catch((err) => {
    logger.error('Secret store unavailable', { error: err });
    return null;
  });
  if (fromStore) return { kind: 'valid', value: fromStore, source: 'store' };

  const fromFile = settings.distributed?.master?.authToken;
  if (fromFile) return { kind: 'valid', value: fromFile, source: 'file' };

  return { kind: 'unavailable', reason: 'no secret in store or config' };
}

Now a caller that receives unavailable marks itself as degraded. It doesn’t sign tokens and surfaces a health check failure instead.

The Token Renewal That Signs With Random Bytes

The token authenticator calls getSecret(), and if that fails, it falls back to random bytes anyway:

let keyStr = await this.secretProvider.getSecret().catch(() => undefined);
if (keyStr == null) {
  this.logger?.debug('Unable to generate a valid token. Disabling.');
  keyStr = random(16);  // this "signed" token will never be accepted by any peer
}
const token = jwt.sign(payload, keyStr);
this.cachedToken = token;  // cached and reused for every future request

The log message says “Disabling,” but nothing gets disabled. The code signs a token with a random key, caches it, and hands it out to every caller for the rest of the process’s life. Workers receive the token, fail to verify it, and log a 401, with nothing to suggest the root cause of the issue.

The explicit contract: if the secret is unavailable, don’t sign anything. Set this.cachedToken = null. Let callers check for null and surface a real health degradation.

The User ID That Regenerates on Every Call

const userId = existing?.username ?? crypto.randomUUID();

When existing is unexpectedly missing, every call generates a brand-new UUID. The user becomes a ghost: every request creates a fresh identity, invisible to deduplication, audit trails, and rate limiters.

The Config Placeholder That Silently Materializes

if (authToken?.token === 'REPLACE_ME') {
  authToken.token = uuidv4();  // silent replacement, no log of the value generated
}

This runs at startup and inside a database migration. If the startup write fails, the generated token is gone.

IV. Swallowed Errors

In one legacy codebase I worked had 656 instances of .catch(NOOP), an empty function attached to a promise rejection that turns any error into undefined and lets execution continue. Many of these were on cleanup paths, which is harmless. But a large number sat on critical data paths like durability, metrics transport, authentication state.

The Durability Guarantee That Wasn’t

A persistent queue exists for exactly one reason: to guarantee that events survive a destination outage. That’s the system’s durability promise.

// PQ flush — the durability guarantee
await this.flushBuffer().catch(NOOP);  // silently swallows disk I/O errors
await this.commit().catch(NOOP);        // silently swallows commit failures

If the flush fails like disk full, I/O error, permission denied, the buffered events are silently gone. The one component whose entire job is preventing data loss is itself a source of silent data loss.

The explicit contract: treat PQ flush errors as fatal to the ingest path. For example, if the flush fails, apply backpressure and pause ingest. Log at error level with the event count at risk and emit a pq_flush_failure metric. Now the operator gets to choose: fix the disk, add capacity, or consciously accept the loss.

Metrics That Vanish

void saasMetrics.sendPacket(packet).catch(NOOP);

Every metrics-send failure is silently swallowed.Dashboards go blank, and nobody knows why.

The Config Load That Treats Corruption as “Empty”

const groups = await conf.loadSystem('internal-groups').catch(() => ({}));

One line here quietly conflates two very different situations:

• “The file doesn’t exist yet”, which is normal on first boot –> return {}
• “The file is corrupt, or a parse error, or permission was denied”, which is a real configuration bug –> also return {}

Either way, the loop over groups never runs. The operator has no way to tell “healthy, no groups configured yet” apart from “broken, groups exist but couldn’t be read.”

Startup That Succeeds Despite Total Failure

export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  try {
    // ... write secrets to all groups ...
  } catch (err) {
    logger.error('failed to sync secrets', { reason: err });
    // swallowed — startup continues, caller receives no signal
  }
}

The function catches everything at the top and resolves successfully no matter what. The caller awaits it and gets no signal that anything went wrong. If the sync fails for every group, the coordinator still starts, workers still connect, and authentication fails across the board but startup “succeeded.”

V. Defensive Defaults

This next category is more subtle. Instead of inventing a random value, the system injects a well-known one like a default credential, a default address, which makes it impossible for downstream code to tell that anything is missing at all.

The Well-Known Default Credential

The shared authentication token has a configuration setting, and by default, the settings loader injects a well-known string whenever nothing is configured:

const { injectDefaultAuthToken = true } = options ?? {};
if (injectDefaultAuthToken && conf.distributed?.master?.authToken == null) {
  conf.distributed.master.authToken = DEFAULT_AUTH_TOKEN;  // 'default-secret'
}

Any caller of getSettings() receives a truthy string for authToken. Code that checks if (authToken) proceeds as if a real token exists. The check passes. Authentication moves forward, using a credential that’s sitting right there in the source cod and every deployment that forgot to override it. The default here is opt-out, not opt-in. Every new call site has to remember to disable the injection.

Workers That Connect to localhost on Config Failure

const server = this.conf.distributed.master
  || { host: 'localhost', port: 5555, authToken: DEFAULT_AUTH_TOKEN };

When a worker fails to load its coordinator address, it silently falls back to localhost:5555 with the default token. On a single-node dev box, this might accidentally work. On a production multi-node deployment, the worker ends up connecting to itself. The symptom looks like a connection timeout, not a config-load failure.

The explicit contract: if conf.distributed.master is missing, throw. A worker cannot function without a coordinator address, and there is no valid default for it. Failing here with a clear message like “coordinator address not configured; set MASTER_URL or add distributed.master to instance.yml“.

VI. Multiple Sources of Truth

This is the pattern that ties everything else together. Every fallback chain creates more than one source of truth and any source of truth that isn’t explicitly designated as the source will eventually drift from the others. In a distributed system, that drift shows up as the hardest class of bug like intermittent or state-dependent failures.

Two Functions, One Secret, Two Different Sources

The bug that inspired this post exists because two functions write the same signing secret to group configs, but read the plaintext from two different physical sources:

A migration populates the token store before syncAllGroupSecrets runs at boot, so the store is meant to be authoritative. But syncAllGroupSecrets predates the store’s existence and still reads straight from the config file. These two sources drift apart after a token rotation or some race condition.

The explicit contract: one source of truth. Both functions read from the store. If the store is unavailable, both fail instead of silent fallback to a secondary source.

The Three-Source Chain Is Three Sources of Truth

Source 1: Encrypted store (authoritative)
  ? (unavailable ? silently falls through)
Source 2: Config file (may hold a stale or default value)
  ? (absent ? silently falls through)
Source 3: Random bytes (structurally valid, semantically useless)

Each fallback quietly downgrades the security posture, and the caller gets back a plain string with no idea which source it came from.

Three branches go into the same sign() call. Only one of them should ever be allowed to reach it, which is the whole argument for making unavailable its own explicit type instead of letting all three collapse into a plain string.

The Cache Nobody Fully Trusts

The settings system keeps a cache for high-availability mode. Some callers pass skipCache: true to bypass it; others don’t, and there’s no documented rule for which is which. This gives the system two sources of truth for the same data: the cache and the disk. If the config file changes between startup and an API call, the API may serve stale data. The skipCache escape hatch is a symptom, not a fix. It means someone stopped trusting the cache’s invalidation and punched a hole through it instead of repairing the underlying mechanism.

The explicit contract: the cache invalidates on every write. Callers never need to know or care whether they’re reading from cache or disk. Remove skipCache as a public option entirely.

VII. Redundant Guards

When a function doesn’t trust its caller’s preconditions, it adds its own guard on top:

// Caller (server.ts):
if (isLeader && featureFlags.check('AUTH_TOKEN_MGMT')) {
  await syncGroupSecrets(conf);
}

// Callee (syncGroupSecrets):
export async function syncGroupSecrets(conf: Configuration): Promise<void> {
  if (!isFreeTier() && !isRunningInSaaS()) return;  // guard 1
  if (!Product.isLeader(settings.distributed?.mode)) return;  // guard 2 (redundant!)
  if (!featureFlags.check('AUTH_TOKEN_MGMT')) return;  // guard 3 (redundant!)
  // ... actual work ...
}

This function lives in a directory named leader/ and is only ever called from the leader startup path, yet it re-checks isLeader internally anyway. The feature flag gets checked at the call site and again inside the function. Three layers of defense against calling this function in the wrong context.

The checks themselves aren’t the problem. It’s what happens when they trip: nothing. The function returns quietly, and the caller gets no signal either way.

The explicit contract: a function either does its job or throws. Preconditions get asserted, not silently absorbed. If the caller already guarantees the precondition, drop the internal check. If the function really can be called from multiple contexts and some of them are invalid, throw on the invalid ones instead of quietly returning.

VIII. “Best-Effort” Writes to State That Isn’t Optional

The most seductive justification for swallowing an error is: “the primary operation already succeeded so we don’t want a secondary failure to undo it.” That reasoning is correct in isolation and catastrophic in aggregate.

The Store Upsert That Swallows Its Own Failure

export async function mirrorTokenToStore(plaintext: string): Promise<void> {
  try {
    await store.upsert({ id: LEGACY_TOKEN_ID, token: encrypt(plaintext) });
  } catch (err) {
    logger.error('failed to mirror token to store', { reason: err });
    // swallowed — the caller sees success
  }
}

The comment above this function explains the intent: it’s “best-effort” so that a store-side failure doesn’t roll back a config file write that already succeeded. That reasoning holds up on its own but config file is now permanently out of sync.

The explicit contract: add reconciliation. On startup, compare the store’s token to the config file’s. If they differ, update the store. Emit a token_store_divergence counter and surface it in a health check similar to reconciliation loops in Kubernetes.

IX. Partial Operations Without a Way Back

The Batch Write That Discards on Failure

const batchTxn = db.transaction((ops) => { ops.forEach(fn => fn()); });
try {
  batchTxn(this.mutationCache.splice(0, maxSize));  // splice removes BEFORE success
} catch (err) {
  logger.error('Batch write failed', err);
  // operations already removed from cache — permanently lost
}

The comment in this code says: operations get removed from the cache regardless of whether the transaction succeeds. But the fix for “one bad operation might stall the queue” ends up being “discard the entire batch, including the good operations.” That isn’t a tradeoff instead it’s silent data loss.

The explicit contract: only remove items from the cache after a confirmed commit. Isolate the failing operation into a dead-letter queue and retry the rest of the batch without it.

The Config Reload Nobody Acknowledges

await conf.triggerReload().catch(NOOP);  // worker continues with old config

After receiving a new config bundle from the coordinator, a worker triggers a reload. If that reload fails, the worker just keeps running the old config. The two sides now disagree about what the worker is actually running.

The explicit contract: report a reload failure back to the coordinator on the next heartbeat. The coordinator marks that worker as “stale config” and can retry or alert. This is how Kubernetes rolling updates work, the controller notices and either retries or halts the rollout.

The Package Install That Saves Despite Partial Failure

for (const op of ops) {
  try {
    switch (op.type) {
      case 'install': await installPackage(op); break;
      case 'uninstall': await uninstallPackage(op); break;
    }
    status.applied.push(op);
  } catch (error) {
    errors.push(error);
  }
}
await this.save(packageManifest);  // saves regardless of how many failed

If three out of five packages install and two fail, the manifest still gets saved with those three. Next startup tries again from this partial state but the failed packages may have left behind lock files or half-written artifacts causing conflicts.

The explicit contract: validate that every operation can succeed before running any of them (a dry-run pass). Execute atomically (ACID transactions).

X. Silent Truncation With No Backpressure

The Metrics Buffer That Silently Drops

Workers piggyback metrics onto heartbeat messages sent to the coordinator, with a hard cap of 100,000 packets. Past that cap, excess metrics are silently dropped without any counter, logs or metrics. This is a nasty failure mode specifically because absence of data is itself meaningful data and silent truncation destroys that signal’s reliability.

The explicit contract: when the buffer nears capacity, reduce granularity instead of dropping outright. When truncation does happen, include metrics_truncated: N in the heartbeat so the coordinator knows its picture is incomplete. Better, instead of piggyback metrics on heartbeats at all, give them their own transport.

The TCP Sender That Zeroes Buffers on Disconnect

// On disconnect: all in-transit events silently lost
this.inTransitBufs = [];
this.bufOffset = 0;
this.bufferEventCount = 0;
this.dropBytes += len;  // only evidence: a counter increment

On a TCP disconnect, every in-transit event gets zeroed out. The only trace left behind is a dropBytes counter buried in internal metrics. Compare that to Kafka’s producer, where unacknowledged messages stay in the producer’s buffer and get retried on reconnect.

The Unbounded Queue That Becomes an OOM

protected queueBatch(): void {
  this.queuedBatches.push({ eventCount, eventsSize, events: this.currentBatch });
  // NO CHECK on length, size, or memory pressure
}

When the output destination is unreachable, failed batches get re-queued, and the queue grows without any bound. Memory climbs until the OOM killer steps in and terminates the process. An unbounded in-memory queue isn’t really a data structure, instead it’s a deferred OOM crash.

The explicit contract: bound the queue. Once it’s full, either apply backpressure to ingest, spill overflow to the persistent queue, or trip a circuit breaker that rejects new events with a typed error. Let the pipeline decide from there: drop, buffer to disk, or pause the source.

XI. Non-Atomic Writes

The Lease File That Can Split-Brain

await writeFile(this.leaseFile, stringify(content));

The failover lease file, the mechanism that’s supposed to guarantee only one coordinator is ever active is written directly with writeFile. On NFS, which is where this system runs in HA mode, writes aren’t atomic. A power failure mid-write leaves behind a truncated or corrupt file. The standby coordinator reads that corrupt lease, fails to parse it, and ends up in an undefined state.

The explicit contract: write to a temp file, fsync, then atomically rename it into place, with a checksum so readers can detect corruption. Every real database does this like SQLite’s WAL.

The Multi-File Config Deploy Without a Journal

Config deployment writes several YAML files in sequence like inputs.yml, outputs.yml, pipelines.yml,, etc. A crash midway through leaves the worker with a partial config and a worker that restarts after a partial deploy loads that inconsistent config.

The explicit contract: write every file to a staging directory first, verify that everything references correctly, then swap atomically like rename the directory, or flip a symlink. This is the same idea behind Docker image layers, Kubernetes ConfigMaps, and Nix store paths.

XII. Six Principles That Cover All of This

Every pattern above breaks one of six well-established principles. They’re standard practice in any system that prioritizes correctness over the appearance of uptime.

1. Fail fast at trust boundaries (Erlang’s “let it crash.”): When a precondition is violated, fail immediately and loudly. Erlang runs telecom infrastructure at 99.9999999% uptime on a philosophy of letting individual processes crash and having a supervisor restart them into known-good state.

2. Make invalid states unrepresentable: If getSecret() can return random(16) as a plain string then every caller is stuck defensively guessing whether it’s “real.” If it returns Secret | Disabled as a discriminated union instead then the type system forces every caller to handle both cases at compile time. I wrote about this pattern in “Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects.”

3. Classify errors (transient vs. fatal): Without classification, every catch block faces an impossible choice: rethrow and break “resilience,” or swallow and hide a real bug. For example, gRPC solves this with status codes like UNAVAILABLE means retry, INVALID_ARGUMENT means don’t, INTERNAL means there’s a bug.

4. Define delivery semantics for critical state: “Fire-and-forget” is fine for debug logs. It’s not fine for persistent queue flushes, token store upserts, or config reloads. If an operation mutates state that downstream code assumes succeeded, it needs at-least-once semantics.

5. One source of truth without fallback chains for critical state: For any given piece of state, there should be exactly one authoritative source. A fallback chain isn’t graceful degradation instead it’s an implicit decision that secondary sources are acceptable substitutes for the truth. If that’s genuinely acceptable, make it explicit with TTLs, version vectors, or consistency levels.

6. Atomic state transitions: State changes should be all-or-nothing: temp file, fsync, atomic rename for files; transactions with rollback for databases; staging plus swap for multi-file deployments.

XIII. Cognitive Load

All this conditional logic and fallback behavior creates the cognitive load, e.g., when any function might silently invent a value, you can’t trust a function’s output without reading its implementation. When errors are swallowed, a successful await no longer means the operation actually succeeded. When defaults get injected into config reads, a non-null value stops meaning “configured.”

Debugging a production incident in a system like this means reading every function in the call chain, understanding every fallback along the way, and reconstructing which code path actually ran. crash tells you exactly where and when an invariant broke. New engineers ask why workers sometimes fail to authenticate after a token rotation, and the honest answer involves reading six functions across four files, understanding a three-source fallback chain. Compare that to: “the store upsert threw on failure, the rotation API returned a 500, the operator re-ran it, it worked.

XIV. Conclusion

Defensive programming isn’t inherently wrong, and neither is Postel’s Law like at the boundary it was designed for. Validating input at system boundaries, handling I/O errors gracefully, protecting against malformed external data are correct applications of defensive thinking. The problem shows up when the same techniques get applied to internal code, inside a system where you control both ends of every interface. The alternative, in short:

• Throw Error when a required state that’s missing: The function refuses to proceed and the caller finds out immediately.
• Explicit return when an optional state that’s missing: null, undefined, Option<T>, a discriminated union.
• Transient failures = retry with backoff: Never .catch(NOOP).
• One source of truth per piece of state: Not a fallback chain that quietly degrades. Not a cache with no real invalidation.
• Bounded queues with backpressure: Not unbounded buffers waiting to OOM.
• Atomic state transitions: Not multi-step operations that can half-finish.
• Reconciliation loops for distributed state: Not one-shot “best-effort” writes that quietly drift apart.

This mud didn’t accumulate overnight, and it won’t disappear overnight either. But every fallback you remove, every error you refuse to swallow will makes the next incident roughly ten times faster to diagnose.

Related Blogs

1. From Big Ball of Mud to Functional Pipeline
2. The Reusability Trap: When DRY Becomes a Liability
3. Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects

Observability is a key part of any infrastructure but I’ve watched teams repeat the same mistakes around measuring availability. For example, they track uptime and watch average latency. They run a TCP health check on port 80 and call it good. Then support learns about the availability issues from customers but the health dashboard shows everything is green. This post covers how to measure availability correctly: what signals to collect, how monitoring tools compute the rolling statistics you see, why percentiles beat averages and what happens to tail latency at scale in microservices.

1. What Availability Actually Means

The textbook definition of availability is uptime, e.g., the fraction of time a service is running. This splits into two independent questions:

Availability = P(request succeeds) AND P(request completes within SLA)

A service can answer every request successfully but take 30 seconds per response then that’s functionally unavailable. Conversely, a service can respond in 5ms but return errors to 50% of requests is also functionally unavailable.

2. User Errors vs Server Errors — Why the Distinction Matters

This is the most commonly conflated measurement in production monitoring. HTTP status codes carry clear semantic meaning that should drive entirely different alert responses:

4xx errors are client/user errors like 400/Bad Request, 401/Unauthorized. 5xx errors means service is failing like 500/Internal Server, 503/Service Unavailable. There is one gray area: client timeouts. If your client times out after 5s waiting for your 10s response, the client sees a 408 or a network error, which look like a 4xx but the root cause is server-side latency. This is why tracking latency separately from error codes is essential.

from prometheus_client import Counter, Histogram

# Track errors with full status code granularity
request_counter = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status_code', 'status_class']
)

latency_histogram = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

def record_request(method: str, endpoint: str, status: int, duration_s: float):
    status_class = f"{status // 100}xx"
    request_counter.labels(
        method=method,
        endpoint=endpoint,
        status_code=str(status),
        status_class=status_class
    ).inc()
    latency_histogram.labels(method=method, endpoint=endpoint).observe(duration_s)


# --- Prometheus queries that actually measure availability ---

# Server error rate (5xx only — excludes client errors)
SERVER_ERROR_RATE = """
sum(rate(http_requests_total{status_class="5xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Availability (only penalize server errors)
AVAILABILITY = """
1 - (
  sum(rate(http_requests_total{status_class="5xx"}[5m]))
  /
  sum(rate(http_requests_total[5m]))
)
"""

# Client error rate (useful to watch, but not availability)
CLIENT_ERROR_RATE = """
sum(rate(http_requests_total{status_class="4xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Latency SLA compliance — fraction of requests completing within 500ms
LATENCY_SLA_COMPLIANCE = """
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
"""

A spike in 4xx that isn’t paired with a 5xx spike is almost certainly a misbehaving client, not your service. Alert on them differently: 5xx pages your on-call, 4xx goes to a ticket queue for review.

3. SLAs, SLOs, and Error Budgets

These three terms are used interchangeably in many organizations and they shouldn’t be.

• SLA (Service Level Agreement) is a contractual commitment to external customers. Violating it has legal or financial consequences. Example: “We guarantee 99.9% availability per calendar month. If we breach this, we issue service credits.”
• SLO (Service Level Objective) is an internal engineering target, usually tighter than the SLA. Example: “We target 99.95% availability.” The gap between SLO and SLA is your buffer.
• Error Budget is what you get to spend before you breach your SLO. For a 99.9% SLO over 30 days:

Total minutes in 30 days = 30 × 24 × 60 = 43,200 minutes
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes

The error budget is your 43.2 minutes. Every minute of downtime spends from it. This reframes the conversation from “is the service up?” to “how fast are we burning through our budget?”

from datetime import datetime, timedelta

class ErrorBudget:
    """
    Track error budget consumption in real time.
    
    Example: 99.9% SLO over 30 days = 43.2 minutes of allowed downtime.
    """
    def __init__(self, slo_target: float, window_days: int = 30):
        self.slo_target = slo_target          # e.g., 0.999 for 99.9%
        self.window_minutes = window_days * 24 * 60
        self.allowed_downtime_minutes = self.window_minutes * (1 - slo_target)
        self.downtime_minutes_spent = 0.0
        self.start_time = datetime.now()

    def record_downtime(self, minutes: float):
        self.downtime_minutes_spent += minutes

    def budget_remaining_minutes(self) -> float:
        return max(0, self.allowed_downtime_minutes - self.downtime_minutes_spent)

    def budget_remaining_pct(self) -> float:
        return (self.budget_remaining_minutes() / self.allowed_downtime_minutes) * 100

    def burn_rate(self) -> float:
        """How fast are we burning budget vs. expected rate? 1.0 = on track, >1.0 = burning fast."""
        elapsed = (datetime.now() - self.start_time).total_seconds() / 60
        expected_spent = (elapsed / self.window_minutes) * self.allowed_downtime_minutes
        if expected_spent == 0:
            return 0.0
        return self.downtime_minutes_spent / expected_spent

    def summary(self) -> str:
        return (
            f"SLO: {self.slo_target*100:.2f}%  |  "
            f"Budget: {self.allowed_downtime_minutes:.1f} min  |  "
            f"Spent: {self.downtime_minutes_spent:.1f} min  |  "
            f"Remaining: {self.budget_remaining_pct():.1f}%  |  "
            f"Burn rate: {self.burn_rate():.2f}x"
        )

# Usage
budget = ErrorBudget(slo_target=0.999, window_days=30)
budget.record_downtime(minutes=12.5)   # incident on day 3
budget.record_downtime(minutes=8.0)    # incident on day 11
print(budget.summary())
# SLO: 99.90%  |  Budget: 43.2 min  |  Spent: 20.5 min  |  Remaining: 52.5%  |  Burn rate: ...

A burn rate above 1.0 means you’ll exceed your error budget before the window closes. Burn rate above 14.4x means you’ll exhaust it within 48 hours, which is a PagerDuty alert.

4. The Health Check Anti-Pattern

I need to address something I’ve seen sink production deployments before we even get to metrics: health checks that only verify the process is listening on a port. A port check tells you the process hasn’t crashed. It tells you nothing about whether the process can serve traffic. I’ve seen this exact scenario: database connection pool was exhausted, port was open, load balancer marked the instance healthy, every request returned a 500. The monitoring was dark green the whole time.

A real health check must exercise the actual request path: connect to dependencies, perform a lightweight but genuine operation, return structured status. In Kubernetes this means a readiness probe hitting a /health endpoint that checks dependency connectivity. Critically, readiness and liveness are different probes:

• Liveness: Is the process deadlocked? If not, keep it alive. If yes, kill and restart it.
• Readiness: Can it serve traffic right now? If not, remove it from the load balancer pool, but don’t kill it.

A process that is alive but not ready (warming up a cache, waiting for a dependency) should fail readiness but pass liveness. Confusing these two causes cascading restarts during startup under load is a failure mode I’ve seen multiple times in prod. See my Zero-Downtime Services on Kubernetes and Istio post for the full treatment.

5. Why Average Latency Lies

Here’s a production story I’ve seen more than once. The team does an efficiency push: optimizes the hot path, ships a 30% improvement in p50 latency. Dashboards celebrate but three weeks later, the p99 is back to where it started. The answer is queuing theory. Consider a server with a queue in front of it. Define utilization P as:

P = arrival rate / service rate

The average number of items in the system in queue plus being served is:

E[N] = P / (1 - P)

This is not a linear relationship. It’s an asymptote that goes vertical as you approach full utilization:

When you make the code faster (higher service-rate), P drops, and you slide left on this curve, i.e., fewer items queuing with lower tail latency. But then traffic grows or you reduce servers to “realize the savings.” P climbs back to where it was, and latency returns with it. The key lesson is that the average latency reflects the fast path but high-percentile latency (p99, p99.9) is extremely sensitive to queue depth. High percentile latency is a leading indicator that you’re approaching overload.

There’s a counterintuitive implication from this: p99 is a terrible way to measure whether your efficiency work succeeded. It’s so sensitive to the queuing nonlinearity that changes in utilization will swamp the signal from your actual code changes. For measuring efficiency, mean latency is actually better because it tracks the true cost of processing one request without queue effects. Use percentiles for alerting and use mean for efficiency measurement.

6. Percentiles From First Principles

Let’s go over percentiles from scratch, because monitoring tools throw around “p50”, “p99”, “p99.9” without ever explaining what they actually represent, and misunderstanding them leads to misreading dashboards. Given a set of N latency measurements, sort them from fastest to slowest. The Nth percentile is the value at position N% in that sorted list.

Latencies (ms): [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
Sorted:          [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
                  ^              ^              ^
                  p10           p50            p90

p50 = 10ms  (50% of requests were at or below this speed)
p90 = 13ms  (90% of requests were at or below this speed)
p99 = 400ms (99% of requests were at or below this speed)

What p99 tells you is: at most 1% of your requests see latency worse than this number. Equivalently, 999 out of every 1000 requests complete faster than p99. The catch is that p99 is a single value and it summarizes nothing about the shape of the distribution between p90 and p99. Latency can get dramatically worse for customers in that range without your p99 alarm firing.

import numpy as np

def explain_percentile(latencies_ms: list[float]):
    """Show what percentiles mean in plain English."""
    arr = np.array(sorted(latencies_ms))
    n = len(arr)
    
    stats = {
        "mean":  np.mean(arr),
        "p50":   np.percentile(arr, 50),
        "p90":   np.percentile(arr, 90),
        "p95":   np.percentile(arr, 95),
        "p99":   np.percentile(arr, 99),
        "p99.9": np.percentile(arr, 99.9),
        "max":   np.max(arr),
    }
    
    print(f"{'Statistic':<10} {'Value':>10}   Plain English")
    print("-" * 65)
    print(f"{'mean':<10} {stats['mean']:>10.1f}ms  Average — hides bimodal distributions")
    print(f"{'p50':<10} {stats['p50']:>10.1f}ms  Half of requests faster than this")
    print(f"{'p90':<10} {stats['p90']:>10.1f}ms  90% of requests faster than this")
    print(f"{'p95':<10} {stats['p95']:>10.1f}ms  95% of requests faster than this")
    print(f"{'p99':<10} {stats['p99']:>10.1f}ms  99% of requests faster than this")
    print(f"{'p99.9':<10} {stats['p99.9']:>10.1f}ms  999/1000 requests faster than this")
    print(f"{'max':<10} {stats['max']:>10.1f}ms  Worst single request (very noisy)")

# Simulate a bimodal latency distribution
# 95% fast requests (cache hit), 5% slow (cache miss + DB query)
import random
random.seed(42)
latencies = [
    random.gauss(10, 2) if random.random() > 0.05 else random.gauss(300, 40)
    for _ in range(1000)
]
explain_percentile(latencies)

Statistic       Value   Plain English
-----------------------------------------------------------------
mean             24.8ms  Average — hides bimodal distributions
p50              10.4ms  Half of requests faster than this
p90              12.1ms  90% of requests faster than this
p95              17.9ms  95% of requests faster than this
p99             302.1ms  99% of requests faster than this
p99.9           375.8ms  999/1000 requests faster than this
max             392.4ms  Worst single request (very noisy)

7. Moving Averages and Rolling Percentiles

When Grafana shows you a p99 or Datadog shows you an error rate, it’s not summing up all-time data. It’s computing over a rolling time window.

Simple Moving Average vs EWMA

A Simple Moving Average (SMA) gives equal weight to every sample in the window:

from collections import deque
import statistics

class SMA:
    """Simple Moving Average — every sample in the window weighted equally."""
    def __init__(self, window: int):
        self.buf = deque(maxlen=window)
    
    def add(self, v: float) -> float:
        self.buf.append(v)
        return statistics.mean(self.buf)

An Exponentially Weighted Moving Average (EWMA) gives more weight to recent samples, fading older ones smoothly:

class EWMA:
    """
    Exponentially Weighted Moving Average.
    alpha: 0 < alpha < 1
    - High alpha (e.g. 0.3): reacts fast, noisier
    - Low alpha  (e.g. 0.05): smoother, slower to detect changes
    
    StatsD uses EWMA for gauge values. Prometheus uses time-window sums.
    """
    def __init__(self, alpha: float = 0.1):
        self.alpha = alpha
        self.value: float | None = None

    def add(self, sample: float) -> float:
        if self.value is None:
            self.value = sample
        else:
            self.value = self.alpha * sample + (1 - self.alpha) * self.value
        return self.value

# Demonstrate: same spike, different alphas
spike_data = [10, 10, 10, 10, 250, 10, 10, 10, 10, 10]
slow_ewma = EWMA(alpha=0.05)
fast_ewma = EWMA(alpha=0.30)

print(f"{'Sample':>8} {'Value':>8} {'alpha=0.05':>10} {'alpha=0.30':>10}")
for i, v in enumerate(spike_data):
    print(f"{i:>8} {v:>8.0f} {slow_ewma.add(v):>10.1f} {fast_ewma.add(v):>10.1f}")

  Sample    Value     alpha=0.05   alpha=0.30
       0       10       10.0       10.0
       1       10       10.0       10.0
       4      250       21.9       82.0   --> fast alpha sees the spike much louder
       5       10       21.3       58.4   --> slow alpha recovers faster
       9       10       18.5       17.2

Rolling Percentile

Computing exact percentiles over a moving window requires keeping raw samples and re-sorting. For production scale, the T-Digest algorithm computes approximate percentiles with bounded memory. Here’s the conceptual version first:

import numpy as np
from collections import deque

class RollingPercentile:
    """
    Rolling percentile over a fixed window of recent samples.
    
    Production note: At high throughput, use T-Digest or DDSketch instead.
    Prometheus uses pre-defined histogram buckets + linear interpolation.
    """
    def __init__(self, window: int, pctile: float):
        self.buf = deque(maxlen=window)
        self.pctile = pctile

    def add(self, v: float) -> float | None:
        self.buf.append(v)
        if len(self.buf) < 2:
            return None
        return float(np.percentile(list(self.buf), self.pctile))

# Show how window size affects sensitivity
import random
random.seed(7)

data = [random.gauss(10, 2) for _ in range(90)] + \
       [random.gauss(200, 20) for _ in range(10)]  # degradation at t=90

p99_small  = RollingPercentile(window=20,  pctile=99)
p99_medium = RollingPercentile(window=100, pctile=99)

print("How window size affects p99 detection of a latency spike:")
print(f"{'t':>4} {'value':>8} {'p99 w=20':>12} {'p99 w=100':>12}")
for t, v in enumerate(data[80:]):   # show the transition region
    small  = p99_small.add(v)
    medium = p99_medium.add(v)
    marker = " --> spike starts" if t == 10 else ""
    if small and medium:
        print(f"{t+80:>4} {v:>8.1f} {small:>12.1f} {medium:>12.1f}{marker}")

Prometheus histogram vs. summary: Prometheus offers two ways to track latency. A Summary computes quantiles client-side over a rolling window but you can’t aggregate across instances. A Histogram records counts in pre-defined buckets and approximates quantiles server-side, which is slightly less accurate, but fully aggregatable. For microservices with multiple replicas, always use Histogram.

8. Trimmed Mean: More Signal, Real Tradeoffs

Here’s the core difference between a percentile and a trimmed mean, using the product review analogy:

100 latency measurements, sorted by speed:

p99  = the single worst measurement in the best 99%
       (the 99th measurement out of 100, sorted fastest-to-slowest)

tm99 = the average of all 99 measurements in the best 99%
       (discard the 1 slowest, average the remaining 99)

tm99 summarizes 99 times more data than p99. That makes it more stable (less spiky under low traffic), harder to game (a gradual degradation can hide between percentile checkpoints, but tm99 will catch it), and more representative of typical customer experience.

• tm99 tracks the average experience of your bulk of customers
• TM(99%:) tracks the average of your slowest 1%; ensures outlier experience doesn’t silently worsen

Together these two numbers cover 100% of your requests with just two metrics.

import numpy as np

def compute_tm_stats(samples: list[float]) -> dict:
    """
    Compute a full suite of trimmed mean statistics.

    Syntax mirrors CloudWatch / AWS Embedded Metrics Format:
      tm99        = TM(0%:99%)  = average of fastest 99%
      TM(99%:)    = TM(99%:100%) = average of slowest 1%  
      TM(1%:99%)  = drop both extremes (handles unbounded latency)
      IQM         = TM(25%:75%) = Interquartile Mean
    """
    arr = np.sort(np.array(samples))
    n = len(arr)

    def tm(lower_pct: float, upper_pct: float) -> float:
        lo = np.percentile(arr, lower_pct)
        hi = np.percentile(arr, upper_pct)
        trimmed = arr[(arr >= lo) & (arr <= hi)]
        return float(np.mean(trimmed)) if len(trimmed) else float('nan')

    return {
        "mean":       float(np.mean(arr)),
        "p50":        float(np.percentile(arr, 50)),
        "p99":        float(np.percentile(arr, 99)),
        "tm99":       tm(0, 99),      # avg of fastest 99%
        "TM(99%:)":   tm(99, 100),    # avg of slowest 1%  --> watch your outliers here
        "TM(1%:99%)": tm(1, 99),      # drop both extremes (use for unbounded latency)
        "IQM":        tm(25, 75),     # interquartile mean
    }

# Scenario: a cache-miss spike where 2% of requests are slow
rng = np.random.default_rng(42)
fast = rng.normal(10, 1.5, 980)
slow = rng.normal(350, 30, 20)
samples = np.concatenate([fast, slow]).tolist()

stats = compute_tm_stats(samples)
print(f"{'Metric':<14} {'Value':>10}   Notes")
print("-" * 65)
for k, v in stats.items():
    notes = {
        "mean":       "Pulled up by slow tail — misleading",
        "p50":        "Median — fine but ignores tail",
        "p99":        "Single value at 99th position",
        "tm99":       "Average of 98% of customers --> primary SLO metric",
        "TM(99%:)":   "Average of slowest 2% --> outlier watchdog",
        "TM(1%:99%)": "Drops both extremes — good for browser metrics",
        "IQM":        "Middle 50% average — robust to both extremes",
    }.get(k, "")
    print(f"{k:<14} {v:>10.1f}ms  {notes}")

Metric              Value   Notes
-----------------------------------------------------------------
mean                16.8ms  Pulled up by slow tail — misleading
p50                  9.9ms  Median — fine but ignores tail
p99                335.2ms  Single value at 99th position
tm99                10.1ms  Average of 98% of customers --> primary SLO metric
TM(99%:)           351.4ms  Average of slowest 2% --> outlier watchdog
TM(1%:99%)          10.1ms  Drops both extremes — good for browser metrics
IQM                  9.8ms  Middle 50% average — robust to both extremes

Bounded vs. unbounded latency:

• Bounded latency (server-side, with request timeouts): use tm99 + TM(99%:). Since latency is capped by your timeout, even the worst measurements are meaningful.
• Unbounded latency (client-side browser metrics, user-perceived time): use TM(1%:99%). A user who closes their laptop mid-request and reopens it days later may log a latency of 230,400 seconds. These shouldn’t contaminate your outlier statistics. Drop the top and bottom extremes.

I have seen in a real-life production services where teams work towards improving p50/median but everything else gets worse. You only find this out when you examine tm95 because latency was consistently worse for a growing number of customers. The key lesson is that percentiles create blind spots “between the checkpoints.” A degradation that affects the 40th–60th percentile range will move neither p25 nor p75 much. Trimmed mean, because it averages across the entire range, catches these shifts. However, trimmed mean has its own blind spot. It deliberately removes the part of the distribution that dominates user experience in fan-out architectures. The right answer is not to choose between percentiles and trimmed mean but use both.

10. Winsorized Mean, Percentile Rank, and IQM

These statistics show up in CloudWatch and modern observability platforms, and they each solve a specific problem.

Winsorized Mean (WM)

Like trimmed mean, but instead of discarding outliers, it replaces them with the boundary value. For wm99:

• Find the value at the 99th percentile (= p99)
• Treat all 1% outliers as if they had exactly that p99 value
• Average all 100% of samples

def winsorized_mean(samples: list[float], lower_pct: float = 0, upper_pct: float = 99) -> float:
    arr = np.array(samples, dtype=float)
    lo = np.percentile(arr, lower_pct)
    hi = np.percentile(arr, upper_pct)
    # Clip: anything below lo becomes lo, anything above hi becomes hi
    winsorized = np.clip(arr, lo, hi)
    return float(np.mean(winsorized))

Winsorized mean gives some weight to outliers without letting extreme values skew the average. The difference between tm99 and wm99 is subtle at high percentages and wm99 will be slightly higher because it includes the outliers rather than dropping them.

Percentile Rank PR()

Percentile rank answers the inverse question from percentile. Percentile says: “What latency value marks the Nth percent?” Percentile rank says: “What percent of requests are below a given latency value?”

If you have an SLA of “respond within 500ms to 99% of users,” you’d normally monitor p99 and check it’s <= 500ms. With Percentile Rank, you instead plot PR(:500ms, i.e., the percentage of requests completing within 500ms and drive that number toward 99% or higher. This is more directly action-oriented: you always know exactly how far below your SLA you are.

def percentile_rank(samples: list[float], threshold: float) -> float:
    """What fraction of samples are at or below threshold?"""
    arr = np.array(samples)
    return float(np.mean(arr <= threshold) * 100)

# Example: SLA is p99 < 500ms
samples_ms = [10, 12, 9, 11, 450, 10, 13, 600, 11, 10]  # small sample
pr_500 = percentile_rank(samples_ms, 500)
print(f"PR(:500ms) = {pr_500:.1f}%  (SLA requires 99%)")
# PR(:500ms) = 90.0%  (SLA requires 99%) — you're 9 percentage points short

IQM (Interquartile Mean)

IQM is simply TM(25%:75%), the average of the middle 50% of samples, discarding the top and bottom 25%. It’s extremely robust to outliers in both directions, useful when you expect noise from both ends of the distribution (e.g., some requests are trivially fast cache hits, others are pathologically slow).

11. The Inspection Paradox: Your Users Experience Worse Than Your Metrics Show

As Marc Brooker’s explained in his blog, this is the most underappreciated gap in distributed systems reliability. For example, say your service has outages with very different durations: some resolve in 30 seconds, but occasionally one runs for 3 hours. Your MTTR (Mean Time to Recovery) might calculate to 5 minutes. But when a user hits your service during an outage, they’re more likely to land in a long outage than a short one because long outages have more time-slots for users to arrive in.

Customer-experienced mean recovery = (1/2) × (MTTR + Variance/MTTR)

The second term is what kills you. If your outage duration has high variance, e.g., fast recovery most of the time, but occasional 3-hour events then that variance term dominates. Your customers experience something dramatically worse than your MTTR.

import random
import math
import statistics

def inspection_paradox_demo(
    median_recovery_min: float,
    p99_recovery_min: float,
    arrivals_per_min: float = 100,
    n_outages: int = 2000
) -> dict:
    """
    Simulate the gap between operator MTTR and customer-experienced recovery.
    
    Key insight: customers are t-weighted samplers of your outage distribution.
    A 10-minute outage gets sampled by ~10x as many clients as a 1-minute outage.
    """
    # Fit lognormal to median and p99
    mu = math.log(median_recovery_min)
    sigma = (math.log(p99_recovery_min) - mu) / 2.326

    server_durations = []
    client_wait_times = []

    for _ in range(n_outages):
        duration = random.lognormvariate(mu, sigma)
        server_durations.append(duration)

        # Clients arrive as a Poisson process during the outage
        t = 0.0
        while True:
            gap = random.expovariate(arrivals_per_min)
            if t + gap > duration:
                break
            # This client arrived at time t, waits until outage ends
            client_wait_times.append(duration - t)
            t += gap

    return {
        "operator_mttr":        statistics.mean(server_durations),
        "operator_p99":         sorted(server_durations)[int(len(server_durations) * 0.99)],
        "customer_mean_wait":   statistics.mean(client_wait_times) if client_wait_times else 0,
        "customer_p99_wait":    sorted(client_wait_times)[int(len(client_wait_times) * 0.99)] if client_wait_times else 0,
        "experience_gap_ratio": (statistics.mean(client_wait_times) / statistics.mean(server_durations)) if client_wait_times else 0,
    }

result = inspection_paradox_demo(
    median_recovery_min=1,    # median outage resolves in 1 minute
    p99_recovery_min=60,      # but 1% of outages take an hour
)

print("Scenario: 1-minute median recovery, 60-minute p99 recovery")
print()
print("What your on-call dashboard shows:")
print(f"  MTTR:              {result['operator_mttr']:.1f} minutes")
print(f"  p99 recovery:      {result['operator_p99']:.1f} minutes")
print()
print("What your customers actually experience:")
print(f"  Mean recovery:     {result['customer_mean_wait']:.1f} minutes")
print(f"  p99 recovery:      {result['customer_p99_wait']:.1f} minutes")
print(f"  Experience gap:    {result['experience_gap_ratio']:.1f}x worse than MTTR")

Scenario: 1-minute median recovery, 60-minute p99 recovery

What your on-call dashboard shows:
  MTTR:              4.9 minutes
  p99 recovery:      56.6 minutes

What your customers actually experience:
  Mean recovery:     60.0 minutes
  p99 recovery:      797.3 minutes
  Experience gap:    12.1x worse than MTTR

This is why tail recovery time matters more than averages suggest. Timeout-and-retry can hide individual request latency, but it cannot hide recovery time. Once a client gets stuck in an outage, retries don’t shorten the outage, they just add load to an already struggling service. The right takeaway: minimize variance in recovery time, not just its mean. Bounded, predictable recovery is far better for customers than fast-average-but-occasional-disaster.

12. Tail Latency Amplifies in Microservices

Modern architectures decompose user requests into many service calls. This creates two topologies, and both amplify tail latency:

Fan-out math: If each service has a 1% probability of a slow response, the probability that at least one is slow when calling N services in parallel is:

P(at least one slow) = 1 - (1 - 0.01)^N

What was a rare 1% tail now affects the majority of user interactions. And here’s the pernicious part: your per-service p99 metric looks perfectly fine. The damage is invisible at the service level, only visible at the user-experience level.

import numpy as np, random

def simulate_fanout(n_backends: int, tail_prob: float = 0.01, n_reqs: int = 20_000):
    """
    Simulate client experience when calling n_backends in parallel.
    Each backend: (1-tail_prob) chance of fast, tail_prob chance of slow.
    """
    results = []
    slow_count = 0
    for _ in range(n_reqs):
        latencies = []
        for _ in range(n_backends):
            if random.random() < tail_prob:
                latencies.append(random.gauss(250, 25))
                slow_count += 1
            else:
                latencies.append(random.gauss(10, 2))
        results.append(max(latencies))  # fan-out: wait for slowest
    
    arr = np.array(results)
    return {
        "p50":  np.percentile(arr, 50),
        "p99":  np.percentile(arr, 99),
        "mean": np.mean(arr),
        "pct_slow_user_requests": np.mean(arr > 50) * 100,
    }

print(f"{'N':>4} {'p50 (ms)':>10} {'p99 (ms)':>10} {'mean (ms)':>10} {'% users hit slow':>18}")
for n in [1, 5, 10, 25, 50, 100]:
    r = simulate_fanout(n)
    print(f"{n:>4} {r['p50']:>10.1f} {r['p99']:>10.1f} {r['mean']:>10.1f} {r['pct_slow_user_requests']:>18.1f}%")

The trimmed mean blind spot revisited. At N=50, nearly 40% of user requests are slow. But your per-service tm99 (averaging the best 99% of individual service calls) still looks great because it’s averaging the fast cluster. This is exactly the case where trimmed mean gives you false comfort. You need explicit end-to-end latency tracking at the user-request level, not just per-service tail tracking.

13. The Pooling Dividend: Why Redundancy Is Non-Linear

Adding servers doesn’t just increase capacity linearly but it also improves latency through pooling. This comes from the Erlang C model in queuing theory. For example, two designs, both handling the same total load:

• Design A: 1 server at 80% utilization
• Design B: 10 servers sharing load, each at 80% utilization

Design A has roughly a 13% chance of any incoming request finding the server busy and joining a queue. Design B has roughly a 3.6% chance. Double the fleet to 20 servers at the same 80% per-server utilization, and the queueing probability drops toward 1%. You’re getting better latency and better tail behavior at the same per-server cost.

import math
from functools import lru_cache

def erlang_c(c: int, rho: float) -> float:
    """
    Erlang C formula: probability an arriving request must queue
    (rather than being served immediately) in an M/M/c system.
    
    c: number of servers
    rho: per-server utilization (0 < rho < 1)
    """
    a = c * rho  # total offered load
    
    @lru_cache(maxsize=None)
    def factorial(n: int) -> int:
        return 1 if n <= 1 else n * factorial(n - 1)
    
    # Sum term for the denominator
    sum_term = sum(a**k / factorial(k) for k in range(c))
    last_term = (a**c / factorial(c)) * (1 / (1 - rho))
    
    ec = last_term / (sum_term + last_term)
    return ec

print("Probability a request must queue before being served:")
print(f"{'Servers':>8} {'Utilization':>12} {'Queue prob':>12}   {'Queue %':>8}")
for c in [1, 2, 5, 10, 20, 50]:
    ec = erlang_c(c=c, rho=0.8)
    print(f"{c:>8} {'80%':>12} {ec:>12.4f}   {ec*100:>7.1f}%")

Probability a request must queue before being served:
 Servers  Utilization   Queue prob    Queue %
       1          80%       0.8000      80.0%
       2          80%       0.7111      71.1%
       5          80%       0.5541      55.4%
      10          80%       0.4092      40.9%
      20          80%       0.2561      25.6%
      50          80%       0.0870       8.7%

Most of the benefit materializes at modest fleet sizes. You don’t need to be at hyperscale to get pooling gains. A fleet of 5-10 servers sharing load through a proper load balancer will have dramatically better tail latency behavior than the same compute running as independent instances.

14. Retries, Circuit Breakers, and the Amplification Trap

Retries protect against transient failures like a GC pause, a brief network glitch, a thundering herd. In past production deployment, I use up to 3 retries with exponential backoff for idempotent read operations. The protection against false positives is real and worthwhile. But retries have a catastrophic failure mode: retry amplification.

A single user request can generate 3 × 3 × 3 = 27 actual requests to a struggling downstream service. This turns a partial overload into a total collapse. I’ve watched this happen in production, e.g., a service that was at 60% capacity receives a burst of retries from a misbehaving upstream and immediately spikes to 200% load, failing every request, causing more retries, a feedback loop.

The mitigations:

import time
import threading
from collections import deque

class RetryBudget:
    """
    Limit total retry rate as a fraction of total traffic.
    If retries exceed the budget, fail fast instead of retrying.
    
    Classic mitigation for retry amplification.
    """
    def __init__(self, budget_fraction: float = 0.10, window_seconds: int = 60):
        self.budget_fraction = budget_fraction
        self.window = window_seconds
        self.total_requests: deque = deque()
        self.retry_requests: deque = deque()
        self._lock = threading.Lock()

    def _prune(self):
        cutoff = time.monotonic() - self.window
        while self.total_requests and self.total_requests[0] < cutoff:
            self.total_requests.popleft()
        while self.retry_requests and self.retry_requests[0] < cutoff:
            self.retry_requests.popleft()

    def record_request(self):
        with self._lock:
            self.total_requests.append(time.monotonic())

    def should_retry(self) -> bool:
        """Returns True if we have retry budget remaining."""
        with self._lock:
            self._prune()
            total = len(self.total_requests)
            retries = len(self.retry_requests)
            if total == 0:
                return True
            current_rate = retries / total
            if current_rate < self.budget_fraction:
                self.retry_requests.append(time.monotonic())
                return True
            return False  # budget exhausted — fail fast, don't amplify


class CircuitBreaker:
    """
    Stop sending requests to a failing downstream.
    Transitions: CLOSED -> OPEN -> HALF_OPEN -> CLOSED
    """
    CLOSED, OPEN, HALF_OPEN = "CLOSED", "OPEN", "HALF_OPEN"

    def __init__(self, failure_threshold: float = 0.5, cooldown_seconds: float = 30):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown_seconds
        self.state = self.CLOSED
        self.failures = 0
        self.total = 0
        self.opened_at: float | None = None

    def call_allowed(self) -> bool:
        if self.state == self.CLOSED:
            return True
        if self.state == self.OPEN:
            if time.monotonic() - self.opened_at > self.cooldown:
                self.state = self.HALF_OPEN
                return True  # let one probe through
            return False  # fail fast
        return True  # HALF_OPEN: let one probe through

    def record_success(self):
        self.failures = 0
        self.total = 0
        self.state = self.CLOSED

    def record_failure(self):
        self.failures += 1
        self.total += 1
        if self.total >= 10 and self.failures / self.total >= self.failure_threshold:
            self.state = self.OPEN
            self.opened_at = time.monotonic()

Hedge requests are often better than retries for latency problems. Instead of waiting for a timeout and retrying, fire a second request after a short delay (say, the p90 latency). Accept whichever responds first, cancel the other. This cuts your tail exposure without amplifying load as aggressively, because typically one of the two requests will succeed quickly.

15. Synthetic Canaries in Production

Error rates and latency percentiles tell you what’s happening to real traffic but only after users are affected. Synthetic canaries fill the gap: background processes that continuously exercise your API end-to-end, giving you availability signal even at 3am when real traffic is low.

Key design decisions from production experience:

• Test the full workflow, not just the health endpoint. A canary for a data API should create, read, update, and delete a record. One for an auth service should issue a token, validate it, and revoke it. Shallow canaries that only call GET /health will miss the exact failures that health check anti-patterns also miss.
• Track first-attempt and final success separately. If your canary succeeds on retry 2 90% of the time, the final success rate looks fine but something is quietly broken. First-attempt success rate catches this.
• Keep canary observability separate from production. Mixing them has two failure modes: canary failures inflate your production error rate, and canary successes can mask production degradation if canaries hit warm caches or a separate code path.
• Account for canary bias. Canaries hit warm caches and have predictable access patterns. Their p99 is almost always better than real user p99. Use canary latency to detect regressions relative to a baseline, not to claim absolute performance numbers.
• Use retries in canaries, but with a limit. Up to 3 retries prevents false positives from transient network blips. But record the retry count per run, e..g, a canary that regularly needs 2+ retries is a signal worth investigating even if it eventually succeeds.

16. Putting It All Together: A Layered Monitoring Strategy

After decades of building and operating distributed systems, here’s the monitoring architecture I’d deploy for any production service from day one:

Closing: The Number That Matters Most

After all of this, the insight that has most changed how I think about availability is this: your users don’t experience your MTTR. They experience a version of it weighted by how long outages last, which skews dramatically toward your worst events. A service with a 1-minute median recovery but occasional 2-hour outages will have customers experiencing something closer to hours, not minutes. The variance in your tail events matters more than the central tendency. This is why the tail cannot be trimmed away from your visibility. Build observability that shows you the tail. Use redundancy and retries but understand how they amplify under pressure. Run canaries that exercise the whole path. Track user errors and server errors separately. Keep SLO burn rate visible so you always know how much budget you’ve spent. And when your customers say the service is slow and your dashboard says everything is green then believe the customers.