Shahzad Bhatti Welcome to my ramblings and rants!

August 25, 2026

Orchestrating Background AI Agents for Software Teams

Filed under: Computing — admin @ 4:25 pm

How to design agents as a graph, run them through a harness that keeps them reliable, sandbox them because you can’t just trust what they do. (This is a follow-up to AI Writes Code, You Own the Design and Declarative AI Coding Agents with an Orchestration System)


I have been using various workflow systems for business process management, data pipelines and batch processing over twenty years. I built a declarative orchestration system over ten years, which was before GitHub Actions, CircleCI, and GitLab CI. The idea was simple: describe your work as a graph of tasks with pipes and filter patterns declaratively, give each task clear inputs and outputs, let a server run them on a schedule or in response to events, and report the results back. Automation has always been the core discipline of software engineering and we built tools for CI/CD, cron jobs, and data pipelines. With AI, your job is changing from the one who executes the steps to a conductor. Instead of manual coding, testing and other tasks, you now design the graph of nodes, define what each node is allowed to do, let agents work on the nodes while you monitor the work where it needs a human hand. You need a harness around each agent so the graph runs reliably, a sandbox for every agent because you fundamentally cannot trust what an LLM decides to do, and a skills layer so the same harness stays general-purpose.

I have seen teams building tightly coupled monolithic agentic systems that includes poorly implemented orchestration, the integrations, the prompts, and the skills, all bolted together. This makes it harder to make changes or extend agent capabilities and skills independently. In this post I will walk through three open source projects: Formicary, the orchestration engine that turns workflows into a graph with the Slack interface; ai-dev-tools, the harness of small scripts that actually do the work; and you-got-skills, the library of skills for SDLC.


Daily Routine

Each morning, you typically have to scan Jira/Github board, check pull requests to review, triage any new blockers, scroll through Slack messages to respond before deciding what to actually work on. AI coding tools have automated the implementation but you still need to know what the team is doing, catching problems, following up on reviews, etc. The basic flaw in most “agentic” setups today is that they’re still fundamentally interactive. You close your laptop and the agent stops. You need a way to keep agents working in background and instead of building systems that make you go pull information, and build agents that push it to you instead. You need somewhere to define the graph of who-triggers-what in a sandbox environment. You need a harness around each step so a flaky script doesn’t take the whole pipeline down. And you need the agent’s behavior to live somewhere editable, not buried in a prompt string, so the system stays general enough to point at a new kind of problem without a rewrite.


The Architecture: Three Tools, Four Layers

This section describes three open source tools: Formicary is the orchestration engine with Slack integration; ai-dev-tools is the harness that actually runs each agent inside a sandbox; and you-got-skills is the knowledge layer that keeps the whole system general-purpose.

Above diagram shows a graph where every box is a node with a job type, a task, a skill and every arrow is an edge that Formicary evaluates at runtime. Designing agents this way, instead of as one long prompt with a loop around it makes the system debuggable. The ai-dev-tools is the harness that runs inside each node and it turns “call an LLM” into “call an LLM inside a container, with a defined timeout, a defined exit-code contract in a sandbox environment. You should not trust an agent’s judgment about what’s safe to run but with sandbox you let the harness enforce the boundary.

Why hand off state through files?

Formicary provides declarative syntax to store artifacts or consume artifacts from a previous task. Every script starts by checking whether its own output already exists:

# Every script starts with this pattern
existing = read_json(config, issue_id, "plan_result.json")
if existing and existing.get("status") == "DONE":
    print("Already done, skipping")
    sys.exit(0)

If a task dies halfway through and gets re-run, it just picks up where it left off. Exit codes are the contract between a script and the orchestrator:

Exit codeMeaningWhat Formicary does
0SuccessMove on to the next task
1Error (worth retrying)Retry with backoff
2Blocked for a human inputPause the job indefinitely
3Not finished yet / waiting on somethingPause, then resume on a trigger or after a delay

The Skills Library

Skills are the knowledge layer that allow general-purpose harness instead of hardcoded to whatever workflow you built. The graph and the sandbox don’t know anything about code review or standups as they just run nodes. A skill is what tells an agent how to do a specific kind of engineering work well. Add a new skill and you’ve extended the system to a new kind of task without touching Formicary or ai-dev-tools at all.

you-got-skills/
??? skills/
    ??? ygs-standup/skill.md
    ??? ygs-risk-scan/skill.md
    ??? ygs-implement/skill.md
    ??? ygs-review-pr/skill.md
    ??? ygs-code-review/skill.md
    ??? ygs-security-review/skill.md
    ??? ygs-sre-review/skill.md
    ??? ygs-learn/skill.md
    ??? ygs-retro/skill.md
    ??? ygs-sprint-plan/skill.md
    ??? ygs-investigate/skill.md
    ??? ygs-qa/skill.md
    ??? ygs-wbs/skill.md
    ??? ygs-estimate/skill.md
    ??? ygs-ship/skill.md
    ??? ygs-triage/skill.md
    ??? ... 28 total

Together they cover the whole lifecycle:

PhaseSkills
Requirementsygs-refine-prd, ygs-review-prd, ygs-refine-trd, ygs-review-trd
Architectureygs-refine-architecture, ygs-review-architecture, ygs-spike
Planningygs-sprint-plan, ygs-wbs, ygs-estimate, ygs-triage
Executionygs-implement, ygs-qa, ygs-ship, ygs-uat
Reviewygs-review-pr, ygs-code-review, ygs-security-review, ygs-sre-review, ygs-api-review, ygs-ui-review
Team intelligenceygs-standup, ygs-risk-scan, ygs-pr-queue, ygs-sync
Learningygs-learn, ygs-retro, ygs-investigate

Skills produce and consume a consistent folder structure inside your project:

your-project/
??? docs/
?   ??? prd/              # Product requirements (YYYY-MM-DD-slug.md)
?   ??? trd/               # Technical designs
?   ??? adr/               # Architecture decisions (NNN-slug.md)
?   ??? spikes/            # Spike findings
?   ??? learnings/         # Learnings pulled from PRs and incidents
??? tasks/
    ??? backlog/            # task-NNN.md
    ??? in-progress/        # moving a file here = starting it
    ??? done/               # moving a file here = finishing it

Status is the folder. No ticket-state dropdown, no transition workflow to configure. mv tasks/backlog/task-042.md tasks/in-progress/ means the task has started. git blame tells you who moved it and when.

Diverge, then converge

The diverage and converge pattern allows running multiple independent LLM passes first (diverge), then merge and rank what came back (converge). A single reviewer may miss something but several independent reviewers will catch most of the issues. The ygs-review-pr skill runs four independent passes in parallel for correctness, security, API surface, and SRE concerns. A finalize step then merges everything and ranks it by severity.

The standup workflow uses the same idea. ygs-standup gathers signals from the issue tracker and from Slack then cross-references the two.


The Standup Workflow

Every weekday at 8am, a cron job wakes up, queries your Jira sprint (or GitHub), reads the open PRs, pulls the last 26 hours of Slack messages from your standup channel, and turns all of it into a brief.

# ai-standup-jira.yaml
job_type: ai-standup-jira
description: "Daily standup brief from Jira sprint + Bitbucket PRs + Slack signals"
cron_trigger: "0 0 8 * * 1-5 *"
max_concurrency: 1
timeout: 900s

tasks:
  - task_type: gather     # pulls Jira issues, Bitbucket PRs, Slack in parallel
  - task_type: synthesize # Claude + ygs-standup + ygs-risk-scan ? standup_brief.md
  - task_type: post       # renders HTML artifact, posts to Slack

The synthesize task calls the ygs-standup skill, which follows a fairly strict protocol:

**Alice:** Closed PROJ-42 (auth fix). Working on PROJ-51 (rate limiter)
— PR open 28h, no review yet. [Slack: "waiting on infra cert renewal"]

**Bob:** No tracker activity in the last 24h. Last Slack message Monday
(3 days ago). [Slack: silent since Monday]

**Carol:** PROJ-55 (data export) marked In Progress, no commits in 4 days.
Blocked label present. [Tracker: blocked since Tuesday]

Every claim traces back to a ticket, a PR, or a Slack message. ygs-risk-scan runs right after and appends a ranked list, using thresholds you can tune:

? HIGH  PROJ-55 blocked, blocks PROJ-60 (in progress, owned by different person)
? MED   PR #142 open 28h, single reviewer, no activity
? MED   Carol: no updates in 4 days, sprint ends Friday

The thresholds live in a shared Markdown file:

SignalDefault severityEscalates to HIGH if…
Issue stale > 3 daysMEDIUMit blocks another issue
Issue stale > 5 daysHIGH
PR open > 2 days, no reviewMEDIUMonly one reviewer assigned
PR open > 4 daysHIGH
Person silent > 2 daysMEDIUMalso no tracker activity
Blocked labelHIGH
Dependency chain: upstream is staleHIGH
Sprint ends in < 2 days, not startedHIGH

You can also trigger the standup on demand from Slack:

@bot standup
@bot risk

Label an Issue, Get Back a Pull Request

You can label any Jira or GitHub issue ai-ready. Every five minutes a cron job checks for newly labeled issues and kicks off a four-task pipeline. When it’s done, there’s an open PR with an implementation and tests, the label has flipped to ai-pr-open.

# ai-gh-issue-picker.yaml
job_type: ai-gh-issue-picker
cron_trigger: "*/5 * * * *"
tasks:
  - task_type: gather-issues
    script:
      - python -m scripts.gh.issue_picker
    on_exit_code:
      2: COMPLETED  # no issues = not an error
  - task_type: submit-jobs
    # Uses formicary template to fan out one ai-gh-implement job per issue
    script:
      - '{{SubmitJobsFromJSON "ai-gh-implement" .IssuesJSON}}'

It has a built-in guard: if 10 or more implement jobs are already running or queued, it skips its turn. That stops someone from labeling 50 issues at once and blowing up the queue. The implementation pipeline itself:

# ai-gh-implement.yaml
job_type: ai-gh-implement
max_concurrency: 5
timeout: 86400s    # 24 hours — some implementations take a while

tasks:
  - task_type: plan
    timeout: 15m
    script:
      - python -m scripts.gh.issue_picker --issue-id {{.IssueNumber}}
      - python -m scripts.gh.plan --issue-id {{.IssueNumber}}
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — needs a human before continuing
    on_completed: implement

  - task_type: implement
    timeout: 45m
    script:
      - python -m scripts.gh.implement --issue-id {{.IssueNumber}}
    on_completed: create-pr

  - task_type: self-review
    timeout: 10m
    script:
      - python -m scripts.review.run --mode self-review
          --issue-id {{.IssueNumber}} --base-branch main
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — critical finding, needs a human before the PR opens
    on_completed: create-pr

  - task_type: create-pr
    timeout: 10m
    script:
      - python -m scripts.gh.create_pr --issue-id {{.IssueNumber}}
    on_completed: poll-pr

  - task_type: poll-pr
    dependencies: [create-pr, poll-pr]  # self-referencing = loop
    script:
      - python -m scripts.gh.poll_pr --issue-id {{.IssueNumber}}
    on_exit_code:
      3: PAUSE_JOB    # PR still open — check back in `delay` seconds
    delay: "{{.PollInterval}}s"  # default 120s

Here’s the full chain:

Let’s walk through what actually happens for a real issue.

Plan

scripts/gh/plan.py calls Claude with up to 50 turns and a prompt that tells it to:

1. Read CLAUDE.md, .cursorrules, or any repo-specific coding guidelines if they exist
2. Discover .claude/skills/ in the repo — if a skill applies, plan to invoke it
3. Before designing new abstractions, search utils/, shared/, common/ for existing utilities
4. Check for monorepo structure
5. Generate a concise plan covering:
   - Task breakdown with complexity estimates (S/M/H/XL)
   - Exact files to create/modify per task
   - Test strategy: write failing tests first, then implement
   - A "Failing Test Spec" section
   - Any risks or blockers
6. Classify overall complexity: S/low (?3 files), M/medium (4-10), H/high (>10)
7. Write the plan to PLANS/{slug}-{issue_id}-plan.md

What it produces:

/workspace/42/
??? issue.json         # issue title, body, labels, assignee
??? plan.md            # human-readable plan
??? plan_result.json   # {"status":"DONE","task_count":3,"total_complexity":"M"}
??? PLANS/
    ??? add-rate-limit-42-plan.md

Implement

scripts/gh/implement.py calls Claude with up to 200 turns. The key rules from the ygs-implement skill:

  • Check for existing utilities before writing new ones.
  • TDD: write the failing test first, then make it pass.
  • One commit per plan task, message format "task: <description>".
  • After all tasks are done, run the full test suite and iterate on failures up to twice.
/workspace/42/
??? impl_result.json   # {"status":"DONE","files_changed":["src/..."],"commits":5,"tests_status":"passing"}
??? branch.txt         # "ai/42-add-rate-limiting"

ygs-implement also uses “ceremony levels” so small tasks don’t get over-engineered:

Light    (1-3 files, <300 lines):   proceed directly, skip checkpoints
Standard (4-8 files, 300-800):      plan mode + checkpoint every 5 files
Heavy    (8+ files, 800+ lines):    flag as oversized, ask user to split

Picking the model by complexity

The plan task classifies overall complexity and writes it to /workspace/plan_complexity.txt. The implement task reads that and picks the right model:

# scripts/common/config.py
COMPLEXITY_MODEL_MAP = {
    "low":    MODEL_BEDROCK_HAIKU,   # ?3 files, simple edits — fast and cheap
    "medium": MODEL_BEDROCK_SONNET,  # default — most issues
    "high":   MODEL_BEDROCK_OPUS,    # complex architecture changes
}

The plan prompt writes a single word (low, medium, or high) to that file, and the implement task reads it back:

# In ai-gh-implement.yaml — implement task
COMPLEXITY=$(cat /workspace/plan_complexity.txt 2>/dev/null || echo "medium")
AI_MODEL="${ANTHROPIC_COMPLEXITY_${COMPLEXITY^^}_MODEL:-${ANTHROPIC_DEFAULT_SONNET_MODEL}}"

AnthropicComplexityLowModel and AnthropicComplexityHighModel are set in your org config by deploy-ai-workflows.sh, and can be overridden per deployment in models.env.

Polling the PR and responding to feedback

Every 2 minutes, the poll task checks the PR for new comments. It only reacts to comments that start with ai-bot. The agent stays out of human-to-human review discussion, and it never responds to its own earlier comments. So when a reviewer writes:

ai-bot please add a test for the rate limit exceeded case

The poll task reads it, applies the feedback, marks the comment handled in processed_comments.json, and keeps polling. Once the PR merges, the learning step kicks off automatically.


PR Review With a Human Gate

Code review usesthe diverge-then-converge pattern to review the code with multiple perspectives like security, architecture, SRE, etc.

# ai-gh-review.yaml
job_type: ai-gh-review
max_concurrency: 10

tasks:
  - task_type: review        # Claude runs ygs-review-pr, writes findings.json
  - task_type: await-feedback # posts Block Kit to Slack, exits 3 ? PAUSE_JOB
  - task_type: finalize       # reads Decision, posts result to PR thread

The review step runs ygs-review-pr with this instruction:

1. Invoke the /ygs-review-pr skill to perform a full PR review
2. After the skill completes, write findings to findings.json
3. Output ONLY this JSON on the last line:
   {"status":"DONE","findings_count":N,"verdict":"APPROVE|REQUEST_CHANGES","summary":"..."}

ygs-review-pr runs four passes in paralle:

  1. Correctness: logic errors, null handling, incomplete enum handling, partial failure, race conditions, off-by-one errors
  2. Security: injection vectors (SQL, command, XSS, SSRF, path traversal), auth/authorization, data exposure across tenant boundaries, etc.
  3. API surface: breaking changes, contract violations, backwards compatibility, versioning
  4. SRE: failure modes, blast radius, observability gaps, rollback safety, resource consumption, dependency risk

Once all four are done, findings get merged and ranked: CRITICAL > HIGH > MEDIUM > LOW, and by confidence within each level. The verdict maps straight off the highest severity found:

Any CRITICAL or HIGH finding  ? REQUEST_CHANGES
Only MEDIUM/LOW findings      ? COMMENT
No findings / only low conf.  ? APPROVE

Deep review: seven domains in one pass

Standard review runs four passes. Deep review adds three more: performance, testing quality, and architecture.

@bot deep review https://github.com/org/repo/pull/42

These all map to the same ai-gh-review (or ai-jira-review) job type, just with ReviewDepth=deep injected as a static job variable:

# workflows.yml — deep-review entry
- name: deep-review
  job_type: ai-gh-review
  triggers: ["deep review", "full review", "arch review"]
  target_kind: github
  extra_params:
    ReviewDepth: "deep"
  description: "Deep 7-domain GitHub PR review: standard + performance, testing quality, architecture"

The workflow reads ReviewDepth and picks the right skill: ygs-review-deep when it’s set to “deep,” ygs-review-pr otherwise. ygs-review-deep is a superset of the standard four passes, plus:

  • Performance: algorithmic complexity, N+1 queries, cache misses, lock contention, unnecessary allocations
  • Testing quality: coverage gaps, brittle assertions, missing edge cases, test-code coupling
  • Architecture: single-responsibility violations, circular dependencies, premature abstractions, missing boundaries

Self-review before the PR even opens

The implement pipeline runs a self-review task right before create-pr. Before the PR exists, the agent reviews its own diff against the base branch:

  - task_type: self-review
    timeout: 10m
    script:
      - python -m scripts.review.run --mode self-review
          --issue-id {{.IssueNumber}} --base-branch {{.BaseBranch}}
    on_exit_code:
      2: PAUSE_JOB    # BLOCKED — critical finding, needs a human before the PR opens
    on_completed: create-pr

This runs ygs-implement in review mode, compares the diff to the original plan, and writes self_review.json. The outcome maps directly to an exit code:

self_review_statusExit codeWhat the pipeline does
APPROVED0Proceed to create-pr
NEEDS_FIX0Claude fixes it inline, then create-pr
BLOCKED2PAUSE_JOB — a human needs to decide before the PR opens

How Slack Messages Turn Into Workflows

Socket Mode lives inside the Formicary queen itself. The queen opens an outbound WebSocket to Slack using your xapp- app-level token. When you mention the bot:

@bot review https://github.com/myorg/myrepo/pull/142

The queen’s SlackService does one thing, deterministically: it strips the mention, takes the first word, and looks it up against a route table in the queen’s config.

# In k8s/formicary-leader.yaml ConfigMap, under slack.routes:
slack:
  routes:
    - triggers: ["review", "pr"]
      job_type: ai-gh-review
      description: "PR review: correctness, security, API, SRE"
    - triggers: ["implement", "build"]
      job_type: ai-jira-implement
      description: "Full pipeline: plan ? implement ? PR"
    - triggers: ["standup", "status", "daily"]
      job_type: ai-standup-jira
      description: "Daily standup brief from Jira"
    - triggers: ["adhoc"]
      job_type: ai-adhoc
      description: "Ad-hoc Claude invocation with any skill"

Everything after the trigger word gets passed through as the Prompt job parameter, verbatim. The queen’s whole job is mapping verb to job type and passing the text along. It does zero AI work of its own. All of that happens inside the ai-dev-tools container once the job actually starts. Once routing resolves, the queen submits the job and replies right in the thread:

Started ai-gh-review (job req-7f3a2) — I'll post updates here.
https://formicary.example.com/dashboard/jobs/requests/req-7f3a2

Replying to the thread resumes a paused job. If a review job is paused waiting on a decision and you reply in that thread, the queen matches it by SlackThreadTs and resumes it with your reply text injected as Prompt.

Registering as a developer

Before Slack commands work for you, you DM the bot your Formicary API token, once:

DM to @bot:
setup eyJhbGc...  (your Formicary API token)

The queen validates the token inline and from that point on, any @bot mention from you has a known Formicary identity behind it.

How multi-tenant isolation actually works, end to end:

When you type @bot review https://github.com/org/repo/pull/42, here’s what the queen does, entirely server-side:

  1. Reads your Slack user ID (U0A1HQL0C9J) off the Socket Mode event.
  2. Looks up slack_user_id = U0A1HQL0C9J in user_configs and finds your Formicary user record, including your UserID and OrganizationID.
  3. Calls SaveJobRequest(qc, req) and the server overwrites request.UserID and request.OrganizationID from that context.
  4. Schedules the job and the ant scheduler first looks for a worker registered under your org_id.

Ant routing: when you connect your laptop as a worker with setup-ant-worker.sh --token <your-token>, the queen reads org_id out of your JWT at connect time and records it on that worker. Your jobs prefer your own worker.

All the commands

What you typeWhat runs
@bot standupDaily brief: per-person status, risks, discussion questions (routes to Jira or GitHub via DEFAULT_TRACKER)
@bot risk / @bot risksRanked sprint risks with a capacity check
@bot prs / @bot open prs / @bot review queueOpen PRs grouped by reviewer status, sorted by age
@bot pr comments <url>All inline feedback and open tasks for a PR
@bot review <github-url>Standard PR review: correctness, security, API, SRE (4 domains)
@bot review <bitbucket-url>Same, for Bitbucket PRs
@bot deep review <url>Deep review: standard 4 domains + performance, testing, architecture (7 domains)
@bot full review <url>Alias for deep review
@bot arch review <url>Alias for deep review
@bot security review <url>OWASP-focused security audit
@bot sre review <url>Failure modes, observability, deploy safety
@bot implement PROJ-123Full pipeline: plan ? implement ? self-review ? PR, for a Jira issue
@bot implement 42Same, for a GitHub issue number (model picked by complexity: Haiku/Sonnet/Opus)
@bot jira-query <term> / @bot qjira <term>Search open Jira issues by keyword, results as a Block Kit table
@bot jira-analyze PROJ-1, PROJ-2Claude analyzes root cause + possible fixes for Jira issues
@bot gh-query <term>Search open GitHub issues by keyword
@bot gh-analyze #123, #456Claude analyzes root cause + possible fixes for GitHub issues
@bot adhoc <free text>Run any Claude skill with a freeform prompt
@bot helpList every command

Ad-hoc Skill Execution

ai-adhoc is a general-purpose runner: any you-got-skills skill can be invoked with a free-form prompt, and the result comes back into your Slack thread.

# ai-adhoc.yaml
job_type: ai-adhoc
max_concurrency: 20
timeout: 1800s

variables:
  Skill:  { type: STRING, required: true }
  Prompt: { type: STRING, required: true }

At runtime the script:

  1. Looks for the skill in a few candidate locations (/workspace/skills, ~/.claude/skills/you-got-skills/skills, ~/workplace/you-got-skills/skills).
  2. Writes .ygs/tracker.yml dynamically from environment variables (Jira or GitHub config, team members, sprint info).
  3. Invokes Claude with the skill content and your prompt.
  4. Strips Markdown formatting from the output so it renders cleanly in Slack.
  5. Posts up to 3000 characters back into the originating thread.

Adding a new shorthand command is just one entry in the queen’s route table:

# k8s/formicary-leader.yaml — slack.routes
slack:
  routes:
    - triggers: ["prs", "open prs", "review queue"]
      job_type: ai-adhoc
      description: "Open PRs grouped by review status"

@bot prs then submits ai-adhoc with Prompt="prs", and the container maps that to the ygs-pr-queue skill. No Python code changes needed.


Querying and Analyzing Issues From Slack

Two commands take you from a Slack message straight to structured Jira insight, no browser required.

@bot query-jira: find issues by keyword

@bot jira-query auth timeout

This submits an ai-jira-query job. The script builds a JQL query scoped to your project and, optionally, your team’s custom field. Results come back as a structured Slack Block Kit table:

Jira issues matching "flaky tests" (5 found)

PROJ-1001  [Bug] Flaky test in auth service                    - clickable link
           Status: In Progress   Priority: High
           Assignee: alice        Date: 2026-07-28

PROJ-995   [Story] Fix race condition in logger test
           Status: To Do         Priority: Medium
           Assignee: bob         Date: 2026-07-21
...

@bot jira-analyze: root cause from issue keys

@bot jira-analyze https://yourorg.atlassian.net/browse/PROJ-1001

This routes to the same ai-jira-query job type. The result posts back to your thread.

@bot gh-query / @bot gh-analyze the GitHub equivalents

Same commands, gh- prefix, for teams on GitHub instead of Jira:

@bot gh-query open authentication bugs
@bot gh-analyze https://github.com/org/repo/issues/42

Team filtering

Both commands automatically filter to your configured team and sprint:

  • JIRA_SPACE (or BITBUCKET_WORKSPACE): the team/area filter value.
  • JIRA_TEAM_FIELD: the Jira custom field name (default EngScrumTeam, resolved to a field ID dynamically).
  • Set JIRA_TEAM_FIELD="" to turn the filter off entirely.

The Learning Loop: Getting Better Over Time

Most agent systems are stateless and every run starts from zero. In our workflow, after every PR merges, learn.py runs automatically as part of the poll-pr task. It reads the PR comments, the implementation artifacts, and the review findings, then invokes ygs-learn:

ygs-learn protocol:
1. Capture: What happened? Why does it matter? Category?
2. Dedup: search docs/learnings/ for similar slug before creating new
3. Write to docs/learnings/YYYY-MM-DD-slug.md

A learning document looks like this:

# Rate limiter key collision when user has multiple active sessions

**Category:** Edge Case
**Date:** 2025-08-03
**Source:** PR review finding, PROJ-123

## Learning
When a user has multiple active sessions, rate limiting by user_id counts across
all sessions. A single slow client can exhaust the budget for all their tabs.

## Evidence
Review comment on PR #142 flagged this. Reproduced locally with two
concurrent sessions against the same account.

## Application
When implementing per-user rate limits, check whether session isolation is
intended. If counts should be per-session, key by session_id not user_id.

Next time an implementation runs, that document is part of the context Claude reads.

ygs-retro runs at sprint end. It reads tasks/done/, recent git history, and the sprint’s accumulated learnings, and asks pointed questions based on what it actually found.

ygs-investigate enforces a debugging discipline: build a feedback loop first, e.g., a failing test, a log line, a REPL session before forming any hypotheses. Rank hypotheses 1 through 5. Instrument one variable at a time with tagged markers.


Trust and Oversight

Though, AI agents have solved most of coding and testing tasks but it still requires human review and feedback. We need a trusted autonomy, with minimal friction. You don’t trust the model to know its own limits; you build a harness that doesn’t need you to. That’s why every agent runs inside a container it can’t escape. You can’t ask an AI agent to self-police or prompt it to be careful. Every decision point is explicit. The agent never merges to main. It opens a branch, opens a PR, and stops there. A human approves and merges. For anything that needs a more formal sign-off, Formicary supports approval workflows with SLAs:

- task_type: security-approval
  method: MANUAL
  approval_policy:
    min_approvals: 1
    sla_deadline: 4h
    timeout_action: ESCALATE
    escalation_recipients: "security-oncall@example.com,vp-eng@example.com"
    escalation_message: "Security approval SLA breached — deployment blocked"
  on_exit_code:
    APPROVED: deploy-prod
    REJECTED: notify-rejected

Secrets live in Kubernetes Secrets, never in ConfigMaps or plain env files. The container runs as non-root (uid 1000). Every artifact is a plain file and every state transition is logged in Formicary.


What Else You Can Build

Everything above is running in production today, but the same architecture supports a much wider range of background agents.

  • Codebase quality agent. A weekly workflow runs ygs-code-review across everything changed in the last week, posts a ranked findings report to Slack, and files tasks in tasks/backlog/ for anything CRITICAL or HIGH.
  • Documentation drift detector. A webhook fires when a PR touching an API handler merges. The workflow checks whether the matching docs were updated.
  • Duplicate abstraction scanner. A periodic workflow compares utility functions across repos owned by different teams and posts “team B has something that looks like what you just built,”.
  • Security posture monitor. Nightly, ygs-security-review runs against everything merged in the last 24 hours that touches auth, authorization, or data access. Findings go to a security channel.
  • Sprint health check, Wednesday afternoons. A mid-sprint cron runs ygs-risk-scan and only posts if it finds something HIGH severity. Most weeks it says nothing.
  • Bug pattern finder. A workflow runs ygs-investigate against recent error logs, proposes the top three hypotheses for each recurring pattern.

Getting Started

Installing the skills locally

git clone https://github.com/bhatti/you-got-skills.git
cd you-got-skills && ./setup

Now any skill runs right in your IDE:

/ygs-standup
/ygs-risk-scan
/ygs-review-pr https://github.com/org/repo/pull/42
/ygs-implement

Loading extra skill repos at runtime

Every job pod can pull in additional skill repos without a rebuild. Set EXTRA_SKILLS_REPOS before running the deploy script.

# 1. Plain URL — sparse-clones only the skills directory (fast, default)
EXTRA_SKILLS_REPOS=https://github.com/myorg/my-skills.git

# 2. Comma-separated — multiple repos in one value, YAML-safe
EXTRA_SKILLS_REPOS="https://github.com/bhatti/you-got-skills.git,skills-cli:nutlope/hallmark"

# 3. JSON array — full control (use for org config; JSON breaks YAML template substitution)
EXTRA_SKILLS_REPOS='[
  {"url": "https://github.com/bhatti/you-got-skills.git", "sparse": false},
  {"url": "nutlope/hallmark", "type": "skills-cli"}
]'

Setup environment variables:

GH_ORG=your-org
GH_REPO=your-repo
GH_TOKEN=ghp_your_token_here
ANTHROPIC_API_KEY=sk-ant-your_key_here
AI_MODEL=claude-sonnet-4-6

# Controls which tracker bare commands like "standup" route to.
# "jira" routes to ai-standup-jira; "github" routes to ai-standup-gh.
DEFAULT_TRACKER=jira

Start formicary server

You can use kubernetes to get Formicary running.

export COMMON_AUTH_JWT_SECRET="<stable-secret-never-rotate>"
export COMMON_AUTH_GOOGLE_CLIENT_ID="<google-client-id>"
export COMMON_AUTH_GOOGLE_CLIENT_SECRET="<google-client-secret>"
export SLACK_BOT_TOKEN="xoxb-..."
export SLACK_APP_TOKEN="xapp-..."

./scripts/deploy-formicary.sh --ec2-ip 10.X.X.X

Connect your ant worker

Jobs run on your own laptop’s local cluster.

./scripts/setup-ant-worker.sh \
  --queen formicary.example.com \
  --token "$FORMICARY_TOKEN"

Deploying the Slack integration

Slack is built into the Formicary queen, so there’s no separate router pod to deploy.

Create a Slack app with Socket Mode

In your Slack app settings:

  1. Enable Socket Mode: generate an xapp- app-level token (scope: connections:write).
  2. Add these bot token scopes under OAuth & Permissions:
ScopePurpose
app_mentions:readReceive @bot mentions
channels:historyRead channel messages
channels:readList channels
chat:writePost messages and Block Kit
groups:historyRead private channel messages
groups:readList private channels
im:historyRead DMs (for the setup registration flow)
im:writeReply in DMs
users:readResolve user display names
  1. Subscribe to bot events under Event Subscriptions:
    • app_mention@bot mentions in channels
    • message.im — DMs, for the setup registration flow
  2. Install to your workspace (needs admin approval if app installs are restricted).

Each developer registers

Anyone who wants Slack commands to work DMs the bot once:

DM to @bot:
setup eyJhbGc...  (Formicary API token from dashboard ? API Tokens)

In any channel the bot’s been invited to:

@bot help          - lists all commands
@bot standup       - runs standup, posts to thread

Summary

The core patterns in this post include declarative pipelines, cron triggers, file-based artifact handoff, exit-code contracts, approval gates. I have used these patterns to automate complex business processing, data pipelines, and CI/CD processes. I am now using it to automate AI backed tasks: a task can read code and form a judgment about it. When the graph, the harness, the sandbox, and the knowledge are four separate layers instead of one tangled system, each one evolves on its own. It allows you to update an environment with configuration, markdown files and configuration. Your job is now to design the graph, define what’s in each node, decide where the sandbox boundary sits, and write down what “good” looks like as a skill. Instead of manually gathering information, you use agents to do the low-level work. You then review the finished product, at the review verdict, at the escalation and decides what matters. That’s conducting, not playing every instrument, and it’s where your judgment actually belongs.


Related Reading

Code

August 16, 2026

Structured Concurrency in Modern Programming Languages Part V: The Coordination Models Behind It All (CSP, Actors, Linda, and async/await)

Filed under: Computing — admin @ 4:37 pm

This is a part of series on structured concurrency: Part I (the general problem and TypeScript), Part II (Erlang and Elixir), Part III (Go and Rust), and Part IV (Kotlin and Swift).

In the earlier parts of this series I delved into how TypeScript, Erlang, Elixir, Go, Rust, Kotlin, and Swift each handle structured concurrency in practice such as spawning tasks, waiting for children to finish, propagating errors, and cancelling work cleanly. But I skipped over the the coordination models these languages are actually built on. For example, Go didn’t invent channels and Erlang didn’t invent actors. Both are engineering ideas that go back to the 1970s and 80s, and once you understand the original model, most of the “gotchas” you hit while using the language stop looking like bugs and start looking like predictable consequences of a design choice made decades ago.

This post explains where each model came from, what it actually guarantees and how structured concurrency sits on top of all of them as a separate concern. It includes PlexSpaces, an actor-and-tuplespace framework I’ve been building in Rust that takes a pragmatic stance on this history, e.g., bounded mailboxes instead of Erlang’s unbounded ones, first-in-first-out matching instead of the classic tuple-space model’s unspecified ordering, and one small API instead of forcing you to learn several calculi at once.

A short timeline

It helps to see these ideas in the order they actually appeared:

  • 1973: Carl Hewitt proposes the actor model: small, isolated units of state that can only talk to each other by sending messages.
  • 1978: Tony Hoare publishes Communicating Sequential Processes (CSP), a mathematical notation (a “process algebra”) with a precise definition of what it means for two processes to synchronize.
  • 1985: David Gelernter publishes Linda, a coordination model built around a shared associative memory (the “tuple space”).
  • 1986: Gul Agha’s book extends the actor model with a fuller algebraic treatment.
  • 1986: Joe Armstrong and colleagues at Ericsson start building Erlang.
  • Early-to-mid 2000s: event-loop async/await goes mainstream: Node.js’s callback-then-promise evolution.
  • 2009: Go ships with goroutines and channels inspired by CSP.
  • 2018 onward structured concurrency (Trio in Python, Kotlin’s coroutines, Swift’s TaskGroup, Java’s StructuredTaskScope) formalizes a simple idea: a spawned task’s lifetime should never outlive the scope that spawned it.

Notice that everything on that list except the last item is about how work talks to other work. Structured concurrency is about a completely different question, i.e., how work’s lifetime gets tracked.

Concurrency and parallelism

Concurrency is a property of how a program is structured: multiple logically independent activities are in progress, possibly interleaved on a single CPU core. Parallelism is a property of execution: things are genuinely happening at the same time, which requires more than one core. You can have concurrency without parallelism like Node.js’s single-threaded event loop juggling many pending requests on one core. You can also have parallelism without concurrency like a tight SIMD loop doing the same arithmetic on many numbers at once has no interleaved independent logic at all. Go’s own documentation defines it as: concurrency is about dealing with lots of things at once, parallelism is about doing lots of things at once. Goroutines give you concurrency; whether that concurrency turns into real parallelism depends on GOMAXPROCS and how many cores are actually available.

This matter for CSP and actors because both are concurrency models and neither one is “more parallel” than the other. What actually differs between them is how they structure communication.

Five models, one spectrum of coupling

Every concurrency model is answering the same underlying question, i.e., how does one unit of work talk to another one.

ModelOriginHow units talkCouplingFormal backing
CSPHoare, 1978Synchronous rendezvous on a named channelTime-coupledFull algebra, checked by tools like FDR
Go-style CSPGo, 2009Channel, synchronous or bufferedTime-decoupled if bufferedNone
Actor modelHewitt 1973 / Erlang 1986Async message to a named addressIdentity-coupled, time-decoupledPartial (Clinger, Agha)
async/awaitNode.js/C#/Python event loopsFuture/promise handleTime-decoupledNone
Linda / tuple spaceGelernter, 1985Tuple matched by contentFully decoupled – no identity, no timingPartial (Klaim’s semantics)
Structured concurrencyTrio/Kotlin/Swift, 2018+Whatever the underlying model usesLifetime-coupled to a scopeNone

CSP: the algebra

Before getting into Go’s implementation, let me explain algebra in CSP. I’ve written before about algebraic effects like resumable exceptions in OCaml 5 / Koka that let a function declare what it needs without saying who provides it. But algebra in CSP is a process algebra: a small set of operators like sequence, choice, parallel composition, hiding with equational laws. Because those laws exist, you can prove two CSP process descriptions behave identically. Tools like FDR (Failures-Divergences Refinement) do this mechanically, e.g., you describe your system as CSP processes, describe a specification as another CSP process, and FDR checks whether the implementation actually refines the spec, across every possible interleaving.

Here’s what that looks like for a scatter-gather pattern, an orchestrator firing off requests to several workers and collecting exactly K responses:

-- Specification: orchestrator collects exactly K results then stops
SPEC = scatter -> (collect -> collect -> collect -> STOP)

-- Implementation: N workers communicate via channels
WORKER(i) = request.i -> response.i -> STOP
SYSTEM = (||| i : {0..4} @ WORKER(i))
         [| {| response |} |]
         COLLECTOR(3)
COLLECTOR(0) = STOP
COLLECTOR(k) = response?i -> COLLECTOR(k-1)

-- FDR checks: assert SYSTEM [T= SPEC (trace refinement)
-- This PROVES: no deadlock, no livelock, exactly K responses collected

A handful of operators do almost all the work here:

CSP OperatorMeaningWhat FDR Proves
P ? QExternal choice: the environment decidesDeadlock-freedom: at least one branch is always available
P ? QInternal choice: the process decides nondeterministicallyLiveness: both branches are eventually reachable
P ? QParallel composition, synchronized on shared eventsNo protocol deadlock between P and Q
P ; QSequential composition: Q starts only after P terminatesTermination: P always reaches STOP
P \ AHiding: internal events in set A become invisibleDivergence-freedom: no infinite internal loops

In other words you can model your protocol in CSP and let FDR check every possible interleaving for you. For the scatter-gather pattern specifically, FDR would catch, automatically, before any code runs:

  • A worker that never responds (a deadlock)
  • A collector that waits for more responses than the workers can ever produce (also a deadlock)
  • A timeout path that accidentally creates an infinite retry loop (a livelock)

Go and Rust can’t do any of this because the as soon as you add a buffer (make(chan int, 5)) or an async boundary, you’ve left the strictly synchronous world that FDR reasons about. Go’s race detector can find data races at runtime, after the fact. Rust’s borrow checker prevents a whole class of shared-state bugs at compile time. But, neither one can prove protocol-level, whole-system deadlock-freedom the way FDR can for pure CSP.

Go’s channels

Hoare’s CSP defines communication as synchronous by construction where a send and its matching receive aren’t two separate events that happen to line up in time but they’re the same event in the algebra. Go’s unbuffered channel matches that faithfully: ch <- x and <-ch really do rendezvous. A buffered channel doesn’t, and that one divergence from the original model explains most of the sharp edges Go developers run into. Also, real CSP processes have no persistent identity beyond the algebra describing them, and channels are closer to anonymous synchronization events than to objects you hold a reference to. Go’s channels, by contrast, are first-class values that you create one, pass it into ten different functions, and any of them can close it. Nothing in the language enforces “exactly one owner, exactly one closer” and this is the seed of several of the gotchas below.

func worker(jobs <-chan int, results chan<- int) {
    for j := range jobs {
        results <- j * j
    }
}

func main() {
    jobs := make(chan int, 5)
    results := make(chan int, 5)
    go worker(jobs, results)

    for i := 1; i <= 5; i++ {
        jobs <- i
    }
    close(jobs) // safe: only the sender closes, and no sends follow

    for i := 0; i < 5; i++ {
        fmt.Println(<-results)
    }

    // jobs <- 6 // panics — sending on a closed channel always panics,
                 // whether or not anything is still listening
}

Three more gotchas that Go’s compiler won’t warn you about:

  • Receiving from a closed channel never panics. It returns the zero value and ok == false immediately instead of blocking.
  • A nil channel blocks forever, on both ends, with no panic. Occasionally this is useful on purpose but if it happens to an uninitialized struct field, it results in permanent hang with no error message pointing you at the cause.
  • Goroutines have no structure by default. go func(){}() creates nothing that ties that goroutine’s lifetime to the caller. A goroutine permanently blocked on a channel operation is invisible to the garbage collector and invisible to Go’s deadlock detector. It causes a partial leak where the program running fine, with one goroutine stuck forever in the background.

One place Go actually stayed close to the algebra: select. Hoare’s algebra has external choice (?) as a first-class operator, and select‘s randomized tie-break among multiple ready cases matches CSP alegbra.

select {
case job := <-jobs:
    handle(job)
case <-ctx.Done():
    return ctx.Err() // structured cancellation, Go-style
default:
    // non-blocking probe — CSP has no built-in default arm,
    // but this is the standard way to build one
}

Best practices that have converged around Go’s channels

  • Confine, don’t share. Exactly one goroutine should own a channel’s write side and be the one to close it.
  • Thread context.Context through every long-running goroutine. A select that never watches ctx.Done() is a goroutine leak waiting to happen.
  • Size buffered channels as semaphores for bounding concurrency (worker pools, rate limiting) instead of letting goroutines fan out unbounded.
  • Use errgroup (or equivalent) for propagating the first error and coordinating cancellation across a group of goroutines, instead of hand-rolling error channels.
  • Treat structured concurrency as the governing principle anyway, even without language support: a goroutine’s lifetime should be scoped to, and never outlive, the function or request that spawned it.
  • Instrument the concurrency itself: race detector in CI, plus metrics and tracing on channel operations and goroutine counts in production because concurrency bugs are nondeterministic and hard to catch with a handful of unit tests.

Actors: isolation you get structurally

Actors give you a different, and in some ways weaker, guarantee than CSP but they give it to you structurally. An actor’s state is private, and it processes exactly one message at a time. There is no data race inside one actor, full stop. Instead of “processes synchronizing on named events,” Hewitt’s model says: everything is an actor. Each actor has a private mailbox (unbounded and asynchronous), private state and three things it’s allowed to do on receiving a message: send messages to other actors, create new actors, and decide how to handle its next message. There’s no synchronous handshake requirement anywhere.

Here’s a worker pool in Erlang, matching the crawler pattern from Part II of this series:

-module(worker_pool).
-export([start_pool/1, dispatch/2, worker_loop/1]).

start_pool(N) ->
    [spawn_link(fun() -> worker_loop(0) end) || _ <- lists:seq(1, N)].

worker_loop(Count) ->
    receive
        {work, Job, From} ->
            From ! {result, do_work(Job)},
            worker_loop(Count + 1);
        {status, From} ->
            From ! {count, Count},
            worker_loop(Count)
        % No catch-all clause yet — see the gotcha below
    end.

dispatch(Pid, Job) ->
    Pid ! {work, Job, self()},
    receive
        {result, R} -> R
    after 5000 ->
        {error, timeout}
    end.

Two Erlang-specific gotchas worth knowing before you ship anything like this:

  • Selective receive skips a non-matching message instead of discarding it. receive scans the mailbox in arrival order against your clauses. Anything that matches none of them just sits there, and the next receive call starts scanning from the front all over again. Left unchecked, this is O(n²) behavior over time as junk quietly accumulates. The fix is a catch-all clause:
worker_loop(Count) ->
    receive
        {work, Job, From} -> ...;
        {status, From} -> ...;
        Other ->
            logger:warning("unexpected message: ~p", [Other]),
            worker_loop(Count)  % drop it, don't let it pile up
    end.
  • Mailboxes have no bound by default. ! never blocks in Erlang and there’s no rendezvous. If a producer outpaces a slower worker, the worker’s mailbox just keeps growing until memory runs out. In practice, you either switch to a blocking gen_server:call for anything where backpressure actually matters, or you monitor process_info(Pid, message_queue_len) yourself and shed load manually.

Supervision is the actor model’s answer to fault structure where a supervisor’s children are linked to it, and a crash triggers a restart strategy instead of taking the whole system down with it. But it is structured fault handling, not structured lifetime tracking. A supervisor doesn’t block waiting for its children to finish instead supervision answers “what happens when a child crashes.” Erlang also provides a location transparency, e.g., an Erlang Pid looks identical whether it points to a local process or one on another node ( Pid ! Msg). But that transparency is syntactic, not operational. A remote send can fail with nodedown or badrpc, latency is never zero so you cannot skip handling the failures that only show up once the mailbox is across a network.

Where actors are simpler than channels

A few structural reasons actors tend to feel simpler in practice than channel-based code:

  1. Ownership is enforced by the design itself. There’s no equivalent of “who’s allowed to write to this channel,”, every interaction is a message dropped into a mailbox that only the receiving actor ever drains.
  2. There’s no close semantics to get wrong. Actors don’t have anything like Go’s send-on-closed-panics / double-close-race. An actor’s lifecycle like start, running, terminated is a small, well-understood state machine, and you can monitor/link actor for detecting unexpected crash.
  3. Failure handling is first-class. Supervision trees and let-it-crash give you a systematic answer to “a worker just crashed, now what?” Go’s answer is recover() scattered wherever someone remembered to put it or manual errgroup/context-cancellation wiring to propagate failure to siblings.
  4. Location transparency. With channels, you need to build remoting capability yourself. With actors, it’s often just a deployment decision.

Where actors are not automatically simpler: mailbox-based concurrency can hide backpressure problems, e.g., an actor with an unbounded mailbox can happily accept messages faster than it processes them and quietly balloon memory. Reasoning about message ordering across several independent actors’ mailboxes is also harder than reasoning about a single shared channel’s FIFO order. CSP’s synchronous rendezvous gives you stronger backpressure for free where an unbuffered send blocks until the receiver is ready (some of modern actor runtimes like Akka support mailbox bounding).

Best practices for actor systems

  • Bound mailboxes and monitor mailbox depth as a first-class metric, e.g., an unbounded mailbox is the actor world’s version of an unbuffered-channel leak.
  • Design supervision hierarchies deliberately like one-for-one, one-for-all, rest-for-one.
  • Keep actor state small and serializable if you ever want migration or persistence.
  • Use location transparency deliberately, not accidentally.

CSP/channels fit use cases when you have a fixed, well-understood pipeline topology like stream-processing stages, worker pools with a known fan-out shape. Actors suite when your system’s topology is dynamic like agents spawning agents and where failure isolation matters. I have built PlexSpaces, an actor-based framework, with facets for durability, supervision, and virtual-actor placement based on these lessons. For example, it provides location transparency, failure isolation, and the backpressure/mailbox-bounding. Here is how an actor lifecycle is managed in PlexSpaces:

async/await

Async/await never got a formal algebra or expressiveness proof. It’s syntactic sugar over futures and promises, running on a single-threaded event loop or a thread-pool-backed task scheduler. Within one event loop, there’s no preemption between await points, which quietly eliminates a lot of classic race conditions but it introduces its own flavor of the “who’s tracking this” problem:

async function processOrder(order) {
  sendConfirmationEmail(order); // fire-and-forget — no await!
  return { status: "accepted" };
}

sendConfirmationEmail here returns a promise nobody is holding onto. If it rejects, nothing catches it and in most runtimes that becomes an unhandled-rejection warning nobody reads. If the process exits before it resolves, it just silently never finishes. Structurally, this is the exact same failure as an unstructured Go goroutine, a unit of work whose lifetime nothing owns. Promise.all and asyncio.gather fix this for the cases you remember to wrap explicitly.

This is also where the “function coloring” problem lives, which I covered in the ADTs and algebraic effects post: once a single function is async, every caller up the chain has to become async too. Algebraic effects unrelated to CSP’s process algebra are one proposed fix: separate what a function needs from who provides it.

Linda Memory Model

Linda coordinates through a shared associative memory called a tuple space, with four operations:

  • out(t): write a tuple, don’t block
  • in(t): block until a tuple matches your template, then atomically remove it
  • rd(t): block until a tuple matches, but leave it there for others
  • eval(t): spawn a computation; its eventual result becomes an ordinary tuple once it finishes

Neither side of a Linda interaction needs to know who the other one is. A producer can out() a tuple long before any consumer even exists. This is “generative communication”, data just floats in the shared space until something matching comes looking for it:

// Pseudocode — classic Linda fan-out/fan-in
for i in 0..n:
    eval(("result", i, compute(i)))    // spawn n concurrent computations

count := 0
while count < n:
    in(("result", ?i, ?r))              // blocking, destructive, matched by content
    collect(r)
    count += 1

Notice there’s no worker identity anywhere in that collector loop at all. This is suitable for use cases like master/worker fan-out, blackboard-style coordination, barrier synchronization by counting tuples as they arrive. Linda has two gotchas of its own:

  • Which matching tuple you get is unspecified. If two tuples both match your template, the classic Linda spec never says which one in() hands you.
  • eval() returns nothing. No handle, no future, no promise object of any kind. The only way to know a spawned computation ever finished is to already know the shape of its result tuple and read it.

These issues prevented Linda from going mainstream but its associative memory primitives are natural for coordination related use cases.

Structured concurrency

Kotlin’s coroutineScope, Swift’s TaskGroup, and Java 21’s StructuredTaskScope bind a spawned task’s lifetime to the lexical scope that spawned it. The scope literally cannot exit until every child has finished whether error or not. None of the five communication models above give you that by default:

ModelWhat tracks a spawned unit’s completion
CSP / GoNothing: go func(){}() has no parent link at all
Actors / ErlangSupervision restarts a crashed child, but nothing blocks waiting for a healthy one to finish
async/awaitNothing, an un-awaited promise just runs, or silently fails
LindaNothing, eval() doesn’t even return a handle to check

This is exactly why structured concurrency reads as an add-on layer rather than another communication model. It’s a lifetime discipline you can apply on top of channels, actors, promises, or tuples, e.g., Trio applies it to async/await, Kotlin applies it to coroutines that might be built on channels.

How PlexSpaces answers these gotchas

Most frameworks inherit one of above models’ specific historical rough edges along with its strengths. PlexSpaces is a actor-and-tuplespace framework I’ve been building, and wrote about in more depth here. It deliberately combines actors and Linda rather than picking one, but it doesn’t reproduce either one’s original sharp edges just for the sake of purity. Here’s the mapping from “gotcha described above” to “the specific fix PlexSpaces makes”:

Gotcha, as described abovePlexSpaces’ pragmatic answer
Erlang mailboxes have no bound, so a fast producer can grow one until memory runs outBounded mailboxes. An actor’s inbox has a real, configurable limit, a producer that outpaces its consumer gets backpressure instead of an unbounded memory leak.
Classic Linda leaves the order of matching tuples unspecified, so which one you get is nondeterministic by designFIFO tuple matching. When more than one tuple matches a template, PlexSpaces returns them in the order they were written, not an arbitrary one, removing nondeterminism-by-specification.
Full CSP requires learning a process algebra; full Linda requires learning a four-primitive calculus bolted onto a host language; Erlang requires learning OTP’s supervision idiomsOne small API surface. Actors expose a handful of primitives like send, ask, and the tuple-space operations.
eval() in classic Linda returns no handle, so a spawned computation’s completion is untracked by defaultBecause the “worker” side of a fan-out/fan-in in PlexSpaces is an ordinary supervised actor rather than a bare eval(), its lifetime is owned by a supervisor even though its result is collected the Linda way.
A crash mid-task loses whatever work was in flight, a concern none of CSP, actors, or Linda’s formulations really addressDurability journaling underneath everything. Messages are journaled at the actor-framework level, below application code, so a crash doesn’t silently lose in-flight work, replay picks the actor back up where it left off.

A fan-out/fan-in worker pool shows the combination directly:

// Coordinator spawns N supervised, bounded-mailbox workers.
for i in 0..n {
    spawn_with_facets(
        &ctx, service_locator.clone(),
        "worker", "default",
        Worker::new(i), vec![],
    ).await?;
}

// Coordinator collects results Linda-style — associative, FIFO,
// no ActorRef needed for any individual worker.
let mut collected = 0;
while collected < n {
    let tuple = ctx.tuple_space()
        .in_(template!["result", Wildcard, Wildcard])
        .await?;
    collect(tuple);
    collected += 1;
}

The workers are ordinary supervised actors, restartable, journaled, isolated, with bounded mailboxes so a slow coordinator can’t be flooded. The result collection is Linda-style associative matching, but FIFO instead of unspecified, so results come back in the order the workers actually produced them rather than in an arbitrary one.

Example: scatter-gather with a timeout, across three models

Comparisons are easier to trust when they’re concrete rather than abstract, so I implemented the same pattern, scatter-gather with a timeout across all three approaches. The problem: fan requests out to N services, collect the first K responses within a deadline, then cancel everything else, guaranteed. This is the pattern underneath every hedged-request system, every parallel-search aggregator, and every timeout-bounded fan-out you’ve seen in production.

Go CSP: the naive version leaks goroutines

The code below shows the mistake almost everyone makes on the first pass like spawning goroutines with no cancellation path at all:

func ScatterGatherNaive(services []time.Duration, firstK int) []ServiceResponse {
    ch := make(chan ServiceResponse, len(services))

    for i, latency := range services {
        go func(id int, lat time.Duration) {
            time.Sleep(lat)
            ch <- ServiceResponse{ServiceID: id, Data: fmt.Sprintf("response-%d", id)}
        }(i, latency)
    }

    results := make([]ServiceResponse, 0, firstK)
    for range firstK {
        results = append(results, <-ch)
    }
    // BUG: N-K goroutines still running in background with no cancellation path
    return results
}

The fix combines context.WithTimeout with errgroup to get a real structured lifetime:

func ScatterGatherStructured(services []time.Duration, firstK int, timeout time.Duration) []ServiceResponse {
    ctx, cancel := context.WithTimeout(context.Background(), timeout)
    defer cancel()

    var mu sync.Mutex
    results := make([]ServiceResponse, 0, firstK)

    g, ctx := errgroup.WithContext(ctx)
    for i, latency := range services {
        g.Go(func() error {
            resp, err := simulateService(ctx, i, latency)
            if err != nil { return nil }
            mu.Lock()
            defer mu.Unlock()
            if len(results) < firstK {
                results = append(results, resp)
                if len(results) >= firstK { cancel() }
            }
            return nil
        })
    }
    _ = g.Wait() // All goroutines done — structured lifetime guarantee
    return results
}

errgroup.Wait() guarantees every goroutine finishes before the function returns, and context.WithTimeout propagates cancellation down to the slow workers, so nothing leaks. The catch: you still have to write that plumbing by hand every time.

Go CSP gotchas, side by side with the CSP algebra:

GotchaWhat happensCSP algebra equivalent
Goroutine leakBlocked goroutines run forever, invisible to GC and deadlock detectorN/A
Nil channel recvBlocks forever – no panic, no warningN/A
Nil channel sendAlso blocks forever silentlyN/A
Send on closedRuntime panic – unrecoverable crashN/A
Recv from closedReturns zero value + ok=false N/A
Buffered vs unbufferedBreaks rendezvous = breaks the formal reasoning about synchronizationBuffered channels aren’t part of original CSP
Select non-determinismA random ready case is chosen when several are readyCSP’s external choice (?) is nondeterministic by design

Runnable, with tests: native/gotchas_test.go — 8 tests demonstrating every row above as failing-then-fixed code.

// Gotcha: Goroutine leak — spawned workers block forever on an unread channel
ch := make(chan int)
for i := 0; i < 10; i++ {
    go func(id int) { ch <- id }(i)  // blocks forever — no reader
}
// 10 goroutines leaked: invisible to GC, invisible to deadlock detector

// Gotcha: Nil channel blocks forever (both directions, no panic)
var ch chan int  // nil
<-ch            // blocks forever on recv
ch <- 42        // blocks forever on send

// Gotcha: Send on closed panics, recv from closed returns zero (asymmetric!)
ch := make(chan int, 1); close(ch)
ch <- 1         // PANIC: send on closed channel
v, ok := <-ch   // v=0, ok=false — no panic, just zero value

// Gotcha: Buffered channel breaks rendezvous
buffered := make(chan int, 5)
buffered <- 1   // sender proceeds without receiver — not CSP anymore

Pure Rust: a Nursery, and select! as a guarded command

Rust gives you ownership-enforced isolation without shared state by default but it doesn’t give you structured lifetime by default either. The Nursery type below binds a group of spawned tasks to a scope explicitly:

pub struct Nursery<T: Send + 'static> {
    join_set: JoinSet<T>,
}

impl<T: Send + 'static> Nursery<T> {
    pub fn spawn<F>(&mut self, future: F)
    where F: Future<Output = T> + Send + 'static {
        self.join_set.spawn(future);
    }

    /// Wait for first K tasks OR timeout — then cancel everything else.
    pub async fn wait_first_k_or_timeout(mut self, k: usize, timeout: Duration) -> Vec<T> {
        let mut results = Vec::with_capacity(k);
        let deadline = tokio::time::Instant::now() + timeout;
        loop {
            if results.len() >= k { break; }
            tokio::select! {
                maybe = self.join_set.join_next() => {
                    match maybe {
                        Some(Ok(value)) => results.push(value),
                        Some(Err(_)) => continue,
                        None => break,
                    }
                }
                _ = tokio::time::sleep_until(deadline) => { break; }
            }
        }
        self.join_set.abort_all(); // Structured cleanup — cancel stragglers
        results
    }
}

tokio::select! maps almost directly onto CSP’s guarded command / external choice operator, whichever branch is ready first wins, and the biased; modifier gives you deterministic priority ordering (unlike Go’s deliberately random tie-break):

let results = scatter_gather_csp(&services, 3, Duration::from_millis(300)).await;
// Nursery guarantees: all children cancelled before scope exits

Ownership prevents shared-state bugs entirely, at compile time. JoinSet::abort_all() gives you a real, guaranteed cancellation. The nursery pattern gets you structured lifetime with essentially no runtime overhead. The catch: there’s no nursery built into the standard library and there’s still no formal algebra backing any of it (no FDR-style prover checking).

Rust CSP gotcha, demonstrated in csp_channels/src/scatter_gather.rs:

// Gotcha: Naive tokio::spawn has no parent link — tasks leak
let mut handles = vec![];
for svc in &services {
    handles.push(tokio::spawn(simulate_service(svc)));
}
// If we return early, spawned tasks run forever — no cancellation

// Fix: JoinSet provides structured lifetime
let mut join_set = JoinSet::new();
for svc in &services { join_set.spawn(simulate_service(svc)); }
// On drop or abort_all(), all tasks are cancelled — guaranteed

PlexSpaces: supervised lifetime plus decoupled collection

This is where actor supervision (structured fault handling) and Linda-style tuple-space coordination (decoupled result collection) get combined. Start with thin Linda-style wrappers over the tuple-space host functions:

// Linda-style thin wrappers over tuplespace host functions
fn linda_out(fields: &[Value]) -> Result<(), String> {
    let request = WriteRequest { tuples: vec![json_to_tuple(fields)?], .. };
    ts_write(&request.encode_to_vec()).map(|_| ())
}

fn linda_in(pattern: &[Value]) -> Result<Option<Vec<Value>>, String> {
    let request = ReadRequest { template: Some(to_pattern(pattern)?), take: true, .. };
    let bytes = ts_take(&request.encode_to_vec())?;
    Ok(decode_response(&bytes)?.first().map(to_json_array))
}

The orchestrator scatters by spawning supervised workers, then sets its own timeout with a self-message:

// Scatter: spawn N workers under supervisor
for i in 0..num_services {
    spawn("actor-csp-wasm", &format!("worker-{i}"), "", &init_json)?;
    send(&worker_id, "cast", &work_payload)?;
}

// Set timeout — send_after fires a collection message to self
send_after(timeout_ms, "cast", &collect_msg)?;

Each worker writes its result to the shared tuple space, with zero knowledge of who’s collecting it:

// Worker: Linda OUT — write result tuple to shared tuplespace
linda_out(&[
    Value::String("result".into()),
    Value::String(request_id.into()),
    Value::Number(service_id.into()),
    Value::String(result_data),
])?;

And gathering reads back whatever arrived in time, then explicitly tells the supervisor to stop the rest:

// Gather: Linda RD-ALL — collect whatever arrived before timeout
let results = linda_rd_all(&["result", request_id, *, *])?;
// Structured cleanup: stop remaining workers via supervisor
for wid in &worker_ids { stop(wid)?; }

Workers never need to know who’s collecting their results, that’s the Linda decoupling doing its job. The supervisor guarantees the worker lifecycle end to end, e.g., a crashed worker restarts automatically under a OneForOne strategy. Bounded mailboxes keep a slow coordinator from getting flooded. FIFO tuple matching makes the collection step deterministic instead of an open question.

The three approaches, side by side

PropertyGo CSP (errgroup)Rust (Nursery/select!)PlexSpaces (actors + Linda)
Cancellationcontext.Cancel() propagatedJoinSet::abort_all()stop() via supervisor
Structured lifetimeg.Wait() blocksNursery scope exitSupervisor manages lifecycle
BackpressureBuffered channel capacityChannel capacityBounded mailbox
Failure handlingerrgroup collects first errorJoinError on abortSupervisor restarts crashed worker
CouplingWorkers know the result channelWorkers know the result typeWorkers only know the tuple shape (Linda)
Formal backingNoneNoneNone (but FIFO + bounded mailboxes removes two classes of nondeterminism)
DistributionSingle process onlySingle process onlyMulti-node, transparently

Full runnable examples with tests live at: examples/rust/embedded/csp_channels, examples/go/apps/csp_structured/native, and examples/rust/apps/actor_csp.

One more actor-model gotcha, demonstrated via supervisor behavior

// Gotcha: Unbounded mailbox — fast producer OOMs the consumer
// Fix: PlexSpaces uses bounded mailboxes with a configurable limit

// Gotcha: No structured lifetime — actors are async, no scope to wait on
// Fix: Supervisor + explicit stop() for child actors after collection

// Gotcha: Orphaned actors — a spawned actor runs forever if nobody stops it
// Fix: OneForOne supervisor manages worker lifecycle; orchestrator calls stop()

The tldr;

  • CSP has real algebra and real tooling (FDR) behind it. Go borrows the vocabulary but drops the proof the when you add a buffer.
  • Actors have partial formal treatment and isolation by construction, but unbounded mailboxes and selective-receive skip are real, sharp edges the model doesn’t protect you from on its own.
  • async/await never had formal backing at all, and its fire-and-forget promise is the exact same “who’s tracking this” bug as an unstructured goroutine.
  • Linda is the most decoupled model on this list and the least adopted. The original spec leaves match order unspecified and spawned work untracked.
  • Structured concurrency isn’t a another concurrency model, instead it’s a lifetime discipline layered.
  • PlexSpaces’ bet is that you don’t have to inherit every historical rough edge along with the good ideas like bounded mailboxes, FIFO matching, and one small unified API let it combine actor supervision with Linda’s decoupled coordination.

The rest of the series

  1. Part I — the general problem, concurrency constructs, and TypeScript
  2. Part II — Erlang and Elixir
  3. Part III — Go and Rust
  4. Part IV — Kotlin and Swift
  5. Building a Durable Actor Framework for Polyglot Serverless Apps
  6. 20+ Production Patterns for Distributed AI Agents Using Actors and TupleSpaces
  7. Building an Agent Harness and Eval Pipeline with Durable Actors
  8. Building a Self-Improving AI Agent with Durable Actors: MiniHermes
  9. Building Mini OpenClaw: Secure AI Agents with Actors, WASM, and Supervision
  10. Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects
  11. Building PlexSpaces: Decades of Distributed Systems Distilled Into One Framework

Code for everything above: github.com/bhatti/PlexSpaces

Powered by WordPress