Shahzad Bhatti Welcome to my ramblings and rants!

Over the past year I’ve built production agentic systems across several domains and shared what I learned along the way: production-grade AI agents with MCP and A2A, a daily minutes assistant with RAG, MCP, and ReAct, rebuilding fintech infrastructure with ReAct and local models, automated PII detection with LangChain and Vertex AI, and API compatibility guardians with LangGraph. I have learned a lot building those systems through trial and error. In this blog, I will share a set of generative and agentic AI patterns I have learned from reading Generative AI Design Patterns and Agentic Design Patterns. I have built hands on python examples from these patterns and built github.com/bhatti/agentic-patterns for running agentic apps locally via Ollama with open-source models (Qwen, DeepSeek, Llama, Mistral). Each pattern in the repo includes a README, working code, real-world use cases, and best practices.

Quick Start

git clone https://github.com/bhatti/agentic-patterns
cd agentic-patterns
pip install -r requirements.txt
ollama pull llama3
cd patterns/logits-masking && python example.py

See SETUP.md for full setup instructions.

Table of Contents

• Category 1: Content & Style Control (Patterns 1–5)
• Category 2: Adding Knowledge / RAG Stack (Patterns 6–12)
• Category 3: LLM Reasoning (Patterns 13–16)
• Category 4: Reliability & Evaluation (Patterns 17–20)
• Category 5: Tools, Agents & Efficiency (Patterns 21–32)
• Category 6: Agentic Behavior Patterns (Patterns 33–50)
• Pattern Comparison Matrix
• Choosing the Right Pattern
• What This Catalog Teaches

Category 1: Content & Style Control

The first five patterns control and optimize content generation, style, and format:

Pattern 1: Logits Masking

Category: Content & Style Control
Use When: You need to enforce constraints during generation (e.g., valid JSON, banned words)

Problem

When generating structured outputs (like JSON, code, or formatted text), language models can produce invalid sequences that don’t conform to required style rules, schemas, or constraints.

Solution

Logits Masking intercepts the model’s token generation process to enforce constraints during sampling. Three key steps:

1. Intercept Sampling — Modify logits before token selection
2. Zero Out Invalid Sequences — Mask invalid tokens (set logits to -inf)
3. Backtracking — Revert to checkpoint if invalid sequence detected

Use Cases

• API response generation (ensure valid JSON)
• Code generation (enforce style guidelines)
• Content moderation (prevent banned words)
• Structured data extraction (match specific formats)

Constraints: Requires access to model logits (not available in all APIs). State tracking can be complex for nested structures. Performance overhead from logits processing.

Tradeoffs:

• ? Prevents invalid generation at source
• ? More efficient than post-processing
• ?? More complex than simple validation
• ?? May limit model creativity

Code Snippet

class JSONLogitsProcessor(LogitsProcessor):
    """Intercept logits and mask invalid JSON tokens."""

    def __call__(self, input_ids, scores):
        # STEP 1: Intercept sampling
        current_text = self.tokenizer.decode(input_ids[0])

        # STEP 2: Zero out invalid sequences
        for token_id in range(scores.shape[-1]):
            if not self._is_valid_json_token(token_id, current_text):
                scores[0, token_id] = float('-inf')  # Mask invalid

        return scores

Full Example: patterns/logits-masking/example.py

Pattern 2: Grammar Constrained Generation

Category: Content & Style Control
Use When: You need outputs that conform to formal grammar specifications

Problem

Language models often produce text that doesn’t conform to required formats, schemas, or grammars. Unlike simple masking, grammar-constrained generation ensures outputs follow formal grammar specifications.

Solution

Grammar Constrained Generation uses formal grammar specifications to guide token generation. Three implementation approaches:

1. Grammar-Constrained Logits Processor — Use EBNF grammar to create processor
2. Standard Data Format — Leverage JSON/XML with existing validators
3. User-Defined Schema — Use custom schemas (JSON Schema, Pydantic)

Use Cases

• API configuration generation (OpenAPI specs)
• Configuration files (YAML, TOML that must parse)
• Database queries (SQL with guaranteed syntax)
• Code generation (must compile/parse)

Constraints: Requires grammar definition or schema. Grammar parsing can be computationally expensive. Complex grammars may limit generation speed.

Tradeoffs:

• ? Guarantees grammatical correctness
• ? Works with existing schema languages
• ?? More complex than simple masking
• ?? May require grammar expertise

Code Snippet

# Option 1: Formal Grammar
grammar = """
root        ::= endpoint_config
endpoint_config ::= "{" ws endpoint_def ws "}"
endpoint_def    ::= '"endpoint"' ws ":" ws endpoint_obj
"""

# Option 2: JSON Schema
schema = {
    "type": "object",
    "required": ["endpoint"],
    "properties": {
        "endpoint": {
            "type": "object",
            "required": ["name", "method", "path"]
        }
    }
}

# Apply grammar constraints during generation
processor = GrammarConstrainedProcessor(grammar, tokenizer)
logits = processor(input_ids, logits)

Full Example: patterns/grammar/example.py

Pattern 3: Style Transfer

Category: Content & Style Control
Use When: You need to transform content from one style to another

Problem

Content often needs to be transformed from one style to another while preserving core information. Manual rewriting is time-consuming and inconsistent.

Solution

Style Transfer uses AI to transform content between styles. Two approaches:

1. Few-Shot Learning — Use example pairs in prompt (no training)
2. Model Fine-Tuning — Fine-tune model on style pairs

Use Cases

• Professional communication (notes to emails)
• Content adaptation (academic to blog posts)
• Brand voice (maintain consistent tone)
• Platform adaptation (different social media styles)

Constraints: Few-shot limited by context window. Fine-tuning requires training data. Style consistency can vary.

Tradeoffs:

• ? Few-shot: Quick, no training needed
• ? Fine-tuning: Better consistency
• ?? Few-shot: May not capture nuances
• ?? Fine-tuning: Requires data collection

Code Snippet

# Option 1: Few-Shot Learning
examples = [
    StyleExample(
        input_text="urgent: need meeting minutes by friday",
        output_text="Subject: Urgent: Meeting Minutes Needed\n\nDear [Recipient],\n\n..."
    )
]

transfer = FewShotStyleTransfer(examples)
result = transfer.transfer_style("quick update: deadline moved")

# Option 2: Fine-Tuning
training_data = [
    {"prompt": "Convert notes to email", "completion": "Professional email..."}
]
fine_tuned_model = fine_tune_model(base_model, training_data)

Full Example: patterns/style-transfer/example.py

Pattern 4: Reverse Neutralization

Category: Content & Style Control
Use When: You need to generate content in a specific personal style that zero-shot can’t capture

Problem

When you need content in a specific, personalized style, zero-shot prompting fails because the model doesn’t know your unique writing style.

Solution

Reverse Neutralization uses a two-stage fine-tuning approach:

1. Generate Neutral Form — Create content in neutral, standardized format
2. Fine-Tune Style Converter — Train model to convert neutral ? your style
3. Inference — Use fine-tuned model for style conversion

Use Cases

• Personal blog writing (technical content to your style)
• Brand voice (consistent voice across content)
• Documentation style (match organization’s style guide)
• Communication templates (your personal email style)

Constraints: Requires fine-tuning. Needs training data (neutral ? style pairs). Two-stage process.

Tradeoffs:

• ? Learns your specific style
• ? Consistent results
• ? Captures personal nuances
• ?? Requires data collection and training
• ?? Less flexible (need retraining to change style)

Code Snippet

# Step 1: Generate neutral form
neutral_generator = NeutralGenerator()
neutral = neutral_generator.generate_neutral("API Authentication")

# Step 2-3: Create training dataset and fine-tune
pairs = [
    StylePair(neutral="Technical doc...", styled="Your blog style...")
]
fine_tuned_model = fine_tune_on_preferences(pairs)

# Step 4: Use fine-tuned model
converter = StyleConverter(fine_tuned_model)
styled = converter.convert_to_style(neutral)

Full Example: patterns/reverse-neutralization/example.py

Pattern 5: Content Optimization

Category: Content & Style Control
Use When: You need to optimize content for specific performance goals (e.g., open rates, conversions)

Problem

When creating content for specific purposes, you need to optimize for outcomes. Traditional A/B testing is limited — it’s manual, time-consuming, and doesn’t learn patterns.

Solution

Content Optimization uses preference-based fine-tuning (DPO) to train a model to generate content that wins in comparisons:

1. Generate Pair — Create two variations from same prompt
2. Compare — Test and pick winner based on metrics
3. Create Dataset — Collect preference pairs (prompt, chosen, rejected)
4. Fine-Tune with DPO — Train model on preferences
5. Use Optimized Model — Generate better-performing content

Use Cases

• Email marketing (optimize subject lines for open rates)
• E-commerce (optimize product descriptions for conversions)
• Social media (optimize posts for engagement)
• Landing pages (optimize copy for sign-ups)

Constraints: Requires preference data collection. DPO fine-tuning is computationally intensive. Need clear optimization metrics.

Tradeoffs:

• ? Learns from all comparisons
• ? Scales to many variations
• ? Model internalizes winning patterns
• ?? Requires training data (100+ pairs)
• ?? More complex than A/B testing

Code Snippet

# Step 1: Generate pair
generator = ContentGenerator()
var_a, var_b = generator.generate_pair("New product launch")

# Step 2: Compare and pick winner
comparator = ContentComparator(optimization_goal="open_rate")
pair = comparator.compare(ContentPair(prompt, var_a, var_b))

# Step 3-4: Create dataset and fine-tune
preferences = [PreferenceExample(prompt, chosen, rejected)]
dpo_trainer = PreferenceTuner()
optimized_model = dpo_trainer.fine_tune(preferences)

# Step 5: Use optimized model
optimized_generator = OptimizedContentGenerator(optimized_model)
result = optimized_generator.generate_optimized("Newsletter")

Full Example: patterns/content-optimization/example.py

Category 2: Adding Knowledge / RAG Stack

Patterns 6–12 augment LLMs with external knowledge sources for accessing up-to-date information, private data, and knowledge beyond the model’s training cutoff.

Pattern 6: Basic RAG (Retrieval-Augmented Generation)

Category: Adding Knowledge
Use When: You need to augment LLM responses with external knowledge sources

Problem

LLMs have three key knowledge limitations:

• Static Knowledge Cutoff — Trained on data up to a specific date
• Model Capacity Limits — Can’t store all knowledge in parameters
• Lack of Private Data Access — No access to internal documents or databases

Solution

Basic RAG uses trusted knowledge sources when generating LLM responses. Two pipelines:

Indexing Pipeline (preparatory):

• Load documents ? Chunk into manageable pieces ? Store in searchable index

Retrieval-Generation Pipeline (runtime):

• Retrieve relevant chunks for query ? Ground prompt with retrieved context ? Generate response using LLM

Use Cases

• Product documentation (answer questions about features/APIs)
• Company knowledge base (query internal wikis/policies)
• Customer support (accurate answers from support docs)
• Research assistance (search through papers/documents)
• Legal/compliance (query regulations/guides)

Tradeoffs:

• ? Access to up-to-date and private knowledge
• ? Can handle large knowledge bases
• ? Transparent (can cite sources)
• ?? Requires indexing infrastructure
• ?? Retrieval quality affects response quality

Code Snippet

# INDEXING PIPELINE
loader = DocumentLoader()
documents = loader.load_documents("product_docs")

splitter = TextSplitter(chunk_size=500, chunk_overlap=50)
chunks = []
for doc in documents:
    chunks.extend(splitter.split_document(doc))

index = Index()
index.add_chunks(chunks)

# RETRIEVAL-GENERATION PIPELINE
retriever = Retriever(index, top_k=3)
generator = RAGGenerator(retriever)

result = generator.generate("How do I authenticate with the API?")
# Returns answer with source citations

Full Example: patterns/basic-rag/example.py

Pattern 7: Semantic Indexing

Category: Adding Knowledge
Use When: You need semantic understanding beyond keywords, or have complex content (images, tables, code)

Problem

Traditional keyword-based indexing has limitations:

• Semantic Understanding — Misses meaning (“car” and “automobile” are different keywords)
• Complex Content — Struggles with images, tables, code blocks, structured data
• Context Loss — Fixed-size chunking breaks up related content
• Multimedia — Can’t effectively index images, videos, or other media

Solution

Semantic Indexing uses embeddings (vector representations) to capture meaning:

1. Embeddings — Encode text/images into fixed vector representations for semantic meaning
2. Semantic Chunking — Divide text into meaningful segments based on semantic content
3. Image/Video Handling — Use OCR or vision models for embedding generation
4. Table Handling — Organize and extract key information from structured data
5. Contextual Retrieval — Preserve context with hierarchical chunking
6. Hierarchical Chunking — Multi-level chunking (document ? section ? paragraph)

Use Cases

• Technical documentation (code examples, API docs, tutorials)
• Research papers (find by concept, not keywords)
• Product catalogs (search by features, not names)
• Multimedia content (images, videos with descriptions)

Code Snippet

# CONCEPT 1: EMBEDDINGS
from sentence_transformers import SentenceTransformer
import math

class EmbeddingGenerator:
    def __init__(self, model_name: str = "all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)

    def generate_embedding(self, text: str) -> List[float]:
        return self.model.encode(text).tolist()

    def cosine_similarity(self, vec1: List[float], vec2: List[float]) -> float:
        dot_product = sum(a * b for a, b in zip(vec1, vec2))
        magnitude1 = math.sqrt(sum(a * a for a in vec1))
        magnitude2 = math.sqrt(sum(a * a for a in vec2))
        return dot_product / (magnitude1 * magnitude2) if magnitude1 * magnitude2 > 0 else 0.0

# CONCEPT 2: SEMANTIC CHUNKING
@dataclass
class SemanticChunk:
    id: str
    text: str
    embedding: Optional[List[float]] = None
    chunk_type: str = "text"  # text, code, table, image
    parent_id: Optional[str] = None
    children_ids: List[str] = None

class SemanticChunker:
    def chunk_by_structure(self, content: str) -> List[SemanticChunk]:
        """Chunk respecting document structure (headers, sections, paragraphs)."""
        chunks = []
        sections = re.split(r'\n(#{2,3}\s+.+?)\n', content)
        current_section = None
        chunk_index = 0
        for i, part in enumerate(sections):
            if part.strip().startswith('#'):
                if current_section:
                    chunks.append(SemanticChunk(id=f"chunk-{chunk_index}", text=current_section))
                    chunk_index += 1
                current_section = part + "\n"
            else:
                current_section = (current_section or "") + part
        if current_section:
            chunks.append(SemanticChunk(id=f"chunk-{chunk_index}", text=current_section))
        return chunks

# CONCEPTS 5 & 6: HIERARCHICAL CHUNKING & CONTEXTUAL RETRIEVAL
class ContextualRetriever:
    def retrieve_with_context(self, query: str, top_k: int = 3,
                              include_context: bool = True) -> List[SemanticChunk]:
        query_embedding = self.embedding_generator.generate_embedding(query)
        scored_chunks = []
        for chunk in self.chunks.values():
            if chunk.embedding:
                similarity = self.embedding_generator.cosine_similarity(
                    query_embedding, chunk.embedding
                )
                scored_chunks.append((similarity, chunk))
        scored_chunks.sort(key=lambda x: x[0], reverse=True)
        top_chunks = [chunk for _, chunk in scored_chunks[:top_k]]
        if include_context:
            contextual_chunks = []
            for chunk in top_chunks:
                contextual_chunks.append(chunk)
                # Add parent for context
                if chunk.parent_id and chunk.parent_id in self.chunks:
                    parent = self.chunks[chunk.parent_id]
                    if parent not in contextual_chunks:
                        contextual_chunks.append(parent)
                # Add children for detail
                for child_id in (chunk.children_ids or []):
                    if child_id in self.chunks:
                        child = self.chunks[child_id]
                        if child not in contextual_chunks:
                            contextual_chunks.append(child)
            return contextual_chunks
        return top_chunks

Full Example: patterns/semantic-indexing/example.py

Pattern 8: Indexing at Scale

Category: Adding Knowledge
Use When: Your RAG system needs to handle large-scale knowledge bases with evolving, time-sensitive information

Problem

RAG systems in production face critical challenges as knowledge bases grow:

• Data Freshness — Recent findings obsolete old guidelines
• Contradictory Content — Multiple versions of information cause confusion
• Outdated Content — Old information remains in index, leading to incorrect answers

Solution

Indexing at Scale uses metadata and temporal awareness:

1. Document Metadata — Use timestamps, version numbers, source information
2. Temporal Tagging — Tag chunks with creation/update dates, expiration dates
3. Contradiction Detection — Identify and prioritize newer over older contradictory content
4. Outdated Content Management — Automatically deprecate or flag outdated information

Code Snippet

@dataclass
class TemporalMetadata:
    created_at: datetime
    updated_at: datetime
    expires_at: Optional[datetime] = None
    version: str = "1.0"
    source: str = ""
    authority: str = "medium"  # high, medium, low

class ContradictionDetector:
    def _resolve_contradiction(self, chunk_a, chunk_b):
        # Prefer newer date
        if chunk_a.metadata.updated_at > chunk_b.metadata.updated_at:
            return "chunk_a"
        # If same date, prefer higher authority
        if chunk_a.metadata.authority > chunk_b.metadata.authority:
            return "chunk_a"
        return "chunk_b"

# KNOWLEDGE BASE WITH TEMPORAL AWARENESS
kb = HealthcareGuidelinesKB()
kb.add_guideline(
    content="CDC recommends masks required in public",
    source="CDC",
    date=datetime(2021, 7, 15),
    authority="high"
)

result = kb.query("Should I wear a mask?", prefer_recent=True)
# Returns most recent guidelines, flags contradictions

Full Example: patterns/indexing-at-scale/example.py

Pattern 9: Index-Aware Retrieval

Category: Adding Knowledge
Use When: Basic RAG fails due to vocabulary mismatches, fine details, or holistic answers requiring multiple concepts

Problem

Users ask questions in natural language (“How do I log in?”), but your API documentation uses technical terminology (“OAuth 2.0 authentication”, “access token”). Basic RAG fails because “log in” ? “authentication” ? “OAuth 2.0”.

Solution

Index-Aware Retrieval uses four advanced retrieval techniques:

1. Hypothetical Document Embedding (HyDE) — Generate hypothetical answer first, then match chunks to that answer
2. Query Expansion — Translate user terms to technical terms used in chunks
3. Hybrid Search — Combine keyword (BM25) and semantic (embedding) search with weighted average
4. GraphRAG — Store documents in graph database, retrieve related chunks after finding initial match

Code Snippet

# TECHNIQUE 1: HYPOTHETICAL DOCUMENT EMBEDDING (HyDE)
class HyDEGenerator:
    def retrieve_with_hyde(self, query: str, chunks: List[DocumentChunk], top_k: int = 3):
        # Step 1: Generate hypothetical answer
        hypothetical_answer = self.generate_hypothetical_answer(query)
        # "To authenticate, use OAuth 2.0 access token..."

        # Step 2: Embed hypothetical answer (not original query)
        hyde_embedding = embedding_generator.generate_embedding(hypothetical_answer)

        # Step 3: Find chunks similar to hypothetical answer
        scored_chunks = []
        for chunk in chunks:
            similarity = cosine_similarity(hyde_embedding, chunk.embedding)
            scored_chunks.append((chunk, similarity))

        return sorted(scored_chunks, key=lambda x: x[1], reverse=True)[:top_k]

# TECHNIQUE 2: QUERY EXPANSION
class QueryExpander:
    def expand_query(self, query: str) -> str:
        term_translations = {
            "log in": ["authentication", "oauth", "access token"],
            "error": ["error code", "status code", "exception"]
        }
        expanded_terms = [query]
        for user_term, tech_terms in term_translations.items():
            if user_term in query.lower():
                expanded_terms.extend(tech_terms)
        return " ".join(expanded_terms)

# TECHNIQUE 3: HYBRID SEARCH (BM25 + Semantic)
class HybridRetriever:
    def retrieve(self, query: str, top_k: int = 5):
        bm25_score = bm25_scorer.score(query, chunk)
        semantic_score = cosine_similarity(query_embedding, chunk.embedding)
        # ? = 0.4 means 40% BM25, 60% semantic
        hybrid_score = 0.4 * bm25_score + 0.6 * semantic_score
        return sorted_chunks_by_score[:top_k]

# TECHNIQUE 4: GRAPHRAG
class GraphRAG:
    def retrieve_related(self, initial_chunk_id: str, depth: int = 1):
        related_ids = graph[initial_chunk_id]
        for _ in range(depth - 1):
            next_level = [graph[rid] for rid in related_ids]
            related_ids.extend(next_level)
        return [chunks[cid] for cid in related_ids]

Full Example: patterns/index-aware-retrieval/example.py

Pattern 10: Node Postprocessing

Category: Adding Knowledge
Use When: Retrieved chunks have issues like ambiguous entities, conflicting content, obsolete information, or are too verbose

Problem

Your RAG system retrieves legal document chunks with issues: ambiguous entities (“Apple” could be company or fruit), conflicting interpretations of the same law, obsolete regulations superseded by new ones, and verbose chunks with only small relevant sections.

Solution

Node Postprocessing improves retrieved chunks through a pipeline:

1. Reranking — Use more accurate models (like BGE) to rerank chunks
2. Hybrid Search — Combine BM25 and semantic retrieval
3. Query Expansion and Decomposition — Expand queries and break into sub-queries
4. Filtering — Remove obsolete, conflicting, or irrelevant chunks
5. Contextual Compression — Extract only relevant parts from verbose chunks
6. Disambiguation — Resolve ambiguous entities and clarify context

Code Snippet

# TECHNIQUE 1: RERANKING (BGE-style Cross-Encoder)
# In production: from sentence_transformers import CrossEncoder
# model = CrossEncoder('BAAI/bge-reranker-base')

# TECHNIQUE 5: CONTEXTUAL COMPRESSION
class ContextualCompressor:
    def compress(self, chunk: DocumentChunk, query: str, max_length: int = 200):
        query_words = set(query.lower().split())
        sentences = chunk.content.split('.')
        relevant_sentences = [
            s for s in sentences
            if query_words & set(s.lower().split())
        ]
        compressed_content = '. '.join(relevant_sentences[:3]) + '.'
        return DocumentChunk(id=chunk.id + "_compressed", content=compressed_content[:max_length])

# TECHNIQUE 6: DISAMBIGUATION
class Disambiguator:
    def disambiguate(self, chunks: List[DocumentChunk], query: str):
        entity_contexts = {
            "apple": {
                "company": ["technology", "iphone", "corporate"],
                "fruit": ["nutrition", "eating", "food"]
            }
        }
        query_words = set(query.lower().split())
        for chunk in chunks:
            for entity, contexts in entity_contexts.items():
                if entity in chunk.content.lower():
                    entity_type = determine_from_context(entity, query_words, chunk.content)
                    if entity_type:
                        chunk.entities.append(f"{entity}:{entity_type}")
        return chunks

# COMPLETE POSTPROCESSING PIPELINE
def query_with_postprocessing(question: str):
    expanded = query_processor.expand_query(question)
    candidates = hybrid_retriever.retrieve(expanded, top_k=10)
    filtered = filter.filter_obsolete([c for c, _ in candidates])
    filtered = filter.filter_by_relevance(candidates, threshold=0.3)
    reranked = reranker.rerank(question, filtered, top_k=5)
    disambiguated = disambiguator.disambiguate([c for c, _ in reranked], question)
    compressed = [compressor.compress(c, question) for c in disambiguated]
    return compressed

Full Example: patterns/node-postprocessing/example.py

Pattern 11: Trustworthy Generation

Category: Adding Knowledge
Use When: RAG systems need to build user trust by preventing hallucination, providing citations, and detecting out-of-domain queries

Problem

Users lose trust because the system answers questions outside its knowledge domain, answers lack citations, and it provides confident answers when retrieval actually failed.

Solution

Trustworthy Generation builds user trust through multiple mechanisms:

1. Out-of-Domain Detection — Detect when knowledge base doesn’t contain relevant information
2. Embedding Distance Checking — Measure similarity between query and retrieved chunks
3. Citations — Provide source citations for all factual claims
4. Self-RAG Workflow — 6-step self-reflective process to verify responses
5. Guardrails — Prevent generation of unsafe or unreliable content

Code Snippet

# OUT-OF-DOMAIN DETECTION
class OutOfDomainDetector:
    def is_out_of_domain(self, query: str, chunks: List[DocumentChunk]) -> Tuple[bool, str]:
        if chunks:
            query_embedding = embedding_generator.generate_embedding(query)
            min_distance = min([
                1 - cosine_similarity(query_embedding, chunk.embedding)
                for chunk in chunks
            ])
            if min_distance > threshold:
                return True, "Query too far from knowledge base"
        if not has_domain_keywords(query):
            return True, "Query lacks domain-specific terminology"
        if not chunks:
            return True, "No relevant chunks found"
        return False, ""

# SELF-RAG WORKFLOW (6 Steps)
class SelfRAGProcessor:
    def process(self, query: str, retrieved_chunks: List[DocumentChunk]):
        # STEP 1: Generate initial response
        initial_response = generate_initial_response(query, retrieved_chunks)
        # STEP 2: Chunk the response
        response_chunks = chunk_response(initial_response)
        # STEP 3: Check whether chunk needs citation
        for chunk in response_chunks:
            chunk.needs_citation = needs_citation(chunk.text)
        # STEP 4: Lookup sources
        for chunk in response_chunks:
            if chunk.needs_citation:
                chunk.sources = lookup_sources(chunk.text, retrieved_chunks)
        # STEP 5: Incorporate citations
        final_response = incorporate_citations(response_chunks)
        # STEP 6: Add warnings
        warnings = generate_warnings(response_chunks)
        return {"response": final_response, "warnings": warnings}

# COMPLETE TRUSTWORTHY GENERATION PIPELINE
def query_with_trustworthiness(question: str):
    is_ood, reason = out_of_domain_detector.is_out_of_domain(question, chunks)
    if is_ood:
        return {"response": f"Cannot answer: {reason}", "out_of_domain": True}
    result = self_rag.process(question, retrieved_chunks)
    passed, reason = guardrails.check(question, result, retrieved_chunks)
    if not passed:
        result["response"] = f"Cannot provide reliable answer: {reason}"
    return result

Full Example: patterns/trustworthy-generation/example.py

Pattern 12: Deep Search

Category: Adding Knowledge
Use When: Complex information needs require iterative retrieval, multi-hop reasoning, or comprehensive research across multiple sources

Problem

Investment analysts need comprehensive research on companies/industries. Basic RAG retrieves a few chunks and provides incomplete answers. They need a system that iteratively explores multiple sources, identifies gaps, and follows up on missing information.

Solution

Deep Search uses an iterative loop that retrieves and thinks until a good enough answer is found or a time/cost budget is exhausted:

Code Snippet

class DeepSearchOrchestrator:
    def __init__(self, budget: Budget):
        self.retriever = MultiSourceRetriever()  # Web, APIs, knowledge bases
        self.reasoner = LLMReasoner()
        self.budget = budget  # Time/cost constraints

    def search(self, query: str, depth: int = 2) -> DeepSearchResult:
        root_section = self._create_section(query)
        sections = [root_section]
        sections_to_expand = [root_section]
        current_depth = 0

        while current_depth < depth:
            current_depth += 1
            exhausted, reason = self.budget.is_exhausted()
            if exhausted:
                break
            next_sections = []
            for section in sections_to_expand:
                gaps = self.reasoner.identify_gaps(query, section.answer, section.sources)
                follow_ups = self.reasoner.generate_follow_ups(query, gaps)
                for follow_up in follow_ups:
                    subsection = self._create_section(follow_up)
                    section.subsections.append(subsection)
                    sections.append(subsection)
                    next_sections.append(subsection)
            sections_to_expand = next_sections
            is_good_enough, quality = self.reasoner.assess_answer_quality(
                query, root_section.answer, sections
            )
            if is_good_enough:
                break

        final_answer = self.reasoner.final_synthesis(query, sections)
        return DeepSearchResult(query, final_answer, sections, self.all_sources)

@dataclass
class Budget:
    max_iterations: int = 5
    max_time_seconds: float = 60.0
    max_cost_dollars: float = 1.0

    def is_exhausted(self) -> Tuple[bool, str]:
        if self.iterations_used >= self.max_iterations:
            return True, "max_iterations"
        if self.time_used >= self.max_time_seconds:
            return True, "max_time"
        if self.cost_used >= self.max_cost_dollars:
            return True, "max_cost"
        return False, ""

# USAGE
analyst = MarketResearchAnalyst()
result = analyst.research(
    query="What factors should I consider when evaluating TechCorp as an investment?",
    max_iterations=10,
    max_time_seconds=30.0
)

Full Example: patterns/deep-search/example.py

Category 3: LLM Reasoning

Patterns 13–16 address reasoning and task specialization: how to get step-by-step or multi-path reasoning from LLMs.

Pattern 13: Chain of Thought (CoT)

Category: LLM Reasoning
Use When: Problems require multistep reasoning, logical deduction, or an auditable reasoning trace

Problem

Foundational models suffer from critical limitations on math, logical deduction, and sequential reasoning:

• Zero-shot often fails when the problem requires multistep reasoning
• Black-box answers with no insight into how the conclusion was reached
• Misinterpretation of rules

Solution

Chain of Thought (CoT) prompts request a step-by-step reasoning process before the final answer. Three variants:

1. Zero-shot CoT — Append “Think step by step” (no examples)
2. Few-shot CoT — Provide example (question ? step-by-step reasoning ? answer). RAG gives fish; few-shot CoT shows how to fish.
3. Auto CoT — Sample questions ? generate reasoning for each with zero-shot CoT ? use as few-shot examples for the actual query

Code Snippet

# VARIANT 1: ZERO-SHOT COT
ZERO_SHOT_COT_SUFFIX = "\n\nThink step by step. Show your reasoning and then state the final conclusion."

def zero_shot_cot(policy: str, case_description: str, question: str, llm=None) -> CoTResult:
    prompt = f"{policy}\n\nCase: {case_description}\n\nQuestion: {question}{ZERO_SHOT_COT_SUFFIX}"
    full_response = llm(prompt)
    return CoTResult(question=question, reasoning=..., conclusion=..., variant="zero_shot")

# VARIANT 2: FEW-SHOT COT — "show how to fish"
FEW_SHOT_EXAMPLES = """
Example 1:
Q: Customer purchased 10 days ago, unopened, has receipt. Eligible for full refund?
A: Step 1: Within 30 days? Yes. Step 2: Unopened? Yes. Step 3: Receipt? Yes.
   Conclusion: Yes, full refund.
"""
def few_shot_cot(policy: str, case_description: str, question: str, llm=None) -> CoTResult:
    prompt = f"{policy}\n\n{FEW_SHOT_EXAMPLES}\n\nNew question:\nQ: {question}\n\nCase: {case_description}\n\nA:"
    return ...

# VARIANT 3: AUTO COT — build few-shot automatically
def auto_cot(policy: str, case_description: str, question: str, num_demos: int = 2, llm=None) -> CoTResult:
    demos = []
    for sample_q in question_pool[:num_demos]:
        response = llm(f"{policy}\n\nQuestion: {sample_q}\n\nThink step by step.")
        demos.append(f"Q: {sample_q}\nA:\n{response}\n")
    prompt = f"{policy}\n\n" + "\n".join(demos) + f"\n\nNew question:\nQ: {question}\n\nCase: {case_description}\n\nA:"
    return ...

# REFUND ELIGIBILITY ADVISOR
advisor = RefundEligibilityAdvisor(policy=REFUND_POLICY)
result = advisor.check_eligibility(case, variant="few_shot")  # zero_shot | few_shot | auto_cot
# result.reasoning, result.conclusion

Full Example: patterns/chain-of-thought/example.py

Pattern 14: Tree of Thoughts (ToT)

Category: LLM Reasoning
Use When: Strategic tasks with multiple plausible paths; single linear CoT is insufficient

Problem

Many tasks that demand strategic thinking cannot be solved by a single multistep reasoning path:

• Single-path limitation — CoT follows one sequence; if that path is wrong, the answer suffers
• Branching decisions — Multiple plausible next steps
• Need for exploration — Best solution often requires exploring several directions

Solution

Tree of Thoughts treats problem-solving as tree search with four components:

1. Thought generation — From current state, generate N possible next steps
2. Path evaluation — Score each partial solution (0–100) for promise
3. Beam search (top K) — Keep only the top K states; prune the rest
4. Summary generation — Produce a concise summary and answer from the best path

Code Snippet

class TreeOfThoughts:
    def generate_thoughts(self, state: str, step: int, problem: str) -> List[str]:
        """Generate N possible next thoughts from current state."""
        return thoughts

    def evaluate_state(self, state: str, problem: str) -> float:
        """Score path promise (0-1). Correctness, progress, potential."""
        return score

    def solve(self, problem: str) -> ToTResult:
        beam = [(0.5, initial_state, [], 0)]
        for step in range(1, self.max_steps + 1):
            candidates = []
            for score, state, path, _ in beam:
                thoughts = self.generate_thoughts(state, step, problem)
                for thought in thoughts:
                    new_state = state + "\nStep N: " + thought
                    new_score = self.evaluate_state(new_state, problem)
                    candidates.append((new_score, new_state, path + [thought], step))
            beam = sorted(candidates, key=lambda x: -x[0])[:self.beam_width]
        best_state, best_path = beam[0]
        summary = self.generate_summary(problem, best_state)
        return ToTResult(..., solution_summary=summary, reasoning_path=best_path)

# INCIDENT ROOT-CAUSE ANALYZER
analyzer = IncidentRootCauseAnalyzer()
result = analyzer.analyze("API latency spiked; DB, cache, dependencies in use.")
# result.solution_summary, result.reasoning_path

Full Example: patterns/tree-of-thoughts/example.py

Pattern 15: Adapter Tuning

Category: LLM Reasoning
Use When: You need a foundation model to perform a specialized task with a small dataset and want to keep base weights frozen while training only a small adapter (e.g., LoRA)

Problem

Incoming tickets must be routed to billing, technical, sales, or general. Prompt-only classification can be brittle. Adapter tuning trains a small task-specific head on a few hundred labeled tickets while keeping the foundation model frozen.

Solution

Adapter tuning (PEFT) has three key aspects:

1. Teaches the foundation model a specialized task — Train on input-output pairs
2. Foundation weights frozen; only a small adapter is updated — LoRA or adapter layers are trained
3. Training dataset can be smaller — Often a few hundred to a few thousand high-quality pairs suffice

Code Snippet

class TicketIntentRouter:
    def __init__(self):
        self._pipeline = Pipeline([
            ("foundation", TfidfVectorizer(max_features=2000)),  # frozen after fit
            ("adapter", LogisticRegression(max_iter=500)),       # only this is "trained"
        ])

    def train(self, examples: List[TicketExample]) -> None:
        texts = [ex.text for ex in examples]
        labels = [ex.intent for ex in examples]
        self._pipeline.fit(texts, labels)

    def predict(self, text: str) -> AdapterTuningResult:
        pred = self._pipeline.predict([text])[0]
        probs = self._pipeline.predict_proba([text])[0]
        return AdapterTuningResult(intent=pred, confidence=float(probs.max()))

router = TicketIntentRouter()
router.train(train_examples)  # 200–2000 (text, intent) pairs
result = router.predict("I was charged twice, please refund.")
# result.intent -> "billing"

Full Example: patterns/adapter-tuning/example.py

Pattern 16: Evol-Instruct

Category: LLM Reasoning
Use When: You need to teach a pretrained model new, complex tasks from private data by evolving simple instructions into harder ones, generating answers, and instruction tuning (SFT/LoRA)

Problem

The company wants a model that answers complex policy questions from internal docs under data privacy. Manually creating thousands of hard (question, answer) pairs is expensive.

Solution

Evol-Instruct in four steps:

1. Evolve instructions — From seed questions, create harder variants: deeper (constraints, hypotheticals), more concrete (“list 3 reasons”), multi-step (combine two questions)
2. Generate answers — For each instruction, produce a high-quality answer (LLM with access to your private context)
3. Evaluate and filter — Score each (instruction, answer) 1–5; keep only examples above a threshold
4. Instruction tuning — SFT on an open-weight model (Llama, Gemma) using the filtered dataset; PEFT/LoRA for efficient training

Code Snippet

# STEP 1: Evolve instructions
def evolve_instructions(seeds: List[str]) -> List[str]:
    # Deeper: add constraints/hypotheticals
    # Concrete: "List 3 reasons...", "What are the steps..."
    # Multi-step: combine two questions
    return all_instructions

# STEP 2: Generate answers (LLM + policy context)
qa_pairs = generate_answers(all_instructions)

# STEP 3: Score and filter (LLM or model; 1-5)
scored = [score_instruction_answer(ia) for ia in qa_pairs]
filtered = [ex for ex in scored if ex.score >= 4]

# STEP 4: SFT-ready dataset (chat format) -> then HuggingFace SFT/LoRA
sft_dataset = [{"messages": [{"role": "user", "content": ex.instruction},
                             {"role": "assistant", "content": ex.answer}]}
               for ex in filtered]
# Train with transformers + peft + trl SFTTrainer

Full Example: patterns/evol-instruct/example.py

Category 4: Reliability & Evaluation

Patterns 17–20 focus on evaluation, safety, and reliability: using LLMs to judge quality, and guard against harmful or off-policy outputs.

Pattern 17: LLM as Judge

Category: Reliability
Use When: You need nuanced evaluation of model or human outputs with scores and justifications to drive feedback loops, filtering, or training

Problem

Teams must evaluate thousands of support replies for helpfulness, tone, accuracy, clarity, and completeness. Human review does not scale; simple metrics (length, keyword match) miss nuance.

Solution

LLM as Judge uses an LLM to score and justify outputs against a scoring rubric. Three options:

1. Prompting — Criteria and instructions in the prompt; LLM returns score (1–5) per criterion and brief justification. Temperature=0 for consistency.
2. ML — Create rubric, collect historical (item, scores) data, train a classification model to replicate the rubric at scale.
3. Fine-tuning — Fine-tune a model as a dedicated judge on your rubric and labeled data.

Code Snippet

SUPPORT_REPLY_CRITERIA = """
- Helpfulness: Addresses the customer's question; actionable next steps.
- Tone: Professional, empathetic.
- Accuracy: Factually correct.
- Clarity: Easy to read; no unnecessary jargon.
- Completeness: Covers the main ask.
"""

def build_judge_prompt(item: str, criteria: str) -> str:
    return f"""You are evaluating a customer support reply. Score 1-5 per criterion with brief justification.
Criteria: {criteria}
Reply: --- {item} ---
Scores:"""

# Invoke judge with temperature=0 for consistency
raw = run_judge(build_judge_prompt(reply))
result = parse_judge_response(raw, reply)
# result.scores -> [CriterionScore(criterion="Helpfulness", score=4, justification="..."), ...]

Full Example: patterns/llm-as-judge/example.py

Pattern 18: Reflection

Category: Reliability
Use When: You invoke the LLM via a stateless API and want it to correct or improve its first response without the user sending a follow-up.

Problem

The API must return a short apology email for a delayed shipment. A single LLM call may omit an order reference, sound generic, or lack a clear next step.

Solution

Reflection: Do not return the first response to the client. (1) First call ? initial response. (2) Evaluate: send initial response to an evaluator; get feedback. (3) Modified prompt: original request + initial response + feedback. (4) Second call ? revised response. Return the revised response.

Code Snippet

def run_reflection(user_prompt: str) -> ReflectionResult:
    initial_response = generate_initial(user_prompt)       # First call
    feedback, notes = evaluate(user_prompt, initial_response)  # Evaluator
    modified_prompt = (
        f"Original request:\n{user_prompt}\n\n"
        f"Your previous response:\n---\n{initial_response}\n---\n\n"
        f"Feedback to apply:\n{feedback}\n\nProduce an improved version."
    )
    revised_response = generate_revised(modified_prompt)  # Second call
    return ReflectionResult(initial_response, feedback, revised_response)
# Return revised_response to client; initial_response is not sent.

Full Example: patterns/reflection/example.py

Pattern 19: Dependency Injection

Category: Reliability
Use When: Developing and testing GenAI apps are nondeterministic, models change quickly, and you need code to be LLM-agnostic; inject LLM and tool calls

Problem

Developing and testing is hard: LLM output is nondeterministic, APIs change, and you want CI and local dev without API keys.

Solution

Dependency Injection: Pass LLM and tool calls into the pipeline as dependencies. Production uses real implementations; tests and dev use mocks that return hardcoded, deterministic results.

Code Snippet

# Pipeline accepts dependencies; no direct LLM calls inside
def run_ticket_pipeline(
    ticket_text: str,
    summarize_fn: Callable[[str], str],
    suggest_action_fn: Callable[[str, str], str],
) -> TicketResult:
    summary = summarize_fn(ticket_text)
    suggested_action = suggest_action_fn(ticket_text, summary)
    return TicketResult(summary=summary, suggested_action=suggested_action)

# Production: real implementations
result = run_ticket_pipeline(ticket, real_summarize, real_suggest_action)

# Tests: mocks (hardcoded, deterministic)
result = run_ticket_pipeline(ticket, mock_summarize, mock_suggest_action)
assert result.summary == "Customer reports an issue..."

Full Example: patterns/dependency-injection/example.py

Pattern 20: Prompt Optimization

Category: Reliability
Use When: You want better results from prompt engineering but changing the foundational model would force repeating all trials so use a repeatable optimization loop with pipeline

Solution

Prompt optimization as four components — (1) Pipeline of steps that use the prompt (prompt is a parameter), (2) Dataset to evaluate on, (3) Evaluator that scores each output, (4) Optimizer that proposes candidates and picks the best by score.

Code Snippet

def run_pipeline(prompt_template: str, ticket: str) -> str:
    return generate_fn(prompt_template, ticket)

dataset = get_dataset()

def evaluate_summary(summary: str, ticket: str) -> float:
    return 0.0  # ... length, key-info, or LLM-as-Judge

best_prompt, best_score = optimize_prompt(
    candidate_prompts=["Summarize in one sentence.", "Write a one-line summary.", ...],
    dataset=dataset,
    run_fn=lambda p, t: run_pipeline(p, t),
    eval_fn=evaluate_summary,
)
# When model changes: re-run optimize_prompt with same dataset/evaluator

Full Example: patterns/prompt-optimization/example.py

Category 5: Tools, Agents & Efficiency

Patterns 21–32 extend LLMs with tool calling, code execution, multi-agent collaboration, and production efficiency techniques.

Pattern 21: Tool Calling

Category: Tools & Agents
Use When: You need the model to act by calling APIs, looking up live order status, searching internal systems

Solution

Bind tools to the model; run a LangGraph with an assistant node (LLM) and ToolNode (executes tools). Conditional routing: if the last message has tool_calls, run tools and loop back.

Code Snippet

from langgraph.graph import END, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode
from langchain_core.tools import tool

@tool
def lookup_order_status(order_id: str) -> str:
    """Look up order in OMS."""
    return '{"status":"shipped",...}'

workflow = StateGraph(MessagesState)
workflow.add_node("assistant", call_model)
workflow.add_node("tools", ToolNode([lookup_order_status]))
workflow.set_entry_point("assistant")
workflow.add_conditional_edges("assistant", route_tools_or_end)
workflow.add_edge("tools", "assistant")
app = workflow.compile()

Full Example: patterns/tool-calling/example.py
Dependencies: pip install -r patterns/tool-calling/requirements.txt; Ollama with a tool-capable model (e.g. llama3.2)

Pattern 22: Code Execution

Category: Tools & Agents
Use When: The task needs an artifact (diagram, plot, query): the model should emit a DSL and a sandbox runs it

Solution

Code execution: Prompt the model for DSL (low temperature). A sandbox writes temp files, runs dot, python (restricted), or a DB driver with timeouts and allowlists. LangGraph can wire generate_dsl ? execute_sandbox as a linear graph.

Code Snippet

from langgraph.graph import END, StateGraph

workflow = StateGraph(CodeExecutionState)
workflow.add_node("generate_dsl", node_generate_dsl)
workflow.add_node("sandbox", execute_in_sandbox)
workflow.set_entry_point("generate_dsl")
workflow.add_edge("generate_dsl", "sandbox")
workflow.add_edge("sandbox", END)
app = workflow.compile()
final = app.invoke({"user_request": "..."})

Full Example: patterns/code-execution/example.py
Dependencies: pip install -r patterns/code-execution/requirements.txt; optional Graphviz (brew install graphviz)

Pattern 23: Multi-Agent Collaboration

Category: Tools & Agents
Use When: Work is multistep, multi-domain, and long-running; a single agent hits cognitive, tool, and tuning limits

Solution

Multi-agent collaboration: Define agents with narrow mandates and clear handoffs. Patterns include hierarchical (planner delegates), prompt chaining (sequential pipelines), peer-to-peer / blackboard (shared store), and parallel execution.

Code Snippet

from langgraph.graph import END, StateGraph

g = StateGraph(MultiAgentState)
g.add_node("plan", node_plan)
g.add_node("technical", node_technical)
g.add_node("compliance", node_compliance)
g.add_node("merge", node_merge)
g.add_node("critic", node_critic)
g.add_node("finalize", node_finalize)
g.set_entry_point("plan")
app = g.compile()
result = app.invoke({"user_request": "..."})

Full Example: patterns/multiagent-collaboration/example.py

Pattern 24: Small Language Model

Category: Efficiency & Deployment
Use When: Frontier models are too large or too expensive to self-host; you want smaller models, distillation, quantization, or faster decoding

Solution

1. Knowledge distillation — Train a student on teacher soft targets; KL divergence aligns token distributions
2. Quantization — 4-bit / 8-bit weights (BitsAndBytesConfig, NF4) shrink footprint
3. Speculative decoding — A small draft model proposes tokens; a large target verifies in parallel

Code Snippet

# Distillation: minimize KL(student || teacher) on teacher softmax + CE on labels
# Quantization: BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", ...)
# Speculative decoding: vLLM speculative_config={"model": draft_id, "num_speculative_tokens": k}

Full Example: patterns/small-language-model/example.py

Pattern 25: Prompt Caching

Category: Efficiency & Deployment
Use When: The same or similar prompts hit your LLM repeatedly and you want lower latency, lower cost, and less load

Solution

1. Client-side exact cache — Hash (model, params, messages) ? store response
2. Framework caches — LangChain InMemoryCache / SQLiteCache via set_llm_cache
3. Semantic cache — Embeddings to match paraphrases; return cached answer if similarity ? threshold
4. Server-side prompt caching — Anthropic / OpenAI may cache eligible long prompts inside the API

Code Snippet

# Exact: sha256(f"{model}\n{prompt}") -> response
# Semantic: cosine(embed(query), embed(cached_prompt)) >= threshold
# LangChain: set_llm_cache(SQLiteCache(database_path="..."))
# Provider: Anthropic cache_control / OpenAI automatic prefix caching (see docs)

Full Example: patterns/prompt-caching/example.py

Pattern 26: Inference Optimization

Category: Efficiency & Deployment
Use When: You self-host LLMs and must maximize throughput, cut latency, and control KV-cache memory

Solution

1. Continuous batching (dynamic batching) — Requests enter and leave at fine granularity; vLLM (PagedAttention) and SGLang reduce padding waste
2. Speculative decoding — Draft + target models (see Pattern 24)
3. Prompt compression — Remove redundancy in system + RAG context to shrink KV footprint

Code Snippet

# Continuous batching: use vLLM / SGLang / TensorRT-LLM — not hand-rolled pad batches
# Speculative decoding: vLLM speculative_config={...} (see Pattern 24)
# Prompt compression: dedupe, summarize, or learned compressors before .generate()

Full Example: patterns/inference-optimization/example.py

Pattern 27: Degradation Testing

Category: Efficiency & Deployment
Use When: You need load testing that matches LLM inference behavior with TTFT, end-to-end latency, token throughput, and RPS under rising concurrency

Key Metrics

• TTFT — Time from request to first token (streaming)
• EERL — End-to-end request latency (wall time to last token)
• Output tokens / second — Generation throughput
• Requests / second — Completed requests per second at a given concurrency

Code Snippet

# Per request: ttft_s, eerl_s, output_tokens -> tok/s = tokens / (eerl_s - ttft_s)
# Aggregate: p95_ttft, p95_eerl, mean tok/s, rps = n / wall_time
# Tools: LLMPerf, LangSmith traces

Full Example: patterns/degradation-testing/example.py

Pattern 28: Long-Term Memory

Category: Memory & Agents
Use When: LLM calls are stateless; you need continuity across sessions with working, episodic, procedural, and semantic memory

Solution

1. Working memory — Recent turns / scratch context (sliding window)
2. Episodic memory — Dated interactions (“what we did”)
3. Procedural memory — Playbooks and tool recipes
4. Semantic memory — Stable facts; typically embedding search (Mem0, custom RAG-over-memories)

Code Snippet

# Mem0: memory.add(messages, user_id=...); memory.search(query, user_id=...)
# Four layers: working (deque), episodic (log), procedural (playbooks), semantic (vector / KV)

Full Example: patterns/long-term-memory/example.py
Dependencies: pip install mem0ai openai chromadb for Mem0-aligned version

Pattern 29: Template Generation

Category: Setting Safeguard
Use When: You need repeatable, reviewable customer-facing text; full free-form generation is too variable or mixes facts with creativity unsafely

Solution

1. Prompt the model to output a template only, with explicit placeholders ([CUSTOMER_NAME], [ORDER_ID], …)
2. Human / comms reviews the template (per locale/product), not every send
3. Fill slots in code; optional second LLM pass only for lint or translation
4. Few-shot examples in the prompt show approved shapes so new templates stay grounded

Code Snippet

# Low temp + few-shot -> template with [SLOT_NAME]
# validate required [ORDER_ID], [CUSTOMER_NAME] present
# fill_template(template, slots_from_crm)

Full Example: patterns/template-generation/example.py

Pattern 30: Assembled Reformat

Category: Setting Safeguard
Use When: A full LLM-generated page can hallucinate high-risk attributes (battery chemistry, hazmat, allergens, medical claims)

Solution

1. Risk registry — chemistry, Wh, hazmat, etc. from structured sources only
2. Assemble deterministic blocks (specs, shipping, legal)
3. Optional LLM for tone/SEO only, conditioned on the assembled facts
4. Validate — banned claims, chemistry contradictions

Code Snippet

# facts = load_pim(sku)  # BatteryChemistry.NIMH, ...
# page = render_compliance_block(facts)  # deterministic
# fluff = llm_marketing(facts)  # constrained; validate_high_risk(page + fluff, facts)

Full Example: patterns/assembled-reformat/example.py

Pattern 31: Self-Check

Category: Setting Safeguard
Use When: You can obtain per-token logprobs from inference and want a statistical signal to flag uncertain or fragile generations for review

Solution

1. Logits ? softmax ? probabilities (p_i)
2. Logprob (log p) for the sampled token (APIs often return this directly)
3. Flag tokens with low (p) or small margin to the second-best token
4. Perplexity on a sequence: PPL = exp(-mean(logprobs))

Code Snippet

# p_i = exp(logprob_i); flag if p_i < threshold
# PPL = exp(-mean(logprobs))  # natural-log probs per token

Full Example: patterns/self-check/example.py

Pattern 32: Guardrails

Category: Setting Safeguard
Use When: You must enforce security, privacy, moderation, and alignment around LLM and RAG systems

Solution

1. Prebuilt — Gemini safety settings; OpenAI Moderation API; hosted provider filters
2. Custom — PII redaction, banned topics, allowlists, regex injection detectors, LLM-as-Judge (Pattern 17)
3. Compose — apply_guardrails(text, scanners) pipeline; scan query, then answer

Code Snippet

# apply_guardrails(user_query, [pii_redact, banned_topic])
# answer = engine.query(sanitized); apply_guardrails(answer, [pii_redact, moderation])

Full Example: patterns/guardrails/example.py

Category 6: Agentic Behavior Patterns

Patterns 33–50 align with Agentic Design Patterns (Antonio Gulli): specialized agent roles, orchestration, and production agentic systems.

Pattern 33: Prompt Chaining

Category: Agentic Orchestration
Use When: A task benefits from sequential decomposition where each LLM call has one job, structured output feeds the next step

Solution

Code Snippet

# state = classify(q); state = decompose(state); state = answer(state); state = format(state)
# Or LangGraph: add_node per step, linear edges

Full Example: patterns/prompt-chaining/example.py

Pattern 34: Routing

Category: Agentic Orchestration
Use When: You must classify or direct each request to the right handler

Solution

1. Rule-based routing for deterministic paths
2. Embedding similarity to handler descriptions or labeled exemplars
3. LLM routing with JSON schema: route, confidence, optional rationale
4. ML classifier on features for scale and SLOs

Code Snippet

# RunnableBranch (langchain_core): (predicate, runnable), ..., default
# Or: rules_first = route_rules(text); if conf < 0.9: route_llm(text)

Full Example: patterns/routing/example.py

Pattern 35: Parallelization

Category: Agentic Orchestration
Use When: Independent subtasks can run together such as research fan-out, analytics partitions, parallel validators

Solution

Code Snippet

# LCEL: RunnableParallel(gather=..., analyze=..., verify=...) | RunnableLambda(merge)
# stdlib: ThreadPoolExecutor; submit each branch; as_completed ? dict

Full Example: patterns/parallelization/example.py

Pattern 36: Learning and Adaptation

Category: Agentic Learning
Use When: Systems must improve from experience such as RL (PPO with clipped surrogate for stability) and preference alignment (DPO without a separate reward model)

Solution

1. RL agents — collect trajectories ? advantage estimates ? PPO-style clipped ratio to limit destructive updates
2. LLM alignment — RLHF path (reward model + PPO) vs DPO (direct policy update from chosen/rejected completions)
3. Online / memory — replay, regularization, retrieval over past successes

Code Snippet

# PPO: clip ratio r to [1-eps, 1+eps]; surrogate min(r*A, clip(r)*A)
# DPO: preference loss on log pi(y_w) - log pi(y_l) vs reference (see TRL / papers)

Full Example: patterns/learning-adaptation/example.py

Pattern 37: Exception Handling and Recovery

Category: Agentic Reliability
Use When: Agents, chains, and tools must survive failures by detecting and classifying errors, and retrying wisely and fallback to degraded paths

Solution

1. Detect — Structured errors, validation, guardrails, timeouts
2. Classify — Transient vs permanent vs policy
3. Handle — Exponential backoff, circuit breaker, fallback model or cache
4. Recover — Idempotent retries, compensation, checkpoint resume

def run_with_fallback(
    primary: Callable[[], T],
    fallback: Callable[[], T],
    is_recoverable: Callable[[BaseException], bool],
) -> T:
    """
    Try ``primary``; on a recoverable exception, invoke ``fallback``.

    Args:
        primary: Preferred code path (e.g. frontier model).
        fallback: Degraded path (e.g. smaller model or cached stub).
        is_recoverable: Whether to use fallback for this exception type.

    Returns:
        Result from primary or fallback.

    Raises:
        Re-raises if the primary fails with a non-recoverable error.
    """
    try:
        return primary()
    except Exception as exc:
        if not is_recoverable(exc):
            raise
        return fallback()

Full Example: patterns/exception-handling-recovery/example.py

Pattern 38: Human-in-the-Loop (HITL)

Category: Agentic Safety
Use When: Automation must yield to people for quality, compliance, or risk

Solution

1. Triggers — Low confidence, high stakes, novel situations, regulatory rules, sampling
2. Review — Queues, rubrics, SLAs, multi-level approval
3. Feedback — Labels and edits ? datasets, policies, routing
4. Orchestration — LangGraph interrupt / human nodes; workflow engines with wait states

Code Snippet

# if stakes == HIGH or conf < tau: enqueue(HumanReviewTicket)
# LangGraph: interrupt_before=[human_node]; resume with Command

Full Example: patterns/human-in-the-loop/example.py

Pattern 39: Agentic RAG (Knowledge Retrieval)

Category: Agentic Knowledge
Use When: You need up-to-date, source-grounded answers with embeddings, semantic search, chunking, vector stores, and advanced variants

Solution

1. Chunk ? embed ? vector DB; measure relevance via cosine / distance metrics
2. Hybrid retrieval (dense + sparse) where lexical match matters
3. Graph RAG for entity-centric queries; agentic RAG for query rewrite, tool retrieval, multi-hop
4. LangChain LCEL / LangGraph for pipelines and cycles

Code Snippet

# LCEL: RunnablePassthrough.assign(context=retriever) | prompt | llm
# LangGraph: retrieve -> grade_documents -> [rewrite_query | generate]

Full Example: patterns/agentic-rag/example.py
See also Patterns 6–12 for depth implementations of each RAG component.

Pattern 40: Resource-Aware Optimization

Category: Agentic Efficiency
Use When: You must optimize LLM and agent workloads for cost, latency, capacity, and graceful degradation

Solution

1. Budgets and tiered models — Estimate $ per request
2. Route by priority and load (Pattern 34)
3. Prune / summarize context; cache (25); smaller models (24)
4. Degrade — Fewer tools, shorter answers, async handoff

Code Snippet

# if budget.remaining() < need: summarize(history) or tier = "small"
# if degradation == MINIMAL: tool_gate.disable_heavy_tools()

Full Example: patterns/resource-aware-optimization/example.py

Pattern 41: Reasoning Techniques (Agentic)

Category: Agentic Reasoning
Use When: You need a structured approach to complex Q&A using CoT, ToT, self-correction, PAL / code-aided reasoning, ReAct, RLVR, debates (CoD), deep research

Solution

Use the technique map in patterns/reasoning-techniques/README.md: CoT (13), ToT (14), Reflection (18), Deep Search (12), ReAct / tools (21), PAL-style code (22), multi-agent debates (23), prompt / workflow optimization (20).

def language_agent_tree_search_stub(
    frontier: list[str],
    expand_fn: Callable[[str], list[str]],
    score_fn: Callable[[str], float],
    beam_width: int = 2,
) -> list[tuple[str, float]]:
    """
    Minimal beam-style selection (stand-in for Language Agent Tree Search).

    LATS in the literature expands **language** states/actions, scores children
    with a value model or LLM critic, and prunes—unlike a flat ToT breadth list.

    Args:
        frontier: Current candidate partial solutions or thoughts.
        expand_fn: Callable taking one candidate, returning child strings.
        score_fn: Callable taking a string, returning higher-is-better score.
        beam_width: Max states to keep after scoring.

    Returns:
        Top ``beam_width`` (candidate, score) pairs.
    """
    children: list[tuple[str, float]] = []
    for node in frontier:
        for ch in expand_fn(node):
            children.append((ch, float(score_fn(ch))))
    children.sort(key=lambda x: x[1], reverse=True)
    return children[:beam_width]

Full Example: patterns/reasoning-techniques/example.py

Pattern 42: Evaluation and Monitoring (Agentic)

Category: Agentic Observability
Use When: You need performance tracking, A/B tests, compliance evidence, latency SLOs, token/cost telemetry, custom quality metrics (LLM-as-judge), and multi-agent traces

Solution

Instrument calls, aggregate SLAs, run experiments with guardrail metrics, store audit evidence, trace multi-agent workflows.

Code Snippet

# trace_id + span per LLM/tool; tokens += pt+ct; export to OTLP
# ab_variant(user_key, "exp", ("a","b")); compare judge_score & p95_latency

Full Example: patterns/evaluation-monitoring/example.py

Pattern 43: Prioritization

Category: Agentic Scheduling
Use When: Competing tasks must be ordered to support queues, cloud jobs, trading paths, security incidents using multi-criteria scores, dynamic re-ranking, and resource-aware scheduling

Solution

Weighted dimensions (urgency, impact, effort, SLA, security), recompute on events, integrate with routing (34) and capacity (40).

Code Snippet

# score = w1*urgency + w2*importance - w3*effort + w4*f(sla) + w5*security

Full Example: patterns/prioritization/example.py

Pattern 44: Memory Management

Category: Agentic State
Use When: Agents need short-term context, long-term persistence, episodic retrieval, procedural playbooks, and privacy-aware storage

Solution

Tier memory (working, episodic, procedural, semantic); extract and retrieve selectively; persist orchestrator state with MemorySaver.

Code Snippet

# LangGraph: compile(..., checkpointer=MemorySaver()); thread_id in config
# External: memory.search(query, user_id=...) for semantic / episodic layers

Full Example: patterns/memory-management/example.py

Pattern 45: Planning and Task Decomposition

Category: Agentic Orchestration
Use When: You need explicit task graphs, dependencies, and valid execution order

Decompose goals into a DAG of subtasks with dependencies. The planner agent determines which tasks to run in parallel vs. sequentially based on dependency analysis.

Full Example: patterns/planning-task-decomposition/example.py

Pattern 46: Goal Setting and Monitoring

Category: Agentic Governance
Use When: SMART goals, progress vs. targets, deviation detection, strategy updates

The goal-monitor agent tracks metrics against defined targets, detects when progress deviates from expected trajectories, and adjusts strategy when needed.

Full Example: patterns/goal-setting-monitoring/example.py

Pattern 47: MCP Integration (Agentic)

Category: Tooling / Integration
Use When: Model Context Protocol servers — discovery, tools/list, tools/call — secure composition with Pattern 21

Model Context Protocol (MCP) provides a standardized interface between agents and external resources. Agents discover available tools at runtime through the protocol, call them with structured inputs, and receive structured outputs.

Full Example: patterns/mcp-integration/example.py

Pattern 48: Inter-Agent Communication

Category: Distributed Agents
Use When: Message envelopes, routing, correlation, capability discovery (A2A-style) with Pattern 23

Agent-to-Agent (A2A) communication defines structured message schemas and communication protocols for inter-agent coordination. Agents send typed messages (task assignments, results, status updates, requests for clarification) through a message bus or shared workspace.

Full Example: patterns/inter-agent-communication/example.py

Pattern 49: Safety Guardian

Category: Safety / Compliance
Use When: Multi-layer defense, risk thresholds, shutdown paths beyond single guardrail scanners (extends Pattern 32)

The safety guardian agent implements three-tier protection: pre-action guardrails (evaluate the proposed action before execution), in-process monitoring (enforce scope and resource constraints during execution), and post-action auditing. Includes prompt injection detection for agents that process external content.

Full Example: patterns/safety-guardian/example.py

Pattern 50: Exploration and Discovery

Category: Search / Learning
Use When: Explore vs. exploit, novel environments, hypothesis cycles (pairs with Patterns 12, 14, 41, 36)

Implements a multi-armed bandit or curiosity-driven strategy that balances exploitation (using known-good approaches) with exploration (trying new approaches to discover if they’re better). Scores update from outcomes, so the agent continuously refines its strategy distribution.

Full Example: patterns/exploration-discovery/example.py

Pattern Comparison Matrix

Choosing the Right Pattern

Takeaways

Here are major takeaways from these agentic patterns:

Enforce constraints early. Logits Masking and Grammar Constrained Generation prevent bad output at the token level. The same logic applies to Guardrails: put them in the runtime layer, not in the system prompt.

RAG is a stack you build layer by layer. Start with Basic RAG. When vocabulary gaps break retrieval, add Semantic Indexing. When contradictions surface, add Indexing at Scale. When queries don’t match chunks, add Index-Aware Retrieval. When retrieved chunks are noisy, add Node Postprocessing. Each pattern fixes the failure mode of the one before.

Structure the reasoning. Chain of Thought, Tree of Thoughts, and ReAct all treat reasoning as something to engineer. Adding “think step by step” costs one line and measurably improves multi-step accuracy. Tree of Thoughts costs more but handles problems where a single reasoning path gets stuck.

You need less data than you think for specialization. Adapter Tuning and Evol-Instruct both produce strong task-specific models from hundreds of examples, not millions. Evolving seed questions into harder variants and filtering by quality gives you a curriculum worth training on. The bottleneck is usually data quality, not quantity.

The operational patterns matter as much as the modeling ones. Prompt Caching, Inference Optimization, and Degradation Testing don’t appear in research papers. They’re the difference between a working demo and a system that holds up under real traffic.

I have added examples for all 50 patterns at: github.com/bhatti/agentic-patterns

Run the setup in ten minutes with SETUP.md, then explore whichever patterns are most relevant to what you’re building.

Technology Stack

• Ollama — Local model serving
• LangChain — LLM orchestration
• CrewAI — Multi-agent systems
• LangGraph — Stateful agent workflows
• HuggingFace — Model hub and transformers
• PyTorch — Direct model access and logits manipulation

In past projects, I saw most engineering teams ran load tests before major launches and rarely at any other time. The assumption is that if a code change is small, performance is probably fine. In practice, that assumption fails regularly. A runtime upgrade can change memory allocation patterns, garbage collection behavior, and connection handling in ways that only appear under load. A third-party library upgrade can introduce synchronous blocking where there was none before. A new database index can shift query planner behavior and affect read latency at scale. None of these surface in functional tests. None of them are visible in code review. They show up under load, in production, usually at the worst possible time.

Performance testing isn’t a pre-launch ceremony. It’s part of how you understand and maintain your system’s behavior as your code evolves, your dependencies change, and your traffic grows. This guide covers the full scope: the test types and what each one tells you, how to design meaningful tests, what metrics to collect, which tools to use, how to handle dependencies in your tests, and how to make this a regular part of your development process rather than a one-time event.

Why Performance Testing Gets Skipped

Often teams skip performance testing due to setup time, cost or slow feedback loop. These constraints are legitimate, but they lead to a familiar outcome where performance problems get discovered in production. Another common pattern I have observed is that many teams don’t have a clear baseline picture of how their application actually behaves. They don’t know their normal memory footprint. They don’t know which code paths are hot. They don’t know at what concurrency level their database connection pool saturates or when their cache hit rate starts degrading. Without a baseline, you can’t detect regressions, you can’t capacity plan accurately, and you can’t tell a normal traffic spike from an actual problem. The goal of performance testing is to know your system well enough to predict how it behaves and catch it when behavior changes unexpectedly.

Performance Testing in the SDLC

The most effective teams don’t treat performance testing as a separate phase instead they integrate it into their regular development process at multiple levels.

• During development: I have found profiling tools like JProbe/Yourkit for Java, pprof for GO, V8 profiler for Node.js, XCode instruments for Swift/Objective-C incredibly useful to find hot code path, memory leaks or concurrency issues.
• During code review: Another common pattern that I have found useful is flagging changes to caching, database queries, serialization, or hot code paths for load testing before merge.
• Nightly CI/CD pipelines: Though, load testing on each commit would be excessive but they can partially run as part of nightly build so that we can fix them before they reach production.
• On a regular schedule: Another option is to run full-scale load and soak tests run on a defined cadence like weekly.
• Before major releases: Comprehensive tests covering all scenarios like average load, peak load, stress, spikes, soak can run against a production-representative environment.
• After significant dependency upgrades: Runtime upgrades, major library version bumps, and infrastructure changes all deserve their own performance test pass.

The Testing Taxonomy

Following are different types of performance tests:

Profiling

Profiling instruments your application during execution and shows you exactly where time and memory are spent like which functions consume CPU, which allocate the most memory, where goroutines or threads block. You can run profiling locally before the code review so that you understand the bottlenecks already exist in your code. Load testing tells you how those bottlenecks behave when many users hit them simultaneously. Most runtimes include profiling support like Go’s pprof, Node.js’s built-in CPU and heap profiler, Python’s cProfile so you can also enable them in a test environment if needed.

Load Testing

Load testing applies a realistic, expected workload and verifies the system meets defined performance targets. The workload mirrors production traffic such as request distribution, concurrency level, and payload shapes. The goal isn’t to break anything. It’s to confirm the system handles its designed workload within acceptable response times and error rates. Any change that could affect throughput like a code change in a hot path, a dependency upgrade, a configuration change, a schema migration should warrant a load test.

Stress Testing

Stress testing pushes load well beyond expected levels to find where the system breaks and how it breaks. At what point does performance degrade? What component fails first? Does the system fail gracefully or catastrophically, or corrupting state? In past projects, I found a practical target in cloud environments is 10x your expected peak load. This accounts for real-world variability: viral traffic events, bot traffic, cascading retries from upstream services, and faster growth than planned. Stress tests also expose whether your failure modes are safe. When your system can’t keep up, what happens? Does it queue requests until it runs out of memory? Does it reject new connections cleanly with meaningful errors? Does retry behavior from clients amplify load turning a recoverable spike into a full outage?

Spike Testing

Spike testing applies an abrupt load increase not a gradual ramp but a sharp jump so that we can learn how the system absorbs and recovers from it. This simulates promotional emails going out, products appearing in news, scheduled batch jobs triggering thousands of concurrent operations, or a mobile app push notification causing a synchronized rush of API calls. The spike testing can identify problems like cold-start latency when new instances initialize, connection pool exhaustion when concurrency jumps faster than the pool replenishes, cache stampedes when many concurrent requests miss cache simultaneously, and auto-scaling lag when the metric-to-action delay is too long. After the spike, watch recovery. Latency should return to baseline. Resource utilization should drop. If it doesn’t, the system is carrying forward pressure that will degrade subsequent traffic.

Soak Testing

Soak testing runs a moderate, sustained load over an extended period from several hours to several days. The load level isn’t extreme; the duration is the point so that it can uncover problems that only occur after a long duration such as:

• Memory leaks: Usage climbs slowly and continuously. The system that runs fine for 30 minutes may run out of heap after 8 hours. This is especially important to test after runtime or library upgrades, which can change allocator behavior.
• Connection leaks: Database or HTTP connections that aren’t properly released accumulate until the pool is exhausted.
• Thread accumulation: Background threads that don’t terminate properly compound over time.
• Disk exhaustion: Log files that aren’t rotated, or temporary files that aren’t cleaned up, fill disk gradually.
• Cache degradation: Caches misconfigured for their access patterns may perform well initially and degrade as the working set evolves.
• GC pressure: Garbage collection that runs cleanly initially can become increasingly frequent and pause-heavy as heap fragmentation grows over time.

Scalability Testing

Scalability testing validates that your system scales up to absorb increasing load and scales back down when load subsides. Cloud infrastructure assumes elastic scaling so scalability testing verifies the assumption. This helps verify that: the metric driving scale-up (CPU, request rate, queue depth) actually reaches its threshold under realistic load. The scaling event actually reduces the pressure that triggered it. Scale-up happens fast enough that users don’t experience degradation during the lag. Scale-down doesn’t trigger an immediate scale-up cycle, creating instability. In practice, auto-scaling especially first scale event can take several minutes so you need to make sure that you have some extra capacity to handle increased load.

Volume Testing

Many performance characteristics change materially as data grows. Index scan times increase. Query planner behavior shifts. Cache hit rates drop as the working set outgrows cache size. Search latency that is acceptable at 50 million records may become unacceptable at 250 million. Test at your current production data volume, then at projected volumes for 1 and 3 years out. The time to address data growth challenges in architecture is before you’re already there.

Recovery Testing

Recovery testing applies an abnormal condition like a dependency failure, a network partition, a resource exhaustion event and measures how long the system takes to return to normal operation. The key questions: does the system recover at all? How long does recovery take? What’s the user-visible impact during the recovery window?

Handling Dependencies in Your Tests

One of the practical decisions in every load test is what to do about dependencies like external APIs, third-party services, internal microservices, payment processors, identity providers, email services, and so on. You have two approaches, and which one you choose depends on what your test is trying to answer.

Mock Dependencies When You’re Focused on Your Own Code

When your goal is to validate your application’s internal performance like memory footprint, CPU usage, throughput of your business logic, efficiency of your data access layer then mocking external dependencies is often the right call. However, you will need to build a well-designed mock that returns realistic response payloads with configurable latency. Mocking lets you:

• Isolate your application’s performance characteristics from the noise of external variability
• Simulate dependency failure modes (timeouts, errors, slow responses) in a controlled way
• Run tests without consuming third-party quotas or generating costs in external systems
• Reproduce specific latency profiles to understand how your code behaves under different dependency performance conditions

Include Real Dependencies When Integration Behavior Matters

When your goal is to validate end-to-end system behavior including the interaction effects between your system and its dependencies then you can use real dependencies or realistic stubs deployed under your control. The reason this matters: under load, dependencies behave differently than they do at idle. For example, higher latency in dependencies can propagate creating back-pressure in your system that a mock would never reveal. Dependencies that are slow, throttled, or unavailable under load can:

• Exhaust your connection pools (connections held open waiting for a slow response)
• Fill your request queues (new requests queueing behind slow in-flight requests)
• Trigger retry storms (your retry logic amplifying load on an already-struggling dependency)
• Surface timeout and circuit-breaker behavior that only activates under real latency conditions

If you include real third-party services in your load test, be explicit about two things: you may consume quota and generate costs, and their performance becomes part of your results. When a dependency is slow, it appears as latency in your own metrics — know what you’re measuring.

A practical middle ground: deploy internal stubs for your external dependencies. A stub is a service you control that returns realistic responses with configurable behavior. Unlike a mock in a test harness, a stub runs as a real service and participates in your actual network topology. It lets you test realistic integration behavior without the unpredictability or cost of real external services.

Watch for Automatic Retry Amplification

Another factor that can skew results from performance testing is automated retries at various layers when a request fails or times out. Under load, this multiplies traffic. If your application generates 400 write operations per second against a dependency, and that dependency starts returning errors, your client may retry each failed request two or three times and suddenly generating 800 to 1,200 operations per second against an already-struggling system. In your load tests, verify that your retry behavior is bounded and doesn’t turn a manageable degradation into a cascading failure. Exponential backoff with jitter, retry budgets, and circuit breakers all exist to prevent this.

Design Your Load Model

Before writing a test script, model the load you intend to generate. A poorly designed load model produces results that feel meaningful but don’t correspond to anything real.

Use Production Traffic Patterns as Your Starting Point

Study your actual production metrics. Identify:

• Average requests per second across a normal operating period
• Peak requests per second during your highest-traffic periods
• Request distribution across endpoints: what percentage of traffic hits each API? Most services have a small number of high-traffic endpoints and many low-traffic ones.
• Read/write ratio: most production services are read-heavy; your load model should reflect that
• Payload characteristics — average request and response sizes
• User session behavior: are users authenticated? Do requests carry session state? Do later requests in a workflow depend on earlier ones?
• Geographic distribution: does your traffic come from one region or many?

Use Stepped Load Progression

Ramp load gradually rather than jumping to peak immediately. A stepped approach produces distinct data points at each level, making it easier to identify where behavior changes.

Hold each step long enough for metrics to stabilize and for any auto-scaling events to complete. If your auto-scaling policy triggers after 5 minutes of sustained high CPU, your steps need to run for at least 7-10 minutes. Steps that are too short produce transient data that doesn’t represent steady-state behavior.

Model Think Time

Real users don’t send requests as fast as possible. They read pages, fill forms, wait for results, and make decisions. Think time like the pause between user actions should be randomized within a realistic range based on observed production behavior. Omitting think time concentrates load artificially, inflates concurrency counts, and produces results that don’t correspond to real user behavior.

Model Transaction Workflows, Not Just Endpoints

A user doesn’t hit /api/checkout. They authenticate, browse products, add items to a cart, enter payment details, and confirm an order. Each step depends on the previous step and carries state forward. Test complete workflows. Measure the whole transaction, not just individual request latency. This reveals which step in the workflow breaks first under load, which is your actual bottleneck. For transactional workflows, count the full transaction as your unit of measurement, not individual requests. A checkout that takes 12 requests and completes in 3 seconds is different from one that requires 12 requests and only completes 60% of the time under load.

The Test Environment

Your test environment is the single largest source of invalid load test results. Get this wrong and every metric, analysis, and conclusion downstream becomes unreliable.

Match Production Infrastructure

The test environment should match production in:

• Instance types, sizes, and counts
• Database configuration: connection pool size, cache allocation, index configuration, replica count
• Caching layers and their sizes (this is a common miss a cache sized to 10% of production will warm and evict very differently)
• Auto-scaling configuration and thresholds
• Load balancer and network configuration
• All service configurations that affect throughput or latency

Pay particular attention to cache sizes. Under-sized caches in test environments produce unrealistically high cache miss rates, which increases database load and makes your results look worse than production will be. Over-sized caches make things look better.

Use Representative Data Volumes

Test environments with small datasets produce misleading results. A database with 1 million rows behaves differently from one with 100 million rows in ways that are significant and non-linear. Index performance, query planner behavior, partition routing, and cache hit rates all change with data volume. Populate your test environment with data that reflects realistic production scale before running meaningful performance tests.

Isolate the Test Environment Completely

I have seen a load test takes down production environment because it shared a common infrastructure. A test environment that shares any infrastructure with production like databases, message queues, caching clusters, network paths, logging infrastructure creates two simultaneous problems: invalid test results (because production traffic contaminates your measurements) and potential production incidents (because your load test contaminates production systems). Shared test environments that connect to production Messaging bus, Kafka, or database clusters have caused outages. Enforce complete isolation.

Account for Test Data Accumulation

Load tests generate real data. After many test runs, your test database accumulates records, logs grow, and storage fills. Plan your test data lifecycle from the start, e.g., how you populate data before tests, whether you clean up between runs, and how you prevent accumulated test data from affecting your test environment’s performance over time.

Document Your Environment Specification

Version-control your environment definition alongside your test scripts. When you compare results across time, you need to know that what changed was the system under test, not the test environment. An environment specification that exists only in someone’s memory cannot be reproduced reliably.

Metrics: Collect the Right Things

Load testing generates a lot of data. The teams that extract the most value don’t collect more metrics, they collect the right metrics and actually analyze them.

Latency

Track percentiles, not averages. Averages hide tail behavior that determines user experience.

• P50 — what the median user experiences
• P90 — your common-case ceiling; nine in ten requests complete within this
• P99 — your near-worst case; one in a hundred users waits this long
• P99.9 — your extreme tail; relevant for high-volume services where 0.1% is still thousands of users

The gap between P50 and P99.9 tells you about consistency. A wide gap means some users experience good performance while others experience unacceptable degradation. Systems under load often hold P50 steady while P99 climbs.

Throughput

• Requests per second: raw system throughput
• Successful transactions per second: throughput filtered by correctness; throughput with a 20% error rate is not good throughput
• Throughput per resource unit: requests per CPU core, per GB of memory helps with capacity planning

Error Rates

• Fault rate — server-side failures (5xx responses)
• Error rate — client rejections, throttled requests, timeouts
• Error distribution — which specific errors, at what load levels

Don’t aggregate errors into a single rate. A 2% error rate composed entirely of timeouts tells you something different from a 2% error rate of connection refused responses. Decompose your error data and correlate specific error types with the load levels at which they appear.

Resource Utilization

Collect these for every component like application servers, databases, caches, message queues, load balancers, and load generators:

• CPU: overall and per-core; watch for single-threaded bottlenecks where overall CPU looks fine but one core is maxed
• Memory: heap usage, GC frequency and pause duration, swap usage; track memory over time in soak tests to detect leaks
• Disk I/O: read and write throughput, queue depth, utilization percentage; relevant for databases and any service that writes logs or temp files
• Network I/O: ingress and egress bytes per second, connection counts, dropped packets
• Thread and connection pool utilization: active threads, queued requests, pool exhaustion events

Application-Level Metrics

• Cache hit/miss/eviction rates: degrading hit rates under load reveal cache sizing or key distribution problems
• Queue depths: growing queues indicate consumers can’t keep pace with producers
• Database connection pool saturation: one of the most common failure modes under load
• GC pause duration and frequency: GC pressure under load causes latency spikes that don’t show up in CPU metrics directly
• Retry rates: high retry rates indicate a dependency is struggling, and may be amplifying load
• Circuit breaker state: how often circuit breakers open under load, and what triggers them

Dependency-Level Metrics

When you include real dependencies in your test, monitor them as carefully as your own service:

• Response latency from each dependency (P50, P99)
• Error rates from each dependency
• Dependency-side resource utilization (if you have access)
• Message bus ingress and egress (if applicable)
• Partition utilization for distributed storage systems

When a dependency is slow or erroring, that signal propagates through your system as elevated latency and errors in your own metrics. You need dependency-level metrics to trace the source.

Availability

Define availability targets before testing:

• Service availability: percentage of requests that succeed
• Per-endpoint availability: some endpoints degrade before others; measure them independently
• Dependency availability: availability of each system your service calls

Business-Level Metrics

The most important metrics are often furthest from the infrastructure:

• Orders completed per minute
• Successful authentication rate
• Payment processing completion rate
• Data write confirmation rate

Infrastructure metrics tell you what the system is doing. Business metrics tell you what users are experiencing. A system where P99 latency stays within SLA but checkout completion drops 15% under load has a problem that infrastructure metrics alone won’t reveal clearly.

Tools

Over the years, I have used various commercial and open source tools like LoadRunner, Grinder, Tsung, etc. that are no longer well maintained. Here are common tools that can be used for load testing:

For Simple Endpoint Testing

• ab (Apache Bench) and Hey: Command-line tools that generate load against a single endpoint. No scripting required, fast to start.
• Vegeta: Generates load at a constant request rate, independent of server response time. This distinction matters: when your server responds slowly, most tools automatically reduce request rate. Vegeta maintains the configured rate as latency climbs, which means you observe back-pressure and degradation accurately.

echo "GET https://api.example.com/users/123" | vegeta attack -rate=500 -duration=60s | vegeta report

• k6: Scripted in JavaScript, distributed as a single Go binary. k6 handles multi-step scenarios natively, supports parameterized test data, models think time, and exposes rich built-in metrics. It integrates with Prometheus, CloudWatch, and Grafana for analysis, and supports threshold-based pass/fail in CI pipelines.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp to 100 users
    { duration: '5m', target: 100 },   // hold at average load
    { duration: '2m', target: 500 },   // ramp to peak
    { duration: '5m', target: 500 },   // hold at peak
    { duration: '1m', target: 1000 },  // spike
    { duration: '2m', target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(99)<500'],  // 99% of requests under 500ms
    http_req_failed: ['rate<0.01'],    // less than 1% error rate
  },
};

export default function () {
  // Step 1: authenticate
  const loginRes = http.post('https://api.example.com/auth/login', {
    username: `user_${__VU % 10000}@example.com`,
    password: 'password',
  });
  check(loginRes, { 'login succeeded': (r) => r.status === 200 });

  const token = loginRes.json('token');

  // Step 2: fetch catalog (read operation)
  const catalogRes = http.get('https://api.example.com/catalog?page=1', {
    headers: { Authorization: `Bearer ${token}` },
  });
  check(catalogRes, { 'catalog loaded': (r) => r.status === 200 });

  sleep(Math.random() * 3 + 1); // think time: 1-4 seconds

  // Step 3: place order (write operation)
  const orderRes = http.post(
    'https://api.example.com/orders',
    JSON.stringify({ item_id: Math.floor(Math.random() * 1000), quantity: 1 }),
    { headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' } }
  );
  check(orderRes, { 'order placed': (r) => r.status === 201 });

  sleep(Math.random() * 2 + 1);
}

• Apache JMeter: Meter supports complex scenarios through a GUI, handles correlation between requests, has a broad plugin ecosystem, and has extensive enterprise adoption.
• Locust: Pure Python, code-defined test scenarios (not XML), a built-in web UI for real-time monitoring, distributed mode via a controller/worker model, and trivially scriptable.

For Distributed Load Generation

AWS Distributed Load Testing: When a single machine can’t generate the volume you need, this solution orchestrates load across multiple instances, accepts JMeter scripts as the test definition, and streams results to time-series storage for analysis. Use it when your bandwidth or TPS requirements exceed what a single load generator can produce.

For Observability During Tests

You can use following monitoring stack to gather performance metrics:

• Prometheus + Grafana: commonly used for infrastructure and application metrics; k6 exports directly to Prometheus
• CloudWatch: native AWS monitoring; integrates with most AWS services and many load testing tools
• Distributed tracing (Jaeger, Zipkin, AWS X-Ray): essential for understanding latency in distributed systems; propagate correlation IDs through every service boundary so you can trace a slow request to the specific component that caused it

Without distributed tracing, diagnosing latency in a multi-service system under load is largely guesswork.

Execution

• Warm Up Before Measuring: JIT compilation, connection pool initialization, cache population, and DNS resolution all affect early request latency. Build a ramp-up period into every test. Discard metrics from the warmup phase. Measure steady-state behavior only.
• Verify Your Load Generator Isn’t the Bottleneck: Before trusting any results, confirm: load generator CPU stays well below saturation (under 70%), network I/O doesn’t approach the bandwidth ceiling, and the tool achieves the TPS you configured not a lower number due to local resource constraints. If you configure 1,000 TPS but the generator only achieves 600, your results reflect the generator’s limits, not your system’s.
• Notify Dependent Teams Before Testing: If your test environment shares any infrastructure with other teams, notify them before running high-volume tests. Unexpected load from your tests against a shared component (a database, a message bus, a routing layer) can cause problems for teams who have no idea a load test is running.
• Run Each Scenario in Isolation First: Test each scenario independently before running combinations. An isolated test that reveals a problem gives you more diagnostic information than a combined test that reveals the same problem buried in noise from other scenarios.
• Don’t Overwrite Previous Results: Each test run should write to a new, timestamped output file. Overwriting results from a previous run is a common mistake when running iterative tests in a loop. You lose the ability to compare across runs.
• Pause Between Runs: Allow the system to fully drain between test iterations like connections close, queues clear, resource utilization returns to baseline. Residual load from one run contaminates the starting conditions of the next.

Common Pitfalls

• Testing a single endpoint and calling it done. A service’s behavior under load isn’t determined by any single endpoint. Test complete workflows, including the paths that matter most to users.
• Ignoring dependencies. When your dependencies are slow or unavailable, your service appears slow. When your service hammers a dependency with load, the dependency may degrade and create a feedback loop. Model dependency behavior explicitly and mock it when you want to isolate your own code, use real or realistic stubs when integration behavior matters.
• Mismatch between test environment and production. Different hardware, different cache sizes, different connection pool limits, different network latency profiles, any of these make test results non-transferable to production. Document your environment specification. Validate that it matches production before trusting results.
• Small data volumes. A test environment with 1% of production data volume produces optimistic results. Populate test data to realistic scale.
• Running load tests once. Performance characteristics change with every code change, every dependency upgrade, and every growth milestone. A load test you ran six months ago tells you about a system that no longer exists.
• Ignoring ramp-down. Verify that resource utilization returns to baseline after load subsides. A system that doesn’t recover cleanly carries forward pressure that degrades subsequent traffic.
• Not collecting metrics from all layers. Application-level metrics without infrastructure metrics leave you guessing about root cause. Infrastructure metrics without application or business-level metrics leave you unable to quantify user impact. Collect all three.
• Stopping tests when something goes wrong instead of analyzing the failure mode. When a stress test surfaces a failure, that’s the point. Note what failed, under what conditions, and how the system behaved. Stopping the test immediately loses the degradation data that tells you whether the failure mode is safe or catastrophic.

Analysis

• Establish a Baseline Before Comparing Anything: Every metric needs a reference point. P99 latency of 300ms is good or bad depending entirely on what P99 looks like at baseline load. Run a baseline test with minimal concurrent users before escalating. Capture that baseline explicitly. Compare every subsequent measurement against it.
• Separate Signal from Noise: A single high-latency data point is noise. A systematic increase in P99 as concurrency crosses 500 users is signal. Look for the pattern: where does behavior change? At what load level? After what duration? What resource metric correlates with the change?
• Trace Latency to Its Source: When you observe elevated latency, resist looking first at application CPU. Latency accumulates in many places: network round trips between services, database query execution, lock contention, GC pause accumulation, connection pool queuing, and downstream dependency latency. Distributed tracing lets you follow a slow request through every component it touched and attribute the latency precisely. Fix the actual source, not the nearest visible symptom.
• Investigate Unexpectedly Good Results: If your system performs better than expected under load, investigate before celebrating. Unexpected improvement often means your test isn’t exercising the paths you intended such as caches warming too aggressively, load not reaching the components you think it is, or test data creating unrealistic access patterns. Results you can’t explain aren’t results you can rely on.
• Generate Comparative Reports: A report listing numbers has limited value. A report comparing those numbers to your baseline, to your previous test run, and to your defined thresholds has significant value. For each metric, capture:
  – Current result
  – Baseline value
  – SLA or target threshold
  – Previous test result (regression or improvement?)
  – Load level at which the metric was captured

Store test results in a queryable format over time.

Building a Continuous Performance Practice

The teams with the most reliable services don’t treat performance testing as a project. They treat it as a discipline with regular cadence.

• Define performance goals and revisit them annually. Goals should include throughput targets, latency percentiles, error rate limits, resource utilization ceilings, and headroom targets (how much capacity should remain available at peak). As your traffic patterns change, your service evolves, and your SLAs tighten, these goals need updating.
• Automate pass/fail thresholds in CI. Encode your performance targets as pipeline gates. A change that increases P99 latency by 40% under load should fail the build, the same way a change that breaks a unit test fails the build.
• Run performance canaries in production. Continuously exercise production endpoints at low volume from monitoring infrastructure. Track latency, error rates, and throughput over time. Detect gradual degradation before users do.
• Assign a performance owner on each service team. Performance improvements don’t happen without someone watching the metrics, reviewing throttling rules, identifying regressions, and driving improvements.
• Review results across time for patterns. Look at all your load test results over the past quarter. Which metrics trend in the wrong direction? Which components appear repeatedly in bottleneck analysis? Patterns across multiple tests reveal systemic issues that any individual test misses.
• Share what you learn. Performance problems and their solutions are valuable organizational knowledge. Document them. Share them across teams. The team dealing with connection pool exhaustion today is probably not the first team to hit that issue.

The Pre-Test Checklist

Before any load test:

• [ ] Test objectives and pass/fail thresholds defined in writing before execution
• [ ] Test environment completely isolated from production
• [ ] Test environment infrastructure matches production configuration (instance types, cache sizes, connection pools, scaling settings)
• [ ] Test data populated to realistic production scale
• [ ] Dependent services decided: mock, stub, or real — with rationale documented
• [ ] Monitoring dashboards active for all components, including load generators
• [ ] Dependent team on-call contacts notified
• [ ] Output file naming prevents overwrites between iterations
• [ ] Previous test results available for comparison

During execution:

• [ ] Baseline captured before escalating load
• [ ] Load generator resource utilization verified (not the bottleneck)
• [ ] Error rates monitored in real time — abnormal errors trigger a pause for investigation
• [ ] Each step held long enough for metrics to stabilize
• [ ] Auto-scaling events logged with timestamps

After execution:

• [ ] Results compared to defined thresholds and previous runs
• [ ] Anomalies investigated before conclusions are drawn
• [ ] Root cause documented for any threshold violations
• [ ] Action items assigned with owners and deadlines
• [ ] Test results stored in versioned, queryable storage
• [ ] Environment cleanup completed (test data, log files, temporary resources)

Putting It Together

Any code change can affect performance. A dependency upgrade, a new index, a configuration tweak, a framework version bump — all of these can change memory footprint, CPU usage, throughput, and latency in ways that don’t appear until you run real load. The only reliable way to catch these changes before they affect users is to make performance testing a routine part of how you build and ship software, not something you do once before a big launch.

Start with profiling to understand where time and memory go in your own code. Add load tests to your CI pipeline to catch regressions early. Run soak tests to find memory and connection leaks. Stress test to 10x your expected peak so you know what your ceiling looks like and how you fail when you hit it. Test with real dependency behavior when integration effects matter, and mock dependencies when you want to isolate your own code.

Collect metrics at every layer such as application, infrastructure, and business so you can connect a latency spike to its root cause and quantify its user impact. Store results over time so you can detect gradual regressions before they become incidents. The goal is to know your system well enough that production behavior matches what you measured in testing.

Over the years, I have watched distributed services evolve through phases I lived through personally such as CORBA, EJB, SOA, REST microservices, containers WebAssembly feels different. It compiles code from any language into a universal binary format, runs it in a sandboxed environment, and delivers near-native performance without containers or language-specific runtimes cluttering your production stack.

When I built PlexSpaces for serverless FaaS applications, I designed its polyglot layer on top of WebAssembly and the WASI Component Model. It allows you to write actors in Python, Rust, Go, or TypeScript, compile them to WASM, and deploy them to the same runtime. The framework handles persistence, fault tolerance, supervision, and scaling regardless of programming language. In this post, I’ll walk you through the core WebAssembly concepts, show how PlexSpaces leverages the Component Model for polyglot development, and demonstrate building, testing, and deploying applications in all four languages. I’ll also show a PlexSpaces Application Server model that lets you deploy entire application bundleslike deploying a WAR file to Tomcat, but with the fault-tolerance of Erlang/OTP built in.

WebAssembly Introduction

WebAssembly launched in 2017 as a browser technology. I ignored it for years — client-side JavaScript ecosystem drama wasn’t something I wanted to track. The server-side story changed everything.

How WebAssembly Executes Code

WebAssembly is a stack-based virtual machine that executes a compact binary instruction format. Every language that compiles to WASM follows the same pipeline:

The WASM binary format encodes typed functions, a linear memory model, and a set of imports and exports. The runtime validates the binary at load time, then executes it using either just-in-time (JIT) compilation or ahead-of-time (AOT) compilation to native machine code. Three properties make this execution model powerful for distributed systems:

Deterministic execution. Given the same inputs, a WASM module produces the same outputs. This property underpins PlexSpaces’ durable execution, which replays journaled messages through the same WASM binary and arrives at the exact same state.

Memory isolation. Each WASM instance gets its own linear memory. One module cannot read, write, or corrupt another module’s memory. No shared-memory race conditions and buffer overflows escaping the sandbox. The runtime enforces these boundaries at the hardware level.

Capability-based security. A WASM module starts with zero capabilities. It cannot access the filesystem, the network, or even a clock unless the host explicitly provides each capability through imported functions. PlexSpaces grants actors exactly the capabilities they need like messaging, key-value storage, tuple spaces.

The Component Model

Early WebAssembly only understood numbers. You passed integers and floats across the boundary, and that was it. The WebAssembly Component Model fixes this limitation by defining rich, typed interfaces that components use to communicate. You can think of it as an IDL (Interface Definition Language) for WASM but one that works across every language. The key building blocks:

• WIT (WebAssembly Interface Types): A language for defining typed function signatures across components. A function defined in WIT can accept strings, records, lists, variants, and enums. WIT bridges the type systems of Rust, Python, Go, and TypeScript into a single, shared contract.
• Components: Self-contained WASM modules that declare their imports (what they need from the host) and exports (what they provide). A Rust component and a Python component that implement the same WIT interface become interchangeable at the binary level.
• WASI (WebAssembly System Interface): The standardized API that gives WASM modules access to system resources like file I/O, networking, clocks, and random number generation within the sandbox. WASI Preview 2 shipped in 2024 with HTTP, filesystem, and socket support. WASI 0.3, released in February 2026, added native async support for concurrent I/O.

Wasm 3.0 and WasmGC

The WebAssembly ecosystem crossed a critical threshold. Wasm 3.0 became the W3C standard in 2025, standardizing nine production features in a single release:

• WasmGC: garbage collection support built into the runtime, eliminating the need for languages like Go, Python, and Java to ship their own GC inside the WASM binary. This shrinks binary sizes and improves performance for GC-dependent languages dramatically.
• Exception handling: structured try/catch at the WASM level, replacing the expensive setjmp/longjmp workarounds that inflated binaries.
• Tail calls: proper tail call optimization for functional programming patterns without stack overflow.
• SIMD (Single Instruction, Multiple Data): vector operations for parallel numeric computation, critical for ML inference and scientific workloads.

For PlexSpaces, WasmGC means Go and Python actors run faster with smaller binaries. SIMD means computational actors like n-body simulations, matrix multiplies, genomics pipelines that process data at near-native throughput inside the sandbox.

What This Means in Practice

You compile a Python actor and a Rust actor to WASM. Both implement the same WIT interface. The runtime loads them identically, calls the same exported functions, and provides the same host capabilities like messaging, key-value storage, tuple spaces, distributed locks. The Python actor handles ML inference; the Rust actor handles high-throughput event processing. They communicate through PlexSpaces message passing without knowing or caring which language sits on the other side.

This is not “Write Once, Run Anywhere” in the old Java sense. This is “Write in Whatever Language Fits, Run Together on the Same Runtime.”

How PlexSpaces Makes It Work

PlexSpaces is a unified distributed actor framework that combines patterns from Erlang/OTP, Orleans, Temporal, and modern serverless architectures into a single abstraction. I described the five foundational pillars in my earlier post: TupleSpace coordination, Erlang/OTP supervision, durable execution, WASM runtime, and Firecracker isolation. Here I focus on the WASM layer and how it enables polyglot development.

Architecture at a Glance

The WIT Contract for Actor

Every actor regardless of source language targets the same WIT world. Here is the simplified world that most polyglot actors use:

// wit/plexspaces-simple-actor/world.wit
package plexspaces:simple-actor@0.1.0;

interface actor {
    // Initialize with JSON config string
    init: func(config-json: string) -> string;

    // Handle a message: route by msg-type, return JSON result
    handle: func(from-actor: string, msg-type: string,
                 payload-json: string) -> string;

    // Snapshot state for persistence
    get-state: func() -> string;

    // Restore state from snapshot
    set-state: func(state-json: string) -> string;
}

interface host {
    // Messaging
    send: func(to: string, msg-type: string, payload-json: string) -> string;
    ask: func(to: string, msg-type: string, payload-json: string,
              timeout-ms: u64) -> string;
    spawn: func(module-ref: string, actor-id: string,
                init-config-json: string) -> string;
    stop: func(actor-id: string) -> string;
    self-id: func() -> string;

    // Erlang/OTP-style linking and monitoring
    link: func(actor-id: string) -> string;
    monitor: func(actor-id: string) -> string;

    // Timers
    send-after: func(delay-ms: u64, msg-type: string,
                     payload-json: string) -> string;

    // Process groups
    pg-join: func(group-name: string) -> string;
    pg-broadcast: func(group-name: string, msg-type: string,
                       payload-json: string) -> string;

    // Key-value store
    kv-get: func(key: string) -> string;
    kv-put: func(key: string, value: string) -> string;
    kv-delete: func(key: string) -> string;
    kv-list: func(prefix: string) -> string;

    // TupleSpace (Linda-style coordination)
    ts-write: func(tuple-json: string) -> string;
    ts-read: func(pattern-json: string) -> string;
    ts-take: func(pattern-json: string) -> string;
    ts-read-all: func(pattern-json: string) -> string;

    // Distributed locks
    lock-acquire: func(tenant-id: string, namespace: string,
                       holder-id: string, lock-name: string,
                       lease-duration-secs: u32, timeout-ms: u64) -> string;
    lock-release: func(lock-id: string, tenant-id: string,
                       namespace: string, holder-id: string,
                       lock-version: string) -> string;

    // Blob storage
    blob-upload: func(blob-id: string, data: string,
                      content-type: string) -> string;
    blob-download: func(blob-id: string) -> string;

    // Logging and time
    log: func(level: string, message: string);
    now-ms: func() -> u64;
}

world actor-world {
    import host;
    export actor;
}

The full-featured actor package adds dedicated WIT interfaces for workflows, channels, durability/journaling, registry/service discovery, HTTP client, and cron scheduling. PlexSpaces also defines specialized worlds that import only the capabilities each actor needs:

This design keeps WASM binaries small. A simple actor that only needs messaging imports two interfaces not thirteen.

Language Toolchains

Each language uses a different compiler to produce WASM, but the output targets the same runtime:

Now let’s build something real in each language.

Getting Started

Before diving into the language examples, set up your development environment.

Prerequisites

• Rust 1.70+ (for building PlexSpaces itself)
• Docker (optional — for the fastest path to a running node)
• One or more WASM compilers for your target languages (see below)

Option 1: Docker Quickstart

Pull and run a PlexSpaces node in seconds:

# Pull the official image
docker pull plexobject/plexspaces:latest

# Run a single node with HTTP API on port 8001
docker run -d \
    --name plexspaces-node \
    -p 8000:8000 \
    -p 8001:8001 \
    -e PLEXSPACES_NODE_ID=node1 \
    -e PLEXSPACES_DISABLE_AUTH=1 \
    plexobject/plexspaces:latest

The node exposes a gRPC endpoint on port 8000 and an HTTP/REST gateway on port 8001 with interactive Swagger UI documentation.

Option 2: Build from Source

git clone https://github.com/bhatti/PlexSpaces.git
cd PlexSpaces

./scripts/server.sh

# Or use the Makefile step by step
make build            # Build all crates
make test             # Run all tests

Install Language Compilers

Install the WASM compiler for each language you plan to use:

# Rust (produces the smallest, fastest WASM)
rustup target add wasm32-wasip2

# Go (pragmatic balance of performance and dev speed)
# macOS:
brew install tinygo
# Also need wasm-tools for component creation:
cargo install wasm-tools

# TypeScript (rapid development, web ecosystem)
npm install -g @bytecodealliance/jco

# Python (ML, data processing, prototyping)
pip install componentize-py

# Optional: WASM binary optimizer (shrinks binaries further)
cargo install wasm-opt

Start the Node and Deploy Your First Actor

# Start a PlexSpaces node (from source)
cargo run --release --bin plexspaces -- start \
    --node-id dev-node \
    --listen-addr 0.0.0.0:8000 \
    --release-config release-config.toml


# Deploy a WASM actor (from any language)
curl -X POST http://localhost:8001/api/v1/applications/deploy \
    -F "application_id=my-app" \
    -F "name=my-actor" \
    -F "version=1.0.0" \
    -F "wasm_file=@my_actor.wasm"

# Send it a message
curl -X POST http://localhost:8001/api/v1/actors/my-app/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "hello", "payload": {}}'

Now let’s build real actors in each language.

Python: A Calculator Actor with the SDK

Python shines for rapid prototyping and data-heavy workloads. The PlexSpaces Python SDK uses decorators (@actor, @handler, state()) that eliminate boilerplate and let you focus on business logic.

The Actor Code

# calculator_actor.py
from plexspaces import actor, state, handler, init_handler


@actor
class Calculator:
    """Calculator actor implementing basic math operations."""

    # Persistent state fields -- survive crashes via journaling
    last_operation: str = state(default=None)
    last_result: float = state(default=None)
    history: list = state(default_factory=list)

    @init_handler
    def on_init(self, config: dict):
        """Initialize calculator with optional config."""
        if "state" in config:
            saved = config["state"]
            self.last_operation = saved.get("last_operation")
            self.last_result = saved.get("last_result")
            self.history = saved.get("history", [])

    @handler("add")
    def add(self, operands: list = None) -> dict:
        """Add operands together."""
        result = sum(operands or [])
        self._record("add", operands, result)
        return {"result": result, "operation": "add"}

    @handler("subtract")
    def subtract(self, operands: list = None) -> dict:
        """Subtract: first operand minus rest."""
        if not operands or len(operands) < 2:
            return {"error": "Subtract requires at least 2 operands"}
        result = operands[0] - sum(operands[1:])
        self._record("subtract", operands, result)
        return {"result": result, "operation": "subtract"}

    @handler("multiply")
    def multiply(self, operands: list = None) -> dict:
        """Multiply all operands."""
        result = 1
        for op in (operands or []):
            result *= op
        self._record("multiply", operands, result)
        return {"result": result, "operation": "multiply"}

    @handler("divide")
    def divide(self, operands: list = None) -> dict:
        """Divide first operand by second."""
        if not operands or len(operands) < 2:
            return {"error": "Divide requires 2 operands"}
        if operands[1] == 0:
            return {"error": "Division by zero"}
        result = operands[0] / operands[1]
        self._record("divide", operands, result)
        return {"result": result, "operation": "divide"}

    @handler("get_history")
    def get_history(self) -> dict:
        """Return calculation history."""
        return {"history": self.history}

    @handler("call", "get_state")
    def get_state_handler(self) -> dict:
        """Snapshot current state."""
        return {
            "last_operation": self.last_operation,
            "last_result": self.last_result,
            "history": self.history,
        }

    def _record(self, operation, operands, result):
        self.last_operation = operation
        self.last_result = result
        self.history.append({
            "operation": operation,
            "operands": operands,
            "result": result,
        })

Notice how the @actor decorator marks the class, state() declares persistent fields that survive crashes, and each @handler("operation") routes incoming messages to the right method. The SDK handles WIT serialization, state checkpointing, and all the plumbing underneath.

Build and Deploy

# Install the Python SDK
pip install -e "sdks/python/[dev]"

# Build WASM using the SDK CLI
plexspaces-py build calculator_actor.py \
    -o calculator_actor.wasm \
    --wit-dir wit/plexspaces-simple-actor

# Deploy the WASM module
curl -X POST http://localhost:8001/api/v1/applications/deploy \
    -F "application_id=calculator-app" \
    -F "name=calculator" \
    -F "version=1.0.0" \
    -F "wasm_file=@calculator_actor.wasm"

Send a Request

curl -X POST http://localhost:8001/api/v1/actors/calculator-app/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "add", "payload": {"operands": [10, 20, 30]}}'

# Response: {"result": 60, "operation": "add"}

The actor processes the request, updates its persistent state, and returns the result. If the node crashes and restarts, the framework replays the journal and restores the calculator’s state .

TypeScript: A Bank Account with Durable State

TypeScript brings type safety and rapid development. The PlexSpaces TypeScript SDK uses an inheritance-based pattern: extend PlexSpacesActor, implement on<Operation>() handlers, and the SDK wires everything to WIT.

The Actor Code

// account_actor.ts
import { PlexSpacesActor } from "@plexspaces/sdk";

interface Transaction {
  type: string;
  amount: number;
  balance_after: number;
}

interface BankAccountState {
  account_id: string;
  balance: number;
  transactions: Transaction[];
}

export class BankAccountActor extends PlexSpacesActor<BankAccountState> {
  getDefaultState(): BankAccountState {
    return { account_id: "", balance: 0, transactions: [] };
  }

  protected override onInit(config: Record<string, unknown>): void {
    this.state.account_id = String(config.account_id ?? "");
    this.state.balance = 0;
    this.state.transactions = [];
  }

  onDeposit(payload: Record<string, unknown>): Record<string, unknown> {
    const amount = Number(payload.amount ?? 0);
    if (amount <= 0) return { error: "invalid_amount" };
    this.state.balance += amount;
    this.state.transactions.push({
      type: "deposit", amount, balance_after: this.state.balance,
    });
    return { status: "ok", balance: this.state.balance };
  }

  onWithdraw(payload: Record<string, unknown>): Record<string, unknown> {
    const amount = Number(payload.amount ?? 0);
    if (amount <= 0) return { error: "invalid_amount" };
    if (amount > this.state.balance) {
      return { error: "insufficient_funds", balance: this.state.balance };
    }
    this.state.balance -= amount;
    this.state.transactions.push({
      type: "withdraw", amount, balance_after: this.state.balance,
    });
    return { status: "ok", balance: this.state.balance };
  }

  onHistory(payload: Record<string, unknown>): Record<string, unknown> {
    const count = Math.min(
      Number(payload.count ?? 5), this.state.transactions.length
    );
    return { transactions: this.state.transactions.slice(-count) };
  }

  onReplay(): Record<string, unknown> {
    let rebuilt = 0;
    for (const tx of this.state.transactions) {
      if (tx.type === "deposit") rebuilt += tx.amount;
      else if (tx.type === "withdraw") rebuilt -= tx.amount;
    }
    return {
      replayed: this.state.transactions.length,
      rebuilt_balance: rebuilt,
      current_balance: this.state.balance,
    };
  }
}

// WIT actor export -- bridges TypeScript class to the WIT interface
const instance = new BankAccountActor();
export const actor = {
  init: (c: string) => instance.init(c),
  handle: (from: string, msg: string, payload: string) =>
    instance.handle(from, msg, payload),
  getState: () => instance.getState(),
  setState: (s: string) => instance.setState(s),
};

The BankAccountActor manages deposits, withdrawals, and transaction history with full durability. The onReplay() handler rebuilds the balance from the transaction log, demonstrating event-sourcing patterns that the framework makes trivial.

Build and Deploy

The TypeScript build uses a three-step pipeline: compile TypeScript, bundle with esbuild, then create a WASM component with jco:

# Install dependencies (SDK is a file: dependency)
npm install

# Compile TypeScript -> JavaScript -> ESM bundle -> WASM component
npm run build              # tsc + esbuild bundle
jco componentize actor_bundle.mjs \
    --wit wit/plexspaces-simple-actor \
    -o account_actor.wasm \
    --disable all

# Deploy to PlexSpaces
curl -X POST http://localhost:8001/api/v1/applications/deploy \
    -F "application_id=bank-app" \
    -F "name=bank-account" \
    -F "version=1.0.0" \
    -F "wasm_file=@account_actor.wasm"

Interact with the Accounts

# Deposit into Alice's account
curl -X POST http://localhost:8001/api/v1/actors/account-alice/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "deposit", "payload": {"amount": 1000}}'
# Response: {"status": "ok", "balance": 1000}

# Withdraw from Alice's account
curl -X POST http://localhost:8001/api/v1/actors/account-alice/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "withdraw", "payload": {"amount": 250}}'
# Response: {"status": "ok", "balance": 750}

# Check transaction history
curl -X POST http://localhost:8001/api/v1/actors/account-alice/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "history", "payload": {"count": 10}}'

Go: An Erlang OTP-Style Rate Limiter

Go delivers a pragmatic balance between performance and developer productivity. The PlexSpaces Go SDK uses an interface-based pattern: implement the Actor interface, embed BaseActor for automatic state serialization, and register your actor for WASM export via plexspaces.Register().

The Actor Code

This example implements a sliding-window rate limiter, the kind you find inside API gateways like NGINX, Kong, or Envoy. Each client gets an independent window with configurable limits:

// rate_limiter.go
package main

import (
    "encoding/json"
    "fmt"
    "github.com/plexobject/plexspaces/sdks/go/plexspaces"
)

type SlidingWindowLimiter struct {
    plexspaces.BaseActor

    WindowSizeMs uint64                    `json:"window_size_ms"`
    MaxRequests  int                       `json:"max_requests"`
    Clients      map[string]*ClientWindow  `json:"clients"`
    TotalChecks  int                       `json:"total_checks"`
    TotalAllowed int                       `json:"total_allowed"`
    TotalDenied  int                       `json:"total_denied"`
}

type ClientWindow struct {
    Timestamps []uint64 `json:"timestamps"`
    Allowed    int      `json:"allowed"`
    Denied     int      `json:"denied"`
}

var host = plexspaces.NewHost()

func NewSlidingWindowLimiter() *SlidingWindowLimiter {
    a := &SlidingWindowLimiter{
        WindowSizeMs: 60000,
        MaxRequests:  100,
        Clients:      make(map[string]*ClientWindow),
    }
    a.SetSelf(a) // enables automatic JSON state serialization
    return a
}

func (s *SlidingWindowLimiter) Init(configJSON string) string {
    var config struct {
        ActorID string         `json:"actor_id"`
        Args    map[string]any `json:"args"`
    }
    json.Unmarshal([]byte(configJSON), &config)

    if args := config.Args; args != nil {
        if v, ok := args["window_size_ms"]; ok {
            s.WindowSizeMs = uint64(v.(float64))
        }
        if v, ok := args["max_requests"]; ok {
            s.MaxRequests = int(v.(float64))
        }
    }

    host.Info(fmt.Sprintf("RateLimiter: window=%dms, max=%d req/window",
        s.WindowSizeMs, s.MaxRequests))
    return ""
}

func (s *SlidingWindowLimiter) Handle(from, msgType, payloadJSON string) string {
    switch msgType {
    case "check_rate":
        return s.checkRate(payloadJSON)
    case "stats":
        return s.getStats()
    default:
        data, _ := json.Marshal(map[string]any{"error": "unknown: " + msgType})
        return string(data)
    }
}

func (s *SlidingWindowLimiter) checkRate(payloadJSON string) string {
    var req struct { ClientID string `json:"client_id"` }
    json.Unmarshal([]byte(payloadJSON), &req)

    window, exists := s.Clients[req.ClientID]
    if !exists {
        window = &ClientWindow{Timestamps: make([]uint64, 0)}
        s.Clients[req.ClientID] = window
    }

    now := host.NowMs()
    cutoff := now - s.WindowSizeMs

    // Slide the window: remove expired timestamps
    var active []uint64
    for _, ts := range window.Timestamps {
        if ts > cutoff { active = append(active, ts) }
    }
    window.Timestamps = active

    // Check the limit
    allowed := len(window.Timestamps) < s.MaxRequests
    if allowed {
        window.Timestamps = append(window.Timestamps, now)
        window.Allowed++; s.TotalAllowed++
    } else {
        window.Denied++; s.TotalDenied++
    }
    s.TotalChecks++

    remaining := s.MaxRequests - len(window.Timestamps)
    if remaining < 0 { remaining = 0 }

    data, _ := json.Marshal(map[string]any{
        "allowed": allowed, "remaining": remaining,
        "limit": s.MaxRequests, "client_id": req.ClientID,
    })
    return string(data)
}

// Register the actor for WASM export -- runs during _initialize,
// before the host calls any exported functions.
func init() {
    plexspaces.Register(NewSlidingWindowLimiter())
}

func main() {}

The Go SDK pattern uses plexspaces.NewHost() to access all host functions (messaging, KV, tuple space, etc.) and plexspaces.Register() in the init() function to wire the actor to the WASM export interface. The comparison to Erlang/OTP maps directly:

Build and Deploy

The Go build uses a three-step TinyGo pipeline: compile to core WASM, embed WIT metadata, then create a WASM component with a WASI adapter:

# Step 1: Compile Go to core WASM
tinygo build -target=wasi -o rate_limiter_core.wasm .

# Step 2: Embed WIT metadata
wasm-tools component embed wit/plexspaces-simple-actor \
    -w actor-world rate_limiter_core.wasm -o rate_limiter_embed.wasm

# Step 3: Create WASM component with WASI adapter
wasm-tools component new rate_limiter_embed.wasm \
    --adapt wasi_snapshot_preview1.reactor.wasm \
    -o rate_limiter.wasm

# Deploy
curl -X POST http://localhost:8001/api/v1/applications/deploy \
    -F "application_id=rate-limiter-app" \
    -F "name=rate-limiter" \
    -F "version=1.0.0" \
    -F "wasm_file=@rate_limiter.wasm"

Each Go example includes a build.sh script that automates this pipeline and resolves the WASI adapter automatically.

Test Rate Limiting

# Check if a client request is allowed
curl -X POST http://localhost:8001/api/v1/actors/rate-limiter/ask \
    -H "Content-Type: application/json" \
    -d '{"message_type": "check_rate", "payload": {"client_id": "api-client-1"}}'
# Response: {"allowed": true, "remaining": 99, "limit": 100, "client_id": "api-client-1"}

# After 100 requests within the window:
# Response: {"allowed": false, "remaining": 0, "limit": 100, "client_id": "api-client-1"}

Rust: A Calculator with Maximum Performance

Rust produces the smallest, fastest WASM binaries. When you need every microsecond like high-frequency trading, real-time event processing, computational pipelines. Rust actors deliver near-native performance with binary sizes under 1MB.

The Actor Code

This calculator uses #![no_std] to eliminate the standard library entirely, producing a tiny, self-contained WASM module:

// lib.rs
#![no_std]
extern crate alloc;

use alloc::vec::Vec;
use core::slice;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "PascalCase")]
pub enum Operation { Add, Subtract, Multiply, Divide }

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CalculatorState {
    calculation_count: u64,
    last_result: Option<f64>,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CalculationRequest {
    operation: Operation,
    operands: Vec<f64>,
}

static mut STATE: CalculatorState = CalculatorState {
    calculation_count: 0, last_result: None,
};

/// Initialize actor with optional persisted state
#[no_mangle]
pub extern "C" fn init(state_ptr: *const u8, state_len: usize) -> i32 {
    unsafe {
        if state_len == 0 {
            STATE = CalculatorState { calculation_count: 0, last_result: None };
            return 0;
        }
        let state_bytes = slice::from_raw_parts(state_ptr, state_len);
        match serde_json::from_slice::<CalculatorState>(state_bytes) {
            Ok(state) => { STATE = state; 0 }
            Err(_) => -1,
        }
    }
}

/// Handle incoming calculation requests
#[no_mangle]
pub extern "C" fn handle_message(
    _from_ptr: *const u8, _from_len: usize,
    type_ptr: *const u8, type_len: usize,
    payload_ptr: *const u8, payload_len: usize,
) -> *const u8 {
    unsafe {
        let msg_type = core::str::from_utf8(
            slice::from_raw_parts(type_ptr, type_len)
        ).unwrap_or("");

        match msg_type {
            "calculate" => {
                let payload = slice::from_raw_parts(payload_ptr, payload_len);
                if let Ok(req) = serde_json::from_slice::<CalculationRequest>(payload) {
                    if let Ok(result) = execute(&req) {
                        STATE.calculation_count += 1;
                        STATE.last_result = Some(result);
                    }
                }
                core::ptr::null()
            }
            _ => core::ptr::null(),
        }
    }
}

fn execute(req: &CalculationRequest) -> Result<f64, &'static str> {
    let (a, b) = (req.operands[0], req.operands[1]);
    match req.operation {
        Operation::Add      => Ok(a + b),
        Operation::Subtract => Ok(a - b),
        Operation::Multiply => Ok(a * b),
        Operation::Divide   => {
            if b == 0.0 { Err("Division by zero") } else { Ok(a / b) }
        }
    }
}

Build and Deploy

rustup target add wasm32-wasip2
cargo build --target wasm32-wasip2 --release

# Optimize the binary further
wasm-opt -Oz --strip-debug \
    target/wasm32-wasip2/release/calculator_wasm_actor.wasm \
    -o calculator_actor.wasm

# Deploy
curl -X POST http://localhost:8001/api/v1/applications/deploy \
    -F "application_id=rust-calc" \
    -F "name=calculator" \
    -F "version=1.0.0" \
    -F "wasm_file=@calculator_actor.wasm"

The resulting binary? Under 200KB. Compare that to a Python actor at 30-40MB or even a TypeScript actor at 1-2MB. When you deploy hundreds of actors per node, those size differences translate directly into memory savings and faster cold starts.

Deploying Applications

One of the patterns I find most compelling that I feel the serverless world has completely neglected is the idea of deploying whole applications, not just individual functions. If you have used Tomcat or JBoss, you understand what I mean. You package your application, hand it to the server, and the server takes care of running it, managing the process lifecycle, enforcing security policies, routing requests, collecting metrics, and handling restarts. You focus on business logic; the server handles the infrastructure cross-cuts. PlexSpaces brings this same model to WASM actors, but with Erlang/OTP’s supervision philosophy underneath. I call this the PlexSpaces Application Server model.

The Application Manifest

Instead of deploying actors one by one via API calls, you define an application bundle — a single manifest that describes your entire application topology: which actors to run, how they supervise each other, what resources they need, and what security policies apply to them.

[supervisor]
strategy = "one_for_one"
max_restarts = 10
max_restart_window_seconds = 60

# ChatRoom actor (Durable Object: one per room)
[[supervisor.children]]
id = "chat-room"
type = "worker"
restart = "permanent"
shutdown_timeout_seconds = 10

[supervisor.children.args]
max_history = "100"

# RateLimiter actor (Durable Object: per-user rate limiting)
[[supervisor.children]]
id = "rate-limiter"
type = "worker"
restart = "permanent"
shutdown_timeout_seconds = 5

[supervisor.children.args]
max_tokens = "5"
refill_rate_ms = "1000"

The runtime validates every WASM module against its declared WIT world, starts the supervision tree from the root down, and begins enforcing all security and resource policies — before your first actor processes its first message. PlexSpaces takes care of most cross cutting concerns like auth token validation, rate limiting, structured logging, trace context propagation, circuit breakers so that you can focus on the business logic.

Supervision and restarts. The manifest’s supervision tree is live. If actor crashes, the supervisor restarts it according to the declared strategy. If it exceeds max_restarts within max_restart_window_seconds, the supervisor escalates to its parent. This is exactly how Erlang/OTP gen_server supervision works.

Comparing Deployment Models

FaaS and Serverless

Here is where PlexSpaces bridges the worlds of actor systems and serverless platforms. Every actor you deploy in any language doubles as a serverless function that you invoke over plain HTTP. No client SDK required. No message queue setup. Just HTTP.

HTTP Invocation Model

PlexSpaces exposes a FaaS-style API that routes HTTP requests to actors using a simple URL pattern:

/api/v1/actors/{tenant}/{namespace}/{actor_type}

The HTTP method determines the invocation pattern:

FaaS in Action

This Rust example shows a FaaS-style webhook handler that receives HTTP POST payloads and stores delivery history — the kind of thing you would build on AWS Lambda or Cloudflare Workers, but here using PlexSpaces SDK annotations:

// Using PlexSpaces Rust SDK annotations (like Python @actor, @handler)
#[gen_server_actor]
struct WebhookHandler {
    deliveries: Vec<WebhookDelivery>,
    total_received: u64,
}

#[plexspaces_handlers]
impl WebhookHandler {
    #[handler("deliver")]
    async fn deliver(&mut self, ctx: &ActorContext, msg: &Message)
        -> Result<Value, BehaviorError>
    {
        let delivery = WebhookDelivery::new(
            ulid::Ulid::new().to_string(), &msg.payload,
        );
        self.deliveries.push(delivery);
        self.total_received += 1;
        Ok(json!({ "status": "received", "total": self.total_received }))
    }

    #[handler("list")]
    async fn list_deliveries(&self, _ctx: &ActorContext, _msg: &Message)
        -> Result<Value, BehaviorError>
    {
        Ok(json!({ "deliveries": self.deliveries, "total": self.total_received }))
    }
}

Invoke this actor over HTTP — no SDK, no message queue, just curl:

# POST a webhook delivery (fire-and-forget)
curl -X POST http://localhost:8001/api/v1/actors/acme-corp/webhooks/webhook_handler \
    -H "Content-Type: application/json" \
    -d '{"event": "order.completed", "order_id": "ORD-12345"}'

# GET recent deliveries (request-reply)
curl "http://localhost:8001/api/v1/actors/acme-corp/webhooks/webhook_handler?action=list"

Multi-Tenant Isolation

The URL path embeds tenant and namespace for built-in multi-tenant isolation. Tenant acme-corp cannot access tenant globex-inc‘s actors. The framework enforces this boundary at the routing layer with JWT-based authentication:

# Tenant A's rate limiter
curl -X POST http://localhost:8001/api/v1/actors/acme-corp/api/rate-limiter \
    -d '{"client_id": "user-123"}'

# Tenant B's rate limiter -- completely isolated state
curl -X POST http://localhost:8001/api/v1/actors/globex-inc/api/rate-limiter \
    -d '{"client_id": "user-456"}'

How PlexSpaces Compares to Traditional FaaS

The critical difference: PlexSpaces actors retain state between invocations. Traditional FaaS platforms treat functions as stateless — you manage state externally in DynamoDB, Redis, or S3. PlexSpaces actors carry durable state inside the actor, persisted via journaling and checkpointing. This eliminates the “stateless function + external state store” tax that adds latency and complexity to every serverless application.

PlexSpaces includes migration examples that show how to port existing Lambda functions, Step Functions workflows, Azure Durable Functions, Cloudflare Workers, and Orleans grains (See examples).

What WebAssembly Gives You

Let me address the obvious question: “Why not just use Docker?” Containers solve many problems well. But as Solomon Hykes, Docker’s creator, said in 2019 when WASI was first announced:

“If WASM+WASI existed in 2008, we wouldn’t have needed to create Docker. That’s how important it is. WebAssembly on the server is the future of computing. A standardized system interface was the missing link. Let’s hope WASI is up to the task!” — Solomon Hykes, March 2019

WebAssembly solves some problems better:

• Startup time. A WASM module instantiates in microseconds. A container takes seconds. When you auto-scale actors in response to load spikes, microsecond cold starts mean your users never notice.
• Memory footprint. A Rust WASM actor uses ~200KB. The equivalent Docker container starts at 50MB minimum (Alpine base image alone). On a single node, you run thousands of WASM actors where you might run dozens of containers.
• Security isolation. WASM sandboxing is capability-based. A module cannot access the filesystem, network, or memory outside its sandbox unless the host explicitly grants each capability through WASI. Containers share a kernel and rely on namespace isolation — a fundamentally larger attack surface.
• True polyglot. With containers, each language gets its own image, runtime, dependency tree, and deployment pipeline. With WASM, all languages produce the same artifact type, run on the same runtime, and share the same deployment pipeline.
• Composability. The Component Model lets you link WASM modules from different languages into a single process. No network calls. No serialization overhead. Direct function invocation across language boundaries. Try that with Docker.

PlexSpaces actually supports both: WASM sandboxing for lightweight actors and Firecracker microVMs for workloads that need full hardware-level isolation. You pick the isolation model per workload, and the framework handles the rest.

Where This Is Heading

The WASM Ecosystem Roadmap

The ecosystem moves fast. Here are the milestones that matter:

• Wasm 3.0 became the W3C standard in September 2025, standardizing nine production features including WasmGC, exception handling, tail calls, and SIMD
• WASI 0.3 shipped in February 2026 with native async support — actors can now handle concurrent I/O without blocking
• WASI 1.0 is on track for late 2026 or early 2027, providing the stability guarantees that enterprise adopters require
• Wasmtime leads the runtime ecosystem with full Component Model and WASI 0.2 support
• Wasmer 6.0 achieved ~95% of native speed on benchmarks
• Docker now runs WASM components alongside containers in Docker Desktop and Docker Engine

The FaaS-Actor Convergence

The most consequential trend is the convergence of serverless FaaS platforms and stateful actor systems. Today these exist as separate categories — AWS Lambda handles stateless functions, Temporal handles durable workflows, Orleans handles virtual actors, and Erlang/OTP handles fault-tolerant supervision. PlexSpaces unifies them into a single abstraction. This convergence accelerates along three axes:

• HTTP-native invocation. Every PlexSpaces actor is already a serverless function, callable over HTTP with automatic routing, multi-tenant isolation, and load balancing. As the WASM ecosystem matures, the cold start advantage (microseconds vs. seconds) makes WASM actors a compelling replacement for traditional Lambda functions, especially at the edge.
• Durable serverless. Traditional FaaS treats functions as stateless. PlexSpaces combines serverless invocation with durable execution — actors retain state, the framework journals every message, and crash recovery replays the journal to restore exact state. This eliminates the “Lambda + DynamoDB + Step Functions” stack that every non-trivial serverless application ends up building.
• Edge-native polyglot. WASM runs everywhere like cloud servers, edge nodes, IoT devices, even browsers. PlexSpaces actors compiled to WASM deploy to any environment that runs wasmtime. A Python ML model runs at the edge. A Rust event processor runs in the cloud. A TypeScript API actor runs in the CDN. All three communicate through the same framework, sharing state through tuple spaces and coordinating through process groups.

Get Started

PlexSpaces is open source. Clone the repository and start building:

git clone https://github.com/bhatti/PlexSpaces.git
cd PlexSpaces

# Quick setup (installs tools, builds, tests)
./scripts/setup.sh

# Or use Docker for the fastest path
docker pull plexobject/plexspaces:latest
docker run -d -p 8000:8000 -p 8001:8001 \
    -e PLEXSPACES_NODE_ID=node1 \
    plexobject/plexspaces:latest

# Explore the examples
ls examples/python/apps/     # calculator, bank_account, chat_room, nbody, ...
ls examples/typescript/apps/  # bank_account, migrating_cloudflare_workers, migrating_orleans
ls examples/go/apps/          # migrating_erlang_otp, migrating_cloudflare_workers, ...
ls examples/rust/apps/        # calculator, nbody, session_manager, ...

# Build and test everything
make all

Each example includes its own app-config.toml, build.sh script, and test instructions. The examples/ directory also contains migration guides from 24+ frameworks like Erlang/OTP, Temporal, Ray, Cloudflare Workers, Orleans, Restate, Azure Durable Functions, AWS Step Functions, wasmCloud, Dapr, and more.

I spent decades wrestling with the same distributed systems problems under different names on different stacks (see my previous blog). Fault tolerance, state management, multi-language support, coordination, serverless invocation, scaling. These problems never change, only the acronyms do. WebAssembly makes the polyglot piece real. The Component Model makes it composable. The application server model makes it deployable in a way that finally lets you focus on what you actually came to write: business logic.

PlexSpaces is available at github.com/bhatti/PlexSpaces. Give it a try and let me know what you think.

Here are the programming languages I’ve used over the last three decades. From BASIC in the late 80s to Rust today, each one taught me something about solving problems with code.

Late 1980s – Early 1990s

• I learned coding with BASIC/QUICKBASIC on Atari and later IBM XT computer in 1980s.
• I learned other languages in college or on my own like C, Pascal, Prolog, Lisp, FORTRAN and PERL.
• In college, I used Icon to build compilers.
• My first job was mainframe work and I used COBOL and CICS for applications, JCL and REXX for scripting and SAS for data processing.
• Later at a physics lab, I used C/C++, Fortran for applications and Python for scripting and glue language.
• I used a number of 4GL languages like dBase, FoxPro, Paradox, Delphi. Later I used Visual Basic and PowerBuilder for building client applications.
• I used SQL and PL/SQL throughout my career for relational databases.

Web Era Mid/Late 1990s

• The web era introduced a number of new languages like HTML, Javascript, CSS, ColdFusion, and Java.
• I used XML/XSLT/XPath/XQuery, PHP, VBScript and ActionScript.
• I used RSS/SPARQL/RDF for buiding semantic web applications.
• I used IDL/CORBA for building distributed systems.

Mobile/Services Era 2000s

• I used Ruby for building web applications, Erlang/Elixir for building concurrent applications.
• I used Groovy for writing tests and R for data analysis.
• When iOS was released, I used Objective-C to build mobile applications.
• In this era, functional languages gained popularity and I used Scala/Haskell/Clojure for some projects.

New Languages Era Mid 2010s

• I started using Go for networking/concurrent applications.
• I started using Swift for iOS applications and Kotlin for Android apps.
• I initially used Flow language from Facebook but then started using TypeScript instead of JavaScript.
• I used Dart for Flutter applications.
• I used GraphQL for some of client friendly backend APIs.
• I used Solidity for Ethereum smart contracts.
• I used Lua as a glue language with Redis, HAProxy and other similar systems.
• I used Rust and became my go to language for highly performant applications.

What Three Decades of Languages Taught Me

1. Every language is a bet on what matters most: Safety vs. speed vs. expressiveness vs. ecosystem vs. hiring.
2. Languages don’t die, they fade: I still see COBOL in production. I still debug Perl scripts. Legacy is measured in decades.
3. The fundamentals never change: Whether it’s BASIC or Rust, you’re still managing state, controlling flow, and abstracting complexity.
4. Polyglotism is a superpower: Each language teaches you a different way to think. Functional programming makes you better at OOP. Systems programming makes you better at scripting.
5. The best language is the one your team can maintain: I’ve seen beautiful Scala codebases become liabilities and ugly PHP applications become billion-dollar businesses.

What’s Next?

I’m watching Zig (Rust without the complexity?) and it’s on my list for next language to learn.

Summary

Introduction

Building distributed systems means confronting failure modes that are nearly impossible to reproduce in development or testing environments. How do you test for metastable failures that only emerge under specific load patterns? How do you validate that your quorum-based system actually maintains consistency during network partitions? How do you catch cross-system interaction bugs when both systems work perfectly in isolation? Integration testing, performance testing, and chaos engineering all help, but they have limitations. For the past few years, I’ve been using simulation to validate boundary conditions that are hard to test in real environments. Interactive simulators let you tweak parameters, trigger failure scenarios, and see the consequences immediately through metrics and visualizations.

In this post, I will share four simulators I’ve built to explore the failure modes and consistency challenges that are hardest to test in real systems:

1. Metastable Failure Simulator: Demonstrates how retry storms create self-sustaining collapse
2. CAP/PACELC Consistency Simulator: Shows the real tradeoffs between consistency, availability, and latency
3. CRDT Simulator: Explores conflict-free convergence without coordination
4. Cross-System Interaction (CSI) Failure Simulator: Reveals how correct systems fail through their interactions

Each simulator is built on research findings and real-world incidents. The goal isn’t just to understand these failure modes intellectually, but to develop intuition through experimentation. All simulators available at: https://github.com/bhatti/simulators.

Part 1: Metastable Failures

The Problem: When Systems Attack Themselves

Metastable failures are particularly insidious because the initial trigger can be small and transient, but the system remains degraded long after the trigger is gone. Research in the metastable failures has shown that traditional fault tolerance mechanisms don’t protect against metastability because the failure is self-sustaining through positive feedback loops in retry logic and coordination overhead. The mechanics are deceptively simple:

1. A transient issue (network blip, brief CPU spike) causes some requests to slow down
2. Slow requests start timing out
3. Clients retry timed-out requests, adding more load
4. Additional load increases coordination overhead (locks, queues, resource contention)
5. Higher overhead increases latency further
6. More timeouts trigger more retries
7. The system is now in a stable degraded state, even though the original trigger is gone

For example, AWS Kinesis experienced a 7+ hour outage in 2020 where a transient metadata mismatch triggered retry storms across the fleet. Even after the original issue was fixed, the retry behavior kept the system degraded. The recovery required externally rate-limiting client retries.

How the Simulator Works

The metastable failure simulator models this feedback loop using discrete event simulation (SimPy). Here’s what it simulates:

Server Model:

• Base latency: Time to process a request with no contention
• Concurrency slope: Additional latency per concurrent request (coordination cost)
• Capacity: Maximum concurrent requests before queueing

# Latency grows linearly with active requests
def current_latency(self):
    return self.base_latency + (self.active_requests * self.concurrency_slope)

Client Model:

• Timeout threshold: When to give up on a request
• Max retries: How many times to retry
• Backoff strategy: Exponential backoff with jitter (configurable)

Load Patterns:

• Constant: Steady baseline load
• Spike: Sudden increase for a duration, then back to baseline
• Ramp: Gradual increase and decrease

Key Parameters to Experiment With:

What You Can Learn:

1. Trigger a metastable failure: Set spike load high, timeout low, disable backoff ? watch P99 latency stay high after spike ends
2. See recovery with backoff: Same scenario but enable exponential backoff ? system recovers when spike ends
3. Understand the tipping point: Gradually increase concurrency slope ? observe when retry amplification begins
4. Test admission control: Set low server capacity ? see benefit of failing fast vs queueing

The simulator tracks success rate, retry count, timeout count, and latency percentiles over time, letting you see exactly when the system tips into metastability and whether it recovers. With this simulator you can validate various prevention strategies such as:

• Exponential backoff with jitter spreads retries over time
• Adaptive retry budgets limit total fleet-wide retries
• Circuit breakers detect patterns and stop retry storms
• Load shedding rejects requests before queues explode

Part 2: CAP and PACELC

The CAP theorem correctly states that during network partitions, you must choose between consistency and availability. However, as Daniel Abadi and others have pointed out, this only addresses partition scenarios. Most systems spend 99.99% of their time in normal operation, where the real tradeoff is between latency and consistency. This is where PACELC comes in:

• If Partition happens: choose Availability or Consistency
• Else (normal operation): choose Latency or Consistency

PACELC provides a more complete framework for understanding real-world distributed databases:

PA/EL Systems (DynamoDB, Cassandra, Riak):

• Partition ? Choose Availability (serve stale data)
• Normal ? Choose Latency (1-2ms reads from any replica)
• Use when: Shopping carts, session stores, high write throughput needed

PC/EC Systems (Google Spanner, VoltDB, HBase):

• Partition ? Choose Consistency (reject operations)
• Normal ? Choose Consistency (5-100ms for quorum coordination)
• Use when: Financial transactions, inventory, anything that can’t be wrong

PA/EC Systems (MongoDB):

• Partition ? Choose Availability (with caveats – unreplicated writes go to rollback)
• Normal ? Choose Consistency (strong reads/writes in baseline)
• Use when: Mixed workloads with mostly consistent needs

PC/EL Systems (PNUTS):

• Partition ? Choose Consistency
• Normal ? Choose Latency (async replication)
• Use when: Read-heavy with timeline consistency acceptable

Quorum Consensus: Strong Consistency with Coordination

When R + W > N (read quorum + write quorum > total replicas), the read and write sets must overlap in at least one node. This overlap ensures that any read sees at least one node with the latest write, providing linearizability.

Example with N=5, R=3, W=3:

• Write to replicas {1, 2, 3}
• Read from replicas {2, 3, 4}
• Overlap at {2, 3} guarantees we see the latest value

Critical Nuances:

R + W > N alone is NOT sufficient for linearizability in practice. You need additional mechanisms: readers must perform read repair synchronously before returning results, and writers must read the latest state from a quorum before writing. “Last write wins” based on wall-clock time breaks linearizability due to clock skew. Sloppy quorums like those used in Dynamo are NOT linearizable because the nodes in the quorum can change during failures. Even R = W = N doesn’t guarantee consistency if cluster membership changes. Google Spanner uses atomic clocks and GPS to achieve strong consistency globally, with TrueTime API providing less than 1ms clock uncertainty at the 99th percentile as of 2023.

How the Simulator Works

The CAP/PACELC simulator lets you explore these tradeoffs by configuring different consistency models and observing their behavior during normal operation and network partitions.

System Model:

• N replica nodes, each with local storage
• Configurable schema for data (to test compatibility)
• Network latency between nodes (WAN vs LAN)
• Optional partition mode (splits cluster)

Consistency Levels:

1. Strong (R+W>N): Quorum reads and writes, linearizable
2. Linearizable (R=W=N): All nodes must respond, highest consistency
3. Weak (R=1, W=1): Single node, eventual consistency
4. Eventual: Async replication, high availability

def get_quorum_size(self, operation_type):
    if self.consistency_level == ConsistencyLevel.STRONG:
        return (self.n_nodes // 2) + 1  # Majority
    elif self.consistency_level == ConsistencyLevel.LINEARIZABLE:
        return self.n_nodes  # All nodes
    elif self.consistency_level == ConsistencyLevel.WEAK:
        return 1  # Any node

Key Parameters:

What You Can Learn:

1. Latency cost of consistency:
   • Run with Strong (R=3,W=3) at network_latency=5ms ? ~15ms operations
   • Same at network_latency=100ms ? ~300ms operations
   • Switch to Weak (R=1,W=1) ? single-digit milliseconds regardless
2. CAP during partitions:
   • Enable partition with Strong consistency ? operations fail (choosing C over A)
   • Enable partition with Eventual ? stale reads but available (choosing A over C)
3. Quorum size tradeoffs:
   • Linearizable (R=W=N) ? single node failure breaks everything
   • Strong (R=W=3 of N=5) ? can tolerate 2 node failures
   • Measure failure rate vs consistency guarantees
4. Geographic distribution:
   • Network latency 10ms (same datacenter) ? quorum cost moderate
   • Network latency 150ms (cross-continent) ? quorum cost severe
   • Observe when you should use eventual consistency for geo-distribution

The simulator tracks write/read latencies, inconsistent reads, failed operations, and success rates, giving you quantitative data on the tradeoffs.

Key Insights from Simulation

The simulator reveals that most architectural decisions are driven by normal operation latency, not partition handling. If you’re building a global system with 150ms cross-region latency, strong consistency means every operation takes 150ms+ for quorum coordination. That’s often unacceptable for user-facing features. This is why hybrid approaches are becoming standard: use strong consistency for critical invariants (financial transactions, inventory), eventual consistency for everything else (user profiles, preferences).

Part 3: CRDTs

CRDTs (Conflict-Free Replicated Data Types) provide strong eventual consistency (SEC) through mathematical guarantees, not probabilistic convergence. They work without coordination, consensus, or concurrency control. CRDTs rely on operations being commutative (order doesn’t matter), merge functions being associative and idempotent (forming a semilattice), and updates being monotonic according to a partial order.

Example: G-Counter (Grow-Only Counter)

class GCounter:
    def __init__(self, replica_id):
        self.counts = {}  # replica_id -> count
    
    def increment(self, amount=1):
        # Each replica tracks its own increments
        self.counts[self.replica_id] = self.counts.get(self.replica_id, 0) + amount
    
    def value(self):
        # Total is sum of all replicas
        return sum(self.counts.values())
    
    def merge(self, other):
        # Take max of each replica's count
        for replica_id, count in other.counts.items():
            self.counts[replica_id] = max(self.counts.get(replica_id, 0), count)

Why this works:

• Each replica only increments its own counter (no conflicts)
• Merge takes max (idempotent: max(a,a) = a)
• Order doesn’t matter: max(max(a,b),c) = max(a,max(b,c))
• Eventually all replicas see all increments ? convergence

CRDT Types

There are two main approaches: State-based CRDTs (CvRDTs) send full local state and require merge functions to be commutative, associative, and idempotent. Operation-based CRDTs (CmRDTs) transmit only update operations and require reliable delivery in causal order. Delta-state CRDTs combine the advantages by transmitting compact deltas.

Four CRDTs in the Simulator:

1. G-Counter: Increment only, perfect for metrics
2. PN-Counter: Increment and decrement (two G-Counters)
3. OR-Set: Add/remove elements, concurrent add wins
4. LWW-Map: Last-write-wins with timestamps

Production systems using CRDTs include Redis Enterprise (CRDBs), Riak, Azure Cosmos DB for distributed data types, and Automerge/Yjs for collaborative editing like Google Docs. SoundCloud uses CRDTs in their audio distribution platform.

Important Limitations

CRDTs only provide eventual consistency, NOT strong consistency or linearizability. Different replicas can see concurrent operations in different orders temporarily. Not all operations are naturally commutative, and CRDTs cannot solve problems requiring atomic coordination like preventing double-booking without additional mechanisms.

The “Shopping Cart Problem”: You can use an OR-Set for shopping cart items, but if two clients concurrently remove the same item, your naive implementation might remove both. The CRDT guarantees convergence to a consistent state, but that state might not match user expectations.

Byzantine fault tolerance is also a concern as traditional CRDTs assume all devices are trustworthy. Malicious devices can create permanent inconsistencies.

How the Simulator Works

The CRDT simulator demonstrates convergence through gossip-based replication. You can watch replicas diverge and converge as they exchange state.

Simulation Model:

• Multiple replica nodes, each with independent CRDT state
• Operations applied to random replicas (simulating distributed clients)
• Periodic “merges” (gossip protocol) with probability merge_probability
• Network delay between merges
• Tracks convergence: do all replicas have identical state?

CRDT Implementations: Each CRDT type has its own semantics:

# G-Counter: Each replica has its own count, merge takes max
def merge(self, other):
    for replica_id, count in other.counts.items():
        self.counts[replica_id] = max(self.counts.get(replica_id, 0), count)

# OR-Set: Elements have unique tags, add always beats remove
def add(self, element, unique_tag):
    self.elements[element].add(unique_tag)

def remove(self, element, observed_tags):
    self.elements[element] -= observed_tags  # Only remove what was observed

# LWW-Map: Latest timestamp wins
def set(self, key, value, timestamp):
    current = self.entries.get(key)
    if current is None or timestamp > current[1]:
        self.entries[key] = (value, timestamp, self.replica_id)

Key Parameters:

What You Can Learn:

1. Convergence speed:
   • Set merge_probability=0.1 ? slow convergence, replicas stay diverged
   • Set merge_probability=0.8 ? fast convergence
   • Understand gossip frequency vs consistency window tradeoff
2. OR-Set semantics:
   • Watch concurrent add/remove ? add wins
   • See how unique tags prevent unintended deletions
   • Compare with naive set implementation
3. LWW-Map data loss:
   • Two replicas set same key concurrently with different values
   • One value “wins” based on timestamp (or replica ID tie-break)
   • Data loss is possible – not suitable for all use cases
4. Network partition tolerance:
   • Low merge probability simulates partition
   • Replicas diverge but operations still succeed (AP in CAP)
   • After “partition heals” (merges resume), all converge
   • No coordination needed, no operations failed

The simulator visually shows replica states over time and convergence status, making abstract CRDT theory concrete.

Key Insights from Simulation

CRDTs trade immediate consistency for availability and partition tolerance. The theoretical guarantees are proven: if all replicas receive all updates (eventual delivery), they will converge to the same state (strong convergence).

But the simulator reveals the practical challenges:

• Merge semantics don’t always match user intent (LWW can lose data)
• Tombstones can grow indefinitely (OR-Set needs garbage collection)
• Causal ordering adds complexity (need vector clocks for some CRDTs)
• Not suitable for operations requiring coordination (uniqueness constraints, atomic updates)

When to use CRDTs:

• High-write distributed counters (page views, analytics)
• Collaborative editing (where eventual consistency is acceptable)
• Offline-first applications (sync when online)
• Shopping carts (with careful semantic design)

When NOT to use CRDTs:

• Bank account balances (need atomic transactions)
• Inventory (can’t prevent overselling without coordination)
• Unique constraints (usernames, reservation systems)
• Access control (need immediate consistency)

Part 4: Cross-System Interaction (CSI) Failures

Research from EuroSys 2023 found that 20% of catastrophic cloud incidents and 37% of failures in major open-source distributed systems are CSI failures – where both systems work correctly in isolation but fail when connected. This is the NASA Mars Climate Orbiter problem: one team used metric units, another used imperial. Both systems worked perfectly. The spacecraft burned up in Mars’s atmosphere because of their interaction.

Why CSI Failures Are Different

Not dependency failures: The downstream system is available, it just can’t process what upstream sends.

Not library bugs: Libraries are single-address-space and well-tested. CSI failures cross system boundaries where testing is expensive.

Not component failures: Each system passes its own test suite. The bug only emerges through interaction.

CSI failures manifest across three planes: Data plane (51% – schema/metadata mismatches), Management plane (32% – configuration incoherence), and Control plane (17% – API semantic violations).

For example, study of Apache Spark-Hive integration found 15 distinct discrepancies in simple write-read testing. Hive stored timestamps as long (milliseconds since epoch), Spark expected Timestamp type. Both worked in isolation, failed when integrated. Kafka and Flink encoding mismatch: Kafka set compression.type=lz4, Flink couldn’t decompress due to old LZ4 library. Configuration was silently ignored in Flink, leading to data corruption for 2 weeks before detection.

Why Testing Doesn’t Catch CSI Failures

Analysis of Spark found only 6% of integration tests actually test cross-system interaction. Most “integration tests” test multiple components of the same system. Cross-system testing is expensive and often skipped. The problem compounds with modern architectures:

• Microservices: More system boundaries to test
• Multi-cloud: Different clouds with different semantics
• Serverless: Fine-grained composition increases interaction surface area

How the Simulator Works

The CSI failure simulator models two systems exchanging data, with configurable discrepancies in schemas, encodings, and configurations.

System Model:

• Two systems (upstream ? downstream)
• Each has its own schema definition (field types, encoding, nullable fields)
• Each has its own configuration (timeouts, retry counts, etc.)
• Data flows from System A to System B with potential conversion failures

Failure Scenarios:

1. Metadata Mismatch (Hive/Spark):
   • System A: timestamp: long
   • System B: timestamp: Timestamp
   • Failure: Type coercion fails ~30% of the time
2. Schema Conflict (Producer/Consumer):
   • System A: encoding: latin-1
   • System B: encoding: utf-8
   • Failure: Silent data corruption
3. Configuration Incoherence (ServiceA/ServiceB):
   • System A: max_retries=3, timeout=30s
   • System B expects: max_retries=5, timeout=60s
   • Failure: ~40% of requests fail due to premature timeout
4. API Semantic Violation (Upstream/Downstream):
   • Upstream assumes: synchronous, thread-safe
   • Downstream is: asynchronous, not thread-safe
   • Failure: Race conditions, out-of-order processing
5. Type Confusion (SystemA/SystemB):
   • System A: amount: float
   • System B: amount: decimal
   • Failure: Precision loss in financial calculations

Implementation Details:

class DataSchema:
    def __init__(self, schema_id, fields, encoding, nullable_fields):
        self.fields = fields  # field_name -> type
        self.encoding = encoding
        
    def is_compatible(self, other):
        # Check field types and encoding
        return (self.fields == other.fields and 
                self.encoding == other.encoding)

class DataRecord:
    def serialize(self, target_schema):
        # Attempt type coercion
        for field, value in self.data.items():
            expected_type = target_schema.fields[field]
            actual_type = self.schema.fields[field]
            
            if expected_type != actual_type:
                # 30% failure on type mismatch (simulating real world)
                if random.random() < 0.3:
                    return None  # Serialization failure
        
        # Check encoding compatibility
        if self.schema.encoding != target_schema.encoding:
            if random.random() < 0.2:  # 20% silent corruption
                return None

Key Parameters:

The simulator doesn’t have many tunable parameters because CSI failures are about specific incompatibilities, not gradual degradation. Each scenario models a real-world pattern.

What You Can Learn:

1. Failure rates: CSI failures often manifest in 20-40% of requests (not 100%)
   • Some requests happen to have compatible data
   • Makes debugging harder (intermittent failures)
2. Failure location:
   • Research shows 69% of CSI fixes go in the upstream system, often in connector modules that are less than 5% of the codebase
   • Simulator shows which system fails (usually downstream)
3. Silent vs loud failures:
   • Type mismatches often crash (loud, easy to detect)
   • Encoding mismatches corrupt silently (hard to detect)
   • Config mismatches cause intermittent timeouts
4. Prevention effectiveness:
   • Schema registry eliminates metadata mismatches
   • Configuration validation catches config incoherence
   • Contract testing prevents API semantic violations

Key Insights from Simulation

The simulator demonstrates that cross-system integration testing is essential but often skipped. Unit tests of each system won’t catch these failures.

Prevention strategies validated by simulation:

1. Write-Read Testing: Write with System A, read with System B, verify integrity
2. Schema Registry: Single source of truth for data schemas, enforced across systems
3. Configuration Coherence Checking: Validate that shared configs match
4. Contract Testing: Explicit, machine-checkable API contracts

Hybrid Consistency Models

Modern systems increasingly use mixed consistency: RedBlue Consistency (2012) marks operations as needing strong consistency (red) or eventual consistency (blue). Replicache (2024) has the server assign final total order while clients do optimistic local updates with rebase. For example: Calendar Application

# Strong consistency for room reservations (prevent double-booking)
def book_conference_room(room_id, time_slot):
    with transaction(consistency='STRONG'):
        if room.is_available(time_slot):
            room.book(time_slot)
            return True
        return False

# CRDTs for collaborative editing (participant lists, notes)
def update_meeting_notes(meeting_id, notes):
    # LWW-Map CRDT, eventual consistency
    meeting.notes.merge(notes)

# Eventual consistency for preferences
def update_user_calendar_color(user_id, color):
    # Who cares if this propagates slowly?
    user_prefs[user_id] = color

Recent theoretical work on the CALM theorem proves that coordination-free consistency is achievable for certain problem classes. Research in 2025 provided mathematical definitions of when coordination is and isn’t required, separating coordination from computation.

What the Simulators Teach Us

Running all four simulators reveals the consistency spectrum:

No “best” consistency model exists:

• Quorums are best when you need linearizability and can tolerate latency
• CRDTs are best when you need high availability and can tolerate eventual consistency
• Neither approach “bypasses” CAP – they make different tradeoffs
• Real systems use hybrid models with different consistency for different operations

Practical Lessons

1. Design for Recovery, Not Just Prevention

The metastable failure simulator shows you can’t prevent all failures. Your retry logic, backoff strategy, and circuit breakers are more important than your happy path code. Validated strategies include:

• Exponential backoff with jitter (spread retries over time)
• Adaptive retry budgets (limit total fleet-wide retries)
• Circuit breakers (detect patterns, stop storms)
• Load shedding (fail fast rather than queue to death)

2. Understand the Consistency Spectrum

The CAP/PACELC simulator demonstrates that consistency is not binary. You need to understand:

• What consistency level do you actually need? (Most operations don’t need linearizability)
• What’s the latency cost? (Quorum reads in cross-region deployment can be 100x slower)
• What happens during partitions? (Can you sacrifice availability or must you serve stale data?)

Decision framework:

• Use strong consistency for: money, inventory, locks, compliance
• Use eventual consistency for: feeds, catalogs, analytics, caches
• Use hybrid models for: most real-world applications

3. Test Cross-System Interactions

The CSI failure simulator reveals that 86% of fixes go into connector modules that are less than 5% of your codebase. This is where bugs hide. Essential tests include:

• Write-read tests (write with System A, read with System B)
• Round-trip tests (serialize/deserialize across boundaries)
• Version compatibility matrix (test combinations)
• Schema validation (machine-checkable contracts)

4. Leverage CRDTs Where Appropriate

The CRDT simulator shows that conflict-free convergence is possible for specific problem types. But you need to:

• Understand the semantic limitations (LWW can lose data)
• Design merge behavior carefully (does it match user intent?)
• Handle garbage collection (tombstones, vector clocks)
• Accept eventual consistency (not suitable for all use cases)

5. Monitor for Sustaining Effects

Metastability, retry storms, and goodput collapse are self-sustaining failure modes. They persist after the trigger is gone. Critical metrics include:

• P99 latency vs timeout threshold (approaching timeout = danger)
• Retry rate vs success rate (high retries = storm risk)
• Queue depth (unbounded growth = admission control needed)
• Goodput vs throughput (doing useful work vs spinning)

Using the Simulators

All four simulators are available at: https://github.com/bhatti/simulators

Installation

git clone https://github.com/bhatti/simulators
cd simulators
pip install -r requirements.txt

Requirements:

• Python 3.7+
• streamlit (web UI)
• simpy (discrete event simulation)
• plotly (interactive visualizations)
• numpy, pandas (data analysis)

Running Individual Simulators

# Metastable failure simulator
streamlit run metastable_simulator.py

# CAP/PACELC consistency simulator
streamlit run cap_consistency_simulator.py

# CRDT simulator
streamlit run crdt_simulator.py

# CSI failure simulator
streamlit run csi_failure_simulator.py

Running All Simulators

python run_all_simulators.py

Conclusion

Building distributed systems means confronting failure modes that are expensive or impossible to reproduce in real environments:

• Metastable failures require specific load patterns and timing
• Consistency tradeoffs need multi-region deployments to observe
• CRDT convergence requires orchestrating concurrent operations across replicas
• CSI failures need exact schema/config mismatches that don’t exist in test environments

Simulators bridge the gap between theoretical understanding and practical intuition:

1. Cheaper than production testing: No cloud costs, no multi-region setup, instant feedback
2. Safer than production experiments: Crash the simulator, not your service
3. More complete than unit tests: See emergent behaviors, not just component correctness
4. Faster iteration: Tweak parameters, re-run in seconds, build intuition through experimentation

What You Can’t Learn Without Simulation

• When does retry amplification tip into metastability? (Depends on coordination slope, timeout, backoff)
• How much does quorum coordination actually cost? (Depends on network latency, replica count, workload)
• Do your CRDT semantics match user expectations? (Depends on merge behavior, conflict resolution)
• Will your schema changes break integration? (Depends on type coercion, encoding, version skew)

The goal isn’t to prevent all failures, that’s impossible. The goal is to understand, anticipate, and recover from the failures that will inevitably occur.

References

Key research papers and resources used in this post:

1. AWS Metastability Research (HotOS 2025) – Sustaining effects and goodput collapse
2. Marc Brooker on DSQL – Practical distributed SQL considerations
3. James Hamilton on Reliable Systems – Large-scale system design
4. CSI Failures Study (EuroSys 2023) – Cross-system interaction failures
5. PACELC Framework – Beyond CAP theorem
6. Marc Brooker on CAP – CAP theorem revisited
7. Anna CRDT Database – Autoscaling with CRDTs
8. Linearizability Paper – Herlihy & Wing’s foundational work
9. Designing Data-Intensive Applications by Martin Kleppmann
10. Distributed Systems Reading Group – MIT CSAIL
11. Jepsen.io – Kyle Kingsbury’s consistency testing
12. Aphyr’s blog – Distributed systems deep dives