Shahzad Bhatti Welcome to my ramblings and rants!

Here are the programming languages I’ve used over the last three decades. From BASIC in the late 80s to Rust today, each one taught me something about solving problems with code.

Late 1980s – Early 1990s

• I learned coding with BASIC/QUICKBASIC on Atari and later IBM XT computer in 1980s.
• I learned other languages in college or on my own like C, Pascal, Prolog, Lisp, FORTRAN and PERL.
• In college, I used Icon to build compilers.
• My first job was mainframe work and I used COBOL and CICS for applications, JCL and REXX for scripting and SAS for data processing.
• Later at a physics lab, I used C/C++, Fortran for applications and Python for scripting and glue language.
• I used a number of 4GL languages like dBase, FoxPro, Paradox, Delphi. Later I used Visual Basic and PowerBuilder for building client applications.
• I used SQL and PL/SQL throughout my career for relational databases.

Web Era Mid/Late 1990s

• The web era introduced a number of new languages like HTML, Javascript, CSS, ColdFusion, and Java.
• I used XML/XSLT/XPath/XQuery, PHP, VBScript and ActionScript.
• I used RSS/SPARQL/RDF for buiding semantic web applications.
• I used IDL/CORBA for building distributed systems.

Mobile/Services Era 2000s

• I used Ruby for building web applications, Erlang/Elixir for building concurrent applications.
• I used Groovy for writing tests and R for data analysis.
• When iOS was released, I used Objective-C to build mobile applications.
• In this era, functional languages gained popularity and I used Scala/Haskell/Clojure for some projects.

New Languages Era Mid 2010s

• I started using Go for networking/concurrent applications.
• I started using Swift for iOS applications and Kotlin for Android apps.
• I initially used Flow language from Facebook but then started using TypeScript instead of JavaScript.
• I used Dart for Flutter applications.
• I used GraphQL for some of client friendly backend APIs.
• I used Solidity for Ethereum smart contracts.
• I used Lua as a glue language with Redis, HAProxy and other similar systems.
• I used Rust and became my go to language for highly performant applications.

What Three Decades of Languages Taught Me

1. Every language is a bet on what matters most: Safety vs. speed vs. expressiveness vs. ecosystem vs. hiring.
2. Languages don’t die, they fade: I still see COBOL in production. I still debug Perl scripts. Legacy is measured in decades.
3. The fundamentals never change: Whether it’s BASIC or Rust, you’re still managing state, controlling flow, and abstracting complexity.
4. Polyglotism is a superpower: Each language teaches you a different way to think. Functional programming makes you better at OOP. Systems programming makes you better at scripting.
5. The best language is the one your team can maintain: I’ve seen beautiful Scala codebases become liabilities and ugly PHP applications become billion-dollar businesses.

What’s Next?

I’m watching Zig (Rust without the complexity?) and it’s on my list for next language to learn.

Summary

Introduction

Building distributed systems means confronting failure modes that are nearly impossible to reproduce in development or testing environments. How do you test for metastable failures that only emerge under specific load patterns? How do you validate that your quorum-based system actually maintains consistency during network partitions? How do you catch cross-system interaction bugs when both systems work perfectly in isolation? Integration testing, performance testing, and chaos engineering all help, but they have limitations. For the past few years, I’ve been using simulation to validate boundary conditions that are hard to test in real environments. Interactive simulators let you tweak parameters, trigger failure scenarios, and see the consequences immediately through metrics and visualizations.

In this post, I will share four simulators I’ve built to explore the failure modes and consistency challenges that are hardest to test in real systems:

1. Metastable Failure Simulator: Demonstrates how retry storms create self-sustaining collapse
2. CAP/PACELC Consistency Simulator: Shows the real tradeoffs between consistency, availability, and latency
3. CRDT Simulator: Explores conflict-free convergence without coordination
4. Cross-System Interaction (CSI) Failure Simulator: Reveals how correct systems fail through their interactions

Each simulator is built on research findings and real-world incidents. The goal isn’t just to understand these failure modes intellectually, but to develop intuition through experimentation. All simulators available at: https://github.com/bhatti/simulators.

Part 1: Metastable Failures

The Problem: When Systems Attack Themselves

Metastable failures are particularly insidious because the initial trigger can be small and transient, but the system remains degraded long after the trigger is gone. Research in the metastable failures has shown that traditional fault tolerance mechanisms don’t protect against metastability because the failure is self-sustaining through positive feedback loops in retry logic and coordination overhead. The mechanics are deceptively simple:

1. A transient issue (network blip, brief CPU spike) causes some requests to slow down
2. Slow requests start timing out
3. Clients retry timed-out requests, adding more load
4. Additional load increases coordination overhead (locks, queues, resource contention)
5. Higher overhead increases latency further
6. More timeouts trigger more retries
7. The system is now in a stable degraded state, even though the original trigger is gone

For example, AWS Kinesis experienced a 7+ hour outage in 2020 where a transient metadata mismatch triggered retry storms across the fleet. Even after the original issue was fixed, the retry behavior kept the system degraded. The recovery required externally rate-limiting client retries.

How the Simulator Works

The metastable failure simulator models this feedback loop using discrete event simulation (SimPy). Here’s what it simulates:

Server Model:

• Base latency: Time to process a request with no contention
• Concurrency slope: Additional latency per concurrent request (coordination cost)
• Capacity: Maximum concurrent requests before queueing

# Latency grows linearly with active requests
def current_latency(self):
    return self.base_latency + (self.active_requests * self.concurrency_slope)

Client Model:

• Timeout threshold: When to give up on a request
• Max retries: How many times to retry
• Backoff strategy: Exponential backoff with jitter (configurable)

Load Patterns:

• Constant: Steady baseline load
• Spike: Sudden increase for a duration, then back to baseline
• Ramp: Gradual increase and decrease

Key Parameters to Experiment With:

What You Can Learn:

1. Trigger a metastable failure: Set spike load high, timeout low, disable backoff ? watch P99 latency stay high after spike ends
2. See recovery with backoff: Same scenario but enable exponential backoff ? system recovers when spike ends
3. Understand the tipping point: Gradually increase concurrency slope ? observe when retry amplification begins
4. Test admission control: Set low server capacity ? see benefit of failing fast vs queueing

The simulator tracks success rate, retry count, timeout count, and latency percentiles over time, letting you see exactly when the system tips into metastability and whether it recovers. With this simulator you can validate various prevention strategies such as:

• Exponential backoff with jitter spreads retries over time
• Adaptive retry budgets limit total fleet-wide retries
• Circuit breakers detect patterns and stop retry storms
• Load shedding rejects requests before queues explode

Part 2: CAP and PACELC

The CAP theorem correctly states that during network partitions, you must choose between consistency and availability. However, as Daniel Abadi and others have pointed out, this only addresses partition scenarios. Most systems spend 99.99% of their time in normal operation, where the real tradeoff is between latency and consistency. This is where PACELC comes in:

• If Partition happens: choose Availability or Consistency
• Else (normal operation): choose Latency or Consistency

PACELC provides a more complete framework for understanding real-world distributed databases:

PA/EL Systems (DynamoDB, Cassandra, Riak):

• Partition ? Choose Availability (serve stale data)
• Normal ? Choose Latency (1-2ms reads from any replica)
• Use when: Shopping carts, session stores, high write throughput needed

PC/EC Systems (Google Spanner, VoltDB, HBase):

• Partition ? Choose Consistency (reject operations)
• Normal ? Choose Consistency (5-100ms for quorum coordination)
• Use when: Financial transactions, inventory, anything that can’t be wrong

PA/EC Systems (MongoDB):

• Partition ? Choose Availability (with caveats – unreplicated writes go to rollback)
• Normal ? Choose Consistency (strong reads/writes in baseline)
• Use when: Mixed workloads with mostly consistent needs

PC/EL Systems (PNUTS):

• Partition ? Choose Consistency
• Normal ? Choose Latency (async replication)
• Use when: Read-heavy with timeline consistency acceptable

Quorum Consensus: Strong Consistency with Coordination

When R + W > N (read quorum + write quorum > total replicas), the read and write sets must overlap in at least one node. This overlap ensures that any read sees at least one node with the latest write, providing linearizability.

Example with N=5, R=3, W=3:

• Write to replicas {1, 2, 3}
• Read from replicas {2, 3, 4}
• Overlap at {2, 3} guarantees we see the latest value

Critical Nuances:

R + W > N alone is NOT sufficient for linearizability in practice. You need additional mechanisms: readers must perform read repair synchronously before returning results, and writers must read the latest state from a quorum before writing. “Last write wins” based on wall-clock time breaks linearizability due to clock skew. Sloppy quorums like those used in Dynamo are NOT linearizable because the nodes in the quorum can change during failures. Even R = W = N doesn’t guarantee consistency if cluster membership changes. Google Spanner uses atomic clocks and GPS to achieve strong consistency globally, with TrueTime API providing less than 1ms clock uncertainty at the 99th percentile as of 2023.

How the Simulator Works

The CAP/PACELC simulator lets you explore these tradeoffs by configuring different consistency models and observing their behavior during normal operation and network partitions.

System Model:

• N replica nodes, each with local storage
• Configurable schema for data (to test compatibility)
• Network latency between nodes (WAN vs LAN)
• Optional partition mode (splits cluster)

Consistency Levels:

1. Strong (R+W>N): Quorum reads and writes, linearizable
2. Linearizable (R=W=N): All nodes must respond, highest consistency
3. Weak (R=1, W=1): Single node, eventual consistency
4. Eventual: Async replication, high availability

def get_quorum_size(self, operation_type):
    if self.consistency_level == ConsistencyLevel.STRONG:
        return (self.n_nodes // 2) + 1  # Majority
    elif self.consistency_level == ConsistencyLevel.LINEARIZABLE:
        return self.n_nodes  # All nodes
    elif self.consistency_level == ConsistencyLevel.WEAK:
        return 1  # Any node

Key Parameters:

What You Can Learn:

1. Latency cost of consistency:
   • Run with Strong (R=3,W=3) at network_latency=5ms ? ~15ms operations
   • Same at network_latency=100ms ? ~300ms operations
   • Switch to Weak (R=1,W=1) ? single-digit milliseconds regardless
2. CAP during partitions:
   • Enable partition with Strong consistency ? operations fail (choosing C over A)
   • Enable partition with Eventual ? stale reads but available (choosing A over C)
3. Quorum size tradeoffs:
   • Linearizable (R=W=N) ? single node failure breaks everything
   • Strong (R=W=3 of N=5) ? can tolerate 2 node failures
   • Measure failure rate vs consistency guarantees
4. Geographic distribution:
   • Network latency 10ms (same datacenter) ? quorum cost moderate
   • Network latency 150ms (cross-continent) ? quorum cost severe
   • Observe when you should use eventual consistency for geo-distribution

The simulator tracks write/read latencies, inconsistent reads, failed operations, and success rates, giving you quantitative data on the tradeoffs.

Key Insights from Simulation

The simulator reveals that most architectural decisions are driven by normal operation latency, not partition handling. If you’re building a global system with 150ms cross-region latency, strong consistency means every operation takes 150ms+ for quorum coordination. That’s often unacceptable for user-facing features. This is why hybrid approaches are becoming standard: use strong consistency for critical invariants (financial transactions, inventory), eventual consistency for everything else (user profiles, preferences).

Part 3: CRDTs

CRDTs (Conflict-Free Replicated Data Types) provide strong eventual consistency (SEC) through mathematical guarantees, not probabilistic convergence. They work without coordination, consensus, or concurrency control. CRDTs rely on operations being commutative (order doesn’t matter), merge functions being associative and idempotent (forming a semilattice), and updates being monotonic according to a partial order.

Example: G-Counter (Grow-Only Counter)

class GCounter:
    def __init__(self, replica_id):
        self.counts = {}  # replica_id -> count
    
    def increment(self, amount=1):
        # Each replica tracks its own increments
        self.counts[self.replica_id] = self.counts.get(self.replica_id, 0) + amount
    
    def value(self):
        # Total is sum of all replicas
        return sum(self.counts.values())
    
    def merge(self, other):
        # Take max of each replica's count
        for replica_id, count in other.counts.items():
            self.counts[replica_id] = max(self.counts.get(replica_id, 0), count)

Why this works:

• Each replica only increments its own counter (no conflicts)
• Merge takes max (idempotent: max(a,a) = a)
• Order doesn’t matter: max(max(a,b),c) = max(a,max(b,c))
• Eventually all replicas see all increments ? convergence

CRDT Types

There are two main approaches: State-based CRDTs (CvRDTs) send full local state and require merge functions to be commutative, associative, and idempotent. Operation-based CRDTs (CmRDTs) transmit only update operations and require reliable delivery in causal order. Delta-state CRDTs combine the advantages by transmitting compact deltas.

Four CRDTs in the Simulator:

1. G-Counter: Increment only, perfect for metrics
2. PN-Counter: Increment and decrement (two G-Counters)
3. OR-Set: Add/remove elements, concurrent add wins
4. LWW-Map: Last-write-wins with timestamps

Production systems using CRDTs include Redis Enterprise (CRDBs), Riak, Azure Cosmos DB for distributed data types, and Automerge/Yjs for collaborative editing like Google Docs. SoundCloud uses CRDTs in their audio distribution platform.

Important Limitations

CRDTs only provide eventual consistency, NOT strong consistency or linearizability. Different replicas can see concurrent operations in different orders temporarily. Not all operations are naturally commutative, and CRDTs cannot solve problems requiring atomic coordination like preventing double-booking without additional mechanisms.

The “Shopping Cart Problem”: You can use an OR-Set for shopping cart items, but if two clients concurrently remove the same item, your naive implementation might remove both. The CRDT guarantees convergence to a consistent state, but that state might not match user expectations.

Byzantine fault tolerance is also a concern as traditional CRDTs assume all devices are trustworthy. Malicious devices can create permanent inconsistencies.

How the Simulator Works

The CRDT simulator demonstrates convergence through gossip-based replication. You can watch replicas diverge and converge as they exchange state.

Simulation Model:

• Multiple replica nodes, each with independent CRDT state
• Operations applied to random replicas (simulating distributed clients)
• Periodic “merges” (gossip protocol) with probability merge_probability
• Network delay between merges
• Tracks convergence: do all replicas have identical state?

CRDT Implementations: Each CRDT type has its own semantics:

# G-Counter: Each replica has its own count, merge takes max
def merge(self, other):
    for replica_id, count in other.counts.items():
        self.counts[replica_id] = max(self.counts.get(replica_id, 0), count)

# OR-Set: Elements have unique tags, add always beats remove
def add(self, element, unique_tag):
    self.elements[element].add(unique_tag)

def remove(self, element, observed_tags):
    self.elements[element] -= observed_tags  # Only remove what was observed

# LWW-Map: Latest timestamp wins
def set(self, key, value, timestamp):
    current = self.entries.get(key)
    if current is None or timestamp > current[1]:
        self.entries[key] = (value, timestamp, self.replica_id)

Key Parameters:

What You Can Learn:

1. Convergence speed:
   • Set merge_probability=0.1 ? slow convergence, replicas stay diverged
   • Set merge_probability=0.8 ? fast convergence
   • Understand gossip frequency vs consistency window tradeoff
2. OR-Set semantics:
   • Watch concurrent add/remove ? add wins
   • See how unique tags prevent unintended deletions
   • Compare with naive set implementation
3. LWW-Map data loss:
   • Two replicas set same key concurrently with different values
   • One value “wins” based on timestamp (or replica ID tie-break)
   • Data loss is possible – not suitable for all use cases
4. Network partition tolerance:
   • Low merge probability simulates partition
   • Replicas diverge but operations still succeed (AP in CAP)
   • After “partition heals” (merges resume), all converge
   • No coordination needed, no operations failed

The simulator visually shows replica states over time and convergence status, making abstract CRDT theory concrete.

Key Insights from Simulation

CRDTs trade immediate consistency for availability and partition tolerance. The theoretical guarantees are proven: if all replicas receive all updates (eventual delivery), they will converge to the same state (strong convergence).

But the simulator reveals the practical challenges:

• Merge semantics don’t always match user intent (LWW can lose data)
• Tombstones can grow indefinitely (OR-Set needs garbage collection)
• Causal ordering adds complexity (need vector clocks for some CRDTs)
• Not suitable for operations requiring coordination (uniqueness constraints, atomic updates)

When to use CRDTs:

• High-write distributed counters (page views, analytics)
• Collaborative editing (where eventual consistency is acceptable)
• Offline-first applications (sync when online)
• Shopping carts (with careful semantic design)

When NOT to use CRDTs:

• Bank account balances (need atomic transactions)
• Inventory (can’t prevent overselling without coordination)
• Unique constraints (usernames, reservation systems)
• Access control (need immediate consistency)

Part 4: Cross-System Interaction (CSI) Failures

Research from EuroSys 2023 found that 20% of catastrophic cloud incidents and 37% of failures in major open-source distributed systems are CSI failures – where both systems work correctly in isolation but fail when connected. This is the NASA Mars Climate Orbiter problem: one team used metric units, another used imperial. Both systems worked perfectly. The spacecraft burned up in Mars’s atmosphere because of their interaction.

Why CSI Failures Are Different

Not dependency failures: The downstream system is available, it just can’t process what upstream sends.

Not library bugs: Libraries are single-address-space and well-tested. CSI failures cross system boundaries where testing is expensive.

Not component failures: Each system passes its own test suite. The bug only emerges through interaction.

CSI failures manifest across three planes: Data plane (51% – schema/metadata mismatches), Management plane (32% – configuration incoherence), and Control plane (17% – API semantic violations).

For example, study of Apache Spark-Hive integration found 15 distinct discrepancies in simple write-read testing. Hive stored timestamps as long (milliseconds since epoch), Spark expected Timestamp type. Both worked in isolation, failed when integrated. Kafka and Flink encoding mismatch: Kafka set compression.type=lz4, Flink couldn’t decompress due to old LZ4 library. Configuration was silently ignored in Flink, leading to data corruption for 2 weeks before detection.

Why Testing Doesn’t Catch CSI Failures

Analysis of Spark found only 6% of integration tests actually test cross-system interaction. Most “integration tests” test multiple components of the same system. Cross-system testing is expensive and often skipped. The problem compounds with modern architectures:

• Microservices: More system boundaries to test
• Multi-cloud: Different clouds with different semantics
• Serverless: Fine-grained composition increases interaction surface area

How the Simulator Works

The CSI failure simulator models two systems exchanging data, with configurable discrepancies in schemas, encodings, and configurations.

System Model:

• Two systems (upstream ? downstream)
• Each has its own schema definition (field types, encoding, nullable fields)
• Each has its own configuration (timeouts, retry counts, etc.)
• Data flows from System A to System B with potential conversion failures

Failure Scenarios:

1. Metadata Mismatch (Hive/Spark):
   • System A: timestamp: long
   • System B: timestamp: Timestamp
   • Failure: Type coercion fails ~30% of the time
2. Schema Conflict (Producer/Consumer):
   • System A: encoding: latin-1
   • System B: encoding: utf-8
   • Failure: Silent data corruption
3. Configuration Incoherence (ServiceA/ServiceB):
   • System A: max_retries=3, timeout=30s
   • System B expects: max_retries=5, timeout=60s
   • Failure: ~40% of requests fail due to premature timeout
4. API Semantic Violation (Upstream/Downstream):
   • Upstream assumes: synchronous, thread-safe
   • Downstream is: asynchronous, not thread-safe
   • Failure: Race conditions, out-of-order processing
5. Type Confusion (SystemA/SystemB):
   • System A: amount: float
   • System B: amount: decimal
   • Failure: Precision loss in financial calculations

Implementation Details:

class DataSchema:
    def __init__(self, schema_id, fields, encoding, nullable_fields):
        self.fields = fields  # field_name -> type
        self.encoding = encoding
        
    def is_compatible(self, other):
        # Check field types and encoding
        return (self.fields == other.fields and 
                self.encoding == other.encoding)

class DataRecord:
    def serialize(self, target_schema):
        # Attempt type coercion
        for field, value in self.data.items():
            expected_type = target_schema.fields[field]
            actual_type = self.schema.fields[field]
            
            if expected_type != actual_type:
                # 30% failure on type mismatch (simulating real world)
                if random.random() < 0.3:
                    return None  # Serialization failure
        
        # Check encoding compatibility
        if self.schema.encoding != target_schema.encoding:
            if random.random() < 0.2:  # 20% silent corruption
                return None

Key Parameters:

The simulator doesn’t have many tunable parameters because CSI failures are about specific incompatibilities, not gradual degradation. Each scenario models a real-world pattern.

What You Can Learn:

1. Failure rates: CSI failures often manifest in 20-40% of requests (not 100%)
   • Some requests happen to have compatible data
   • Makes debugging harder (intermittent failures)
2. Failure location:
   • Research shows 69% of CSI fixes go in the upstream system, often in connector modules that are less than 5% of the codebase
   • Simulator shows which system fails (usually downstream)
3. Silent vs loud failures:
   • Type mismatches often crash (loud, easy to detect)
   • Encoding mismatches corrupt silently (hard to detect)
   • Config mismatches cause intermittent timeouts
4. Prevention effectiveness:
   • Schema registry eliminates metadata mismatches
   • Configuration validation catches config incoherence
   • Contract testing prevents API semantic violations

Key Insights from Simulation

The simulator demonstrates that cross-system integration testing is essential but often skipped. Unit tests of each system won’t catch these failures.

Prevention strategies validated by simulation:

1. Write-Read Testing: Write with System A, read with System B, verify integrity
2. Schema Registry: Single source of truth for data schemas, enforced across systems
3. Configuration Coherence Checking: Validate that shared configs match
4. Contract Testing: Explicit, machine-checkable API contracts

Hybrid Consistency Models

Modern systems increasingly use mixed consistency: RedBlue Consistency (2012) marks operations as needing strong consistency (red) or eventual consistency (blue). Replicache (2024) has the server assign final total order while clients do optimistic local updates with rebase. For example: Calendar Application

# Strong consistency for room reservations (prevent double-booking)
def book_conference_room(room_id, time_slot):
    with transaction(consistency='STRONG'):
        if room.is_available(time_slot):
            room.book(time_slot)
            return True
        return False

# CRDTs for collaborative editing (participant lists, notes)
def update_meeting_notes(meeting_id, notes):
    # LWW-Map CRDT, eventual consistency
    meeting.notes.merge(notes)

# Eventual consistency for preferences
def update_user_calendar_color(user_id, color):
    # Who cares if this propagates slowly?
    user_prefs[user_id] = color

Recent theoretical work on the CALM theorem proves that coordination-free consistency is achievable for certain problem classes. Research in 2025 provided mathematical definitions of when coordination is and isn’t required, separating coordination from computation.

What the Simulators Teach Us

Running all four simulators reveals the consistency spectrum:

No “best” consistency model exists:

• Quorums are best when you need linearizability and can tolerate latency
• CRDTs are best when you need high availability and can tolerate eventual consistency
• Neither approach “bypasses” CAP – they make different tradeoffs
• Real systems use hybrid models with different consistency for different operations

Practical Lessons

1. Design for Recovery, Not Just Prevention

The metastable failure simulator shows you can’t prevent all failures. Your retry logic, backoff strategy, and circuit breakers are more important than your happy path code. Validated strategies include:

• Exponential backoff with jitter (spread retries over time)
• Adaptive retry budgets (limit total fleet-wide retries)
• Circuit breakers (detect patterns, stop storms)
• Load shedding (fail fast rather than queue to death)

2. Understand the Consistency Spectrum

The CAP/PACELC simulator demonstrates that consistency is not binary. You need to understand:

• What consistency level do you actually need? (Most operations don’t need linearizability)
• What’s the latency cost? (Quorum reads in cross-region deployment can be 100x slower)
• What happens during partitions? (Can you sacrifice availability or must you serve stale data?)

Decision framework:

• Use strong consistency for: money, inventory, locks, compliance
• Use eventual consistency for: feeds, catalogs, analytics, caches
• Use hybrid models for: most real-world applications

3. Test Cross-System Interactions

The CSI failure simulator reveals that 86% of fixes go into connector modules that are less than 5% of your codebase. This is where bugs hide. Essential tests include:

• Write-read tests (write with System A, read with System B)
• Round-trip tests (serialize/deserialize across boundaries)
• Version compatibility matrix (test combinations)
• Schema validation (machine-checkable contracts)

4. Leverage CRDTs Where Appropriate

The CRDT simulator shows that conflict-free convergence is possible for specific problem types. But you need to:

• Understand the semantic limitations (LWW can lose data)
• Design merge behavior carefully (does it match user intent?)
• Handle garbage collection (tombstones, vector clocks)
• Accept eventual consistency (not suitable for all use cases)

5. Monitor for Sustaining Effects

Metastability, retry storms, and goodput collapse are self-sustaining failure modes. They persist after the trigger is gone. Critical metrics include:

• P99 latency vs timeout threshold (approaching timeout = danger)
• Retry rate vs success rate (high retries = storm risk)
• Queue depth (unbounded growth = admission control needed)
• Goodput vs throughput (doing useful work vs spinning)

Using the Simulators

All four simulators are available at: https://github.com/bhatti/simulators

Installation

git clone https://github.com/bhatti/simulators
cd simulators
pip install -r requirements.txt

Requirements:

• Python 3.7+
• streamlit (web UI)
• simpy (discrete event simulation)
• plotly (interactive visualizations)
• numpy, pandas (data analysis)

Running Individual Simulators

# Metastable failure simulator
streamlit run metastable_simulator.py

# CAP/PACELC consistency simulator
streamlit run cap_consistency_simulator.py

# CRDT simulator
streamlit run crdt_simulator.py

# CSI failure simulator
streamlit run csi_failure_simulator.py

Running All Simulators

python run_all_simulators.py

Conclusion

Building distributed systems means confronting failure modes that are expensive or impossible to reproduce in real environments:

• Metastable failures require specific load patterns and timing
• Consistency tradeoffs need multi-region deployments to observe
• CRDT convergence requires orchestrating concurrent operations across replicas
• CSI failures need exact schema/config mismatches that don’t exist in test environments

Simulators bridge the gap between theoretical understanding and practical intuition:

1. Cheaper than production testing: No cloud costs, no multi-region setup, instant feedback
2. Safer than production experiments: Crash the simulator, not your service
3. More complete than unit tests: See emergent behaviors, not just component correctness
4. Faster iteration: Tweak parameters, re-run in seconds, build intuition through experimentation

What You Can’t Learn Without Simulation

• When does retry amplification tip into metastability? (Depends on coordination slope, timeout, backoff)
• How much does quorum coordination actually cost? (Depends on network latency, replica count, workload)
• Do your CRDT semantics match user expectations? (Depends on merge behavior, conflict resolution)
• Will your schema changes break integration? (Depends on type coercion, encoding, version skew)

The goal isn’t to prevent all failures, that’s impossible. The goal is to understand, anticipate, and recover from the failures that will inevitably occur.

References

Key research papers and resources used in this post:

1. AWS Metastability Research (HotOS 2025) – Sustaining effects and goodput collapse
2. Marc Brooker on DSQL – Practical distributed SQL considerations
3. James Hamilton on Reliable Systems – Large-scale system design
4. CSI Failures Study (EuroSys 2023) – Cross-system interaction failures
5. PACELC Framework – Beyond CAP theorem
6. Marc Brooker on CAP – CAP theorem revisited
7. Anna CRDT Database – Autoscaling with CRDTs
8. Linearizability Paper – Herlihy & Wing’s foundational work
9. Designing Data-Intensive Applications by Martin Kleppmann
10. Distributed Systems Reading Group – MIT CSAIL
11. Jepsen.io – Kyle Kingsbury’s consistency testing
12. Aphyr’s blog – Distributed systems deep dives

Introduction

Over the years, I have seen countless data breaches leaking private personal data of customers. For example, Equifax exposed 147 million Americans’ SSNs and birth dates; Facebook leaked 533 million users’ personal details; Yahoo lost 3 billion accounts. This risk of leaking personal data is not unique to large companies but most companies play security chicken. They bet on luck that we haven’t been breached yet, so we must be fine. In many cases, companies don’t even know what PII they have, where it lives, or who can access it.

Unrestrained Production Access

Here’s what I have seen in most companies where I worked: DevOps teams with unrestricted access to production databases “for debugging.” Support engineers who can browse any customer’s SSN, medical records, or financial data. That contractor from six months ago who still has production credentials. Engineers who can query any table, any field, anytime. I’ve witnessed the consequences firsthand:

• Customer service reps browsing financial data of large customers “out of curiosity”
• APIs that return PII data without proper authorization policies
• Devops or support receives permanent permissions to access production data instead of time-bound or customer specific based on the underlying issue
• Engineers accidentally logging credit card numbers in plaintext

This violates OWASP’s principle of least privilege—grant only the minimum access necessary. But there’s an even worse problem: most companies can’t even identify which fields contain PII. They often don’t have policies on how to protect different kind of PII data based on risks.

The Scale Problem

In modern architectures, manual PII identification is impossible:

• Hundreds of microservices, each with dozens of data models
• Tens of thousands of API endpoints
• Constant schema evolution as teams ship daily
• Our single customer proto had 84 fields—multiply that by hundreds of services

Traditional approaches—manual reviews, compliance audits, security questionnaires—can’t keep up. By the time you’ve reviewed everything, the schemas have already changed.

Enter Agentic AI: From 0% to 92% PII Detection

I have been applying AI assistants and agents to solve complex problems for a while and I have been thinking about how can we automatically detect PII? Not just obvious fields like “ssn” or “credit_card_number,” but the subtle ones—employee IDs that could be cross-referenced. I then built an AI-powered system that uses LangChain, LangGraph, and Vertex AI to scan every proto definition, identify PII patterns, and classify sensitivity levels. Though iterative development, I went from:

• 0% accuracy: Naive prompt (“find PII fields”)
• 45% accuracy: Basic rules without specificity
• 92%+ accuracy: Iterative prompt engineering with explicit field mappings

It’s not perfect, but it’s infinitely better than the nothing most companies have.

The Real Problem: It’s Not Just About Compliance

Let me share some uncomfortable truths about PII in modern systems:

The Public API Problem

We had list APIs returning customer data like this:

{
  "customers": [
    {
      "id": "cust_123",
      "name": "John Doe",
      "email": "john@example.com",
      "ssn": "123-45-6789",
      "date_of_birth": "1990-01-15",
      "credit_score": 750,
    }
  ]
}

Someone with the API access could list all customers and capture their private data like ssn and date_of_birth.

The Internal Access Problem

One recurring issue I found with internal access is giving carte blanche access (often permanent) to devops environment or production database for debugging. In other cases, support team needed customer data for tickets. But did they need to see following PII data for all customers:

• Social Security Numbers?
• Medical records?
• Credit card numbers?
• Salary information?

Of course not. I saw often the list APIs return this PII data for all customers or calling GetAccount gave you everything without proper authorization policies.

The Compliance Nightmare

The government regulations like GDPR, CCPA, HIPAA, PCI-DSS have been growing but each has different rules about what constitutes PII, how it should be protected, and what happens if you leak it. Manual compliance checking is impossible at scale.

The RBAC Isn’t Enough Problem

I’ve spent years building authorization systems, believing RBAC was the answer. I wrote about it in Building a Hybrid Authorization System for Granular Access Control and created multiple authorization solutions like:

• PlexRBAC – A comprehensive RBAC library for Java/Scala with dynamic role hierarchies
• PlexRBACJS – JavaScript implementation with fine-grained permissions
• SaaS_RBAC – Multi-tenant RBAC with organization-level isolation

These systems can enforce incredibly sophisticated access controls. They can handle role inheritance, permission delegation, contextual access rules. But here’s what I learned the hard way: RBAC is useless if you don’t know what data needs protection. First, you need to identify PII. Then you can enforce field-level authorization.

The Solution: AI-Powered PII Detection with Proto Annotations

I built an Agentic AI based automation that:

1. Automatically scans all proto definitions for PII
2. Classifies sensitivity levels (HIGH, MEDIUM, LOW, PUBLIC)
3. Generates appropriate annotations for enforcement
4. Integrates with CI/CD to prevent PII leaks before deployment

Here’s what it looks like in action:

Before: Unmarked PII Everywhere

message Account {
  string id = 1;
  string first_name = 2;
  string ssn = 3;  // No indication this is sensitive!
  string email = 4;
  string credit_card_number = 5;  // Just sitting there, unprotected
  repeated string medical_conditions = 6;  // HIPAA violation waiting to happen
}

After: Fully Annotated with Sensitivity Levels

message Account {
  option (pii.v1.message_sensitivity) = HIGH;

  string id = 1 [
    (pii.v1.sensitivity) = LOW,
    (pii.v1.pii_type) = CUSTOMER_ID
  ];

  string first_name = 2 [
    (pii.v1.sensitivity) = LOW,
    (pii.v1.pii_type) = NAME
  ];

  string ssn = 3 [
    (pii.v1.sensitivity) = HIGH,
    (pii.v1.pii_type) = SSN
  ];

  string email = 4 [
    (pii.v1.sensitivity) = MEDIUM,
    (pii.v1.pii_type) = EMAIL_PERSONAL
  ];

  string credit_card_number = 5 [
    (pii.v1.sensitivity) = HIGH,
    (pii.v1.pii_type) = CREDIT_CARD
  ];

  repeated string medical_conditions = 6 [
    (pii.v1.sensitivity) = HIGH,
    (pii.v1.pii_type) = MEDICAL_RECORD
  ];
}

Now our authorization system knows exactly what to protect!

Architecture: How It All Works

The system uses a multi-stage pipeline combining LangChain, LangGraph, and Vertex AI:

Technical Implementation Deep Dive

1. The LangGraph State Machine

I used LangGraph to create a deterministic workflow for PII detection:

from langgraph.graph import StateGraph, END
from typing import TypedDict, List, Optional, Dict, Any
from langchain_google_vertexai import ChatVertexAI
from pydantic import BaseModel, Field

class PiiDetectionState(TypedDict):
    """State for PII detection workflow"""
    proto_file: str
    proto_content: str
    parsed_proto: Dict[str, Any]
    llm_analysis: Optional[ProtoAnalysis]
    final_report: Optional[PiiDetectionReport]
    annotated_proto: Optional[str]
    errors: List[str]

class PiiDetector:
    def __init__(self, model_name: str = "gemini-2.0-flash-exp"):
        self.llm = ChatVertexAI(
            model_name=model_name,
            project=PROJECT_ID,
            location=LOCATION,
            temperature=0.1,  # Low temperature for consistent classification
            max_output_tokens=8192,
            request_timeout=120  # Handle large protos
        )
        self.workflow = self._create_workflow()

    def _create_workflow(self) -> StateGraph:
        """Create the LangGraph workflow"""
        workflow = StateGraph(PiiDetectionState)

        # Add nodes for each step
        workflow.add_node("parse_proto", self._parse_proto_node)
        workflow.add_node("analyze_pii", self._analyze_pii_node)
        workflow.add_node("generate_annotations", self._generate_annotations_node)
        workflow.add_node("create_report", self._create_report_node)

        # Define the flow
        workflow.set_entry_point("parse_proto")
        workflow.add_edge("parse_proto", "analyze_pii")
        workflow.add_edge("analyze_pii", "generate_annotations")
        workflow.add_edge("generate_annotations", "create_report")
        workflow.add_edge("create_report", END)

        return workflow.compile()

    async def _analyze_pii_node(self, state: PiiDetectionState) -> PiiDetectionState:
        """Analyze PII using LLM with retry logic"""
        max_retries = 3
        retry_delay = 2

        for attempt in range(max_retries):
            try:
                # Create structured output chain
                analysis_chain = self.llm.with_structured_output(ProtoAnalysis)

                # Create the analysis prompt
                prompt = self.create_pii_detection_prompt(state['parsed_proto'])

                # Get LLM analysis
                result = await analysis_chain.ainvoke(prompt)

                if result:
                    state['llm_analysis'] = result
                    return state

            except Exception as e:
                if attempt < max_retries - 1:
                    await asyncio.sleep(retry_delay)
                    continue
                else:
                    state['errors'].append(f"LLM analysis failed: {str(e)}")

        return state

2. Pydantic Models for Structured Output

I used Pydantic to ensure consistent, structured responses from the LLM:

class FieldAnalysis(BaseModel):
    """Analysis of a single proto field for PII"""
    field_name: str = Field(description="The name of the field")
    field_path: str = Field(description="Full path like Message.field")
    contains_pii: bool = Field(description="Whether field contains PII")
    sensitivity: str = Field(description="HIGH, MEDIUM, LOW, or PUBLIC")
    pii_type: Optional[str] = Field(default=None, description="Type of PII")
    reasoning: str = Field(description="Explanation for classification")

class MessageAnalysis(BaseModel):
    """Analysis of a proto message"""
    message_name: str = Field(description="Name of the message")
    overall_sensitivity: str = Field(description="Highest sensitivity in message")
    fields: List[FieldAnalysis] = Field(description="Analysis of each field")

class ProtoAnalysis(BaseModel):
    """Complete analysis of a proto file"""
    messages: List[MessageAnalysis] = Field(description="All analyzed messages")
    services: List[ServiceAnalysis] = Field(default_factory=list)
    summary: AnalysisSummary = Field(description="Overall statistics")

3. The Critical Prompt Engineering

I found that the key to accurate PII detection is in the prompt. Here’s a battle-tested prompt that achieves 92%+ accuracy after many trial and errors:

def create_pii_detection_prompt(self) -> str:
    """Create the prompt for PII detection"""
    return """You are an expert in data privacy and PII detection.
    Analyze the Protocol Buffer definition and identify ALL fields that contain PII.

    STRICT Classification Rules - YOU MUST FOLLOW THESE EXACTLY:

    1. HIGH Sensitivity (MAXIMUM PROTECTION REQUIRED):
       ALWAYS classify these field names as HIGH:
       - ssn, social_security_number ? HIGH + SSN
       - tax_id, tin ? HIGH + TAX_ID
       - passport_number, passport ? HIGH + PASSPORT
       - drivers_license, driving_license ? HIGH + DRIVERS_LICENSE
       - bank_account_number ? HIGH + BANK_ACCOUNT
       - credit_card_number ? HIGH + CREDIT_CARD
       - credit_card_cvv ? HIGH + CREDIT_CARD
       - medical_record_number ? HIGH + MEDICAL_RECORD
       - health_insurance_id ? HIGH + HEALTH_INSURANCE
       - medical_conditions ? HIGH + MEDICAL_RECORD
       - prescriptions ? HIGH + MEDICAL_RECORD
       - password_hash, password ? HIGH + PASSWORD
       - api_key ? HIGH + API_KEY
       - salary, annual_income ? HIGH + null

    2. MEDIUM Sensitivity:
       - email, personal_email ? MEDIUM + EMAIL_PERSONAL
       - phone, mobile_phone ? MEDIUM + PHONE_PERSONAL
       - home_address ? MEDIUM + ADDRESS_HOME
       - date_of_birth, dob ? MEDIUM + DATE_OF_BIRTH
       - username ? MEDIUM + USERNAME
       - ip_address ? MEDIUM + IP_ADDRESS
       - device_id ? MEDIUM + DEVICE_ID
       - geolocation (latitude, longitude) ? MEDIUM + null

    3. LOW Sensitivity:
       - first_name, last_name, middle_name ? LOW + NAME
       - gender ? LOW + GENDER
       - work_email ? LOW + EMAIL_WORK
       - work_phone ? LOW + PHONE_WORK
       - job_title ? LOW + null
       - employer_name ? LOW + null

    4. PUBLIC (non-PII):
       - id (if system-generated)
       - status, created_at, updated_at
       - counts, totals, metrics

    IMPORTANT: Analyze EVERY SINGLE FIELD. Do not skip any.
    """

3. Handling the Gotchas

During development, I faced several challenges that required creative solutions:

Challenge 1: Multi-line Proto Annotations

Proto files often have annotations spanning multiple lines:

string ssn = 3 [
    (pii.v1.sensitivity) = HIGH,
    (pii.v1.pii_type) = SSN
];

Solution: Parse with look-ahead:

def extract_annotations(self, lines: List[str]) -> Dict:
    i = 0
    while i < len(lines):
        if '[' in lines[i]:
            # Collect until we find ']'
            annotation_text = lines[i]
            j = i + 1
            while j < len(lines) and '];' not in annotation_text:
                annotation_text += ' ' + lines[j]
                j += 1
            # Now parse the complete annotation
            self.parse_annotation(annotation_text)
            i = j
        else:
            i += 1

Challenge 2: Context-Dependent Classification

A field named id could be:

• PUBLIC if it’s a system-generated UUID
• LOW if it’s a customer ID that could be used for lookups
• MEDIUM if it’s an employee ID with PII implications

Solution: Consider the message context:

def classify_with_context(self, field_name: str, message_name: str) -> str:
    if message_name in ['Customer', 'User', 'Account']:
        if field_name == 'id':
            return 'LOW'  # Customer ID has some sensitivity
    elif message_name in ['System', 'Config']:
        if field_name == 'id':
            return 'PUBLIC'  # System IDs are not PII
    return self.default_classification(field_name)

Challenge 3: Handling Nested Messages and Maps

Real protos have complex structures:

message Account {
    map<string, string> metadata = 100;  // Could contain anything!
    repeated Address addresses = 101;
    Location last_location = 102;
}

Solution: Recursive analysis with inheritance:

def analyze_field(self, field: Field, parent_sensitivity: str = 'PUBLIC'):
    if field.type == 'map':
        # Maps could contain PII
        return 'MEDIUM' if parent_sensitivity != 'HIGH' else 'HIGH'
    elif field.is_message:
        # Analyze the referenced message
        message_sensitivity = self.analyze_message(field.message_type)
        return max(parent_sensitivity, message_sensitivity)
    else:
        return self.classify_field(field.name)

Real-World Testing

I tested the system on a test customer account proto with 84 fields. Here’s what happened:

Before: Original Proto Without Annotations

syntax = "proto3";

package pii.v1;

// Account represents a user account - NO PII ANNOTATIONS
message Account {
    // System fields
    string id = 1;
    string account_number = 2;
    AccountStatus status = 3;
    google.protobuf.Timestamp created_at = 4;
    google.protobuf.Timestamp updated_at = 5;

    // Personal information - UNPROTECTED PII!
    string first_name = 10;
    string last_name = 11;
    string middle_name = 12;
    string date_of_birth = 13;  // Format: YYYY-MM-DD
    string gender = 14;

    // Contact information - MORE UNPROTECTED PII!
    string email = 20;
    string personal_email = 21;
    string work_email = 22;
    string phone = 23;
    string mobile_phone = 24;
    string work_phone = 25;

    // Government IDs - CRITICAL PII EXPOSED!
    string ssn = 40;
    string tax_id = 41;
    string passport_number = 42;
    string drivers_license = 43;
    string national_id = 44;

    // Financial information - HIGHLY SENSITIVE!
    string bank_account_number = 50;
    string routing_number = 51;
    string credit_card_number = 52;
    string credit_card_cvv = 53;
    string credit_card_expiry = 54;
    double annual_income = 55;
    int32 credit_score = 56;

    // Medical information - HIPAA PROTECTED!
    string medical_record_number = 70;
    string health_insurance_id = 71;
    repeated string medical_conditions = 72;
    repeated string prescriptions = 73;

    // Authentication - SECURITY CRITICAL!
    string username = 80;
    string password_hash = 81;
    string security_question = 82;
    string security_answer = 83;
    string api_key = 84;
    string access_token = 85;

    // Device information
    string ip_address = 90;
    string device_id = 91;
    string user_agent = 92;
    Location last_location = 93;

    // Additional fields
    map<string, string> metadata = 100;
    repeated string tags = 101;
}

service AccountService {
    // All methods exposed without sensitivity annotations!
    rpc CreateAccount(CreateAccountRequest) returns (Account);
    rpc GetAccount(GetAccountRequest) returns (Account);
    rpc UpdateAccount(UpdateAccountRequest) returns (Account);
    rpc DeleteAccount(DeleteAccountRequest) returns (google.protobuf.Empty);
    rpc ListAccounts(ListAccountsRequest) returns (ListAccountsResponse);
    rpc SearchAccounts(SearchAccountsRequest) returns (SearchAccountsResponse);
}

After: AI-Generated Annotations (92.3% Accuracy!)

syntax = "proto3";

import "api/proto/pii/v1/sensitivity.proto";

// Account represents a user account - FULLY ANNOTATED WITH PII SENSITIVITY
message Account {
    option (pii.v1.message_sensitivity) = HIGH;

    // System fields
    string id = 1 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = CUSTOMER_ID];
    string account_number = 2 [(pii.v1.sensitivity) = MEDIUM];
    AccountStatus status = 3;  // Enum - no PII
    google.protobuf.Timestamp created_at = 4;  // PUBLIC
    google.protobuf.Timestamp updated_at = 5;  // PUBLIC

    // Personal information - PROPERLY CLASSIFIED
    string first_name = 10 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = NAME];
    string last_name = 11 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = NAME];
    string middle_name = 12 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = NAME];
    string date_of_birth = 13 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = DATE_OF_BIRTH];
    string gender = 14 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = GENDER];

    // Contact information - MEDIUM SENSITIVITY
    string email = 20 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = EMAIL_PERSONAL];
    string personal_email = 21 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = EMAIL_PERSONAL];
    string work_email = 22 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = EMAIL_WORK];
    string phone = 23 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = PHONE_PERSONAL];
    string mobile_phone = 24 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = PHONE_PERSONAL];
    string work_phone = 25 [(pii.v1.sensitivity) = LOW, (pii.v1.pii_type) = PHONE_WORK];

    // Government IDs - ALL HIGH SENSITIVITY ?
    string ssn = 40 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = SSN];
    string tax_id = 41 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = TAX_ID];
    string passport_number = 42 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = PASSPORT];
    string drivers_license = 43 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = DRIVERS_LICENSE];
    string national_id = 44 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = NATIONAL_ID];

    // Financial information - ALL HIGH SENSITIVITY ?
    string bank_account_number = 50 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = BANK_ACCOUNT];
    string routing_number = 51 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = ROUTING_NUMBER];
    string credit_card_number = 52 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = CREDIT_CARD];
    string credit_card_cvv = 53 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = CREDIT_CARD];
    string credit_card_expiry = 54 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = CREDIT_CARD];
    double annual_income = 55 [(pii.v1.sensitivity) = HIGH];
    int32 credit_score = 56 [(pii.v1.sensitivity) = HIGH];

    // Medical information - ALL HIGH SENSITIVITY ?
    string medical_record_number = 70 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = MEDICAL_RECORD];
    string health_insurance_id = 71 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = HEALTH_INSURANCE];
    repeated string medical_conditions = 72 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = MEDICAL_RECORD];
    repeated string prescriptions = 73 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = MEDICAL_RECORD];

    // Authentication - ALL HIGH SENSITIVITY ?
    string username = 80 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = USERNAME];
    string password_hash = 81 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = PASSWORD];
    string security_question = 82 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = PASSWORD];
    string security_answer = 83 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = PASSWORD];
    string api_key = 84 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = API_KEY];
    string access_token = 85 [(pii.v1.sensitivity) = HIGH, (pii.v1.pii_type) = API_KEY];

    // Device information - MEDIUM SENSITIVITY
    string ip_address = 90 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = IP_ADDRESS];
    string device_id = 91 [(pii.v1.sensitivity) = MEDIUM, (pii.v1.pii_type) = DEVICE_ID];
    string user_agent = 92 [(pii.v1.sensitivity) = LOW];
    Location last_location = 93;  // Location message handled separately

    // Additional fields
    map<string, string> metadata = 100 [(pii.v1.sensitivity) = MEDIUM];
    repeated string tags = 101;  // PUBLIC
}

// Service methods also get sensitivity annotations
service AccountService {
    rpc CreateAccount(CreateAccountRequest) returns (Account) {
        option (pii.v1.method_sensitivity) = HIGH;
        option (pii.v1.audit_pii_access) = true;
    }

    rpc GetAccount(GetAccountRequest) returns (Account) {
        option (pii.v1.method_sensitivity) = HIGH;
        option (pii.v1.audit_pii_access) = true;
    }

    // ... all methods properly annotated
}

Results: 92.3% Accuracy!

Here’s the actual output from our final test run:

Testing PII detection on: ../api/proto/pii/v1/account_without_annotations.proto
================================================================================

================================================================================
PII DETECTION REPORT
================================================================================
Total Fields Analyzed: 84
PII Fields Detected: 57
Non-PII Fields: 27

Fields by Sensitivity Level:
  HIGH: 22 fields
  MEDIUM: 22 fields
  LOW: 13 fields
  PUBLIC: 27 fields

HIGH Sensitivity Fields (22):
  • Account.ssn ? SSN
  • Account.tax_id ? TAX_ID
  • Account.passport_number ? PASSPORT
  • Account.drivers_license ? DRIVERS_LICENSE
  • Account.national_id ? NATIONAL_ID
  • Account.bank_account_number ? BANK_ACCOUNT
  • Account.routing_number ? ROUTING_NUMBER
  • Account.credit_card_number ? CREDIT_CARD
  • Account.credit_card_cvv ? CREDIT_CARD
  • Account.annual_income ? null
  • Account.credit_score ? null
  • Account.salary ? null
  • Account.medical_record_number ? MEDICAL_RECORD
  • Account.health_insurance_id ? HEALTH_INSURANCE
  • Account.medical_conditions ? MEDICAL_RECORD
  • Account.prescriptions ? MEDICAL_RECORD
  • Account.password_hash ? PASSWORD
  • Account.security_question ? PASSWORD
  • Account.security_answer ? PASSWORD
  • Account.api_key ? API_KEY
  • Account.access_token ? API_KEY
  • CreateAccountRequest.account ? null

[Additional fields by sensitivity level...]

================================================================================

Annotated proto saved to: output/account_with_detected_annotations.proto

================================================================================
VERIFICATION: Comparing with Reference Implementation
================================================================================

Field Annotations:
  ? Correct: 60
  ? Incorrect: 5
  ??  Missing: 0
  ? Extra: 0

Message Annotations:
  ? Correct: 8
  ? Incorrect: 0
  ??  Missing: 1

Method Annotations:
  ? Correct: 0
  ? Incorrect: 6
  ??  Missing: 0

Overall Field Accuracy: 92.3%
? VERIFICATION PASSED (>=80% accuracy)

Note: The LLM may classify some fields differently based on context.

================================================================================
SUMMARY
================================================================================
Total fields analyzed: 84
PII fields detected: 57

Fields by sensitivity level:
  HIGH: 22 fields
  MEDIUM: 22 fields
  LOW: 13 fields
  PUBLIC: 27 fields

Test completed successfully!

The system correctly identified:

• ? 100% of HIGH sensitivity fields (SSNs, credit cards, medical records)
• ? 95% of MEDIUM sensitivity fields (personal emails, phone numbers, addresses)
• ? 85% of LOW sensitivity fields (names, work emails, job titles)
• ? 100% of PUBLIC fields (IDs, timestamps, enums)

Why 92.3% Accuracy Matters

1. Perfect HIGH Sensitivity Detection: The system caught 100% of the most critical PII – SSNs, credit cards, medical records. These are the fields that can destroy lives if leaked.
2. Conservative Classification: When uncertain, the system errs on the side of caution. It’s better to over-protect a field than to expose PII.
3. Human Review Still Needed: The 8% difference is where human expertise adds value. The AI does the heavy lifting, humans do the fine-tuning.
4. Continuous Improvement: Every correction teaches the system. Our accuracy improved from 0% to 45% to 92% through iterative refinement.

Integration with Field-Level Authorization

I also built a prototype for enforcing field-level authorization and masking PII data outside this project but here is a general approach for enforcement of PII protection policies and masking response fields:

Step 1: Generate Authorization Rules

def generate_authz_rules(proto_with_annotations: str) -> Dict:
    """Generate authorization rules from annotated proto"""
    rules = {}

    for field in parse_annotated_proto(proto_with_annotations):
        if field.sensitivity == 'HIGH':
            rules[field.path] = {
                'required_roles': ['admin', 'compliance_officer'],
                'required_scopes': ['pii.high.read'],
                'audit': True,
                'mask_in_logs': True
            }
        elif field.sensitivity == 'MEDIUM':
            rules[field.path] = {
                'required_roles': ['support', 'admin'],
                'required_scopes': ['pii.medium.read'],
                'audit': True,
                'mask_in_logs': False
            }

    return rules

Step 2: Runtime Enforcement

// In your gRPC interceptor
func (i *AuthzInterceptor) UnaryInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    // Get user's roles and scopes
    user := auth.UserFromContext(ctx)

    // Check field-level permissions
    response, err := handler(ctx, req)
    if err != nil {
        return nil, err
    }

    // Filter response based on PII annotations
    filtered := i.filterResponse(response, user)

    return filtered, nil
}

func (i *AuthzInterceptor) filterResponse(
    response interface{},
    user *auth.User,
) interface{} {
    // Use reflection to check each field's annotation
    v := reflect.ValueOf(response)
    for i := 0; i < v.NumField(); i++ {
        field := v.Type().Field(i)

        // Get PII annotation from proto
        sensitivity := getPIISensitivity(field)

        // Check if user has permission
        if !user.HasPermission(sensitivity) {
            // Mask or remove the field
            v.Field(i).Set(reflect.Zero(field.Type))
        }
    }

    return response
}

Step 3: The Magic Moment

Here is an example response from an API with PII data that enforces proper PII data protection:

// Before: Everything exposed
{
  "customer": {
    "name": "John Doe",
    "ssn": "123-45-6789",  // They see this!
    "credit_card": "4111-1111-1111-1111"  // And this!
  }
}

// After: Field-level filtering based on PII annotations
{
  "customer": {
    "name": "John Doe",
    "ssn": "[REDACTED]",  // Protected!
    "credit_card": "[REDACTED]"  // Protected!
  }
}

CI/CD Integration: Catching PII Before Production

This tool can be easily integrated with CI/CD pipelines to identify PII data if proper annotations are missing:

# .github/workflows/pii-detection.yml
name: PII Detection Check

on:
  pull_request:
    paths:
      - '**/*.proto'

jobs:
  detect-pii:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          pip install -r check-pii-automation/requirements.txt

      - name: Detect PII in Proto Files
        env:
          GCP_PROJECT: ${{ secrets.GCP_PROJECT }}
        run: |
          cd check-pii-automation

          # Scan all proto files
          for proto in $(find ../api/proto -name "*.proto"); do
            echo "Scanning $proto"
            python pii_detector.py "$proto" \
              --output "output/$(basename $proto)" \
              --json "output/$(basename $proto .proto).json"
          done

      - name: Check for Unannotated PII
        run: |
          # Fail if HIGH sensitivity PII found without annotations
          for report in check-pii-automation/output/*.json; do
            high_pii=$(jq '.fields[] | select(.sensitivity == "HIGH" and .annotated == false)' $report)
            if [ ! -z "$high_pii" ]; then
              echo "? ERROR: Unannotated HIGH sensitivity PII detected!"
              echo "$high_pii"
              exit 1
            fi
          done

      - name: Generate Security Report
        if: always()
        run: |
          python check-pii-automation/generate_security_report.py \
            --input output/ \
            --output security_report.md

      - name: Comment on PR
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const report = fs.readFileSync('security_report.md', 'utf8');

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: report
            });

Advanced Features: Learning and Adapting

1. Custom PII Patterns

As, every organization has unique PII, we can support custom patterns:

# custom_pii_rules.yaml
custom_patterns:
  - name: "employee_badge_number"
    pattern: "badge_.*|.*_badge_id"
    sensitivity: "MEDIUM"
    pii_type: "EMPLOYEE_ID"

  - name: "internal_customer_reference"
    pattern: "cust_ref_.*|customer_reference"
    sensitivity: "LOW"
    pii_type: "CUSTOMER_ID"

  - name: "biometric_data"
    pattern: "fingerprint.*|face_.*|retina_.*"
    sensitivity: "HIGH"
    pii_type: "BIOMETRIC"

2. Context-Aware Classification

We can also learn from the codebase:

class ContextAwarePiiDetector:
    def __init__(self):
        self.context_rules = self.learn_from_codebase()

    def learn_from_codebase(self):
        """Learn patterns from existing annotated protos"""
        patterns = {}

        # Scan all existing annotated protos
        for proto_file in glob.glob("**/*.proto"):
            annotations = self.extract_annotations(proto_file)

            for field, annotation in annotations.items():
                # Learn the pattern
                if field not in patterns:
                    patterns[field] = []
                patterns[field].append({
                    'context': self.get_message_context(field),
                    'sensitivity': annotation['sensitivity']
                })

        return patterns

    def classify_with_learned_context(self, field_name: str, context: str):
        """Use learned patterns for classification"""
        if field_name in self.context_rules:
            # Find similar contexts
            for rule in self.context_rules[field_name]:
                if self.context_similarity(context, rule['context']) > 0.8:
                    return rule['sensitivity']

        return self.default_classification(field_name)

3. Incremental Learning from Corrections

Also, we can apply a RLHF (Reinforcement learning from human feedback) based mechanism to learn from human corrects a classification:

def record_correction(self, field: str, ai_classification: str, human_correction: str):
    """Learn from human corrections"""
    correction_record = {
        'field': field,
        'ai_said': ai_classification,
        'human_said': human_correction,
        'context': self.get_full_context(field),
        'timestamp': datetime.now()
    }

    # Store in vector database for RAG
    self.knowledge_base.add_correction(correction_record)

    # Update prompt if pattern emerges
    if self.count_similar_corrections(field) > 3:
        self.update_classification_rules(field, human_correction)

Results: What We Achieved

Before the System

• Hours of manual review for each proto change
• No systematic way to track PII across services
• Compliance audits were nightmares

After Implementation

• Automated detection in under 30 seconds
• Complete PII inventory across all services
• Compliance reports generated automatically
• 92%+ accuracy in classification

Performance Optimization: From 0% to 92%

Above journey to 92% accuracy wasn’t straightforward. Here’s how it was improved:

Iteration 1: Generic Prompt (0% Accuracy)

# Initial naive approach
prompt = "Find PII fields in this proto and classify their sensitivity"
# Result: LLM returned None or generic responses

Iteration 2: Basic Rules (45% Accuracy)

# Added basic rules but not specific enough
prompt = """
Classify fields as:
- HIGH: Very sensitive data
- MEDIUM: Somewhat sensitive
- LOW: Less sensitive
"""
# Result: Everything classified as MEDIUM

Iteration 3: Explicit Field Mapping (92% Accuracy)

# The breakthrough: explicit field name patterns
prompt = """
STRICT Classification Rules - YOU MUST FOLLOW THESE EXACTLY:

1. HIGH Sensitivity:
   ALWAYS classify these field names as HIGH:
   - ssn, social_security_number ? HIGH + SSN
   - credit_card_number ? HIGH + CREDIT_CARD
   [... explicit mappings ...]
"""
# Result: 92.3% accuracy!

Key Performance Improvements

1. Retry Logic with Exponential Backoff

   for attempt in range(max_retries):
       try:
           result = await self.llm.ainvoke(prompt)
           if result:
               return result
       except RateLimitError:
           delay = 2 ** attempt  # 2, 4, 8 seconds
           await asyncio.sleep(delay)

2. Request Batching for Multiple Files

   async def batch_process(proto_files: List[Path]):
       # Process in batches of 5 to avoid rate limits
       batch_size = 5
       for i in range(0, len(proto_files), batch_size):
           batch = proto_files[i:i+batch_size]
           tasks = [detect_pii(f) for f in batch]
           results = await asyncio.gather(*tasks)
           # Add delay between batches
           await asyncio.sleep(2)

3. Caching for Development

   @lru_cache(maxsize=100)
   def get_cached_analysis(proto_hash: str):
       # Cache results during development/testing
       return previous_analysis

Lessons Learned: The Hard Way

1. Start with High-Value PII

Don’t try to classify everything at once. Start with:

• Government IDs (SSN, passport)
• Financial data (credit cards, bank accounts)
• Medical information
• Authentication credentials

Get these right first, then expand.

2. False Positives Are Better Than False Negatives

We tuned for high recall (catching all PII) over precision. Why? It’s better to over-classify a field as sensitive than to leak an SSN.

3. Context Matters More Than Field Names

A field called data could be anything. Look at:

• The message it’s in
• Surrounding fields
• Comments in the proto
• How it’s used in code

4. Make Annotations Actionable

Don’t just mark fields as “sensitive”. Specify:

• Exact sensitivity level (HIGH/MEDIUM/LOW)
• PII type (SSN, CREDIT_CARD, etc.)
• Required protections (encryption, masking, audit)

5. Integrate Early in Development

The best time to annotate PII is when the field is created, not after it’s in production. Make PII detection part of proto creation and API review process.

Getting Started

Here is how you can start with protecting your customers’ data:

Step 1: Install and Configure

# Clone the repository
git clone https://github.com/bhatti/todo-api-errors.git
cd todo-api-errors/check-pii-automation

# Set up Python environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Configure GCP
export GCP_PROJECT=your-project-id
export GCP_REGION=us-central1

# Authenticate with Google Cloud
gcloud auth application-default login

Step 2: Run Your First Scan

# Scan a proto file
python pii_detector.py path/to/your/file.proto \
  --output annotated.proto \
  --json report.json

# Review the report
cat report.json | jq '.fields[] | select(.sensitivity == "HIGH")'

Step 3: Real-World Example

Here’s a complete example using our test proto:

# 1. Scan the proto without annotations
python pii_detector.py ../api/proto/pii/v1/account_without_annotations.proto \
  --output output/account_annotated.proto \
  --json output/report.json

# 2. View the detection summary
echo "=== PII Detection Summary ==="
cat output/report.json | jq '{
  total_fields: .total_fields,
  pii_detected: .pii_fields,
  high_sensitivity: [.fields[] | select(.sensitivity == "HIGH") | .field_path],
  accuracy: "\(.pii_fields) / \(.total_fields) = \((.pii_fields / .total_fields * 100 | floor))%"
}'

# 3. Compare with reference implementation
python test_pii_detection.py

# 4. View the annotated proto
head -50 output/account_annotated.proto

Expected output:

=== PII Detection Summary ===
{
  "total_fields": 84,
  "pii_detected": 57,
  "high_sensitivity": [
    "Account.ssn",
    "Account.tax_id",
    "Account.credit_card_number",
    "Account.medical_record_number",
    "Account.password_hash"
  ],
  "accuracy": "57 / 84 = 67%"
}

Verification Results:
? Correct Classifications: 60
Overall Accuracy: 92.3%

Step 4: Integrate with CI/CD

Add the GitHub Action above to your repository. Start with warnings, then move to blocking deployments.

Step 5: Implement Field-Level Authorization

Use the annotations to enforce access control in your services. Start with the highest sensitivity fields.

Step 6: Monitor and Improve

Track false positives/negatives. Update custom rules. Share learnings with your team.

Conclusion: Privacy as Code

I have learned that manual API reviews are insufficient to evaluate risks of sensitive field when dealing with hundreds of services. Also, this responsibility can’t all be delegated to developers as it requires collaboration and feedback from security, legal and product teams. We need tooling and automated processes that understand and protect PII automatically. Every new field, every API change, every refactor is a chance for PII to leak. But with AI-powered detection, we can make privacy protection as automatic as running tests. The system we built isn’t perfect – 92% accuracy means we still miss 8% of PII. But it’s infinitely better than the 0% we were catching before.

The code is at https://github.com/bhatti/todo-api-errors. Star it, fork it, break it, improve it.

Resources and References

• Project Repository
• LangChain Documentation
• LangGraph Guide
• Vertex AI Documentation
• GDPR Compliance Guide
• CCPA Requirements
• My RBAC Libraries

Back in 1974, physicist Richard Feynman gave a graduation speech at Caltech about something he called “cargo cult science.” He told a story about islanders in the South Pacific who, after World War II, built fake airstrips and control towers out of bamboo. They’d seen cargo planes land during the war and figured if they recreated what they saw—runways, headsets, wooden antennas—the planes would come back with supplies. They copied the appearance but missed the substance. The planes never came. Feynman used this to describe bad research—studies that look scientific on the surface but lack real rigor. Researchers going through the motions without understanding what makes science actually work.

Software engineering does the exact same thing. I’ve been doing this long enough to see the pattern repeat everywhere: teams adopt tools and practices because that’s what successful companies use, without asking if it makes sense for them. Google uses monorepos? We need a monorepo. Amazon uses microservices? Time to split our monolith. Kubernetes is what “real” companies use? Better start writing YAML.

In my previous post, I wrote about how layers of abstraction have made software too complex. This post is about a related problem: we’re not just dealing with necessary complexity—we’re making things worse by cargo culting what other companies do. We build the bamboo control towers and wonder why the planes don’t land. This is cargo cult software development, and I am sharing what I’ve learned here.

Executive Stack Envy

Executives suffer from massive stack envy. The executive reads about scalability of Kafka so suddenly we need Kafka. Never mind that we already have RabbitMQ and IBM MQSeries running just fine. Then another executive decides Google Pub/Sub is “more cloud native.” Now we have four message queues. Nobody provides guidance on how to use any of them. I watched teams struggle with poisonous messages for weeks. They’d never heard of dead letter queues.

On the database side, it’s the same pattern. In the early 2000s, I saw everyone rushed to adopt object-oriented databases like Versant and ObjectStore but they were proved to be short lived. At one company, leadership bet everything on a graph database. When customers came, scalability collapsed. We spent the next six years migrating away—not because migration was inherently hard, but because engineers built an overly complex migration architecture. Classic pattern: complexity for promotion, not for solving problems.

Meanwhile, at another company: we already had CloudSQL. Some teams moved to AlloyDB. Then an executive discovered Google Spanner. Now we have three databases. Nobody can explain why. Nobody knows which service uses which. At one company, we spent five years upgrading everything to gRPC. Created 500+ services. Nobody performance tested any of it until a large customer signed up. That’s when we discovered the overhead—gRPC serialization, microservice hops, network calls—it all compounded.

The Sales Fiction

Sales promised four nines availability, sub-100ms latency, multi-region DR. “Netflix-like reliability.” Reality? Some teams couldn’t properly scale within a single region. The DR plan was a wiki page nobody tested. Nobody understood the dependencies.

The Complexity Tax

Every service needs monitoring, logging, deployment pipelines, load balancing, service mesh config. Every network call adds latency and failure modes. Every distributed transaction risks inconsistency [How Abstraction is Killing Software: A 30-Year Journey Through Complexity].

The Monorepo That Ate Our Productivity

At one company, leadership decided we needed a monorepo “because Google uses one.” They’d read about how Google Chrome’s massive codebase benefited from having all dependencies in one place. What they missed was that Google has hundreds of engineers dedicated solely to tooling support.

Our reality? All services—different languages, different teams—got crammed into one repository. The promise was better code sharing. The result was forced dependency alignment that broke builds constantly. A simple package update in one service would cascade failures across unrelated services. Build times ballooned to over an hour and engineers spent endless hours fighting the build system.

The real kicker: most of our services only needed to communicate through APIs. We could have used service interfaces, but instead we created compile-time dependencies where none should have existed. At my time at Amazon, we handled shared code with live version dependencies that would trigger builds only when actually affected. There are alternatives—we just didn’t explore them.

Blaze Builds and the Complexity Tax

The same organization then adopted Bazel (Google’s open-sourced version of Blaze). Again, the reasoning was “Google uses it, so it must be good.” Nobody asked whether our small engineering team needed the same build system as Google’s tens of thousands of engineers. Nobody calculated the learning curve cost. Nobody questioned whether our relatively simple microservices needed this level of build sophistication. The complexity tax was immediate and brutal. New engineers took weeks to understand the build system. Simple tasks became complicated. Debugging build failures required specialized knowledge that only a few people possessed. We’d traded a problem we didn’t have for a problem we couldn’t solve.

The Agile Cargo Cult

I’ve watched dozens of companies claim they’re “doing Agile” while missing every principle that makes Agile work. They hold standups, run sprints, track velocity—all the visible rituals. The results? Same problems as before, now with more meetings.

Standups That Aren’t

At one company, “daily standups” lasted 30 minutes. Each developer gave a detailed status report to their manager. Everyone else mentally checked out waiting their turn. Nobody coordinated. It was a status meeting wearing an Agile costume.

The Velocity Obsession

Another place tracked velocity religiously. Management expected consistent story points every sprint. When velocity dropped, teams faced uncomfortable questions about “productivity.” Solution? Inflate estimates. Break large stories into tiny ones. The velocity chart looked great. The actual delivery? Garbage. Research shows teams game metrics when measured on internal numbers instead of customer value.

Product Owners Who Aren’t

I’ve seen “Product Owners” who were actually project managers in disguise. They translated business requirements into user stories. Never talked to customers. Couldn’t make product decisions. Spent their time tracking progress and managing stakeholders. Without real product ownership, teams build features nobody needs. The Agile ceremony continues, the product fails.

Copying Without Understanding

The pattern is always the same: read about Spotify’s squads and tribes, implement the structure, wonder why it doesn’t work. They copied the org chart but missed the culture of autonomy, the customer focus, the experimental mindset. Or they send everyone to a two-day Scrum certification. Teams return with a checklist of activities—sprint planning, retrospectives, story points—but no understanding of why these matter. They know the mechanics, not the principles.

Why It Fails

The academic research identified the problem: teams follow practices without understanding the underlying principles. They cancel meetings when the Scrum Master is absent (because they’re used to managers running meetings). They bring irrelevant information to standups (because they think it’s about reporting, not coordinating). They wait for task assignments instead of self-organizing (because autonomy is scary). Leadership mandates “Agile transformation” without changing how they make decisions or interact with teams. They want faster delivery and better predictability—the outcomes—without the cultural changes that enable those outcomes.

The Real Problem

True Agile requires empowering teams to make decisions. Most organizations aren’t ready for that. They create pseudo-empowerment: teams can choose how to implement predetermined requirements. They can organize their work as long as they hit the deadlines. They can self-manage within tightly controlled boundaries.

Platform Engineering and the Infrastructure Complexity Trap

Docker and Kubernetes are powerful tools. They solve real problems. But here’s what nobody talks about: they add massive complexity, and most organizations don’t have the expertise to handle it. I watched small startup adopt Kubernetes. They could have run on services directly EC2 instances. Instead, they had a three-node cluster, service mesh, ingress controllers, the whole nine yards.

Platform Teams That Made Things Worse

Platform engineering was supposed to make developers’ lives easier. Instead, I’ve watched platform teams split by technology—the Kubernetes team, the Terraform team, the CI/CD team—each making things harder. The pattern was consistent: they’d either expose raw complexity or build leaky abstractions that constrained without simplifying. One platform team exposed raw Kubernetes YAML to developers, expecting them to become Kubernetes experts overnight.

The fundamental problem? Everyone had to understand Kubernetes, Istio, Terraform, and whatever else the platform team used. The abstractions leaked everywhere. And the platform teams didn’t understand what the application teams were actually building—they’d never worked with the gRPC services they were supposed to support. The result was bizarre workarounds. One team found Istio was killing their long-running database queries during deployments. Their solution? Set terminationDrainDuration to 2 hours. They weren’t experts in Istio, so instead of fixing the real problem—properly implementing graceful shutdown with query cancellation—they just cranked the timeout to an absurd value.

When something broke, nobody could tell if it was the app or the platform. Teams burned days or weeks debugging through countless layers of abstraction.

The Microservices Cargo Cult

Every company wants microservices now. It’s modern, it’s scalable, it’s what Amazon does. I’ve watched this pattern repeat across multiple companies. They split monoliths into microservices and get all the complexity without any of the benefits. Let me tell you what I’ve seen go wrong.

Idempotency? Never Heard of It

At one company, many services didn’t check for duplicate requests resulting in double charges or incorrect balances. Classic non-atomic check-then-act: check if transaction exists, then create it—two separate database calls. Race condition waiting to happen. Two requests hit simultaneously, both check, both see nothing, both charge the customer. Same pattern everywhere I looked. I wrote about these antipatterns in How Duplicate Detection Became the Dangerous Impostor of True Idempotency.

The Pub/Sub Disaster

At another place, Google Pub/Sub had an outage. Publishers timed out, retried their events. When Pub/Sub recovered, both original and retry got delivered—with different event IDs. Duplicate events everywhere. Customer updates applied twice. Transactions processed multiple times. The Events Service was built for speed, not deduplication. Each team handled duplicates their own way. Many didn’t handle them at all. We spent days manually finding data drift and fixing it. No automated reconciliation, no detection—just manual cleanup after the fact.

No Transaction Boundaries

Simple database joins became seven network calls across services. Create order -> charge payment -> allocate inventory -> update customer -> send notification. Each call a potential failure point. Something fails midway? Partial state scattered across services. No distributed transactions, no sagas, just hope. I explained proper implementation of transaction boundaries in Transaction Boundaries: The Foundation of Reliable Systems.

Missing the Basics

But the real problem was simpler than that. I’ve seen services deployed without:

• Proper health checks. Teams reused the same shallow check for liveness and readiness. Kubernetes routed traffic to pods that weren’t ready.
• Monitoring and alerts. Services ran in production with no alarms. We’d find out about issues from customer complaints.
• Dependency testing. Nobody load tested their dependencies. Scaling up meant overwhelming downstream services that couldn’t handle the traffic.
• Circuit breakers. One slow service took down everything calling it. No timeouts, no fallbacks.
• Graceful shutdown. Deployments dropped requests because nobody coordinated shutdown timeouts between application, Istio, and Kubernetes.
• Distributed tracing. Logs scattered across services with no correlation IDs. Debugging meant manually piecing together what happened from nine different log sources.
• Backup and recovery. Nobody tested their disaster recovery until disaster struck.

The GRPC Disaster Nobody Talks About

Another organization went all-in on GRPC for microservices. The pitch was compelling: better performance, strongly typed interfaces, streaming support. What could go wrong? Engineers copied GRPC examples without understanding connection management. Nobody grasped how GRPC’s HTTP/2 persistent connections work or the purpose of connection pooling. Services would start before the Istio sidecar was ready. Application tries an outbound GRPC call—ECONNREFUSED. Pod crashes, Kubernetes restarts it, repeat. The fix was one annotation nobody added: sidecar.istio.io/holdApplicationUntilProxyStarts: "true".

Shutdown was worse. Kubernetes sends SIGTERM, Istio sidecar shuts down immediately, application still draining requests. Dropped connections everywhere. The fix required three perfectly coordinated timeout values:

• Application shutdown: 40s
• Istio drain: 45s
• Kubernetes grace period: 65s

Load balancing was a disaster. HTTP/2 creates one persistent connection and multiplexes all requests through it. Kubernetes’ round-robin load balancing works at the connection level. Result? All traffic to whichever pod got the first connection. Health checks were pure theater. Teams copied the same probe definition for both liveness and readiness. Even distinct probes were “shallow”—a database ping that doesn’t validate the service can actually function. Services marked “ready” that immediately 500’d on real traffic.

The HTTP-to-GRPC proxy layer? Headers weren’t properly mapped between protocols. Auth tokens got lost in translation. Customer-facing errors were cryptic GRPC status codes instead of meaningful messages. I ended up writing detailed guides on GRPC load balancing in Kubernetes, header mapping, and error handling. These should have been understood before adoption, not discovered through production failures.

The Caching Silver Bullet That Shot Us in the Foot

“Just add caching” became the answer to every performance problem. Database slow? Add Redis. API slow? Add CDN. At one company, platform engineering initially didn’t support Redis. So application teams spun up their own clusters. No standards. No coordination. Just dozens of Redis instances scattered across environments, each configured differently. Eventually, platform engineering released Terraform modules for Redis. Problem solved, right? Wrong. They provided the infrastructure with almost no guidance on how to use it properly. Teams treated it as a magic performance button.

What Actually Happened

Teams started caching without writing fault-tolerant code. One service had Redis connection timeouts set to 30 seconds. When Redis became unavailable, every request waited 30 seconds to fail. The cascading failures took down the entire application. Another team cached massive objects—full customer balances, assets, events, transactions, etc. Their cache hydration on startup took 10 minutes. Every deploy meant 10 minutes of degraded performance while the cache warmed up. Auto-scaling was useless because new pods weren’t ready to serve traffic. Nobody calculated cache invalidation complexity. Nobody considered memory costs. Nobody thought about cache coherency across regions.

BiModal Hell

The worst part? BiModal logic. Cache hit? Fast. Cache miss? Slow. Cold cache? Everything’s slow until it warms up. This obscured real problems—race conditions, database failures—because performance was unpredictable. Was it slow because of a cache miss or because the database was dying? Nobody knew. I’ve documented more of these war stories—cache poisoning, thundering herds, memory leaks, security issues with unencrypted credentials. The pattern was always the same: reach for caching before understanding the actual problem.

Infrastructure as Code: The Code That Wasn’t

“We do infrastructure as code” was the proud claim at multiple companies I’ve worked at. The reality? Terraform or AWS CloudFormation templates existed, sure. But some of the infrastructure was still being created through admin console, modified through scripts, and updated through a mix of manual processes and half-automated pipelines. The worst part was the configuration drift. Each environment—dev, staging, production—was supposedly identical. In reality, they’d diverged so much that bugs would appear in production that were impossible to reproduce in staging. The CI/CD pipelines for application code ran smoothly, but infrastructure changes were often applied manually or through separate automation. Database migrations lived completely outside the deployment pipeline, making rollbacks impossible. One failed migration meant hours of manual recovery.

The Platform Engineering “Solution” That Made Everything Worse

At one platform engineering org, they provided reusable Terraform modules but required each application team to maintain their own configs for every environment. The modules covered maybe 50% of what teams actually needed, so teams built custom solutions, and created snowflakes. The whole point—consistency and maintainability—was lost.

The brilliant solution? A manager built a UI to abstract away Terraform entirely. Just click some buttons!It was a masterclass in leaky abstractions. You couldn’t do anything sophisticated, but when it broke, you had to understand both the UI’s logic AND the generated Terraform to debug it. The UI became a lowest-common-denominator wrapper inadequate for actual needs. I’ve seen AWS CDK provide excellent abstraction over CloudFormation—real programming language power with the ability to drop down to raw resources when needed. That’s proper abstraction: empowering developers, not constraining them. This UI understood nothing about developer needs. It was cargo cult thinking: “Google has internal tools, so we should build internal tools!” I’ve learned: engineers prefer CLI or API approaches to tooling. It’s scriptable, automatable, and fits into workflows. But executives see broken tooling and think the solution is slapping a UI on it—lipstick on a pig. It never works.

The Config Drift Nightmare

We claimed to practice “config as code.” Reality? Our config was scattered across:

• Git repos (three different ones)
• AWS Parameter Store
• Environment variables set manually
• Hardcoded in Docker images
• Some in a random database table
• Feature flags in LaunchDarkly
• Secrets in three different secret managers

Dev environment had different configs than staging, which was different from production. Not by design—by entropy. Each environment had been hand-tweaked over years by different engineers solving different problems. Infrastructure changes were applied manually to environments through separate processes, completely bypassing synchronization with application code. Database migrations lived in four different directory structures across services, no standard anywhere.

Feature flags were even worse. Some teams used LaunchDarkly, others ZooKeeper, none integrated with CI/CD. Instead of templating configs or inheriting from a base, we maintained duplicate configs for every single environment. Copy-paste errors meant production regularly went down from missing or wrong values.

Feature Flags: When the Safety Net Becomes a Trap

I have seen companies buy expensive solutions like LaunchDarkly but fail to provide proper governance and standards. Google’s outage showed exactly what happens: a new code path protected by a feature flag went untested. When enabled, a nil pointer exception took down their entire service globally. The code had no error handling. The flag defaulted to ON. Nobody tested the actual conditions that would trigger the new path. I’ve seen the same pattern repeatedly. Teams deploy code behind flags, flip them on in production, and discover the code crashes. The flag was supposed to be the safety mechanism—it became the detonator. Following are a few common issues related to feature flags that I have observed:

No Integration

Flag changes weren’t integrated with our deployment pipeline. We treated them as configuration, not code. When problems hit, we couldn’t roll back cleanly. We’d deploy old code with new flag states, creating entirely new failure modes. No canary releases for flags. Teams would flip a flag for 100% of traffic instantly. No phased rollout. No monitoring the impact first. Just flip it and hope.

Misuse Everywhere

Teams used flags for everything: API endpoints, timeout values, customer tier logic. The flag system became a distributed configuration database. Nobody planned for LaunchDarkly being unavailable.

I’ve documented these antipatterns extensively—inadequate testing, no peer review, missing monitoring, zombie flags that never get removed. The pattern is always the same: treat flags as toggles instead of critical infrastructure that needs the same rigor as code.

The Observability Theater

At one company, they had a dedicated observability team monitoring hundreds of services across tens of thousands of endpoints. Sounds like the right approach, doesn’t it? The reality was they couldn’t actually monitor at that scale, so they defaulted to basic liveness checks. Is the service responding with 200 OK? Great, it’s “monitored.” We didn’t have synthetic health probes so customers found these issues before the monitoring did. Support tickets were our most reliable monitoring system.

Each service needed specific SLOs, custom metrics, detailed endpoint monitoring. Instead, we got generic dashboards and alerts that fired based on a single health check for all operations of a service. The solution was obvious: delegate monitoring ownership to service teams while the platform team provides tools and standards.

The Security Theater Performance

We had SOC2 compliance, which sales loved to tout. Reality? Internal ops and support had full access to customer data—SSNs, DOBs, government IDs—with zero guardrails and no auditing. I saw list APIs returned everything including SSNs, dates of birth, driver’s license numbers—all in the response. No field-level authorization. Teams didn’t understand authentication vs authorization. OAuth? Refresh tokens? “Too complicated.” They’d issue JWT tokens with 12-24 hour expiration. Session hijacking waiting to happen. Some teams built custom authorization solutions. Added 500ms latency to every request because they weren’t properly integrated with data sources. Overly complex permission systems that nobody understood. When they inevitably broke, services went down.

The Chicken Game

Most companies play security chicken. Bet on luck rather than investment. “We haven’t been breached yet, so we must be fine.” Until they’re not. The principle of least privilege? Never heard of it. I saw everyone in Devops teams gets admin access because it’s easier than managing permissions properly.

AI Makes It Worse

With AI, security got even sloppier. I’ve seen agentic AI code that completely bypasses authorization. The AI has credentials, the AI can do anything. No concept of user context or permissions. The Salesloft breach showed exactly what happens: their AI chatbot stored authentication tokens for hundreds of services—Salesforce, Slack, Google Workspace, AWS, Azure, OpenAI. Attackers stole them all. One breach, access to everything. Standards like MCP (Model Context Protocol) aren’t designed with security in mind. They give companies a false sense of security while creating massive attack surfaces. AI agents with broad access, minimal auditing, no principle of least privilege.

Training vs Reality

But we had mandatory security training! Eight hours of videos about not clicking phishing links. Nothing about secure coding, secret management, access control, or proper authentication. Nothing about OAuth flows, token rotation, or session management. We’d pass audits because we had the right documents. Incident response plans nobody tested. Encryption “at rest” that was just AWS defaults we never configured.

The On-Call Horror Show

Let me tell you about the most broken on-call setup I’ve seen. The PagerDuty escalation went: Engineer -> Head of Engineering. That’s it. No team lead, no manager, just straight from IC to executive.

The Escalation Disaster

New managers? Not in the escalation chain. Senior engineers? Excluded. Other teams skipped layers entirely—engineer to director, bypassing everyone in between. When reorganizations happened, escalation paths didn’t get updated. People left, new people joined, and PagerDuty kept paging people who’d moved to different teams or left the company entirely. Nobody had proper governance. No automated compliance checks. Escalation policies drifted until they bore no resemblance to the org chart.

Missing the Basics

Many services had inadequate SLOs and alerts defined. Teams would discover outages from customer complaints because there was no monitoring. The services that did have alerts? Engineers ignored them. Lower environment alerts went to Slack channels nobody read. Critical errors showed up in staging logs, but no one looked. The same errors would hit production weeks later, and everyone acted surprised. “This never happened before!” It did. In dev. In staging. Nobody checked.

Runbooks and Shadowing

I have seen many teams didn’t keep runbooks up to date. New engineers got added to on-call rotations without shadowing experienced people. One person knew how to handle each class of incident. When they were unavailable, everyone else fumbled through it.

We had the tool the “best” companies used, so we thought we must be doing it right.

The Remote Work Hypocrisy

I’ve been working remotely since 2015, long before COVID made it mainstream. When everyone went remote in 2020, I thought finally companies understood that location doesn’t determine productivity. Then came the RTO (Return to Office) mandates. CEOs talked about “collaboration” and “culture” while most team members were distributed across offices anyway. Having 2 out of 10 team members in the same office doesn’t create collaboration—it creates resentment.

I watched talented engineers leave rather than relocate. Companies used RTO as voluntary layoffs, losing their best people who had options. The cargo cult here? Copying each other’s RTO policies without examining their own situations.

Startups with twenty people and no proper office facilities demanded RTO because big tech was doing it. They had no data on productivity impact, no plan for making office time valuable, just blind imitation of companies with completely different contexts.

The AI Gold Rush

The latest cargo cult is AI adoption. CEOs mandate “AI integration” without thinking through actual use cases. I’ve watched this play out repeatedly.

The Numbers Don’t Lie

95% of AI pilots fail at large companies. McKinsey found 42% of companies using generative AI abandoned projects with “no significant bottom line impact.” But executives already got their stock bumps and bonuses before anyone noticed.

What Actually Fails

I’ve seen companies roll out AI tools with zero training. No prompt engineering guidance. No standardized tools—just a chaotic mess of ChatGPT, Claude, Copilot, whatever people found online. No policies. No metrics. Result? People tried it, got mediocre results, concluded AI was overhyped. The technology wasn’t the problem—the deployment was. Budget allocation is backwards. Companies spend 50%+ on flashy sales and marketing AI while back-office automation delivers the highest ROI. Why? Investors notice the flashy stuff.

The Code Quality Disaster

Here’s what nobody talks about: AI is producing mountains of shitty code. Most teams haven’t updated their SDLC to account for AI-generated code. Senior engineers succeed with AI; junior engineers don’t. Why? Because writing code was never the bottleneck—design and architecture are. You need skill to write proper prompts and critically review output. I’ve used Copilot since before ChatGPT, then Claude, Cursor, and a dozen others. They all have the same problems: limited context windows mean they ignore existing code. They produce syntactically correct code that’s architecturally wrong.

I’ve been using Claude Code extensively. Even with detailed plans and design docs, long sessions lose track of what was discussed. Claude thinks something is already implemented when it isn’t. Or ignores requirements from earlier in the conversation. The context window limitation is fundamental.

Cargo Cult Adoption

I’ve worked at companies where the CEO mandated AI adoption without defining problems to solve. People got promoted for claiming “AI adoption” with useless demos. Hackathon demos are great for learning—actual production integration is completely different. Teams write poor abstractions instead of using battle-tested frameworks like LangChain and LangGraph. They forget to sanitize inputs when using CrewAI. They deploy agents without proper context engineering, memory architecture, or governance.

At one company I worked at, we deployed AI agents without proper permission boundaries—no safeguards to ensure different users got different answers based on their access levels. The Salesforce breach showed what happens when you skip this step. Companies were reusing the same auth tokens in AI prompts and real service calls. No separation between what the AI could access and what the user should see.

The 5% That Work

The organizations that succeed do it differently:

• Buy rather than build (67% success rate vs 33%)
• Start narrow and deep—one specific problem done well
• Focus on workflow integration, not flashy features
• Actually train people on how to use the tools
• Define metrics before deployment

The Productivity Theater

Companies announce layoffs and credit AI, but the details rarely add up. IBM’s CEO claimed AI replaced HR workers—viral posts said 8,000 jobs. Reality? About 200 people, and IBM’s total headcount actually increased. Klarna was more honest. Their CEO publicly stated AI helped shrink their workforce 40%—from 5,527 to 3,422 employees. But here’s the twist: they’re now hiring humans back because AI-driven customer service quality tanked. Builder.ai became a $1.5 billion unicorn claiming their AI “Natasha” automated coding. Turned out it was 700 Indian developers manually writing code while pretending to be AI. The company filed for bankruptcy in May 2025 after exposing not just the fake AI, but $220 million in fake revenue through accounting fraud. Founders had already stepped down.

Why This Is Dangerous

Unlike previous tech hype, AI actually works for narrow tasks. That success gets extrapolated into capabilities that don’t exist. As ACM notes about cargo cult AI, we’re mistaking correlation for causation, statistical patterns for understanding. AI can’t establish causality. It can’t reason from first principles. It can’t ask “why.” These aren’t bugs—they’re fundamental limitations of current approaches. The most successful AI deployments treat it as a tool requiring proper infrastructure: context management, semantic layers, memory architecture, governance. The 95% that fail skip all of this and wonder why their chatbot doesn’t work.

Breaking Free from the Cult

After years of watching this pattern, I’ve learned to recognize the warning signs:

The Name Drop: “Google/Amazon/Netflix does it this way” The Presentation: Slick slides, no substance The Resistance: Questioning is discouraged The Metrics: Activity over outcomes The Evangelists: True believers who’ve never seen it fail

The antidote is simple but not easy:

1. Ask Why: Not just why others do it, but why you should
2. Start Small: Pilot programs reveal problems before they metastasize
3. Measure Impact: Real metrics, not vanity metrics
4. Listen to Skeptics: They often see what evangelists miss
5. Accept Failure: Admitting mistakes early is cheaper than denying them

The Truth About Cargo Cult Culture

After living through all this, I’ve realized cargo cult software engineering isn’t random. It’s systematic. It starts at the top with executives who believe that imitating success is the same as achieving it. They hire from big tech not for expertise, but for credibility. “We have ex-Google engineers!” becomes the pitch, even if those engineers were junior PMs who never touched the systems they’re now supposed to recreate.

These executives enable sales and marketing to sell fiction. “Fake it till you make it” becomes company culture. Engineering bears the burden of making lies true, burning out in the process. The engineers who point out that the emperor has no clothes get labeled as “not team players.” The saddest part? Some of these companies could have been successful with honest, appropriate technology choices. But they chose cosplay over reality, form over function, complexity over simplicity.

The Way Out

I’ve learned to spot these situations in interviews now. When they brag about their tech stack before mentioning what problem they solve, I run. When they name-drop companies instead of explaining their architecture, I run. When they say “we’re the Uber of X” or “we’re building the next Google,” I run fast.

The antidote isn’t just asking “why” – it’s demanding proof. Show me the metrics that prove Kubernetes saves you money. Demonstrate that microservices made you faster. Prove that your observability actually prevents outages. Most can’t, because they never measured before and after. They just assumed newer meant better, complex meant sophisticated, and copying meant competing.

Your context is not Google’s context. Your problems are not Amazon’s problems. And that’s okay. Solve your actual problems with boring, appropriate technology. Your customers don’t care if you use Kubernetes or Kafka or whatever this week’s hot technology is. They care if your shit works. Stop building bamboo airports. Start shipping working software.

Introduction

I’ve been in software development for decades, and if there’s one lesson that’s been burned into my memory through countless production incidents, it’s this: innocuous-looking API changes have an uncanny ability to break everything. You’re getting alerts—an API change that sailed through testing is breaking production. Customer support is calling. You’re coordinating an emergency rollback, wondering how your tests missed this entirely.

The Problem We Keep Facing

Throughout my career, I’ve watched teams struggle with the same challenge: API evolution shouldn’t be a game of Russian roulette. Yet “safe” changes repeatedly pass tests only to break production. Unit testing doesn’t catch the subtle semantic changes that break client integrations. For years, I’ve been building tools to solve this. I created PlexMockServices for API mocking, then evolved it into api-mock-service with full mock and contract testing support. These tools have saved us from many production incidents. I have also written about various testing methodologies for validating APIs such as:

• Mocking and Fuzz Testing Distributed Micro Services with Record/Play, Templates and OpenAPI Specifications
• Property-based and Generative testing for Microservices
• Consumer-driven and Producer-generated Contract Testing for REST APIs

When gRPC and Protocol Buffers arrived, I thought we’d finally solved it. Tools like Buf excel at catching wire-level protocol changes—remove a field, Buf catches it. But here’s what I discovered: Buf and similar tools only see part of the picture.

The Blind Spots

Traditional static analysis tools understand syntax but not semantics. They catch structural changes but miss:

• Fields made required through validation rules—wire-compatible, but every client fails
• Fields that were “always” populated until you made them conditional
• Error messages that clients parse with regex
• Sort orders that changed, breaking customer dashboards
• Default values that shifted behavior

With enough users, all observable behaviors will be depended upon—that’s Hyrum’s Law. The challenge isn’t just detecting changes; it’s understanding their impact from every consumer’s perspective.

Enter Agentic AI

Over the past year, I’ve been experimenting with combining static analysis tools like Buf with the contextual understanding of Large Language Models. Not to replace traditional tools, but to augment them—to catch what they structurally cannot see. In this blog, I’ll show you how to build an intelligent API guardian using LangChain and LangGraph—an agentic AI system that:

• Orchestrates multiple tools (Git, Buf, LLMs) in coordinated workflows
• Understands not just what changed, but what it means
• Catches both wire-level and semantic breaking changes
• Explains why something breaks and how to fix it
• Makes autonomous deployment decisions based on comprehensive analysis

Let me show you how we built this system and how you can implement it for your APIs. Those emergency customer calls about broken integrations might just become a thing of the past.

Architecture Overview: The Intelligent Pipeline

The key insight behind this approach is that no single tool can catch all breaking changes. Static analyzers like Buf excel at structural validation but can’t reason about semantics. LLMs understand context and business logic but lack the deterministic guarantees of rule-based systems. The solution? Combine them in an orchestrated pipeline where each component contributes its strengths.

What I’ve built is an intelligent pipeline that layers multiple detection strategies:

• Buf provides fast, deterministic detection of wire-level protocol violations
• LangGraph orchestrates a stateful workflow that coordinates all the analysis steps
• LangChain manages the LLM interactions, handling prompts, retries, and structured output parsing
• Vertex AI/Gemini brings semantic understanding to analyze what changes actually mean for API consumers

Here’s how these components work together in practice:

Setting Up the Environment

Let’s walk through setting up this system step by step. We’ll use a sample Todo API project as our example.

Prerequisites

# Clone the sample repository
git clone https://github.com/bhatti/todo-api-errors.git
cd todo-api-errors/check-api-break-automation

# Create Python virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Installing Buf

Buf is essential for proto file analysis:

# macOS
brew install bufbuild/buf/buf

# Linux
curl -sSL "https://github.com/bufbuild/buf/releases/latest/download/buf-Linux-x86_64" -o /usr/local/bin/buf
chmod +x /usr/local/bin/buf

# Verify installation
buf --version

Configuring Google Cloud and Vertex AI

1. Set up GCP Project:

# Install gcloud CLI if not already installed
# Follow: https://cloud.google.com/sdk/docs/install

# Authenticate
gcloud auth application-default login

# Set your project
gcloud config set project YOUR_PROJECT_ID

2. Enable Vertex AI API:

gcloud services enable aiplatform.googleapis.com

3. Create Configuration File:

# Create .env file
cat > .env << EOF
GCP_PROJECT=your-project-id
GCP_REGION=us-central1
VERTEX_AI_MODEL=gemini-2.0-flash-exp
EOF

Implementation Deep Dive

The LangGraph State Machine

Our implementation uses LangGraph to create a deterministic workflow for analyzing API changes:

Here’s the core LangGraph implementation:

from langgraph.graph import StateGraph, MessagesState
from typing import TypedDict, List, Dict, Any
import logging

class CompatibilityState(TypedDict):
    """State for the compatibility checking workflow"""
    workspace_path: str
    proto_files: List[str]
    git_diff: str
    buf_results: Dict[str, Any]
    ai_analysis: Dict[str, Any]
    final_report: Dict[str, Any]
    can_deploy: bool

class CompatibilityChecker:
    def __init__(self, project_id: str, model_name: str = "gemini-2.0-flash-exp"):
        self.logger = logging.getLogger(__name__)
        self.project_id = project_id
        self.model = self._initialize_llm(model_name)
        self.workflow = self._build_workflow()

    def _build_workflow(self) -> StateGraph:
        """Build the LangGraph workflow"""
        workflow = StateGraph(CompatibilityState)

        # Add nodes for each step
        workflow.add_node("load_protos", self.load_proto_files)
        workflow.add_node("get_diff", self.get_git_diff)
        workflow.add_node("buf_check", self.run_buf_analysis)
        workflow.add_node("ai_analysis", self.run_ai_analysis)
        workflow.add_node("generate_report", self.generate_report)

        # Define the flow
        workflow.add_edge("load_protos", "get_diff")
        workflow.add_edge("get_diff", "buf_check")
        workflow.add_edge("buf_check", "ai_analysis")
        workflow.add_edge("ai_analysis", "generate_report")

        # Set entry point
        workflow.set_entry_point("load_protos")
        workflow.set_finish_point("generate_report")

        return workflow.compile()

Intelligent Prompt Engineering

The key to accurate breaking change detection lies in the prompt design. Here’s our approach:

def create_analysis_prompt(self, diff: str, buf_results: dict) -> str:
    """Create a comprehensive prompt for the LLM"""
    return f"""
    You are an API compatibility expert analyzing protobuf changes.

    CONTEXT:
    - This is a production API with existing consumers
    - Breaking changes can cause service outages
    - We follow semantic versioning principles

    STATIC ANALYSIS RESULTS:
    {json.dumps(buf_results, indent=2)}

    GIT DIFF:
    ```
    {diff}
    ```

    ANALYZE THE FOLLOWING:
    1. Wire-level breaking changes (trust buf results completely)
    2. Semantic breaking changes:
       - Required fields added without defaults
       - Field removals (always breaking)
       - Type changes that lose precision
       - Enum value removals or reordering

    3. Behavioral concerns:
       - Fields that might be parsed by consumers
       - Error message format changes
       - Ordering or filtering logic changes

    CRITICAL RULES:
    - If buf reports breaking changes, mark them as is_breaking=true
    - Field removal is ALWAYS breaking (severity: HIGH)
    - Adding REQUIRED fields is breaking (severity: MEDIUM-HIGH)
    - Be conservative - when in doubt, flag as potentially breaking

    OUTPUT FORMAT:
    Return a JSON object with this structure:
    {{
        "changes": [...],
        "overall_severity": "NONE|LOW|MEDIUM|HIGH|CRITICAL",
        "can_deploy": true|false,
        "recommendations": [...]
    }}
    """

Real-World Example: When Buf Missed Half the Problem

Let me show you exactly why we need AI augmentation with a concrete example. I’m going to intentionally break a Todo API in two different ways to demonstrate the difference between what traditional tools catch versus what our AI-enhanced system detects.

The Original Proto File

message Task {
  string id = 1;
  string title = 2;
  string description = 3;  // This field will be removed
  bool completed = 4;
  google.protobuf.Timestamp created_at = 5;
  google.protobuf.Timestamp updated_at = 6;
  repeated string tags = 7;
  TaskPriority priority = 8;
  string assignee_id = 9;
  google.protobuf.Timestamp due_date = 10;
  repeated Comment comments = 11;
}

The Modified Proto File

message Task {
  string id = 1;
  string title = 2;
  // REMOVED: string description = 3;
  bool completed = 4;
  google.protobuf.Timestamp created_at = 5;
  google.protobuf.Timestamp updated_at = 6;
  repeated string tags = 7;
  TaskPriority priority = 8;
  string assignee_id = 9;
  google.protobuf.Timestamp due_date = 10;
  repeated Comment comments = 11;

  // NEW REQUIRED FIELD ADDED:
  TaskMetadata metadata = 12 [(validate.rules).message.required = true];
}

message TaskMetadata {
  string created_by = 1;
  int64 version = 2;
  map<string, string> labels = 3;
}

What Buf Detected

When we ran buf breaking --against '.git#branch=main', Buf only detected one breaking change:

api/proto/todo/v1/todo.proto:83:3:Field "3" with name "description" on message "Task" was deleted.

Why did Buf miss the second breaking change? Because adding a field with [(validate.rules).message.required = true] is an application-level annotation, not a wire-protocol breaking change. Buf focuses on wire compatibility – it doesn’t understand application-level validation rules.

What Our AI-Enhanced System Detected

Here’s the actual output from our tool:

2025-10-14 18:29:11,388 - __main__ - INFO - Collecting git diffs...
2025-10-14 18:29:11,392 - __main__ - INFO - Analyzing with LLM...
2025-10-14 18:29:14,471 - __main__ - INFO - Generating final report...
================================================================================
API BACKWARD COMPATIBILITY REPORT
================================================================================
Timestamp: 2025-10-14T18:29:14.471705
Files Analyzed: api/proto/todo/v1/todo.proto
Total Changes: 2
Breaking Changes: 2
Overall Severity: HIGH
Can Deploy: NO

DETECTED CHANGES:
----------------------------------------
1. Removed field 'description'
   Location: api/proto/todo/v1/todo.proto:83
   Category: field_removal
   Breaking: YES
   Severity: HIGH
   Recommendation: Consider providing a migration path for clients relying on this field.

2. Added required field 'metadata'
   Location: api/proto/todo/v1/todo.proto:136
   Category: field_addition
   Breaking: YES
   Severity: HIGH
   Recommendation: Ensure all clients are updated to include this field before deployment.

LLM ANALYSIS:
----------------------------------------
The changes include the removal of the 'description' field and the addition of a required
'metadata' field, both of which are breaking changes.

================================================================================
2025-10-14 18:29:14,472 - __main__ - INFO - JSON report saved to results/non_breaking.json

The “Aha!” Moment

This is exactly the scenario I warned about in my presentation. Here’s what happened:

1. Buf did its job – It caught the field removal. That’s wire-level breaking change detection working as designed.
2. But Buf has blind spots – It completely missed the required field addition because [(validate.rules).message.required = true] is an application-level annotation. To Buf, it’s just another optional field on the wire.
3. The AI understood context – Our LLM looked at that validation rule and immediately recognized: “Hey, this server is going to reject any request without this field. That’s going to break every existing client!”

Think about it – if we had only relied on Buf, we would have deployed thinking we fixed the one breaking change. Then boom – production down because no existing client sends the new metadata field. This is precisely why we need AI augmentation. It’s not about replacing Buf – it’s about catching what Buf structurally cannot see.

Beyond This Example

This pattern repeats across many scenarios that static analysis misses:

• Validation rules that make previously optional behavior mandatory
• Fields that were always populated but are now conditional
• Changes to default values that alter behavior
• Error message format changes (clients parse these!)
• Response ordering changes (someone always depends on order)
• Rate limiting or throttling policy changes
• Authentication requirements that changed

Integrating with CI/CD

The tool can be integrated into your CI/CD pipeline:

# .github/workflows/api-compatibility.yml
name: API Compatibility Check

on:
  pull_request:
    paths:
      - '**/*.proto'

jobs:
  check-breaking-changes:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Need full history for comparison

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install Buf
        run: |
          curl -sSL "https://github.com/bufbuild/buf/releases/latest/download/buf-Linux-x86_64" -o /usr/local/bin/buf
          chmod +x /usr/local/bin/buf

      - name: Install dependencies
        run: |
          pip install -r check-api-break-automation/requirements.txt

      - name: Run compatibility check
        env:
          GCP_PROJECT: ${{ secrets.GCP_PROJECT }}
        run: |
          cd check-api-break-automation
          python api_compatibility_checker.py \
            --workspace .. \
            --against origin/main \
            --output results/pr-check.json

      - name: Comment PR with results
        if: always()
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('check-api-break-automation/results/pr-check.json'));

            const comment = `## ? API Compatibility Check Results

            **Can Deploy**: ${results.can_deploy ? '? Yes' : '? No'}
            **Severity**: ${results.overall_severity}
            **Breaking Changes**: ${results.summary.total_breaking_changes}

            ${results.can_deploy ? '' : '### ?? Breaking Changes Detected\n' + results.recommendations.join('\n')}
            `;

            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });

Advanced Features: RAG and MCP in Action

1. RAG (Retrieval-Augmented Generation): Learning from Past Mistakes

One of the most powerful aspects of our system is how it learns from history. Here’s how RAG actually works in our implementation:

from langchain.vectorstores import Chroma
from langchain.embeddings import VertexAIEmbeddings
from langchain.schema import Document

class BreakingChangeKnowledgeBase:
    """RAG system that learns from past breaking changes"""

    def __init__(self, project_id: str):
        self.embeddings = VertexAIEmbeddings(
            model_name="textembedding-gecko@003",
            project=project_id
        )
        # Store historical breaking changes in vector database
        self.vector_store = Chroma(
            collection_name="api_breaking_changes",
            embedding_function=self.embeddings,
            persist_directory="./knowledge_base"
        )

    def index_breaking_change(self, change_data: dict):
        """Store a breaking change incident for future reference"""
        doc = Document(
            page_content=f"""
            Proto Change: {change_data['diff']}
            Breaking Type: {change_data['type']}
            Customer Impact: {change_data['impact']}
            Resolution: {change_data['resolution']}
            """,
            metadata={
                "severity": change_data['severity'],
                "date": change_data['date'],
                "service": change_data['service'],
                "prevented": change_data.get('caught_before_prod', False)
            }
        )
        self.vector_store.add_documents([doc])

    def find_similar_changes(self, current_diff: str, k: int = 5):
        """Find similar past breaking changes"""
        results = self.vector_store.similarity_search_with_score(
            current_diff,
            k=k,
            filter={"severity": {"$in": ["HIGH", "CRITICAL"]}}
        )
        return results

# How it's used in the main checker:
class CompatibilityChecker:
    def __init__(self, project_id: str):
        self.knowledge_base = BreakingChangeKnowledgeBase(project_id)

    def run_ai_analysis(self, state: dict):
        """Enhanced AI analysis using RAG"""
        # Find similar past incidents
        similar_incidents = self.knowledge_base.find_similar_changes(
            state['git_diff']
        )

        # Build context from past incidents
        historical_context = ""
        if similar_incidents:
            historical_context = "\n\nSIMILAR PAST INCIDENTS:\n"
            for doc, score in similar_incidents:
                if score > 0.8:  # High similarity
                    historical_context += f"""
                    - Previous incident: {doc.metadata['date']}
                      Impact: {doc.page_content}
                      This suggests high risk of similar issues.
                    """

        # Include historical context in prompt
        enhanced_prompt = f"""
        {self.base_prompt}

        {historical_context}

        Based on historical patterns, pay special attention to similar past issues.
        """

        return self.llm.invoke(enhanced_prompt)

2. Model Context Protocol (MCP) Integration

MCP allows our AI to interact with external tools seamlessly. Here’s the actual implementation:

# mcp_server.py - MCP server for API compatibility tools
from mcp.server import MCPServer
from mcp.tools import Tool, ToolResult
import subprocess
import json

class APICompatibilityMCPServer(MCPServer):
    """MCP server exposing API compatibility tools to AI agents"""

    def __init__(self):
        super().__init__("api-compatibility-checker")
        self.register_tools()

    def register_tools(self):
        """Register all available tools"""

        @self.tool("buf_lint")
        async def buf_lint(proto_path: str) -> ToolResult:
            """Run buf lint on proto files"""
            result = subprocess.run(
                ["buf", "lint", proto_path],
                capture_output=True,
                text=True
            )
            return ToolResult(
                success=result.returncode == 0,
                output=result.stdout,
                error=result.stderr
            )

        @self.tool("buf_breaking")
        async def buf_breaking(proto_path: str, against: str = "main") -> ToolResult:
            """Check for breaking changes using buf"""
            cmd = [
                "buf", "breaking",
                "--against", f".git#branch={against}",
                "--path", proto_path
            ]
            result = subprocess.run(cmd, capture_output=True, text=True)

            # Parse breaking changes
            breaking_changes = []
            for line in result.stdout.splitlines():
                if line.strip():
                    breaking_changes.append(self.parse_buf_output(line))

            return ToolResult(
                success=True,
                data={
                    "has_breaking": len(breaking_changes) > 0,
                    "changes": breaking_changes,
                    "raw_output": result.stdout
                }
            )

        @self.tool("check_consumer_contracts")
        async def check_contracts(service: str, version: str) -> ToolResult:
            """Check if change breaks consumer contracts"""
            # This connects to our contract testing system
            contracts = self.load_consumer_contracts(service)
            violations = []

            for contract in contracts:
                if not self.validate_contract(contract, version):
                    violations.append({
                        "consumer": contract["consumer"],
                        "expectation": contract["expectation"],
                        "impact": "Contract violation detected"
                    })

            return ToolResult(
                success=True,
                data={
                    "total_consumers": len(contracts),
                    "violations": violations,
                    "safe_to_deploy": len(violations) == 0
                }
            )

        @self.tool("generate_migration_guide")
        async def generate_migration(breaking_changes: list) -> ToolResult:
            """Generate migration guide for breaking changes"""
            guide = self.create_migration_steps(breaking_changes)
            return ToolResult(
                success=True,
                data={"migration_guide": guide}
            )

# How LangChain uses MCP tools:
from langchain.agents import create_mcp_agent
from langchain_mcp import MCPToolkit

# Initialize MCP toolkit
mcp_toolkit = MCPToolkit(
    server_url="http://localhost:8080",  # MCP server endpoint
    available_tools=["buf_lint", "buf_breaking", "check_consumer_contracts"]
)

# Create agent with MCP tools
agent = create_mcp_agent(
    llm=llm,
    tools=mcp_toolkit.get_tools(),
    system_prompt="""
    You are an API compatibility expert. Use the available MCP tools to:
    1. Run buf lint and breaking checks
    2. Verify consumer contracts
    3. Generate migration guides when needed

    Always check consumer contracts after detecting breaking changes.
    """
)

# Usage in the main workflow
class CompatibilityChecker:
    def __init__(self):
        self.mcp_agent = agent

    def comprehensive_check(self, proto_path: str):
        """Run comprehensive compatibility check using MCP tools"""

        # Let the agent orchestrate the tools
        result = self.mcp_agent.invoke({
            "input": f"""
            Analyze {proto_path} for breaking changes:
            1. Run buf lint first
            2. Check breaking changes against main branch
            3. If breaking changes found, check consumer contracts
            4. Generate migration guide if needed
            """
        })

        return result

3. How RAG + MCP Work Together

Here’s the magic – combining RAG’s historical knowledge with MCP’s tool access:

class IntelligentAPIGuardian:
    """Combines RAG and MCP for comprehensive analysis"""

    def analyze_change(self, proto_diff: str):
        # Step 1: Use MCP to run all tools
        mcp_results = self.mcp_agent.invoke({
            "input": f"Analyze this diff: {proto_diff}"
        })

        # Step 2: Use RAG to find similar past incidents
        historical_data = self.knowledge_base.find_similar_changes(proto_diff)

        # Step 3: Combine insights
        combined_analysis = self.llm.invoke(f"""
        Current change analysis from tools:
        {mcp_results}

        Historical patterns from similar changes:
        {historical_data}

        Synthesize a comprehensive risk assessment considering both
        current tool results and historical precedents.

        If historical data shows issues that tools didn't catch,
        flag them as "HISTORICAL_RISK" items.
        """)

        # Step 4: Store this analysis for future RAG queries
        if combined_analysis['has_breaking_changes']:
            self.knowledge_base.index_breaking_change({
                'diff': proto_diff,
                'type': combined_analysis['breaking_type'],
                'impact': combined_analysis['impact'],
                'resolution': combined_analysis['recommendations'],
                'severity': combined_analysis['severity'],
                'date': datetime.now(),
                'caught_before_prod': True
            })

        return combined_analysis

The Power of This Combination:

• MCP gives us real-time tool access – running buf, checking contracts, generating migrations
• RAG gives us institutional memory – learning from every incident, getting smarter over time
• Together they catch issues that neither could find alone

For example, RAG might recall “last time we added a required field to Task, the mobile team’s app crashed because they cache responses for 24 hours” – something no static tool would know, but crucial for preventing an outage.

Testing the System

Here’s a complete walkthrough of testing the system:

# 1. First, verify your setup
python test_simple.py

# Output should show:
# ? All core modules imported successfully
# ? Proto file found
# ? Proto modifier works - 12 test scenarios available
# ? Buf integration initialized successfully
# ? GCP_PROJECT configured: your-project-id
# ? Vertex AI connection verified

# 2. Make breaking changes to the proto file
python proto_modifier.py ../api/proto/todo/v1/todo.proto \
  --scenario remove_field

python proto_modifier.py ../api/proto/todo/v1/todo.proto \
  --scenario add_required_field

# 3. Run the compatibility checker
python api_compatibility_checker.py \
  --workspace .. \
  --against '.git#branch=main' \
  --output results/breaking_changes.json

# 4. Review the detailed report
cat results/breaking_changes.json | jq '.'

Lessons Learned and Best Practices

1. Combine Multiple Analysis Methods: Static analysis catches structure, AI catches semantics
2. Use Conservative Defaults: When uncertain, flag as potentially breaking
3. Provide Clear Explanations: Developers need to understand why something is breaking
4. Version Your Prompts: Treat prompts as code – version and test them
5. Monitor LLM Costs: Use caching and optimize prompt sizes
6. Implement Gradual Rollout: Start with warnings before blocking deployments
7. Build Team Trust Gradually: Don’t start by blocking deployments. Run in shadow mode first, report findings alongside Buf results, and let teams see the value before enforcement. Track false positives and tune your prompts based on real feedback.
8. Document Your Prompts: Your prompt engineering is as critical as your code. Version control your prompts, document why certain instructions exist, and treat them as first-class artifacts that need testing and review.

The Power of Agentic AI

What makes this approach “agentic” rather than just AI-assisted?

1. Autonomous Decision Making: The system doesn’t just flag issues – it makes decisions whether API changes can deployed
2. Multi-Step Reasoning: It performs complex analysis chains without human intervention
3. Tool Integration: It orchestrates multiple tools (Git, Buf, LLMs) to achieve its goal
4. Contextual Understanding: It considers historical patterns and project-specific rules
5. Actionable Output: It provides specific remediation steps, not just warnings

Future Enhancements

The roadmap for this tool includes:

1. Multi-Protocol Support: Extend beyond protobuf/gRPC to OpenAPI and GraphQL
2. Behavioral Testing: Integration with contract testing frameworks
3. Auto-Migration Generation: Create migration scripts for breaking changes
4. Client SDK Updates: Automatically update client libraries
5. Performance Impact Analysis: Predict performance implications of changes

Known Limitations: This system excels at catching semantic and behavioral changes, but it’s not perfect. It can’t predict how undocumented client implementations behave, can’t catch changes in external dependencies your API relies on, and can’t guarantee zero false positives. Human judgment remains essential—especially for nuanced cases where breaking changes might be intentional and necessary.

Conclusion

Throughout my decades in software development, I’ve learned that API compatibility isn’t just about wire protocols and field numbers. It’s about understanding how our users actually depend on our APIs—all the documented behaviors, the undocumented quirks, and yes, even the bugs they’ve built workarounds for. Traditional static analysis tools like Buf are essential—they catch structural breaking changes with perfect precision. But as we’ve seen with the required field example, they can’t reason about semantic changes, business context, or application-level validation rules. That’s where AI augmentation transforms the game. By combining Buf’s deterministic analysis with an LLM’s contextual understanding through LangChain and LangGraph, we’re not just catching more bugs—we’re fundamentally changing how we think about API evolution.

The complete implementation, including all the code and configurations demonstrated in this article, is available at: https://github.com/bhatti/todo-api-errors. Fork it, experiment with it, break it, improve it.

Resources and References

• LangChain Documentation
• LangGraph Guide
• Buf Documentation
• Vertex AI/Gemini Models
• Protocol Buffers Best Practices
• API Versioning Strategies

Postel’s Law: “Be conservative in what you send, liberal in what you accept” – but with Agentic AI, we can be intelligent about both.

Hyrum’s Law: “With a sufficient number of users, all observable behaviors will be depended upon” – which is why we need AI to catch the subtle breaking changes that static analysis misses.

I’ve worked at organizations where engineers would sneak changes into production, bypassing CI/CD pipelines, hoping nobody would notice if something broke. I’ve also worked at places where engineers would openly discuss a failed experiment at standup and get help fixing it. The difference wasn’t the engineers—it was psychological safety. Research on psychological safety, particularly from high-stakes industries like healthcare, tells us something counterintuitive: teams with high psychological safety don’t have fewer incidents. They have better outcomes because people speak up about problems early.

Software engineering isn’t life-or-death medicine, but the principle holds: in blame cultures, I’ve watched talented engineers:

• Deploy sketchy changes outside normal hours to avoid oversight
• Ignore monitoring alerts, hoping they’re false positives
• Blame infrastructure, legacy code, or “the previous team” rather than examining their contributions
• Build elaborate workarounds instead of fixing root causes

These behaviors don’t just hurt morale—they actively degrade reliability. Post mortems in blame cultures become exercises in creative finger-pointing and CYA documentation.

In learning cultures, post mortems are gold mines of organizational knowledge. The rule of thumb I’ve seen work best: if you’re unsure whether something deserves a post mortem, write one anyway—at least internally. Not every post mortem needs wide distribution, and some (especially those with security implications) shouldn’t be shared externally. But the act of writing crystallizes learning.

The Real Problem: Post Mortem Theater

Here’s what nobody talks about: many organizations claim to value post mortems but treat them like bureaucratic checklists. I’ve seen hundreds of meticulously documented post mortems that somehow don’t prevent the same incidents from recurring. This is what I call “post mortem theater”—going through the motions without actual learning.

Shallow vs. Deep Analysis

Shallow analysis stops at the proximate cause:

• “The database connection pool was exhausted”
• “An engineer deployed buggy code”
• “A dependency had high latency”

Deep analysis asks uncomfortable questions:

• “Why don’t we load test with production-scale data? What makes it expensive to maintain realistic test environments?”
• “Why did code review and automated tests miss this? What’s our philosophy on preventing bugs vs. recovering quickly?”
• “Why do we tolerate single points of failure in our dependency chain? What would it take to build resilience?”

The difference between these approaches determines whether you’re learning or just documenting.

Narrow vs. Systemic Thinking

Narrow analysis fixes the immediate problem:

• Add monitoring for connection pool utilization
• Add a specific test case for the bug that escaped
• Increase timeout values for the slow dependency

Systemic analysis asks meta questions:

• “How do we systematically identify what we should be monitoring? Do we have a framework for this?”
• “What patterns in our testing gaps led to this escape? Are we missing categories of testing?”
• “What’s our philosophy on dependency management and resilience? Should we rethink our architecture?”

I’ve seen teams play post mortem bingo—hitting the same squares over and over. “No monitoring.” “Insufficient tests.” “Deployed during peak traffic.” “Rollback was broken.” When you see repeated patterns, you’re not learning from incidents—you’re collecting them. I have also written about common failures in distributed systems that can show up in recurring incidents if they are not properly addressed.

Understanding Complex System Failures

Modern systems fail in ways that defy simple “root cause” thinking. Consider a typical outage:

Surface story: Database connection pool exhausted
Deeper story:

• A code change increased query volume 10x
• Load testing used 1/10th production data and missed it
• Connection pool monitoring didn’t exist
• Alerts only monitored error rates, not resource utilization
• Manual approval processes delayed rollback by 15 minutes
• Staging environment configuration drifted from production

Which of these is the “root cause”? All of them. None of them individually would have caused an outage, but together they created a perfect storm. This is why I cringe when post mortems end with “human error” as the root cause. It’s not wrong—humans are involved in everything—but it’s useless. The question is: why was the error possible? What systemic changes make it impossible, or at least improbable?

You can think of this as the Swiss Cheese Model of failure: your system has multiple layers of defense (code review, testing, monitoring, gradual rollout, alerting, incident response). Each layer has holes. Most of the time, the holes don’t align and problems get caught. But occasionally, everything lines up perfectly and a problem slips through all layers. That’s your incident. This mental model is more useful than hunting for a single root cause because it focuses you on strengthening multiple layers of defense.

When to Write a Post Mortem

Always write one:

• Customer-facing service disruptions
• SLA/SLO breaches
• Security incidents (keep these separate, limited distribution)
• Complete service outages
• Incidents requiring emergency escalation or multiple teams

Strongly consider:

• Near-misses that made you think “we got lucky”
• Interesting edge cases with valuable lessons
• Internal issues that disrupted other teams
• Process breakdowns causing significant project delays

The litmus test: If you’re debating whether it needs a post mortem, write at least an internal one. The discipline of writing forces clarity of thinking.

Post Mortem Ownership

Who should own writing it? The post mortem belongs to the team that owns addressing the root cause, not necessarily the team that triggered the incident or resolved it. If root cause is initially unclear, it belongs with whoever is investigating. If investigation reveals the root cause lies elsewhere, reassign it.

The Anatomy of an Effective Post Mortem

Title and Summary: The Elevator Pitch

Your title should state the customer-facing problem, not the cause.

Good: “Users unable to complete checkout for 23 minutes in US-EAST”
Bad: “Connection pool exhaustion caused outage”

Your summary should work as a standalone email to leadership. Include:

• Brief service context (one sentence)
• Timeline with time zones
• Quantified customer impact
• How long it lasted (from first customer impact to full recovery)
• High-level cause
• How it was resolved
• Communication sent to customers (if applicable)

Timeline: The Narrative Spine

A good timeline tells the story of what happened, including system events and human decisions. Important: Your timeline should start with the first trigger that led to the problem (e.g., a deployment, a configuration change, a traffic spike), not just when your team got paged. The timeline should focus on the actual event start and end, not just your team’s perception of it.

All times PST

14:32 - Deployment begins in us-west-2
14:38 - Error rates spike to 15%
14:41 - Automated alerts fire
14:43 - On-call engineer begins investigation
14:47 - Customer support escalates: users reporting checkout failures
14:52 - Incident severity raised to SEV-1
15:03 - Root cause identified: connection pool exhaustion
15:07 - Rollback initiated
15:22 - Customer impact resolved, errors back to baseline

Key practices:

• Start with the root trigger, not when you were notified
• Consistent time zones throughout
• Bold major milestones and customer-facing events
• Include detection, escalation, and resolution times`
• No gaps longer than 10-15 minutes without explanation
• Use roles (“on-call engineer”) not names
• Include both what the system did and what people did

Metrics: Show, Don’t Just Tell

Visual evidence is crucial. Include graphs showing:

• Error rates during the incident
• The specific resource that failed (connections, CPU, memory)
• Business impact metrics (orders, logins, API calls)
• Comparison with normal operation

For complex incidents involving multiple services, include a simple architecture diagram showing the relevant components and their interactions. This helps readers understand the failure chain without needing deep knowledge of your system.

Make graphs comparable:

• Same time range across all graphs
• Label your axes with units (milliseconds, percentage, requests/second)
• Vertical lines marking key events
• Include context before and after the incident
• Embed actual screenshots, not just links that will break

Don’t do this:

• Include 20 graphs because you can
• Use different time zones between graphs
• Forget to explain what the graph shows and why it matters

Service Context and Glossary

If your service uses specialized terminology or acronyms, add a brief glossary section or spell out all acronyms on first use. Your post mortem should be readable by engineers from other teams. For complex incidents, consider including:

• Brief architecture overview (what are the key components?)
• Links to related items (monitoring dashboards, deployment records, related tickets)
• Key metrics definitions if not standard

Customer Impact: Get Specific

Never write “some customers were affected” or “significant impact.” Quantify everything:

Instead of: “Users experienced errors”
Write: “23,000 checkout attempts failed over 23 minutes, representing approximately $89,000 in failed transactions”

Instead of: “API latency increased”
Write: “P95 latency increased from 200ms to 3.2 seconds, affecting 15,000 API calls”

If you can’t get exact numbers, explain why and provide estimates with clear caveats.

Root Cause Analysis: Going Deeper

Use numbered lists (not bullets) for your Five Whys so people can easily reference them in discussions (“Why #4 seems incomplete…”). Use the Five Whys technique, but don’t stop at five if you need more. Start with the customer-facing problem and keep asking why:

1. Why did customers see checkout errors?
   -> Application servers returned 500 errors

2. Why did application servers return 500 errors?
   -> They couldn't connect to the database

3. Why couldn't they connect?
   -> Connection pool was exhausted

4. Why was the pool exhausted?
   -> New code made 10x more queries per request

5. Why didn't we catch this in testing?
   -> Staging uses 1/10th production data

6. Why is staging data volume so different?
   -> We haven't prioritized staging environment investment

Branch your analysis for multiple contributing factors. Number your branches (1.1, 1.2, etc.) to maintain traceability:

Primary Chain (1.x):

Why did customers see checkout errors?
 did customers see checkou Application servers returned 500 errors
[...]

Branch A - Detection (2.x):

Why did detection take 12 minutes?
-> We only monitor error rates, not resource utilization
Why don't we monitor resource utilization?
-> We haven't established a framework for what to monitor

Branch B - Mitigation (3.x):

Why did rollback take 15 minutes after identifying the cause?
-> Manual approval was required for production rollbacks
Why is manual approval required during emergencies?
-> Our process doesn't distinguish between routine and emergency changes

Never stop at:

• “Human error”
• “Process failure”
• “Legacy system”

Keep asking why until you reach actionable systemic changes.

Incident Response Analysis

This section examines how you handled the crisis during the incident, not how to prevent it. This is distinct from post-incident analysis (root causing) which happens after. Focus on the temporal sequence of events:

• Detection: How did you discover the problem? Automated alerts, customer reports, accidental discovery? How could you have detected it sooner?
• Diagnosis: How long from “something’s wrong” to “we know what’s wrong”? What information or tools would have accelerated diagnosis?
• Mitigation: How long from diagnosis to resolution? What would have made recovery faster?
• Blast Radius: What percentage of customers/systems were affected? How could you have reduced the blast radius? Consider:
  • Would cellular architecture have isolated the failure?
  • Could gradual rollout have limited impact?
  • Did failure cascade to dependent systems unnecessarily?
  • Would circuit breakers have prevented cascade?

For each phase, ask: “How could we have cut this time in half?” And for blast radius: “How could we have cut the affected population in half?”

Post-Incident Analysis vs Real-Time Response

Be clear about the temporal distinction in your post mortem:

• Incident Response Analysis = What happened DURING the incident
  • How we detected, diagnosed, and mitigated
  • Time-critical decisions under pressure
  • Effectiveness of runbooks and procedures
• Post-Incident Analysis = What happened AFTER to understand root cause
  • How we diagnosed the underlying cause
  • What investigation techniques we used
  • How long root cause analysis took

This distinction matters because improvements differ: incident response improvements help you recover faster from any incident; post-incident improvements help you understand failures more quickly.

Lessons Learned: Universal Insights

Number your lessons learned (not bullets) so they can be easily referenced and linked to action items. Lessons should be broadly applicable beyond your specific incident:
1. Bad lesson learned: “We need connection pool monitoring”
Good lesson learned: “Services should monitor resource utilization for all constrained resources, not just error rates”
2. Bad lesson learned: “Load testing failed to catch this”
Good lesson learned: “Test environments that don’t reflect production characteristics will systematically miss production-specific issues”

Connect each lesson to specific action items by number reference (e.g., “Lesson #2 ? Action Items #5, #6”).

Action Items: Making Change Happen

This is where post mortems prove their value. Number your action items and explicitly link them to the lessons learned they address. Every action item needs:

• Clear description: Not “improve monitoring” but “Add CloudWatch alarms for RDS connection pool utilization with thresholds at 75% (warning) and 90% (critical)”
• Specific owner: A person’s name, not a team name
• Realistic deadline: Most should complete within 45 days
• Priority level:
  • High for root cause fixes and issues that directly caused customer impact
  • Medium for improvements to detection/mitigation
  • Low for nice-to-have improvements
• Link to lesson learned: “Addresses Lesson #2”
• Avoid action items that start with “investigate.” That’s not an action item—it’s procrastination. Do the investigation during the post mortem process and commit to specific changes.

Note: Your lessons learned should be universal principles that other teams could apply. Your action items should be specific changes your team will make. If your lessons learned just restate your action items, you’re missing the bigger picture.

Common Patterns That Indicate Shallow Learning

When you see the same issues appearing in multiple post mortems, you have a systemic problem:

• Repeated monitoring gaps -> You don’t have a framework for determining what to monitor
• Repeated test coverage issues -> Your testing philosophy or practices need examination
• Repeated “worked in staging, failed in prod” -> Your staging environment strategy is flawed
• Repeated manual process errors -> You’re over-relying on human perfection
• Repeated deployment-related incidents -> Your deployment pipeline needs investment

These patterns are your organization’s immune system telling you something. Listen to it.

Common Pitfalls

After reading hundreds of post mortems, here are the traps I see teams fall into:

• Writing for Insiders Only: Your post mortem should be readable by someone from another team. Explain your system’s architecture briefly, spell out acronyms, and assume your reader is smart but unfamiliar with your specific service.
• Action Items That Start with “Investigate”: “Investigate better monitoring” is not an action item – it’s a placeholder for thinking you haven’t done yet. During the post mortem process, do the investigation and commit to specific changes.
• Stopping at “Human Error”: If your Five Whys ends with “the engineer made a mistake,” you haven’t gone deep enough. Why was that mistake possible? What system changes would prevent it?
• The Boil-the-Ocean Action Plan: Post mortems aren’t the place for your three-year architecture wish list. Focus on targeted improvements that directly address the incident’s causes and can be completed within a few months.

Ownership and Follow-Through

Here’s something that separates good teams from great ones: they actually complete their post mortem action items.

• Assign clear ownership: Every action item needs a specific person (not a team) responsible for completion. That person might delegate the work, but they own the outcome.
• Set realistic deadlines: Most action items should be completed within 45 days. If something will take longer, either break it down or put it in your regular backlog instead.
• Track relentlessly: Use whatever task tracking system your team prefers, but make action item completion visible. Review progress in your regular team meetings.
• Close the loop: When action items are complete, update the post mortem with links to the changes made. Future readers (including future you) will thank you.

Making Post Mortems Part of Your Culture

• Write them quickly: Create the draft within 24 hours while memory is fresh. Complete the full post mortem within 14 days.
• Get outside review (critical step): An experienced engineer from another team—sometimes called a “Bar Raiser”—should review for quality before you publish. The reviewer should check:
  • Would someone from another team understand and learn from this?
  • Are the lessons learned actually actionable?
  • Did you dig deep enough in your root cause analysis?
  • Are your action items specific and owned?
  • Does the incident response analysis identify concrete improvements?
• Draft status: Keep the post mortem in draft/review status for at least 24 hours to gather feedback from stakeholders. Account for holidays and time zones for distributed teams.
• Make them visible: Share widely (except security-sensitive ones) so other teams learn from your mistakes.
• Customer communication: For customer-facing incidents, document what communication was sent:
  • Status page updates
  • Support team briefings
  • Proactive customer notifications
  • Post-incident follow-up
• Track action items relentlessly: Use whatever task system you have. Review progress in regular meetings.
• Review for patterns: Monthly or quarterly, look across all post mortems for systemic issues.
• Celebrate learning: In team meetings, highlight interesting insights from post mortems. Make clear that thorough post mortems are valued, not punishment.
• Train your people: Writing good post mortems is a skill. Share examples of excellent ones and give feedback.

Security-Sensitive Post Mortems

Some incidents involve security implications, sensitive customer data, or information that shouldn’t be widely shared. These still need documentation, but with appropriate access controls:

• Create a separate, access-controlled version
• Document what happened and how to prevent it
• Share lessons learned (without sensitive details) more broadly
• Work with your security team on appropriate distribution

The learning is still valuable—it just needs careful handling.

The Long Game

Post mortems are how organizations build institutional memory. They’re how you avoid becoming that team that keeps making the same mistakes. They’re how you onboard new engineers to the reality of your systems. Most importantly, they’re how you shift from a culture of blame to a culture of learning.

When your next incident happens—and it will—remember you’re not just fixing a problem. You’re gathering intelligence about how your system really behaves under stress. You’re building your team’s capability to handle whatever comes next. Write the post mortem you wish you’d had during the incident. Be honest about what went wrong. Be specific about what you’ll change. Be generous in sharing what you learned.

Your future self, your teammates, and your customers will all benefit from it. And remember: if you’re not sure whether something deserves a post mortem, write one anyway. The discipline of analysis is never wasted.