Shahzad Bhatti Welcome to my ramblings and rants!

Observability is a key part of any infrastructure but I’ve watched teams repeat the same mistakes around measuring availability. For example, they track uptime and watch average latency. They run a TCP health check on port 80 and call it good. Then support learns about the availability issues from customers but the health dashboard shows everything is green. This post covers how to measure availability correctly: what signals to collect, how monitoring tools compute the rolling statistics you see, why percentiles beat averages and what happens to tail latency at scale in microservices.

1. What Availability Actually Means

The textbook definition of availability is uptime, e.g., the fraction of time a service is running. This splits into two independent questions:

Availability = P(request succeeds) AND P(request completes within SLA)

A service can answer every request successfully but take 30 seconds per response then that’s functionally unavailable. Conversely, a service can respond in 5ms but return errors to 50% of requests is also functionally unavailable.

2. User Errors vs Server Errors — Why the Distinction Matters

This is the most commonly conflated measurement in production monitoring. HTTP status codes carry clear semantic meaning that should drive entirely different alert responses:

4xx errors are client/user errors like 400/Bad Request, 401/Unauthorized. 5xx errors means service is failing like 500/Internal Server, 503/Service Unavailable. There is one gray area: client timeouts. If your client times out after 5s waiting for your 10s response, the client sees a 408 or a network error, which look like a 4xx but the root cause is server-side latency. This is why tracking latency separately from error codes is essential.

from prometheus_client import Counter, Histogram

# Track errors with full status code granularity
request_counter = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status_code', 'status_class']
)

latency_histogram = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

def record_request(method: str, endpoint: str, status: int, duration_s: float):
    status_class = f"{status // 100}xx"
    request_counter.labels(
        method=method,
        endpoint=endpoint,
        status_code=str(status),
        status_class=status_class
    ).inc()
    latency_histogram.labels(method=method, endpoint=endpoint).observe(duration_s)


# --- Prometheus queries that actually measure availability ---

# Server error rate (5xx only — excludes client errors)
SERVER_ERROR_RATE = """
sum(rate(http_requests_total{status_class="5xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Availability (only penalize server errors)
AVAILABILITY = """
1 - (
  sum(rate(http_requests_total{status_class="5xx"}[5m]))
  /
  sum(rate(http_requests_total[5m]))
)
"""

# Client error rate (useful to watch, but not availability)
CLIENT_ERROR_RATE = """
sum(rate(http_requests_total{status_class="4xx"}[5m]))
/
sum(rate(http_requests_total[5m]))
"""

# Latency SLA compliance — fraction of requests completing within 500ms
LATENCY_SLA_COMPLIANCE = """
sum(rate(http_request_duration_seconds_bucket{le="0.5"}[5m]))
/
sum(rate(http_request_duration_seconds_count[5m]))
"""

A spike in 4xx that isn’t paired with a 5xx spike is almost certainly a misbehaving client, not your service. Alert on them differently: 5xx pages your on-call, 4xx goes to a ticket queue for review.

3. SLAs, SLOs, and Error Budgets

These three terms are used interchangeably in many organizations and they shouldn’t be.

• SLA (Service Level Agreement) is a contractual commitment to external customers. Violating it has legal or financial consequences. Example: “We guarantee 99.9% availability per calendar month. If we breach this, we issue service credits.”
• SLO (Service Level Objective) is an internal engineering target, usually tighter than the SLA. Example: “We target 99.95% availability.” The gap between SLO and SLA is your buffer.
• Error Budget is what you get to spend before you breach your SLO. For a 99.9% SLO over 30 days:

Total minutes in 30 days = 30 × 24 × 60 = 43,200 minutes
Allowed downtime = 43,200 × (1 - 0.999) = 43.2 minutes

The error budget is your 43.2 minutes. Every minute of downtime spends from it. This reframes the conversation from “is the service up?” to “how fast are we burning through our budget?”

from datetime import datetime, timedelta

class ErrorBudget:
    """
    Track error budget consumption in real time.
    
    Example: 99.9% SLO over 30 days = 43.2 minutes of allowed downtime.
    """
    def __init__(self, slo_target: float, window_days: int = 30):
        self.slo_target = slo_target          # e.g., 0.999 for 99.9%
        self.window_minutes = window_days * 24 * 60
        self.allowed_downtime_minutes = self.window_minutes * (1 - slo_target)
        self.downtime_minutes_spent = 0.0
        self.start_time = datetime.now()

    def record_downtime(self, minutes: float):
        self.downtime_minutes_spent += minutes

    def budget_remaining_minutes(self) -> float:
        return max(0, self.allowed_downtime_minutes - self.downtime_minutes_spent)

    def budget_remaining_pct(self) -> float:
        return (self.budget_remaining_minutes() / self.allowed_downtime_minutes) * 100

    def burn_rate(self) -> float:
        """How fast are we burning budget vs. expected rate? 1.0 = on track, >1.0 = burning fast."""
        elapsed = (datetime.now() - self.start_time).total_seconds() / 60
        expected_spent = (elapsed / self.window_minutes) * self.allowed_downtime_minutes
        if expected_spent == 0:
            return 0.0
        return self.downtime_minutes_spent / expected_spent

    def summary(self) -> str:
        return (
            f"SLO: {self.slo_target*100:.2f}%  |  "
            f"Budget: {self.allowed_downtime_minutes:.1f} min  |  "
            f"Spent: {self.downtime_minutes_spent:.1f} min  |  "
            f"Remaining: {self.budget_remaining_pct():.1f}%  |  "
            f"Burn rate: {self.burn_rate():.2f}x"
        )

# Usage
budget = ErrorBudget(slo_target=0.999, window_days=30)
budget.record_downtime(minutes=12.5)   # incident on day 3
budget.record_downtime(minutes=8.0)    # incident on day 11
print(budget.summary())
# SLO: 99.90%  |  Budget: 43.2 min  |  Spent: 20.5 min  |  Remaining: 52.5%  |  Burn rate: ...

A burn rate above 1.0 means you’ll exceed your error budget before the window closes. Burn rate above 14.4x means you’ll exhaust it within 48 hours, which is a PagerDuty alert.

4. The Health Check Anti-Pattern

I need to address something I’ve seen sink production deployments before we even get to metrics: health checks that only verify the process is listening on a port. A port check tells you the process hasn’t crashed. It tells you nothing about whether the process can serve traffic. I’ve seen this exact scenario: database connection pool was exhausted, port was open, load balancer marked the instance healthy, every request returned a 500. The monitoring was dark green the whole time.

A real health check must exercise the actual request path: connect to dependencies, perform a lightweight but genuine operation, return structured status. In Kubernetes this means a readiness probe hitting a /health endpoint that checks dependency connectivity. Critically, readiness and liveness are different probes:

• Liveness: Is the process deadlocked? If not, keep it alive. If yes, kill and restart it.
• Readiness: Can it serve traffic right now? If not, remove it from the load balancer pool, but don’t kill it.

A process that is alive but not ready (warming up a cache, waiting for a dependency) should fail readiness but pass liveness. Confusing these two causes cascading restarts during startup under load is a failure mode I’ve seen multiple times in prod. See my Zero-Downtime Services on Kubernetes and Istio post for the full treatment.

5. Why Average Latency Lies

Here’s a production story I’ve seen more than once. The team does an efficiency push: optimizes the hot path, ships a 30% improvement in p50 latency. Dashboards celebrate but three weeks later, the p99 is back to where it started. The answer is queuing theory. Consider a server with a queue in front of it. Define utilization P as:

P = arrival rate / service rate

The average number of items in the system in queue plus being served is:

E[N] = P / (1 - P)

This is not a linear relationship. It’s an asymptote that goes vertical as you approach full utilization:

When you make the code faster (higher service-rate), P drops, and you slide left on this curve, i.e., fewer items queuing with lower tail latency. But then traffic grows or you reduce servers to “realize the savings.” P climbs back to where it was, and latency returns with it. The key lesson is that the average latency reflects the fast path but high-percentile latency (p99, p99.9) is extremely sensitive to queue depth. High percentile latency is a leading indicator that you’re approaching overload.

There’s a counterintuitive implication from this: p99 is a terrible way to measure whether your efficiency work succeeded. It’s so sensitive to the queuing nonlinearity that changes in utilization will swamp the signal from your actual code changes. For measuring efficiency, mean latency is actually better because it tracks the true cost of processing one request without queue effects. Use percentiles for alerting and use mean for efficiency measurement.

6. Percentiles From First Principles

Let’s go over percentiles from scratch, because monitoring tools throw around “p50”, “p99”, “p99.9” without ever explaining what they actually represent, and misunderstanding them leads to misreading dashboards. Given a set of N latency measurements, sort them from fastest to slowest. The Nth percentile is the value at position N% in that sorted list.

Latencies (ms): [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
Sorted:          [5, 7, 8, 9, 10, 11, 12, 13, 250, 400]
                  ^              ^              ^
                  p10           p50            p90

p50 = 10ms  (50% of requests were at or below this speed)
p90 = 13ms  (90% of requests were at or below this speed)
p99 = 400ms (99% of requests were at or below this speed)

What p99 tells you is: at most 1% of your requests see latency worse than this number. Equivalently, 999 out of every 1000 requests complete faster than p99. The catch is that p99 is a single value and it summarizes nothing about the shape of the distribution between p90 and p99. Latency can get dramatically worse for customers in that range without your p99 alarm firing.

import numpy as np

def explain_percentile(latencies_ms: list[float]):
    """Show what percentiles mean in plain English."""
    arr = np.array(sorted(latencies_ms))
    n = len(arr)
    
    stats = {
        "mean":  np.mean(arr),
        "p50":   np.percentile(arr, 50),
        "p90":   np.percentile(arr, 90),
        "p95":   np.percentile(arr, 95),
        "p99":   np.percentile(arr, 99),
        "p99.9": np.percentile(arr, 99.9),
        "max":   np.max(arr),
    }
    
    print(f"{'Statistic':<10} {'Value':>10}   Plain English")
    print("-" * 65)
    print(f"{'mean':<10} {stats['mean']:>10.1f}ms  Average — hides bimodal distributions")
    print(f"{'p50':<10} {stats['p50']:>10.1f}ms  Half of requests faster than this")
    print(f"{'p90':<10} {stats['p90']:>10.1f}ms  90% of requests faster than this")
    print(f"{'p95':<10} {stats['p95']:>10.1f}ms  95% of requests faster than this")
    print(f"{'p99':<10} {stats['p99']:>10.1f}ms  99% of requests faster than this")
    print(f"{'p99.9':<10} {stats['p99.9']:>10.1f}ms  999/1000 requests faster than this")
    print(f"{'max':<10} {stats['max']:>10.1f}ms  Worst single request (very noisy)")

# Simulate a bimodal latency distribution
# 95% fast requests (cache hit), 5% slow (cache miss + DB query)
import random
random.seed(42)
latencies = [
    random.gauss(10, 2) if random.random() > 0.05 else random.gauss(300, 40)
    for _ in range(1000)
]
explain_percentile(latencies)

Statistic       Value   Plain English
-----------------------------------------------------------------
mean             24.8ms  Average — hides bimodal distributions
p50              10.4ms  Half of requests faster than this
p90              12.1ms  90% of requests faster than this
p95              17.9ms  95% of requests faster than this
p99             302.1ms  99% of requests faster than this
p99.9           375.8ms  999/1000 requests faster than this
max             392.4ms  Worst single request (very noisy)

7. Moving Averages and Rolling Percentiles

When Grafana shows you a p99 or Datadog shows you an error rate, it’s not summing up all-time data. It’s computing over a rolling time window.

Simple Moving Average vs EWMA

A Simple Moving Average (SMA) gives equal weight to every sample in the window:

from collections import deque
import statistics

class SMA:
    """Simple Moving Average — every sample in the window weighted equally."""
    def __init__(self, window: int):
        self.buf = deque(maxlen=window)
    
    def add(self, v: float) -> float:
        self.buf.append(v)
        return statistics.mean(self.buf)

An Exponentially Weighted Moving Average (EWMA) gives more weight to recent samples, fading older ones smoothly:

class EWMA:
    """
    Exponentially Weighted Moving Average.
    alpha: 0 < alpha < 1
    - High alpha (e.g. 0.3): reacts fast, noisier
    - Low alpha  (e.g. 0.05): smoother, slower to detect changes
    
    StatsD uses EWMA for gauge values. Prometheus uses time-window sums.
    """
    def __init__(self, alpha: float = 0.1):
        self.alpha = alpha
        self.value: float | None = None

    def add(self, sample: float) -> float:
        if self.value is None:
            self.value = sample
        else:
            self.value = self.alpha * sample + (1 - self.alpha) * self.value
        return self.value

# Demonstrate: same spike, different alphas
spike_data = [10, 10, 10, 10, 250, 10, 10, 10, 10, 10]
slow_ewma = EWMA(alpha=0.05)
fast_ewma = EWMA(alpha=0.30)

print(f"{'Sample':>8} {'Value':>8} {'alpha=0.05':>10} {'alpha=0.30':>10}")
for i, v in enumerate(spike_data):
    print(f"{i:>8} {v:>8.0f} {slow_ewma.add(v):>10.1f} {fast_ewma.add(v):>10.1f}")

  Sample    Value     alpha=0.05   alpha=0.30
       0       10       10.0       10.0
       1       10       10.0       10.0
       4      250       21.9       82.0   --> fast alpha sees the spike much louder
       5       10       21.3       58.4   --> slow alpha recovers faster
       9       10       18.5       17.2

Rolling Percentile

Computing exact percentiles over a moving window requires keeping raw samples and re-sorting. For production scale, the T-Digest algorithm computes approximate percentiles with bounded memory. Here’s the conceptual version first:

import numpy as np
from collections import deque

class RollingPercentile:
    """
    Rolling percentile over a fixed window of recent samples.
    
    Production note: At high throughput, use T-Digest or DDSketch instead.
    Prometheus uses pre-defined histogram buckets + linear interpolation.
    """
    def __init__(self, window: int, pctile: float):
        self.buf = deque(maxlen=window)
        self.pctile = pctile

    def add(self, v: float) -> float | None:
        self.buf.append(v)
        if len(self.buf) < 2:
            return None
        return float(np.percentile(list(self.buf), self.pctile))

# Show how window size affects sensitivity
import random
random.seed(7)

data = [random.gauss(10, 2) for _ in range(90)] + \
       [random.gauss(200, 20) for _ in range(10)]  # degradation at t=90

p99_small  = RollingPercentile(window=20,  pctile=99)
p99_medium = RollingPercentile(window=100, pctile=99)

print("How window size affects p99 detection of a latency spike:")
print(f"{'t':>4} {'value':>8} {'p99 w=20':>12} {'p99 w=100':>12}")
for t, v in enumerate(data[80:]):   # show the transition region
    small  = p99_small.add(v)
    medium = p99_medium.add(v)
    marker = " --> spike starts" if t == 10 else ""
    if small and medium:
        print(f"{t+80:>4} {v:>8.1f} {small:>12.1f} {medium:>12.1f}{marker}")

Prometheus histogram vs. summary: Prometheus offers two ways to track latency. A Summary computes quantiles client-side over a rolling window but you can’t aggregate across instances. A Histogram records counts in pre-defined buckets and approximates quantiles server-side, which is slightly less accurate, but fully aggregatable. For microservices with multiple replicas, always use Histogram.

8. Trimmed Mean: More Signal, Real Tradeoffs

Here’s the core difference between a percentile and a trimmed mean, using the product review analogy:

100 latency measurements, sorted by speed:

p99  = the single worst measurement in the best 99%
       (the 99th measurement out of 100, sorted fastest-to-slowest)

tm99 = the average of all 99 measurements in the best 99%
       (discard the 1 slowest, average the remaining 99)

tm99 summarizes 99 times more data than p99. That makes it more stable (less spiky under low traffic), harder to game (a gradual degradation can hide between percentile checkpoints, but tm99 will catch it), and more representative of typical customer experience.

• tm99 tracks the average experience of your bulk of customers
• TM(99%:) tracks the average of your slowest 1%; ensures outlier experience doesn’t silently worsen

Together these two numbers cover 100% of your requests with just two metrics.

import numpy as np

def compute_tm_stats(samples: list[float]) -> dict:
    """
    Compute a full suite of trimmed mean statistics.

    Syntax mirrors CloudWatch / AWS Embedded Metrics Format:
      tm99        = TM(0%:99%)  = average of fastest 99%
      TM(99%:)    = TM(99%:100%) = average of slowest 1%  
      TM(1%:99%)  = drop both extremes (handles unbounded latency)
      IQM         = TM(25%:75%) = Interquartile Mean
    """
    arr = np.sort(np.array(samples))
    n = len(arr)

    def tm(lower_pct: float, upper_pct: float) -> float:
        lo = np.percentile(arr, lower_pct)
        hi = np.percentile(arr, upper_pct)
        trimmed = arr[(arr >= lo) & (arr <= hi)]
        return float(np.mean(trimmed)) if len(trimmed) else float('nan')

    return {
        "mean":       float(np.mean(arr)),
        "p50":        float(np.percentile(arr, 50)),
        "p99":        float(np.percentile(arr, 99)),
        "tm99":       tm(0, 99),      # avg of fastest 99%
        "TM(99%:)":   tm(99, 100),    # avg of slowest 1%  --> watch your outliers here
        "TM(1%:99%)": tm(1, 99),      # drop both extremes (use for unbounded latency)
        "IQM":        tm(25, 75),     # interquartile mean
    }

# Scenario: a cache-miss spike where 2% of requests are slow
rng = np.random.default_rng(42)
fast = rng.normal(10, 1.5, 980)
slow = rng.normal(350, 30, 20)
samples = np.concatenate([fast, slow]).tolist()

stats = compute_tm_stats(samples)
print(f"{'Metric':<14} {'Value':>10}   Notes")
print("-" * 65)
for k, v in stats.items():
    notes = {
        "mean":       "Pulled up by slow tail — misleading",
        "p50":        "Median — fine but ignores tail",
        "p99":        "Single value at 99th position",
        "tm99":       "Average of 98% of customers --> primary SLO metric",
        "TM(99%:)":   "Average of slowest 2% --> outlier watchdog",
        "TM(1%:99%)": "Drops both extremes — good for browser metrics",
        "IQM":        "Middle 50% average — robust to both extremes",
    }.get(k, "")
    print(f"{k:<14} {v:>10.1f}ms  {notes}")

Metric              Value   Notes
-----------------------------------------------------------------
mean                16.8ms  Pulled up by slow tail — misleading
p50                  9.9ms  Median — fine but ignores tail
p99                335.2ms  Single value at 99th position
tm99                10.1ms  Average of 98% of customers --> primary SLO metric
TM(99%:)           351.4ms  Average of slowest 2% --> outlier watchdog
TM(1%:99%)          10.1ms  Drops both extremes — good for browser metrics
IQM                  9.8ms  Middle 50% average — robust to both extremes

Bounded vs. unbounded latency:

• Bounded latency (server-side, with request timeouts): use tm99 + TM(99%:). Since latency is capped by your timeout, even the worst measurements are meaningful.
• Unbounded latency (client-side browser metrics, user-perceived time): use TM(1%:99%). A user who closes their laptop mid-request and reopens it days later may log a latency of 230,400 seconds. These shouldn’t contaminate your outlier statistics. Drop the top and bottom extremes.

I have seen in a real-life production services where teams work towards improving p50/median but everything else gets worse. You only find this out when you examine tm95 because latency was consistently worse for a growing number of customers. The key lesson is that percentiles create blind spots “between the checkpoints.” A degradation that affects the 40th–60th percentile range will move neither p25 nor p75 much. Trimmed mean, because it averages across the entire range, catches these shifts. However, trimmed mean has its own blind spot. It deliberately removes the part of the distribution that dominates user experience in fan-out architectures. The right answer is not to choose between percentiles and trimmed mean but use both.

10. Winsorized Mean, Percentile Rank, and IQM

These statistics show up in CloudWatch and modern observability platforms, and they each solve a specific problem.

Winsorized Mean (WM)

Like trimmed mean, but instead of discarding outliers, it replaces them with the boundary value. For wm99:

• Find the value at the 99th percentile (= p99)
• Treat all 1% outliers as if they had exactly that p99 value
• Average all 100% of samples

def winsorized_mean(samples: list[float], lower_pct: float = 0, upper_pct: float = 99) -> float:
    arr = np.array(samples, dtype=float)
    lo = np.percentile(arr, lower_pct)
    hi = np.percentile(arr, upper_pct)
    # Clip: anything below lo becomes lo, anything above hi becomes hi
    winsorized = np.clip(arr, lo, hi)
    return float(np.mean(winsorized))

Winsorized mean gives some weight to outliers without letting extreme values skew the average. The difference between tm99 and wm99 is subtle at high percentages and wm99 will be slightly higher because it includes the outliers rather than dropping them.

Percentile Rank PR()

Percentile rank answers the inverse question from percentile. Percentile says: “What latency value marks the Nth percent?” Percentile rank says: “What percent of requests are below a given latency value?”

If you have an SLA of “respond within 500ms to 99% of users,” you’d normally monitor p99 and check it’s <= 500ms. With Percentile Rank, you instead plot PR(:500ms, i.e., the percentage of requests completing within 500ms and drive that number toward 99% or higher. This is more directly action-oriented: you always know exactly how far below your SLA you are.

def percentile_rank(samples: list[float], threshold: float) -> float:
    """What fraction of samples are at or below threshold?"""
    arr = np.array(samples)
    return float(np.mean(arr <= threshold) * 100)

# Example: SLA is p99 < 500ms
samples_ms = [10, 12, 9, 11, 450, 10, 13, 600, 11, 10]  # small sample
pr_500 = percentile_rank(samples_ms, 500)
print(f"PR(:500ms) = {pr_500:.1f}%  (SLA requires 99%)")
# PR(:500ms) = 90.0%  (SLA requires 99%) — you're 9 percentage points short

IQM (Interquartile Mean)

IQM is simply TM(25%:75%), the average of the middle 50% of samples, discarding the top and bottom 25%. It’s extremely robust to outliers in both directions, useful when you expect noise from both ends of the distribution (e.g., some requests are trivially fast cache hits, others are pathologically slow).

11. The Inspection Paradox: Your Users Experience Worse Than Your Metrics Show

As Marc Brooker’s explained in his blog, this is the most underappreciated gap in distributed systems reliability. For example, say your service has outages with very different durations: some resolve in 30 seconds, but occasionally one runs for 3 hours. Your MTTR (Mean Time to Recovery) might calculate to 5 minutes. But when a user hits your service during an outage, they’re more likely to land in a long outage than a short one because long outages have more time-slots for users to arrive in.

Customer-experienced mean recovery = (1/2) × (MTTR + Variance/MTTR)

The second term is what kills you. If your outage duration has high variance, e.g., fast recovery most of the time, but occasional 3-hour events then that variance term dominates. Your customers experience something dramatically worse than your MTTR.

import random
import math
import statistics

def inspection_paradox_demo(
    median_recovery_min: float,
    p99_recovery_min: float,
    arrivals_per_min: float = 100,
    n_outages: int = 2000
) -> dict:
    """
    Simulate the gap between operator MTTR and customer-experienced recovery.
    
    Key insight: customers are t-weighted samplers of your outage distribution.
    A 10-minute outage gets sampled by ~10x as many clients as a 1-minute outage.
    """
    # Fit lognormal to median and p99
    mu = math.log(median_recovery_min)
    sigma = (math.log(p99_recovery_min) - mu) / 2.326

    server_durations = []
    client_wait_times = []

    for _ in range(n_outages):
        duration = random.lognormvariate(mu, sigma)
        server_durations.append(duration)

        # Clients arrive as a Poisson process during the outage
        t = 0.0
        while True:
            gap = random.expovariate(arrivals_per_min)
            if t + gap > duration:
                break
            # This client arrived at time t, waits until outage ends
            client_wait_times.append(duration - t)
            t += gap

    return {
        "operator_mttr":        statistics.mean(server_durations),
        "operator_p99":         sorted(server_durations)[int(len(server_durations) * 0.99)],
        "customer_mean_wait":   statistics.mean(client_wait_times) if client_wait_times else 0,
        "customer_p99_wait":    sorted(client_wait_times)[int(len(client_wait_times) * 0.99)] if client_wait_times else 0,
        "experience_gap_ratio": (statistics.mean(client_wait_times) / statistics.mean(server_durations)) if client_wait_times else 0,
    }

result = inspection_paradox_demo(
    median_recovery_min=1,    # median outage resolves in 1 minute
    p99_recovery_min=60,      # but 1% of outages take an hour
)

print("Scenario: 1-minute median recovery, 60-minute p99 recovery")
print()
print("What your on-call dashboard shows:")
print(f"  MTTR:              {result['operator_mttr']:.1f} minutes")
print(f"  p99 recovery:      {result['operator_p99']:.1f} minutes")
print()
print("What your customers actually experience:")
print(f"  Mean recovery:     {result['customer_mean_wait']:.1f} minutes")
print(f"  p99 recovery:      {result['customer_p99_wait']:.1f} minutes")
print(f"  Experience gap:    {result['experience_gap_ratio']:.1f}x worse than MTTR")

Scenario: 1-minute median recovery, 60-minute p99 recovery

What your on-call dashboard shows:
  MTTR:              4.9 minutes
  p99 recovery:      56.6 minutes

What your customers actually experience:
  Mean recovery:     60.0 minutes
  p99 recovery:      797.3 minutes
  Experience gap:    12.1x worse than MTTR

This is why tail recovery time matters more than averages suggest. Timeout-and-retry can hide individual request latency, but it cannot hide recovery time. Once a client gets stuck in an outage, retries don’t shorten the outage, they just add load to an already struggling service. The right takeaway: minimize variance in recovery time, not just its mean. Bounded, predictable recovery is far better for customers than fast-average-but-occasional-disaster.

12. Tail Latency Amplifies in Microservices

Modern architectures decompose user requests into many service calls. This creates two topologies, and both amplify tail latency:

Fan-out math: If each service has a 1% probability of a slow response, the probability that at least one is slow when calling N services in parallel is:

P(at least one slow) = 1 - (1 - 0.01)^N

What was a rare 1% tail now affects the majority of user interactions. And here’s the pernicious part: your per-service p99 metric looks perfectly fine. The damage is invisible at the service level, only visible at the user-experience level.

import numpy as np, random

def simulate_fanout(n_backends: int, tail_prob: float = 0.01, n_reqs: int = 20_000):
    """
    Simulate client experience when calling n_backends in parallel.
    Each backend: (1-tail_prob) chance of fast, tail_prob chance of slow.
    """
    results = []
    slow_count = 0
    for _ in range(n_reqs):
        latencies = []
        for _ in range(n_backends):
            if random.random() < tail_prob:
                latencies.append(random.gauss(250, 25))
                slow_count += 1
            else:
                latencies.append(random.gauss(10, 2))
        results.append(max(latencies))  # fan-out: wait for slowest
    
    arr = np.array(results)
    return {
        "p50":  np.percentile(arr, 50),
        "p99":  np.percentile(arr, 99),
        "mean": np.mean(arr),
        "pct_slow_user_requests": np.mean(arr > 50) * 100,
    }

print(f"{'N':>4} {'p50 (ms)':>10} {'p99 (ms)':>10} {'mean (ms)':>10} {'% users hit slow':>18}")
for n in [1, 5, 10, 25, 50, 100]:
    r = simulate_fanout(n)
    print(f"{n:>4} {r['p50']:>10.1f} {r['p99']:>10.1f} {r['mean']:>10.1f} {r['pct_slow_user_requests']:>18.1f}%")

The trimmed mean blind spot revisited. At N=50, nearly 40% of user requests are slow. But your per-service tm99 (averaging the best 99% of individual service calls) still looks great because it’s averaging the fast cluster. This is exactly the case where trimmed mean gives you false comfort. You need explicit end-to-end latency tracking at the user-request level, not just per-service tail tracking.

13. The Pooling Dividend: Why Redundancy Is Non-Linear

Adding servers doesn’t just increase capacity linearly but it also improves latency through pooling. This comes from the Erlang C model in queuing theory. For example, two designs, both handling the same total load:

• Design A: 1 server at 80% utilization
• Design B: 10 servers sharing load, each at 80% utilization

Design A has roughly a 13% chance of any incoming request finding the server busy and joining a queue. Design B has roughly a 3.6% chance. Double the fleet to 20 servers at the same 80% per-server utilization, and the queueing probability drops toward 1%. You’re getting better latency and better tail behavior at the same per-server cost.

import math
from functools import lru_cache

def erlang_c(c: int, rho: float) -> float:
    """
    Erlang C formula: probability an arriving request must queue
    (rather than being served immediately) in an M/M/c system.
    
    c: number of servers
    rho: per-server utilization (0 < rho < 1)
    """
    a = c * rho  # total offered load
    
    @lru_cache(maxsize=None)
    def factorial(n: int) -> int:
        return 1 if n <= 1 else n * factorial(n - 1)
    
    # Sum term for the denominator
    sum_term = sum(a**k / factorial(k) for k in range(c))
    last_term = (a**c / factorial(c)) * (1 / (1 - rho))
    
    ec = last_term / (sum_term + last_term)
    return ec

print("Probability a request must queue before being served:")
print(f"{'Servers':>8} {'Utilization':>12} {'Queue prob':>12}   {'Queue %':>8}")
for c in [1, 2, 5, 10, 20, 50]:
    ec = erlang_c(c=c, rho=0.8)
    print(f"{c:>8} {'80%':>12} {ec:>12.4f}   {ec*100:>7.1f}%")

Probability a request must queue before being served:
 Servers  Utilization   Queue prob    Queue %
       1          80%       0.8000      80.0%
       2          80%       0.7111      71.1%
       5          80%       0.5541      55.4%
      10          80%       0.4092      40.9%
      20          80%       0.2561      25.6%
      50          80%       0.0870       8.7%

Most of the benefit materializes at modest fleet sizes. You don’t need to be at hyperscale to get pooling gains. A fleet of 5-10 servers sharing load through a proper load balancer will have dramatically better tail latency behavior than the same compute running as independent instances.

14. Retries, Circuit Breakers, and the Amplification Trap

Retries protect against transient failures like a GC pause, a brief network glitch, a thundering herd. In past production deployment, I use up to 3 retries with exponential backoff for idempotent read operations. The protection against false positives is real and worthwhile. But retries have a catastrophic failure mode: retry amplification.

A single user request can generate 3 × 3 × 3 = 27 actual requests to a struggling downstream service. This turns a partial overload into a total collapse. I’ve watched this happen in production, e.g., a service that was at 60% capacity receives a burst of retries from a misbehaving upstream and immediately spikes to 200% load, failing every request, causing more retries, a feedback loop.

The mitigations:

import time
import threading
from collections import deque

class RetryBudget:
    """
    Limit total retry rate as a fraction of total traffic.
    If retries exceed the budget, fail fast instead of retrying.
    
    Classic mitigation for retry amplification.
    """
    def __init__(self, budget_fraction: float = 0.10, window_seconds: int = 60):
        self.budget_fraction = budget_fraction
        self.window = window_seconds
        self.total_requests: deque = deque()
        self.retry_requests: deque = deque()
        self._lock = threading.Lock()

    def _prune(self):
        cutoff = time.monotonic() - self.window
        while self.total_requests and self.total_requests[0] < cutoff:
            self.total_requests.popleft()
        while self.retry_requests and self.retry_requests[0] < cutoff:
            self.retry_requests.popleft()

    def record_request(self):
        with self._lock:
            self.total_requests.append(time.monotonic())

    def should_retry(self) -> bool:
        """Returns True if we have retry budget remaining."""
        with self._lock:
            self._prune()
            total = len(self.total_requests)
            retries = len(self.retry_requests)
            if total == 0:
                return True
            current_rate = retries / total
            if current_rate < self.budget_fraction:
                self.retry_requests.append(time.monotonic())
                return True
            return False  # budget exhausted — fail fast, don't amplify


class CircuitBreaker:
    """
    Stop sending requests to a failing downstream.
    Transitions: CLOSED -> OPEN -> HALF_OPEN -> CLOSED
    """
    CLOSED, OPEN, HALF_OPEN = "CLOSED", "OPEN", "HALF_OPEN"

    def __init__(self, failure_threshold: float = 0.5, cooldown_seconds: float = 30):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown_seconds
        self.state = self.CLOSED
        self.failures = 0
        self.total = 0
        self.opened_at: float | None = None

    def call_allowed(self) -> bool:
        if self.state == self.CLOSED:
            return True
        if self.state == self.OPEN:
            if time.monotonic() - self.opened_at > self.cooldown:
                self.state = self.HALF_OPEN
                return True  # let one probe through
            return False  # fail fast
        return True  # HALF_OPEN: let one probe through

    def record_success(self):
        self.failures = 0
        self.total = 0
        self.state = self.CLOSED

    def record_failure(self):
        self.failures += 1
        self.total += 1
        if self.total >= 10 and self.failures / self.total >= self.failure_threshold:
            self.state = self.OPEN
            self.opened_at = time.monotonic()

Hedge requests are often better than retries for latency problems. Instead of waiting for a timeout and retrying, fire a second request after a short delay (say, the p90 latency). Accept whichever responds first, cancel the other. This cuts your tail exposure without amplifying load as aggressively, because typically one of the two requests will succeed quickly.

15. Synthetic Canaries in Production

Error rates and latency percentiles tell you what’s happening to real traffic but only after users are affected. Synthetic canaries fill the gap: background processes that continuously exercise your API end-to-end, giving you availability signal even at 3am when real traffic is low.

Key design decisions from production experience:

• Test the full workflow, not just the health endpoint. A canary for a data API should create, read, update, and delete a record. One for an auth service should issue a token, validate it, and revoke it. Shallow canaries that only call GET /health will miss the exact failures that health check anti-patterns also miss.
• Track first-attempt and final success separately. If your canary succeeds on retry 2 90% of the time, the final success rate looks fine but something is quietly broken. First-attempt success rate catches this.
• Keep canary observability separate from production. Mixing them has two failure modes: canary failures inflate your production error rate, and canary successes can mask production degradation if canaries hit warm caches or a separate code path.
• Account for canary bias. Canaries hit warm caches and have predictable access patterns. Their p99 is almost always better than real user p99. Use canary latency to detect regressions relative to a baseline, not to claim absolute performance numbers.
• Use retries in canaries, but with a limit. Up to 3 retries prevents false positives from transient network blips. But record the retry count per run, e..g, a canary that regularly needs 2+ retries is a signal worth investigating even if it eventually succeeds.

16. Putting It All Together: A Layered Monitoring Strategy

After decades of building and operating distributed systems, here’s the monitoring architecture I’d deploy for any production service from day one:

Closing: The Number That Matters Most

After all of this, the insight that has most changed how I think about availability is this: your users don’t experience your MTTR. They experience a version of it weighted by how long outages last, which skews dramatically toward your worst events. A service with a 1-minute median recovery but occasional 2-hour outages will have customers experiencing something closer to hours, not minutes. The variance in your tail events matters more than the central tendency. This is why the tail cannot be trimmed away from your visibility. Build observability that shows you the tail. Use redundancy and retries but understand how they amplify under pressure. Run canaries that exercise the whole path. Track user errors and server errors separately. Keep SLO burn rate visible so you always know how much budget you’ve spent. And when your customers say the service is slow and your dashboard says everything is green then believe the customers.

A self-guided path for junior and mid-level engineers whose core skills are quietly eroding

I have observed a contrast productivity gap watching a senior engineer use an agentic coding tool and watching a junior engineer use the same tool. The senior engineer moves faster, catches more problems, and produces better outcomes, while the junior engineer often ships code that looks finished but quietly breaks at the seams. The reason is not the tool, it is what each engineer brings to the tool.

Senior engineers are more effective with agentic AI because they have already built the skills that make AI useful: writing precise specifications, designing systems that hold together under real load, spotting code smells in a diff, understanding trade-offs between correctness and performance, maintaining the conceptual integrity of a codebase across hundreds of changes. These skills aren’t separate from coding experience, they are products of it, built up over years of writing code, breaking things, debugging production incidents, and internalizing the consequences of design decisions.

Junior and mid-level engineers haven’t built those skills yet, which used to be fine because the path to building them was clear: you wrote code, made mistakes, got reviewed by someone who caught what you missed, and learned. Repeat for several years. The trouble is that agentic coding short-circuits exactly that path. When an agent generates the code, a junior engineer faces a key problem: they cannot reliably distinguish between code that is actually correct and code that is plausibly correct. Agentic code looks right in the narrow context where it was generated, the function compiles, the tests pass, the logic seems sound. But a system is not a collection of locally correct functions. It is a web of interacting decisions, constraints, and invariants that only hold together if someone understands the whole. Senior engineers have built that whole-system mental model through years of implementation.

Some companies have responded to this by stopping junior engineer hiring entirely, reasoning that agents can now fill entry-level roles. This is a serious mistake, and a slow-moving disaster. It optimizes for short-term output while eliminating the pipeline through which every senior and principal engineer is eventually produced. Today’s junior engineers are tomorrow’s architects. When companies stop hiring and developing them, they are consuming the seed corn and they will feel it in three to five years when there are no experienced engineers left to review what the agents produce.

The risk doesn’t stop at junior engineers. Senior engineers face a subtler version of the same problem. When you stop writing code regularly, the skills built through writing it, the intuition for design, the eye for code smells, the ability to hold a large system in your head begin to decay. Specification writing becomes abstract rather than grounded. Architecture decisions lose their connection to implementation reality. Code review gets shallower because you’re no longer maintaining the mental model of how things fit together. The most important thing being lost is not any individual skill but shared understanding, what Fred Brooks called conceptual integrity, the coherence of design philosophy across an entire system that only exists when the people building it have deeply internalized how it works.

This post is about what to do about all of it. How to deliberately build the skills that agentic coding doesn’t hand you. How to maintain the skills you’ve built, as the nature of the work changes. And how to grow from junior to senior to principal in an era when the traditional feedback loop between design and implementation has been broken.

How Engineers Used to Grow

For decades, the career path followed a recognizable arc. You joined at entry level, wrote code, broke things, got your code torn apart in review, made better mistakes, and gradually developed what researchers call tacit understanding, the ability to look at a system and feel what is wrong before you can fully articulate why.

The Dreyfus model of skill acquisition describes this progression across five stages:

The Japanese martial arts concept of Shu Ha Ri mirrors this exactly. First you follow the form faithfully (Shu learn the rules). Then you find the exceptions and break with tradition (Ha question the rules). Then form dissolves into natural action (Ri transcend the rules). You cannot skip stages. The competent engineer who writes their first distributed system will make mistakes the expert would never make because they haven’t yet built the mental model that only comes from doing the work and suffering the consequences.

What made this work was consequence. Writing code gave you direct feedback. A missing lock caused a race condition. A clever abstraction became unmaintainable by the third person to touch it. A shared base class six levels deep broke four products when a parent changed. These lessons were visceral, and they stuck. The Dreyfus model would say you accumulated the situational exposure that moves you from rule-following to intuition.

Books codified what masters had learned. The Pragmatic Programmer showed how to develop craft. Code Complete provided the vocabulary for code quality. A Philosophy of Software Design showed what makes modules deep or shallow. The Mythical Man-Month showed why adding engineers to a late project makes it later, coordination cost, not coding hours, drives timelines. Research quoted there put coding at roughly 14% of total project effort. Requirements, design, testing, debugging, coordination, documentation, and operations consumed the rest. Agentic coding has compressed that 14% toward zero. The other 86% remains entirely human.

The Disruption: Design and Build Are No Longer Learned Together

In traditional software development, design and build were not two separate activities. They were one activity experienced from two angles. When you wrote the code yourself, you felt every consequence of your design decisions in real time. A bad abstraction made your own implementation painful. A missing transaction boundary caused a bug you personally had to trace to its source at midnight. You didn’t just observe these consequences. You lived them, and that is what made the lessons stick.

Agentic coding severs this connection. The engineer writes a specification, the agent produces code, and the engineer reviews the result. This workflow feels productive. It is often highly productive, for engineers who already have the judgment to specify well and review rigorously. But for engineers who are still building that judgment, it removes the primary mechanism that builds engineering judgment.

The specific failure mode for junior and mid-level engineers is the plausibility trap. Agent-generated code looks correct in the narrow local context: functions are clean, tests pass, the logic holds for the cases the spec described. What the code often lacks is correctness at the system level, the consistency guarantees that span service boundaries, the failure modes that only appear under concurrent load, the invariants that hold only if you understand the domain well enough to define them. A senior engineer reviewing that code has a whole-system mental model built through years of implementation experience. They feel when something is off even before they can articulate why. A junior engineer doesn’t have that model yet, and reviewing agent-generated code without it is like trying to spot a structural flaw in a building you’ve never seen the blueprints for.

Bertrand Meyer, in his analysis in Communications of the ACM makes this point precisely: AI-generated code creates a dangerous psychological bias because it is significantly harder to spot a subtle logical flaw in well-structured generated code than in the messy human-written code reviewers are used to. Cleanliness produces false confidence. Agents write plausible code. Plausible is not the same as correct, and the gap between the two is exactly where junior engineers without deep mental models get stuck.

For senior engineers, the risk is different but equally real. Specification, design, architecture, code review, and debugging are not static skills you acquire once and keep forever. They are maintained through practice through the practice of building systems, not just reviewing them. When senior engineers stop writing code regularly, their design intuitions gradually lose contact with implementation reality. Architectural decisions start floating free from the constraints that make them achievable. Specifications become abstract rather than grounded in how things actually work. Code review gets shallower because the reviewer is no longer holding a live mental model of how the system fits together under pressure. The decay is slow and invisible until it isn’t. I had previously seen this decay when principal/staff stopped writing code and became pure architects but agentic coding is making it more prevalent.

In AI Writes Code. You Own the Design. Here’s How to Keep It That Way, I described how AI agents resemble offshore teams more than co-located colleagues: they have a narrow context window, they lack shared understanding of your codebase, they produce locally correct work that misses the bigger picture, and they have no memory between sessions. Every session starts from zero. Amazon AWS teams learned this the hard way, AI-generated code that looked right, passed review, and then caused production incidents. Their response was to significantly tighten review policies. When a production incident costs customers millions or exposes a security breach, you cannot file a bug against Cursor or Claude Code. The engineer who approved the change is accountable.

What’s at stake, underneath all of this, is shared understanding. Fred Brooks called it conceptual integrity, the coherence of design philosophy that runs through an entire system. Conceptual integrity doesn’t live in a document. It lives in the heads of engineers who have thought deeply about the system, implemented parts of it themselves, debugged its failures, and built up a shared mental model of how the pieces fit. That shared model is what gets lost when design and implementation are permanently separated. It is also the most important and hardest-to-recover thing a team can lose. Code can be rewritten. Conceptual integrity, once gone, takes years to rebuild. Instead, the system accumulates exactly what Brooks warned against: many locally reasonable decisions that don’t cohere into a coherent whole.

We cannot give up on junior and mid-level engineers developing real skills, even as agents handle more of the typing. Code reviews by senior engineers help but we mostly learn by doing, not by watching. Engineers still need to understand how code works, how it fits into the larger system, and what the trade-offs mean under real conditions. What we build and what we understand are not separate. Pulling them apart entirely is a quality risk the whole team will pay for, slowly at first and then all at once.

The Two Skill Trees

Engineering growth has always required two parallel tracks. Agentic coding affects each differently.

Hard skills are the technical capabilities: designing systems, understanding trade-offs, debugging complex failures, recognizing code smells, reasoning about correctness under concurrency, and mastering both functional and non-functional requirements. The traditional path built these incrementally through writing code, breaking things, and fixing them. Agentic coding removes that feedback loop without replacing it.

Soft skills are the interpersonal and organizational capabilities: writing clearly, building consensus, managing ambiguity, estimating honestly, communicating with non-technical stakeholders, mentoring others, and owning outcomes across an entire project lifecycle. These have always mattered for growth from mid-level to senior and from senior to principal. Agentic coding hasn’t reduced their importance, it has raised the bar, because the differentiating value of a senior engineer shifts away from code production and toward judgment, communication, and design thinking.

Both tracks require deliberate practice. The diagram below shows the full skill landscape across career levels, mapped to the hard/soft split.

The Career Levels in Detail

Before getting to specific advice, it helps to have a clear picture of what each level actually requires.

Junior: You own software components and work on well-defined problems. You produce high-quality code under guidance, learn from review feedback. You collaborate across the full development lifecycle including code, tests, deployment, documentation but you rely on peers and managers for guidance on design. You are expected to deliver reliably within a clear scope, and you actively seek to learn.

Mid-level: You are an autonomous contributor, owning features, not just components. You design software solutions for difficult problems, though you still seek guidance on architectural strategy. You coach junior engineers. You make priority trade-offs between feature work and operational work. You participate meaningfully in code reviews not just catching bugs, but providing direction.

Senior: You lead multi-engineer projects and own team-level architecture. You work on complex problems with multiple conflicting constraints. You write for both technical and non-technical audiences. You solve problems that don’t yet have a defined technology strategy. You balance short-term delivery against long-term architectural health. You become a force multiplier and your presence should make everyone meaningfully better.

Staff / Principal: You lead across an organization, not just a team. You define technical strategy and roadmaps that span multiple teams. You take on intrinsically hard problems like major bottlenecks and undefined high-impact opportunities. You align teams toward cohesive technical visions. You earn influence through credibility and results, not title.

Junior engineers are largely tactical. Seniors span tactical and operational. Staffs/Principals are primarily operational and strategic.

Hard Skills: What to Build Deliberately

1. Learn to Write Specifications

The most immediately practical hard skill in the agentic era is writing precise specifications. Agents produce what you specify. Vague specifications produce code that fills gaps with training-data assumptions, which may or may not match your domain. This is a learnable craft.

I wrote you-got-skills framework to demonstrate how to build specification skills with use of RFC 2119 discipline. Write a one-page spec before prompting an agent, every time. For significant features, use the full design document structure I previously shared: problem statement, proposal with trade-offs, alternatives considered, non-functional requirements, and rollout plan. The act of writing this forces you to make decisions you were previously leaving implicit.

2. Build a Design Sense

One of the clearest failure patterns in junior and mid-level engineers using agentic coding is an underdeveloped design sense. They can describe what they want. They struggle to explain why one design is better than another, or to recognize when generated code silently violates conceptual integrity, which is identified in The Mythical Man-Month as the single most important property of a well-designed system.

Build this sense deliberately. Read A Philosophy of Software Design and practice the deletion test: if you deleted this module, where would the complexity go? Deep modules with small interfaces earn their place. Shallow pass-throughs add indirection without value. AI defaults to shallow modules, lots of small classes, each delegating to the next. Learning to recognize this pattern and push back on it is a concrete skill you can develop right now.

In Applying Domain-Driven Design and Clean/Hexagonal Architecture to MicroServices, I shared how Domain-Driven Design can employed for an application architecture. When AI generates code for your domain, it has no idea what your domain means. Practice making invalid states unrepresentable. Sum types that enumerate valid states, state machines that encode valid transitions, parse-don’t-validate at boundaries, These design patterns matter more in the agentic era because the compiler becomes your code reviewer when humans can’t catch everything. When AI generates code within a well-typed system, category errors that would slip through casual review become compile errors.

Study the Stable Dependencies Principle: depend in the direction of stability. As illustrated in the reusability trap analysis, the most expensive bugs often don’t come from duplicated code, they come from code shared prematurely. Recognizing when DRY has become a liability is a senior-engineer skill that requires real practice to develop.

3. Develop a Nose for Code Smells and Code Review

Reviewing AI-generated code is not casual reading. Agents write clean, plausible code. The bugs that slip through are not obvious, they’re missing idempotency tokens, race conditions that appear only under concurrent load, enum values that propagate without being handled by all consumers.

Build a structured review practice. Apply two explicit passes. The first pass looks for correctness and security: logic errors (off-by-one, null handling, TOCTOU races), security holes (injection, missing auth checks, hardcoded secrets), data loss risks, and error swallowing. The second pass looks for design: are modules deep or shallow? Are invalid states representable in the type system? Does this code separate commands from queries? Is the complexity justified by the actual problem, or has the agent added abstractions for a feature used by twelve people?

Practice this on every pull request you review, whether AI-generated or human-written. The structured passes build the intuition that experienced engineers call a “nose for code smells”.

4. Master Non-Functional Requirements

Most junior engineers understand functional requirements. Senior engineers understand non-functional requirements , how reliably, under what conditions, and with what failure behavior. This is arguably the most important distinction on the path from mid-level to senior.

When you read a feature request, train yourself to immediately ask: what is the latency budget, and at what percentile? What is the consistency model between these two data stores? What happens if this operation half-succeeds? What’s the blast radius if this component fails completely? What happens at 10x current load? These questions are what agents cannot answer from a vague prompt. In Failures in MicroService Architecture, I shared a number of production issues that I experienced with distributed systems. You can apply the outbox pattern, circuit breaker, retry with jitter, bulkheads and other patterns to remedy common production issues.

5. Keep Your Hands in the Code

Agentic coding creates pressure to delegate implementation entirely. Resist it. You do not build judgment about systems you have never built yourself. Write the spike yourself before committing to full design. Implement the critical path at least once, even if an agent later handles the boilerplate. Trace the execution of generated code in a debugger until you understand what it actually does before approving it for production.

This matters most for debugging. When something fails in production, the mental model you’ve built through implementation is what lets you form hypotheses quickly. Engineers who have only reviewed AI-generated code without deeply understanding it will struggle to diagnose the failures that code produces. The CACM analysis shows that AI-generated code introduces logical and concurrency bugs in clean-looking code that humans find harder to spot than equivalent bugs in messy human-written code.

Furthermore, as agentic coding produces more and more code we don’t fully own mentally, understanding decay sets in. Storey calls this cognitive debt. A team can have low technical debt while sitting on a mountain of cognitive debt where no one can confidently predict the impact of a change. Over time, no single engineer holds the complete picture of how the system works. This makes production incidents progressively harder to diagnose. Keeping your hands in the code, owning critical-path implementations, using agents to explain generated code you don’t immediately understand.

6. Learn Formal Methods Basics

One underappreciated direction for junior/mid-level engineers is to begin learning specification and verification techniques, not as academic exercises but as practical tools for the agentic era. I shared my experience applying TLA+ for specifications in Beyond Vibe Coding: Using TLA+ and Executable Specifications with Claude. But, you can start with property-based testing: instead of writing examples, write invariants that your system must maintain regardless of input. Start with static analysis tools and learn to interpret what they find. Write explicit pre-conditions and post-conditions for complex functions, even as comments. These habits build the specification discipline that makes your agent prompts more precise and your reviews more effective.

Soft Skills: What Separates Mid-Level from Senior from Principal

7. Write with Precision and Clarity

Writing is the highest-leverage soft skill for any engineer who wants to grow. Design documents, post-mortems, and stakeholder communications all require the same underlying capability: translating technical thinking into prose that creates shared understanding.

Practice this deliberately. Write a design document for every significant thing you build, using the full structure described in How Not to Write a Design Document: problem statement, proposal, trade-offs, alternatives considered, non-functional requirements, rollout plan. Show it to a senior engineer. Ask what questions it fails to answer. Design documents are also how you develop design skills. A bad design doc does exactly what a bad design does: it makes the solution sound inevitable, skips trade-offs, and pushes hard questions into implementation. That feels fast until production starts collecting interest on every shortcut.

8. Bring Clarity to Ambiguity

The most important skill a senior engineer develops is the ability to look at a fuzzy problem and make it concrete. This is the single most valued contribution that humans still provide in the agentic era: not the code, but the thinking that makes the code correct.

Ambiguity reduction works in both directions. On the problem side: understand the actual customer need before finalizing a solution, push back on specs that describe a solution rather than a problem, ask what the real constraint is. On the solution side: identify which design decisions are reversible versus which are one-way doors. Practice this in every design review, every planning discussion, every incident retrospective.

9. Build Alignment and Consensus

The transition from proficient to expert in any domain requires operating at the social level: building consensus among people with competing interests, aligning a technical direction through an organization that has other priorities, and navigating disagreements constructively. The trust equation from Maister et al. shows that trust has four components: credibility, reliability, intimacy (safety), and self-orientation (does it serve the system or you?). Engineers who lose influence at the senior and principal level almost always fail on the fourth element. Proposals that come across as serving “my architecture” rather than “our actual problem” collapse trust fast.

Build alignment by listening before proposing. Spend time understanding what actually hurts the team before advocating for a technical direction. Frame proposals in terms of reduced toil, reduced uncertainty instead of architectural purity. Find a long-standing pain and solve it visibly. The Aikido principle from Jerry Weinberg applies here: center, enter, turn. First be aware of yourself and what you want to accomplish. Then enter the world of the other person. Then together turn the energy in a more effective direction.

10. Communicate Upward in Business Terms

Translating technical decisions into business impact separates senior engineers from principals. The ability to tell a VP concisely, what the risk is, and what it costs to address it. Learn the metrics that matter to leadership: revenue impact, customer retention, incident cost, deployment frequency, engineer productivity. Practice expressing technical proposals in those terms. This is the failure mode highlighted in How Senior Engineers Lose Trust: communicating technical complexity without translating it into business impact, focusing on engineering outputs.

11. Estimate Honestly and Decompose Work Well

Engineers who consistently underestimate erode trust. Engineers who consistently overestimate become known as blockers. Honest estimation with explicit uncertainty ranges, clear assumptions, and candid identification of the biggest risks is a key skill.

Three practices make estimation better. First, decompose into vertical slices, not horizontal layers. A vertical slice cuts through all layers and produces something independently demoable. Horizontal slicing delays feedback as you don’t know if the feature works until the last layer is complete. Second, use three-point estimation for commitments: (Best + 4×MostLikely + Worst) / 6, and present ranges rather than single numbers. Capacity is never 100%. Budget explicitly for KTLO like operational work, incident response, and technical debt.

12. Own Outcomes Beyond Your Code

The clearest signal of an engineer ready for senior responsibility is willingness to own the work nobody wants to do: the failing test that has been skipped for months, the runbook that was never written, the technical debt accumulating in the corner nobody touches, the onboarding documentation that every new hire struggles with. This is what some call being the janitor, taking responsibility for team health and code health. It builds organizational trust faster than any individual feature. Own incidents that aren’t yours. When a production problem occurs on your team, treat it as your problem regardless of who wrote the code. In Writing Post Mortems That Actually Make You Better: A Practitioner’s Guide, I explained how to use the Five Whys and the Swiss Cheese model for documenting incident post-mortems.

13. Become a Go-To Person

Focused expertise builds the kind of reputation that earns you higher-impact work. The path to being a go-to person has three branches: project ownership, technology expertise, and domain expertise. Pick one to start. Host a learning session on something you know well. Write about it internally. Help others who are stuck on it.

14. Mentor Others

Teaching is one of the fastest ways to consolidate your own understanding. When you explain a design decision to a junior engineer, you discover exactly what you do and don’t understand. When you give code review feedback that helps someone see a flaw they missed, you sharpen your own eye.

In the agentic era, junior engineers need mentorship more than ever because the traditional mechanism of learning through building and breaking code is less available. Senior engineers who help juniors understand why AI-generated code works the way it does, how to critique it structurally, and how to reason about trade-offs are providing something genuinely important. The psychological safety research from Google’s Project Aristotle applies here: teams where members feel safe raising concerns, asking questions, and challenging designs outperform teams where they don’t. You build that culture one mentoring conversation at a time.

The T-Shape and Broken Comb Model

The most useful framework for thinking about hard skill investment is the T-shape: one area of genuine depth combined with broad familiarity across adjacent areas (the horizontal bar). As engineers progress toward principal level, the shape often becomes what practitioners call a broken comb, multiple verticals of depth across different domains, connected by broad horizontal understanding. A principal engineer might go deep in distributed systems, in observability, and in the security model of their specific domain, while maintaining enough breadth to lead design conversations across the full stack.

A Concrete Self-Guided Growth Plan

Here is a practical, time-bounded path for engineers at each stage.

If you are a junior engineer (0–3 years):

• Write a one-page spec before prompting an agent. Compare what the agent produced to what you specified.
• Ask to implement at least one non-trivial feature entirely yourself, even if it takes longer.
• Read The Pragmatic Programmer and Code Complete.
• Request structured feedback on every code review you submit.
• Use agents to explain generated code you don’t understand.

If you are a mid-level engineer (3–6 years):

• Write a full design document for the next significant feature you build. Share it with a senior engineer and ask specifically what questions it fails to answer.
• Own one domain on your team completely: its documentation, its monitoring, its failure modes, its onboarding.
• Start hosting one internal learning session per quarter on something you know well. Write it up afterward.
• Apply a structured two-pass review to every pull request you review. Track what you catch over a month.
• Read A Philosophy of Software Design. Apply the deletion test and bounded context thinking to your current codebase.
• Write one post-mortem per incident using the Five Whys structure.

If you are a senior engineer aiming for staff/principal:

• Lead one project that coordinates work across multiple engineers. Own the design and run the design review. Drive the post-project retrospective.
• Translate one technical proposal into business impact language: metrics, incident cost, customer effect.
• Mentor junior engineers specifically in how to critically evaluate AI-generated code.
• Identify the most painful systemic problem on your team, the thing everyone complains about and nobody fixes. Fix it, document it, and share what you learned.

Ongoing, at every level:

Keep your hands in the code. The fraction of code that engineers write themselves will keep shrinking, but understanding what the code does, how it fits the larger system, and what its failure modes are requires someone who can read it critically, reason about it deeply, and debug it under pressure.

What We Cannot Give Up

There is real pressure in many organizations to reduce engineering involvement in requirements, design, and review to automate the entire lifecycle. This deserves serious assessment. Agents accelerate delivery but they do not absorb accountability. When code fails in production, the customer doesn’t care whether the bug was introduced by a human or a model. The engineer who approved it is responsible. Code review, even partially automated still requires human engineers who understand the system well enough to know what they’re reviewing. Junior engineers who bypass the developmental stages that build that understanding will produce reviews that miss what matters. Organizations that accept this trade-off in exchange for short-term velocity will eventually pay compounding interest.

The goal is not to resist agentic coding. The productivity gains are real and the trend is irreversible. The goal is to keep all three in check: technical debt in the code, cognitive debt in the team’s shared understanding, and intent debt in the artifacts. Agentic coding, used carelessly, accelerates all three simultaneously.

Further Reading

• AI Writes Code. You Own the Design.
• How Not to Write a Design Document
• Writing Post Mortems That Actually Make You Better: A Practitioner’s Guide
• The Reusability Trap: When DRY Becomes a Liability
• Key Takeaways from Leading Effective Engineering Teams
• Insights from Become a Great Engineering Leader
• Highlights from The Engineering Executive’s Primer
• you-got-skills structured SDLC skill framework for agentic development

I have worked for both large tech companies and startups. Two patterns kept showing up across every company I worked at startup and large company alike that both punish the engineers doing the right thing.

At startups, the pressure is entirely on shipping features. Engineers who move fast and ship constantly get rewarded. Security, observability, scalability become “future problems.” The engineers who slow down to build things properly, who push back on cutting corners, get treated as obstacles. The corners get cut anyway. When the system eventually breaks under load or gets breached, nobody connects it back to the decisions made two years earlier. The engineers who raised concerns are long gone or drowned out.

At large companies, a different trap. Ship something clean with simple design, solid implementation, few follow-up bugs and people move on. Nobody notices the problems that didn’t happen. Nobody gets promoted for the outages that never occurred. But ship something overengineered, watch it fall apart in production, spend months firefighting and suddenly you’re a hero. The tech lead who pushed patches at 2am gets noticed. Management reads the complexity as evidence of a hard problem solved. The tech lead gets promoted and moves to the next team. The engineers left behind inherit the mess.

Same outcome, different path. In both cases, the engineers who built things well are invisible. The ones who created the problems or thrived on them get ahead.

Essential vs. Accidental Complexity

In The Mythical Man-Month, Fred Brooks defined two kinds of complexity. Essential complexity is the irreducible difficulty built into the problem domain itself. Accidental complexity is the difficulty we add through poor abstractions, unnecessary coupling, and artificial layers. Larry Tesler’s Law of Conservation of Complexity says essential complexity can’t be eliminated, only moved. Push it out of the user interface and it lands in your middleware.

What most companies reward the accidental kind. Many moving parts, multiple failure modes, a fleet of services with their own deployment pipelines as these look like a hard problem solved by smart engineers. A system that just works, simply and reliably, signals nothing. The people who built it must have been working on something easy. I saw this repeatedly at larger companies. Senior engineers with years of incremental, principled improvements couldn’t get promoted because their work wasn’t considered “complex enough.” The implicit rule was clear: elegance doesn’t get you promoted.

War Stories

The database migration that became a platform. At a large tech company, we needed a simple migration from one database to another but it turned into a real-time data synchronization system. Suddenly there were shadow testing components, reconciliation pipelines, anti-entropy jobs for fixing discrepancies, and runbooks for each failure mode. The project stretched from months into years. The original problem, move data from A to B, never required any of it. But the complexity generated headcount, resources, and career advancement that a clean migration would never have produced.

The microservices migration that never finished. A monolith-to-microservices transition ran so long the team ended up maintaining both systems simultaneously. The migration date kept slipping. Nobody could tell you which services were fully cut over. The codebase became a graveyard of abandoned halfway points. Years of engineering time consumed, several promotions justified. The engineers who eventually inherited it had no idea what was intentional and what was just never cleaned up.

The Erlang rewrite. At a FinTech company, senior executive decided to rewrite an order management system from Java to Erlang, not for a specific technical reason, but because Erlang was interesting. Brooks called this the second-system effect: when engineers rewrite something they think they now understand, they pile in everything they held back the first time. The effort was far larger than anyone expected. Management abandoned it partway through. The team was left with two halves of the same system in two different languages, domain knowledge split across both.

The Go rewrite. The same executive years later decided to rewrite a Java financial system in Go because Go was what the industry was talking about. Years passed, the migration stalled. Some parts in Go, most still in Java. The team gave up. Meanwhile the actual urgent problems like data consistency, observability, performance at scale went unaddressed because everyone’s attention was on the rewrite. Nobody owned the full picture of dependencies or understood the consistency guarantees. Meanwhile, sales sold the system as a low-latency and four nine availability but in practice it was based on false illusion due to poor observability.

The postscript at that second company: when AI became the new shiny thing, the pattern played out again. Engineers who built flashy demos got promoted. The people fixing real infrastructure problems had nothing visible to show.

Conceptual Integrity Breaks Down as Organizations Grow

In the original Mythical Man-Month, Brooks argued that the most important property a system can have is conceptual integrity, one coherent design philosophy, with someone who holds the whole system in mind and says no to things that don’t fit. His prescription was a chief architect with real authority over what goes in and what stays out. That works when one person can still comprehend the system. As organizations grow and systems get divided among teams, nobody has that view anymore. Each team makes locally reasonable decisions. Accidental complexity accumulates not from individual mistakes but from the disconnect between groups who can’t see each other’s work.

Cross-cutting concerns like security, authentication, observability are where this gets dangerous fastest. I saw one system where authentication behaved differently depending on whether you were on-premises or in the cloud, and whether you were hitting the control plane or data plane. Secrets in some places, JWTs in others, config files in some environments, environment variables in others, a wall of conditional logic tying it together. No single person understood the whole thing. That mess led to a significant security breach and customer churn. Nobody designed it. It grew, one locally reasonable decision at a time.

Two Different Failure Modes

Startups and large companies both get this wrong, but for opposite reasons.

Startups are under pressure to ship customer-facing features. Security, observability, performance, operational burden become “future problems.” Sometimes that’s the right call. A startup that dies building the perfect architecture ships nothing. But the technical debt from ignored non-functionals doesn’t disappear. It accumulates, and it usually arrives all at once right when the company is trying to scale. That’s the worst possible time to deal with it.

Large companies have the opposite problem. The incentive structure rewards visible complexity. Tech leads propose ambitious architectures, staff up around them, ship something complicated, and move to the next team before the consequences mature. The engineers who inherit the system didn’t choose the design, can’t fully explain it, and can’t safely simplify it because they don’t understand what each piece is actually doing.

In both cases, the people who make the architectural decisions aren’t around to live with them. That gap between decision and consequence is the core of the problem.

The Goldilocks Principle

The approach that actually works is simpler than it sounds: start with the least complex architecture that handles the real requirements. Add complexity only when something forces you to.

Not simple for its own sake, e.g., if the domain genuinely requires distributed coordination, the design should say so. But the default should be: prove the complexity is necessary before building it. “This is how I’ve seen it done at bigger companies” and “this technology is interesting” are not justifications. Neither is designing for scale you don’t have. I’ve watched teams build for ten million users when they had ten thousand, then spend two years maintaining infrastructure that served no real requirement.

Vertical slices enforce this discipline. When you ship thin, end-to-end cuts of real functionality that a user can actually touch then you find out fast whether your design is right. The feedback loop is short. A wrong assumption costs a week, not six months. You can correct before the mistake becomes load-bearing.

AI Accelerates This Problem

With tools like Claude Code and Cursor, the implementation bottleneck is largely gone. A team using AI assistants can build a distributed system with five services in the time it used to take to build one. That’s progress if the design is right. If the incentive structure still rewards accidental complexity, AI just produces it faster.

In When Copying Kills Innovation: My Journey Through Software’s Cargo Cult Problem, I shared the cargo-cult behavior like adding components because they look sophisticated happens at higher velocity now. An AI agent given a vague prompt and no design constraints defaults to patterns common in its training data. That means microservices when a monolith would do, event buses when a direct call would do, five abstractions where two would do.

As I wrote in AI Writes Code. You Own the Design., the thinking parts like the what and why can’t be delegated to an agent. AI handles the how. Engineers who can identify essential complexity, strip the accidental kind, and hold a design together are more valuable now than before. But only if the organization’s reward structure reflects that.

How Do You Fix the Reward Structure?

I don’t have a clean answer. But here’s where the levers are.

• Reward outcomes, not artifacts. Most promotion processes credit visible artifacts: the design doc for a complex system, the heroic incident response, the fleet of services owned. The outcomes that actually matter, a system that stayed up for two years, a migration that finished in six weeks, a design that five new engineers understood on day one are harder to see and usually go uncredited. Engineering leaders have to explicitly define what good engineering looks like and measure it over time horizons long enough to see consequences.
• Make accountability follow decisions. Connect tech leads to the consequences of their architectural choices twelve to eighteen months later. Not as punishment as designs fail for unforeseeable reasons. But an engineer who never sees what their decisions cost never updates their model. Right now the feedback loop doesn’t exist for most people who make these calls.
• Credit the “no.” The engineers who prevent bad architectures from being built are the hardest to recognize. The bad system was never built, so there’s nothing to point to. If you want more of this behavior, name it explicitly and credit it explicitly. Otherwise the rational move for any ambitious engineer is to propose the complex thing and let someone else clean it up.
• Add a simplicity lens to design reviews. Most design reviews ask: will this work? Fewer ask: is this more complex than it needs to be? Formally asking “what would we remove without losing essential functionality?” changes the conversation. The burden of proof shifts to adding a component, not removing one.

The Conversation Worth Having

Brooks wrote that conceptual integrity is the most important consideration in system design. What the book doesn’t address is that most organizations are structured to undermine it like rewarding the engineers who add complexity and moving them on before they face the consequences. The engineers who hold the line against unnecessary moving parts, who ship systems that work quietly for years, who say “we don’t need this” and mean it are doing some of the hardest work in software. In most companies, they’re not the ones getting promoted.

With AI accelerating the implementation layer, the judgment required to distinguish essential from accidental complexity matters more than it ever has. If the reward structure doesn’t change to reflect that, we’ll just build the wrong things faster.

Related reading:

• How Abstraction Is Killing Software.
• When Copying Kills Innovation.
• AI Writes Code. You Own the Design.