I have seen some systems never crash, they start cleanly, swallow every error, and keep running no matter what goes wrong. In my experience, they are also the hardest systems to debug, the most dangerous to operate, and the most expensive to maintain. I worked on a similar legacy system for distributed data platform that routed events between hundreds of thousands of nodes. It had zero unhandled exceptions in production. It also had silent authentication failures, invisible data loss, and configuration divergence that took days to diagnose. This is a follow-up to my earlier posts on building an observability platform in Rust, why DRY becomes a liability, and making bad state impossible with ADTs. Those posts were about the type system but this one is about a habit of mind that no type system fixes on its own: the instinct to catch every error with some fallback or default behavior.
The culprit in that system was never a lack of error handling. It was error handling, applied in the wrong places, for the wrong reasons. I call it defensive programming as a religion: every function protects itself against every possible invalid input by inventing a fallback. Missing config? Use a default. Secret unavailable? Generate a random one. Database write failed? Log it and move on. The result is a system that looks healthy on every dashboard while quietly corrupting its own state underneath. In this post, I will walk through the patterns I found, why each one causes more damage than the crash, and what an alternative looks like instead. The whole argument rests on one idea: Every fallback creates a new source of truth, and two sources of truth always drift apart.
I. Why a Fallback Is Worse Than It Looks
It’s tempting to think of a fallback as just “hiding an error.” It’s worse than that. When a function invents a value because the real one is missing, that invented value doesn’t stay hidden, instead it becomes a fact in the system. From that moment on, the system is carrying two truths: the one that should exist but doesn’t, and the one that got made up and does. These two truths never stay in sync and when they drift apart, the failure almost never shows up where the fallback happened. Instead, it shows up somewhere else entirely, in a component with no obvious connection to the code that invented the value. For example, you’ll spend hours in the authentication layer before realizing the signing key was randomly generated at startup by a config migration function three layers away. To be clear about what I mean by “fallback,” I am not talking about:
- Validating input at system boundaries: checking what a user typed, sanitizing data from outside. Correct and necessary.
- Graceful degradation with an explicit signal: returning a typed
Degradedstate that the caller can see and react to. - Retrying transient I/O failures with backoff. Standard practice.
I’m talking about code that silently invents state when the real state is missing, and then carries on as if nothing happened. Three things make a fallback harmful:
- It hides the root cause. The missing value was the bug. The fallback makes it disappear.
- It persists the invented value. Once it’s written to disk or sent over the wire, every future operation has to succeed against a value that was never correct in the first place.
- It relocates the symptom. The failure surfaces hours later, in a different component, in a different log file, with no visible thread connecting it back to the startup code that invented the wrong value.
I am not advocating “crash on every error.” It just means: a function that requires X must fail when X is absent and it must never invent X.
II. Postel’s Law
There’s a reason smart engineers build these fallback-heavy systems: they’re following a respected principle: “be conservative in what you send, be liberal in what you accept.” that Jon Postel wrote as guidance for TCP implementations. That advice made a lot of sense in its original context. But this principle leaked out of the protocol layer and turned into a general design philosophy. Engineers started applying “be liberal in what you accept” to function signatures, config loading, and communication between services inside a system they fully control. A function that takes string | undefined and silently substitutes a random value gets called “being robust.” A startup sequence that swallows errors and keeps going gets called “being tolerant.”
Inside your own system, that assumption doesn’t hold. For example, you can fix the sender because you own the caller. But when you own the migration script that’s supposed to write the auth token and you “liberally accept” a missing auth token by inventing a random one instead, you’re not enabling interoperability with an outside party instead you’re hiding a bug in code you wrote. This misapplication creates a ratchet effect. Every “liberal” receiver makes it harder to notice problems at the source. If every function tolerates missing input, the function that’s supposed to supply that input has no pressure to get it right. People also tend to forget that Postel’s Law has a second half: “be conservative in what you send.”
The corrected version for internal systems is this: be strict with components you control, and liberal only at the boundaries where you genuinely can’t fix the sender like external APIs, user input, third-party integrations. Inside your own codebase, strictness isn’t fragility. A function that rejects invalid input tells you exactly where the bug lives.
III. Inventing Values: The Most Dangerous Pattern
This is the category that caused the most damage and I have seen countless bugs due to this anti-pattern. For example, the code generates a random value, assigns it to a security-critical field, and moves on as if that field were properly populated. The invented value becomes a durable fact somewhere and it’s always wrong.
The Archetype: Inventing a Secret
A function runs at startup and writes a signing secret into every worker group’s configuration. Workers and the coordinator use this secret to authenticate each other. If two groups end up with different values, every authentication attempt between them fails silently, showing up as delivery failures instead of auth failures.
// The bug: if authToken is absent, invent a UUID and persist it const authToken = settings.distributed?.master?.authToken; const plaintext = authToken != null && authToken.length > 0 ? authToken : randomUUID(); // <-- this line breaks authentication across the cluster
When authToken is missing from the merged settings, which is a perfectly valid state on a fresh install but this function generates a randomUUID() and writes it as the signing secret for every group it touches. Each call produces a different UUID. Meanwhile the token store writes yet another value through a completely different code path. The function reports success for every group it writes to. The symptom 401 errors between nodes shows up minutes or hours later, in worker logs, pointing investigators toward the wrong layer entirely. This is the archetype of the whole problem: if the required input is missing, invent something plausible-looking and keep going. The fix is three lines:
const authToken = settings.distributed?.master?.authToken;
if (authToken == null || authToken.length === 0) {
logger.warn('authToken absent from settings; cannot write signing secrets');
return; // do not proceed; do not invent
}
The “Disabled” Sentinel That Looks Just Like a Real Key
A secret provider has a three-source fallback chain: encrypted store, config file, random bytes. That last fallback is supposed to act as a “poison key” that never validates:
async getSecret(): Promise<string> {
// Source 1: encrypted store
const fromStore = await secretsMgr.get(KEY_ID).catch(() => null);
if (fromStore) return fromStore;
// Source 2: config file (which may itself contain a well-known default!)
const fromFile = settings.distributed?.master?.authToken;
if (fromFile) return fromFile;
// Source 3: "disable" by returning random bytes — looks perfectly valid to the caller
return random(16);
}
The caller gets back a plain string in all three cases. It has no way to tell “a real secret from the store” apart from “a well-known default from the config file” apart from “random bytes that will never work.” It signs a token with whatever string it received and sends it off. This is a type-system failure because the return type Promise<string> squashes three semantically different outcomes into one shape.
The explicit contract fixes this at the type level:
type SecretResult =
| { kind: 'valid'; value: string; source: 'store' | 'file' }
| { kind: 'unavailable'; reason: string };
async getSecret(): Promise<SecretResult> {
const fromStore = await secretsMgr.get(KEY_ID).catch((err) => {
logger.error('Secret store unavailable', { error: err });
return null;
});
if (fromStore) return { kind: 'valid', value: fromStore, source: 'store' };
const fromFile = settings.distributed?.master?.authToken;
if (fromFile) return { kind: 'valid', value: fromFile, source: 'file' };
return { kind: 'unavailable', reason: 'no secret in store or config' };
}
Now a caller that receives unavailable marks itself as degraded. It doesn’t sign tokens and surfaces a health check failure instead.
The Token Renewal That Signs With Random Bytes
The token authenticator calls getSecret(), and if that fails, it falls back to random bytes anyway:
let keyStr = await this.secretProvider.getSecret().catch(() => undefined);
if (keyStr == null) {
this.logger?.debug('Unable to generate a valid token. Disabling.');
keyStr = random(16); // this "signed" token will never be accepted by any peer
}
const token = jwt.sign(payload, keyStr);
this.cachedToken = token; // cached and reused for every future request
The log message says “Disabling,” but nothing gets disabled. The code signs a token with a random key, caches it, and hands it out to every caller for the rest of the process’s life. Workers receive the token, fail to verify it, and log a 401, with nothing to suggest the root cause of the issue.
The explicit contract: if the secret is unavailable, don’t sign anything. Set this.cachedToken = null. Let callers check for null and surface a real health degradation.
The User ID That Regenerates on Every Call
const userId = existing?.username ?? crypto.randomUUID();
When existing is unexpectedly missing, every call generates a brand-new UUID. The user becomes a ghost: every request creates a fresh identity, invisible to deduplication, audit trails, and rate limiters.
The Config Placeholder That Silently Materializes
if (authToken?.token === 'REPLACE_ME') {
authToken.token = uuidv4(); // silent replacement, no log of the value generated
}
This runs at startup and inside a database migration. If the startup write fails, the generated token is gone.
IV. Swallowed Errors
In one legacy codebase I worked had 656 instances of .catch(NOOP), an empty function attached to a promise rejection that turns any error into undefined and lets execution continue. Many of these were on cleanup paths, which is harmless. But a large number sat on critical data paths like durability, metrics transport, authentication state.
The Durability Guarantee That Wasn’t
A persistent queue exists for exactly one reason: to guarantee that events survive a destination outage. That’s the system’s durability promise.
// PQ flush — the durability guarantee await this.flushBuffer().catch(NOOP); // silently swallows disk I/O errors await this.commit().catch(NOOP); // silently swallows commit failures
If the flush fails like disk full, I/O error, permission denied, the buffered events are silently gone. The one component whose entire job is preventing data loss is itself a source of silent data loss.
The explicit contract: treat PQ flush errors as fatal to the ingest path. For example, if the flush fails, apply backpressure and pause ingest. Log at error level with the event count at risk and emit a pq_flush_failure metric. Now the operator gets to choose: fix the disk, add capacity, or consciously accept the loss.
Metrics That Vanish
void saasMetrics.sendPacket(packet).catch(NOOP);
Every metrics-send failure is silently swallowed.Dashboards go blank, and nobody knows why.
The Config Load That Treats Corruption as “Empty”
const groups = await conf.loadSystem('internal-groups').catch(() => ({}));
One line here quietly conflates two very different situations:
- “The file doesn’t exist yet”, which is normal on first boot –> return
{} - “The file is corrupt, or a parse error, or permission was denied”, which is a real configuration bug –> also return
{}
Either way, the loop over groups never runs. The operator has no way to tell “healthy, no groups configured yet” apart from “broken, groups exist but couldn’t be read.”
Startup That Succeeds Despite Total Failure
export async function syncGroupSecrets(conf: Configuration): Promise<void> {
try {
// ... write secrets to all groups ...
} catch (err) {
logger.error('failed to sync secrets', { reason: err });
// swallowed — startup continues, caller receives no signal
}
}
The function catches everything at the top and resolves successfully no matter what. The caller awaits it and gets no signal that anything went wrong. If the sync fails for every group, the coordinator still starts, workers still connect, and authentication fails across the board but startup “succeeded.”
V. Defensive Defaults
This next category is more subtle. Instead of inventing a random value, the system injects a well-known one like a default credential, a default address, which makes it impossible for downstream code to tell that anything is missing at all.
The Well-Known Default Credential
The shared authentication token has a configuration setting, and by default, the settings loader injects a well-known string whenever nothing is configured:
const { injectDefaultAuthToken = true } = options ?? {};
if (injectDefaultAuthToken && conf.distributed?.master?.authToken == null) {
conf.distributed.master.authToken = DEFAULT_AUTH_TOKEN; // 'default-secret'
}
Any caller of getSettings() receives a truthy string for authToken. Code that checks if (authToken) proceeds as if a real token exists. The check passes. Authentication moves forward, using a credential that’s sitting right there in the source cod and every deployment that forgot to override it. The default here is opt-out, not opt-in. Every new call site has to remember to disable the injection.
Workers That Connect to localhost on Config Failure
const server = this.conf.distributed.master
|| { host: 'localhost', port: 5555, authToken: DEFAULT_AUTH_TOKEN };
When a worker fails to load its coordinator address, it silently falls back to localhost:5555 with the default token. On a single-node dev box, this might accidentally work. On a production multi-node deployment, the worker ends up connecting to itself. The symptom looks like a connection timeout, not a config-load failure.
The explicit contract: if conf.distributed.master is missing, throw. A worker cannot function without a coordinator address, and there is no valid default for it. Failing here with a clear message like “coordinator address not configured; set MASTER_URL or add distributed.master to instance.yml“.
VI. Multiple Sources of Truth
This is the pattern that ties everything else together. Every fallback chain creates more than one source of truth and any source of truth that isn’t explicitly designated as the source will eventually drift from the others. In a distributed system, that drift shows up as the hardest class of bug like intermittent or state-dependent failures.
Two Functions, One Secret, Two Different Sources
The bug that inspired this post exists because two functions write the same signing secret to group configs, but read the plaintext from two different physical sources:
| Function | Reads from | Runs when |
|---|---|---|
syncAllGroupSecrets | Config file (instance.yml) | Startup for every group |
syncNewGroupSecret | Encrypted token store | Group creation for one group |
A migration populates the token store before syncAllGroupSecrets runs at boot, so the store is meant to be authoritative. But syncAllGroupSecrets predates the store’s existence and still reads straight from the config file. These two sources drift apart after a token rotation or some race condition.
The explicit contract: one source of truth. Both functions read from the store. If the store is unavailable, both fail instead of silent fallback to a secondary source.
The Three-Source Chain Is Three Sources of Truth
Source 1: Encrypted store (authoritative) ? (unavailable ? silently falls through) Source 2: Config file (may hold a stale or default value) ? (absent ? silently falls through) Source 3: Random bytes (structurally valid, semantically useless)
Each fallback quietly downgrades the security posture, and the caller gets back a plain string with no idea which source it came from.
Three branches go into the same sign() call. Only one of them should ever be allowed to reach it, which is the whole argument for making unavailable its own explicit type instead of letting all three collapse into a plain string.
The Cache Nobody Fully Trusts
The settings system keeps a cache for high-availability mode. Some callers pass skipCache: true to bypass it; others don’t, and there’s no documented rule for which is which. This gives the system two sources of truth for the same data: the cache and the disk. If the config file changes between startup and an API call, the API may serve stale data. The skipCache escape hatch is a symptom, not a fix. It means someone stopped trusting the cache’s invalidation and punched a hole through it instead of repairing the underlying mechanism.
The explicit contract: the cache invalidates on every write. Callers never need to know or care whether they’re reading from cache or disk. Remove skipCache as a public option entirely.
VII. Redundant Guards
When a function doesn’t trust its caller’s preconditions, it adds its own guard on top:
// Caller (server.ts):
if (isLeader && featureFlags.check('AUTH_TOKEN_MGMT')) {
await syncGroupSecrets(conf);
}
// Callee (syncGroupSecrets):
export async function syncGroupSecrets(conf: Configuration): Promise<void> {
if (!isFreeTier() && !isRunningInSaaS()) return; // guard 1
if (!Product.isLeader(settings.distributed?.mode)) return; // guard 2 (redundant!)
if (!featureFlags.check('AUTH_TOKEN_MGMT')) return; // guard 3 (redundant!)
// ... actual work ...
}
This function lives in a directory named leader/ and is only ever called from the leader startup path, yet it re-checks isLeader internally anyway. The feature flag gets checked at the call site and again inside the function. Three layers of defense against calling this function in the wrong context.
The checks themselves aren’t the problem. It’s what happens when they trip: nothing. The function returns quietly, and the caller gets no signal either way.
The explicit contract: a function either does its job or throws. Preconditions get asserted, not silently absorbed. If the caller already guarantees the precondition, drop the internal check. If the function really can be called from multiple contexts and some of them are invalid, throw on the invalid ones instead of quietly returning.
VIII. “Best-Effort” Writes to State That Isn’t Optional
The most seductive justification for swallowing an error is: “the primary operation already succeeded so we don’t want a secondary failure to undo it.” That reasoning is correct in isolation and catastrophic in aggregate.
The Store Upsert That Swallows Its Own Failure
export async function mirrorTokenToStore(plaintext: string): Promise<void> {
try {
await store.upsert({ id: LEGACY_TOKEN_ID, token: encrypt(plaintext) });
} catch (err) {
logger.error('failed to mirror token to store', { reason: err });
// swallowed — the caller sees success
}
}
The comment above this function explains the intent: it’s “best-effort” so that a store-side failure doesn’t roll back a config file write that already succeeded. That reasoning holds up on its own but config file is now permanently out of sync.
The explicit contract: add reconciliation. On startup, compare the store’s token to the config file’s. If they differ, update the store. Emit a token_store_divergence counter and surface it in a health check similar to reconciliation loops in Kubernetes.
IX. Partial Operations Without a Way Back
The Batch Write That Discards on Failure
const batchTxn = db.transaction((ops) => { ops.forEach(fn => fn()); });
try {
batchTxn(this.mutationCache.splice(0, maxSize)); // splice removes BEFORE success
} catch (err) {
logger.error('Batch write failed', err);
// operations already removed from cache — permanently lost
}
The comment in this code says: operations get removed from the cache regardless of whether the transaction succeeds. But the fix for “one bad operation might stall the queue” ends up being “discard the entire batch, including the good operations.” That isn’t a tradeoff instead it’s silent data loss.
The explicit contract: only remove items from the cache after a confirmed commit. Isolate the failing operation into a dead-letter queue and retry the rest of the batch without it.
The Config Reload Nobody Acknowledges
await conf.triggerReload().catch(NOOP); // worker continues with old config
After receiving a new config bundle from the coordinator, a worker triggers a reload. If that reload fails, the worker just keeps running the old config. The two sides now disagree about what the worker is actually running.
The explicit contract: report a reload failure back to the coordinator on the next heartbeat. The coordinator marks that worker as “stale config” and can retry or alert. This is how Kubernetes rolling updates work, the controller notices and either retries or halts the rollout.
The Package Install That Saves Despite Partial Failure
for (const op of ops) {
try {
switch (op.type) {
case 'install': await installPackage(op); break;
case 'uninstall': await uninstallPackage(op); break;
}
status.applied.push(op);
} catch (error) {
errors.push(error);
}
}
await this.save(packageManifest); // saves regardless of how many failed
If three out of five packages install and two fail, the manifest still gets saved with those three. Next startup tries again from this partial state but the failed packages may have left behind lock files or half-written artifacts causing conflicts.
The explicit contract: validate that every operation can succeed before running any of them (a dry-run pass). Execute atomically (ACID transactions).
X. Silent Truncation With No Backpressure
The Metrics Buffer That Silently Drops
Workers piggyback metrics onto heartbeat messages sent to the coordinator, with a hard cap of 100,000 packets. Past that cap, excess metrics are silently dropped without any counter, logs or metrics. This is a nasty failure mode specifically because absence of data is itself meaningful data and silent truncation destroys that signal’s reliability.
The explicit contract: when the buffer nears capacity, reduce granularity instead of dropping outright. When truncation does happen, include metrics_truncated: N in the heartbeat so the coordinator knows its picture is incomplete. Better, instead of piggyback metrics on heartbeats at all, give them their own transport.
The TCP Sender That Zeroes Buffers on Disconnect
// On disconnect: all in-transit events silently lost this.inTransitBufs = []; this.bufOffset = 0; this.bufferEventCount = 0; this.dropBytes += len; // only evidence: a counter increment
On a TCP disconnect, every in-transit event gets zeroed out. The only trace left behind is a dropBytes counter buried in internal metrics. Compare that to Kafka’s producer, where unacknowledged messages stay in the producer’s buffer and get retried on reconnect.
The Unbounded Queue That Becomes an OOM
protected queueBatch(): void {
this.queuedBatches.push({ eventCount, eventsSize, events: this.currentBatch });
// NO CHECK on length, size, or memory pressure
}
When the output destination is unreachable, failed batches get re-queued, and the queue grows without any bound. Memory climbs until the OOM killer steps in and terminates the process. An unbounded in-memory queue isn’t really a data structure, instead it’s a deferred OOM crash.
The explicit contract: bound the queue. Once it’s full, either apply backpressure to ingest, spill overflow to the persistent queue, or trip a circuit breaker that rejects new events with a typed error. Let the pipeline decide from there: drop, buffer to disk, or pause the source.
XI. Non-Atomic Writes
The Lease File That Can Split-Brain
await writeFile(this.leaseFile, stringify(content));
The failover lease file, the mechanism that’s supposed to guarantee only one coordinator is ever active is written directly with writeFile. On NFS, which is where this system runs in HA mode, writes aren’t atomic. A power failure mid-write leaves behind a truncated or corrupt file. The standby coordinator reads that corrupt lease, fails to parse it, and ends up in an undefined state.
The explicit contract: write to a temp file, fsync, then atomically rename it into place, with a checksum so readers can detect corruption. Every real database does this like SQLite’s WAL.
The Multi-File Config Deploy Without a Journal
Config deployment writes several YAML files in sequence like inputs.yml, outputs.yml, pipelines.yml,, etc. A crash midway through leaves the worker with a partial config and a worker that restarts after a partial deploy loads that inconsistent config.
The explicit contract: write every file to a staging directory first, verify that everything references correctly, then swap atomically like rename the directory, or flip a symlink. This is the same idea behind Docker image layers, Kubernetes ConfigMaps, and Nix store paths.
XII. Six Principles That Cover All of This
Every pattern above breaks one of six well-established principles. They’re standard practice in any system that prioritizes correctness over the appearance of uptime.
1. Fail fast at trust boundaries (Erlang’s “let it crash.”): When a precondition is violated, fail immediately and loudly. Erlang runs telecom infrastructure at 99.9999999% uptime on a philosophy of letting individual processes crash and having a supervisor restart them into known-good state.
2. Make invalid states unrepresentable: If getSecret() can return random(16) as a plain string then every caller is stuck defensively guessing whether it’s “real.” If it returns Secret | Disabled as a discriminated union instead then the type system forces every caller to handle both cases at compile time. I wrote about this pattern in “Making Bad State Impossible: A Practical Guide to ADTs and Algebraic Effects.”
3. Classify errors (transient vs. fatal): Without classification, every catch block faces an impossible choice: rethrow and break “resilience,” or swallow and hide a real bug. For example, gRPC solves this with status codes like UNAVAILABLE means retry, INVALID_ARGUMENT means don’t, INTERNAL means there’s a bug.
4. Define delivery semantics for critical state: “Fire-and-forget” is fine for debug logs. It’s not fine for persistent queue flushes, token store upserts, or config reloads. If an operation mutates state that downstream code assumes succeeded, it needs at-least-once semantics.
5. One source of truth without fallback chains for critical state: For any given piece of state, there should be exactly one authoritative source. A fallback chain isn’t graceful degradation instead it’s an implicit decision that secondary sources are acceptable substitutes for the truth. If that’s genuinely acceptable, make it explicit with TTLs, version vectors, or consistency levels.
6. Atomic state transitions: State changes should be all-or-nothing: temp file, fsync, atomic rename for files; transactions with rollback for databases; staging plus swap for multi-file deployments.
XIII. Cognitive Load
All this conditional logic and fallback behavior creates the cognitive load, e.g., when any function might silently invent a value, you can’t trust a function’s output without reading its implementation. When errors are swallowed, a successful await no longer means the operation actually succeeded. When defaults get injected into config reads, a non-null value stops meaning “configured.”
Debugging a production incident in a system like this means reading every function in the call chain, understanding every fallback along the way, and reconstructing which code path actually ran. crash tells you exactly where and when an invariant broke. New engineers ask why workers sometimes fail to authenticate after a token rotation, and the honest answer involves reading six functions across four files, understanding a three-source fallback chain. Compare that to: “the store upsert threw on failure, the rotation API returned a 500, the operator re-ran it, it worked.
XIV. Conclusion
Defensive programming isn’t inherently wrong, and neither is Postel’s Law like at the boundary it was designed for. Validating input at system boundaries, handling I/O errors gracefully, protecting against malformed external data are correct applications of defensive thinking. The problem shows up when the same techniques get applied to internal code, inside a system where you control both ends of every interface. The alternative, in short:
- Throw Error when a required state that’s missing: The function refuses to proceed and the caller finds out immediately.
- Explicit return when an optional state that’s missing:
null,undefined,Option<T>, a discriminated union. - Transient failures = retry with backoff: Never
.catch(NOOP). - One source of truth per piece of state: Not a fallback chain that quietly degrades. Not a cache with no real invalidation.
- Bounded queues with backpressure: Not unbounded buffers waiting to OOM.
- Atomic state transitions: Not multi-step operations that can half-finish.
- Reconciliation loops for distributed state: Not one-shot “best-effort” writes that quietly drift apart.
This mud didn’t accumulate overnight, and it won’t disappear overnight either. But every fallback you remove, every error you refuse to swallow will makes the next incident roughly ten times faster to diagnose.