Fire in the Lobby. Anyone Copy?

Your users have been working around the same broken widget for four months. They refresh it twice, open a second tab, copy the number out by hand, and tell the new hire, “yeah, that part never works right.”

Nobody got paged.

The API was up. The queues were moving. Error rates looked normal. Every system engineering had decided to be afraid of was reporting green.

That’s how you end up with a trash fire in the lobby while every smoke detector in the building is wired to the server room.

Sleep-protection architecture

In every shop I’ve worked in, the same thing is true: developers quietly decide end-user failure tolerances, and nobody else is ever consulted.

I have an admin page in production right now that only works on the second load. The first render fires a job I hung on page load, because I had a few hours to ship it, and I never went back to move it behind a proper initiator. It gets used maybe twice a year. Everyone who touches it knows to refresh. That is not a bug. It’s a decision I made about someone else’s time, and never wrote down anywhere they could see it.

Here’s how it happens more generally. The last thing any dev wants is to be woken up in the middle of the night. So the alerting stack gets pointed at the things the developer is worried about. Sentry, uptime monitors, all of it. And they’re usually worried for one reason: engineering weaknesses they already know exist. The pager architecture is sleep-protection architecture. It maps the developer’s anxieties, not the user’s workflow.

Meanwhile, the widget your paying users look at every single day is more important than your email-queue throughput or your landing-page load time. But the widget has no alert on it, because the widget never scared anyone on the engineering team.

Organizations instrument the failures they already understand. The failures that cost the most usually happen outside that model.

Acquisition latency is not workflow latency

“If your site isn’t instant, you lose customers.” The studies behind this are real, and I’m not disputing them. I’d point out what they measured: conversion on acquisition surfaces. Search results, product pages, checkout. That finding then gets carried, unmodified, into contexts nobody measured.

Of course load time affects conversion once a site becomes meaningfully slow. But there’s a difference between removing an obvious delay and endlessly shaving milliseconds off a page that already loads within the normal variance of a modern connection.

Users don’t experience your landing page in a laboratory. They experience it through cellular signal, crowded Wi-Fi, an aging phone, a busy browser, and an internet full of inconsistent response times. A modest delay gets attributed to the environment long before anyone interprets it as evidence of bad engineering. Nobody concludes your company is incompetent because a page took 1.4 seconds instead of 1.1.

Inside the product, the attribution flips. A visitor loads your marketing page once. A paying user opens the same widget two hundred times a day. They know what normal feels like. They notice when it hesitates. A small delay, repeated across an essential workflow, is friction imposed directly on their work, and unlike the landing page, they know exactly whose fault it is.

That’s the distinction the headline numbers erase. Both are measured in milliseconds; they are not the same good. At some point the landing page no longer has a perceptible problem to solve. Meanwhile, the widget your customers live in can stay slow for years, because nobody ever attached a conversion study to it.

Failure enters at the boundaries

Your own code, queues, and pipelines can be tested constantly, at scales ordinary use will never reach. Their inputs can be controlled. Their state can be inspected. Their outputs can be compared against defined expectations.

You can test a user interface. You can observe a person using it, run experiments, and interview them afterward. But you cannot test the user in the same sense. You cannot inspect the state that produced their behavior. You cannot know that they understood the task, brought the same context they would bring to real work, behaved naturally while being watched, or can truthfully and accurately explain afterward why they did what they did. Their explanation is not the internal state. It is one more output from the black box.

That black box arrives carrying a mental model assembled from documentation, old habits, workplace folklore, prior software, onboarding, and promises made elsewhere in the business. A sales process optimized around overcoming objections rather than diagnosing fit will inject expectations no product team ever agreed to support. The promise made on that call is external data feeding the user’s mental model, and it is exactly as unreliable as any third-party API.

Then that person is inside your app, uploading their email for reasons nobody can reconstruct, angry they can’t do something that never existed, and no dashboard anywhere is watching for it.

I’ve told plenty of prospects, “you don’t need my thing. You need X instead.” I’d rather lose the deal than spend two years being blamed for an outcome I was never built to produce. A yes only means something when a no was genuinely available.

Where I’d point the pager instead

Point it at the user working harder than they should have to:

• Repeated attention — the same record opened over and over
• Retries — the same action attempted more than once in a row
• Abandoned workflows — sequences entered and left incomplete
• Improvised workarounds — export-edit-reimport, second tabs, manual recalculation

These signals don’t tell you what the user is thinking. The black box doesn’t open. They tell you where product reality and customer expectation have separated, and where a human needs to find out why.

At the start, you assume. You have to. The user arrives with a real problem and some belief that your product can solve it. You built the system knowing what it does and how it does it. When a user repeatedly tries to make it do something else, the mismatch is itself a signal, and one of two things is probably true.

The first is that the product is missing a capability the user is naturally reaching for. Their repeated attempts are demand showing up before it has a feature request attached.

The second, and more common, is that the user has attached your product to the wrong problem. The need may be real, but their understanding of it, or the expectation created by sales, onboarding, documentation, or prior software, has led them into a workflow your system was never meant to support.

In either case, somebody should intervene. The job of customer success is not to teach the user which buttons to press until the ticket disappears. It is to back up far enough to understand the problem the customer believes they are solving. Then the company can make an honest decision: build the missing capability, show them the workflow that already exists, or help them move to the product they actually need.

Nobody likes the third answer, but sometimes it’s the correct one, and occasionally the best one for you. A customer fundamentally mismatched to the product is not a retention victory. Keeping them means preserving their frustration, consuming support capacity, and eventually being blamed for an outcome you were never built to produce.

One planning note that falls out of this: in CRUD-heavy or data-heavy systems, the workflow that matters will surface as a clear outlier. Capacity-plan around credible high-percentile workflows and the power users whose behavior previews tomorrow’s load. Not today’s average.

What this looks like in practice

Concretely: on the timing platform I build, a pipeline break produces two different experiences on purpose.

Race officials are alerted immediately and loudly, because they can act. Hold the call, fall back to manual, tell the tower. Spectators see the leaderboard slow down, with a message saying the data is stuck in the pipes, not lost. Same fault, two audiences, two designs.

The naive version of that is one generic error state for everyone. The common version is worse: a spinner that keeps spinning and tells the user nothing, which is a product quietly lying to the person who trusted it.

That decision was made at design time, before a single person had used the system. It could not have been retrofitted cheaply. Adding it later means threading a new state through every layer that touches the stream. It’s a seam, not a feature, and seams are nearly free before you build and nearly impossible afterward.

Now the disclosure, because a piece about honest instrumentation should be honestly instrumented itself: the engineering half of this is proven. One system, validated at 10,000 concurrent connections, then operated across a three-day race weekend without incident. Roughly 24 hours of live running. The user-side half, pointing the pager at retries and workarounds, has mostly lost the roadmap fight in every system I didn’t own, which, you’ll notice, is exactly the failure this piece describes. Read that half as design reasoning rather than a battle report. I’m building it into my own platform next; I’ll report back.

The point

Code is the part you can subject to repeatable measurement. Test it brutally, at scales ordinary use should never reach, so that routine engineering failure consumes as little human attention as possible. Then spend that attention on the component you cannot exhaustively model: the person carrying a mental model you didn’t write, doing things you didn’t foresee.

Things will break at the boundaries. When they do, route the alarm correctly. Fail loud at the operator, who can act on it. Fail soft at the audience, who can’t. Users were never the right people to find out first. For most of the internet’s history, they were just the only monitoring anyone actually deployed.