From Prototype to Production: What Breaks First

What actually changes when a prototype meets real users

A prototype answers one question: can this be built at all. It runs on a laptop, for one user who knows which buttons not to press, against a few dozen hand-typed rows. Taking a prototype to production invalidates all three conditions in a week.

The failures then arrive in a reliable order, sequenced by how much traffic and data each needs to surface. Authorisation breaks on day one: the first real user is the first person who is not you. Query performance breaks around month two, when a table passes the point at which a sequential scan stops being free. Concurrency bugs break the moment two people click the same button inside the same second. None of this indicts the prototype — one that handled it all would have cost four times as much and answered the same question.

Every item here has a known cause and fix. What makes them dangerous is that a prototype gives no signal any of them exist — it passes every test you thought to run, because you wrote them and you were the only user.

What breaks first when taking a prototype to production: identity and access

Prototype auth takes one of three shapes: a hard-coded user, a JWT signed with a secret that also lives in the repository, or a token in localStorage with no expiry. When a token cannot expire or be revoked, the only way to end a session is rotating the signing secret, which logs everyone out — so nobody does it, and the offboarded employee keeps access. A token in localStorage is readable by any script on the page, turning one compromised dependency into account takeover.

The dangerous half is the part prototypes almost never have. A prototype checks whether someone is logged in; production must check whether this actor may touch this record. When it does not, a customer who edits an integer in a URL sees another customer’s invoice. Broken object-level authorisation sits at the top of the OWASP API Security Top 10 for good reason: it is discoverable by incrementing a number, and unlike a slow query it gives no warning first.

The fix is structural rather than a bigger library: every read and write passes through one check taking both actor and resource, close to the data rather than scattered across route handlers. Row-level security or a scoped query builder works; remembering to add the tenant filter to each new endpoint does not.

Why the database is fine at 50 rows and dead at 500,000

Postgres will scan a fifty-row table faster than it would use an index on it, and the planner is right to do so, so a prototype gives you no feedback about missing indexes. Once the working set outgrows cache the application does not degrade gracefully: slow queries hold connections, the pool empties, and unrelated requests start timing out.

Two problems get conflated. One is missing indexes on the columns you filter, sort and join by — foreign keys are the usual omission, since Postgres indexes the primary key automatically but not the referencing side. The other is the N+1, where an ORM issues one query for a list and one more per row. Over a local socket, 201 queries at a fraction of a millisecond each are invisible; across an availability zone boundary at a few milliseconds each, the same page takes over a second to render.

EXPLAIN ANALYZE
SELECT * FROM invoices
WHERE organisation_id = $1
ORDER BY created_at DESC
LIMIT 50;
--  Seq Scan on invoices  ->  Filter: (organisation_id = $1)
--  Rows Removed by Filter: 498   -- cheap now, linear later

-- CONCURRENTLY: an index build, not an outage.
CREATE INDEX CONCURRENTLY invoices_org_created_idx
  ON invoices (organisation_id, created_at DESC);

• Set log_min_duration_statement to a threshold you would be embarrassed by, so slow queries announce themselves rather than arriving as complaints.
• Enable pg_stat_statements and read it fortnightly. It ranks by total time, which is the number that matters, not worst single execution.
• Assert query counts in tests for busy endpoints. Almost nothing else catches an N+1 reintroduced by a refactor.
• Cap page size server-side, and prefer keyset pagination to OFFSET on long lists — OFFSET 10000 makes the database walk ten thousand rows to discard them.

The failure order, with blast radius and fix

In the order they surface; blast radius decides the sequencing.

What the open internet does to an unprotected endpoint

Obscurity stopped being a control years ago: every publicly trusted TLS certificate is written to public, searchable Certificate Transparency logs, so a hostname is discoverable minutes after you issue its certificate. Scanners follow, and nothing needs to link to your endpoint for it to receive traffic.

Rate limiting pays for itself fastest, because unauthenticated traffic now converts directly into money. Any route triggering a metered call on your behalf — a model inference, an SMS, a geocoding lookup — is a route where someone else spends your budget in a loop. Limit per identity as well as per IP: an IP limit falls to a proxy pool, an identity limit to repeated signups.

File uploads combine three failure modes. The size limit belongs at the reverse proxy, so a 4 GB body is rejected before your process buffers it. Content-Type and extension are client-supplied, so validate the bytes and re-encode images. And serving user content from your own domain means an uploaded HTML file executes with your cookies in scope; a separate domain or a signed object-storage URL removes that.

Why you hear about the outage from a customer, not a dashboard

A prototype’s error handling is a person watching a terminal. Production removes the person: the exception happens in a background worker at three in the morning, and the only trace is a customer who abandoned the signup form. Teams debate observability tooling for a fortnight and ship none of it, when the useful minimum is a day.

• Exception capture carrying release, tenant and request, alerting on the first occurrence of a new error signature rather than on volume — volume alerts miss the bug hitting only the twelve customers who matter.
• Structured logs with a request identifier propagated through every service and queued job, so one identifier reconstructs the path.
• Four alerts, not forty: error rate, p95 latency, queue depth or replication lag, and a synthetic check of the journey that produces revenue.
• An uptime check running outside your own infrastructure — a health check hosted on the thing that is down reports nothing at all.
• Point-in-time recovery plus one rehearsed, timed restore.

Concurrency, time zones and money: the quiet correctness bugs

Race conditions never appear in logs as errors, because nothing errored. Check-then-act is the pattern: the handler queries for an existing booking, finds none, inserts one. Two requests milliseconds apart both find nothing and both insert. Application code cannot fix this alone, since the requests may be in different processes on different machines. The invariant must live where all writers converge.

-- The constraint is the control; application logic is convenience.
ALTER TABLE bookings
  ADD CONSTRAINT bookings_slot_unique UNIQUE (resource_id, starts_at);

INSERT INTO bookings (resource_id, starts_at, customer_id)
VALUES ($1, $2, $3)
ON CONFLICT (resource_id, starts_at) DO NOTHING
RETURNING id;
-- Zero rows means somebody else won the race.

For anything reaching outside your database — charging a card, sending a message — the control is a caller-generated idempotency key, which is why Stripe accepts an Idempotency-Key header on writes. A client retrying after a timeout otherwise cannot know whether the first attempt succeeded, and a retry storm becomes a billing incident.

Time and money are where prototypes make the decisions that are expensive to reverse. Store timestamps as timestamptz in UTC and keep the IANA zone when the zone is part of the intent: a recurring nine o’clock meeting is a local time plus a zone, not an instant, and collapsing it to UTC moves it by an hour twice a year. Money as a float is worse, since IEEE 754 cannot represent 0.1 exactly and errors compound across sums. Store integer minor units with an ISO 4217 code, and note that not every currency has two decimals: JPY has none, KWD three.

How to sequence taking a prototype to production without stalling the roadmap

Hardening gets deferred because it arrives as one undifferentiated block called technical debt, which competes badly against features. Split it by blast radius and most becomes schedulable.

1. Before the first external user: the authorisation check, secrets out of the repository and rotated, backups with one rehearsed restore, exception tracking, body size limits at the edge. Here the downside is unrecoverable rather than embarrassing.
2. In the first fortnight: rate limiting on public and metered routes, idempotency keys on external writes, slow query logging, request identifiers in logs, migrations under version control.
3. In the first ninety days: an index audit driven by pg_stat_statements rather than intuition, N+1 removal with query-count assertions, and the money and time representations corrected while the tables are still small enough to backfill in one maintenance window.

The last two get more expensive every week they wait, for a reason worth stating: money and time representation are not code changes, they are data migrations. Altering a column takes an afternoon; backfilling four million rows whose original currency was never recorded takes a project, and some of that information may no longer exist.

For most prototypes the first two groups are a fortnight of focused work, not a quarter. If you have something customers are about to touch and want a second pair of eyes over the list first, a hardening pass is straightforward to scope — we are happy to talk through what that looks like on your stack.