Scaling Platform Services and Improving Resilience

Executive summary

How I lead performance and resilience work that spans several services and teams: observe before acting, measure where time really goes, fix the contention, isolate workloads and model capacity so the next limit is anticipated rather than discovered.

Context and constraints

The slowest parts of a data-intensive system are rarely where people expect. As data and usage grow, batch processing, reporting and query-heavy features begin competing for the same resources, and tight coupling turns a local slowdown into a system-wide one.

The temptation is to reach for the optimisation you already know. The discipline is to find where time is really being spent, fix what actually matters, and leave behind a shape that keeps scaling rather than a set of one-off patches. When the work spans several services and teams, it is as much a coordination problem as a technical one.

Constraints

• The platform stays live. Improvements ship incrementally, behind evidence, without a freeze on feature work.
• Shared ownership. Bottlenecks cross service and team boundaries, so priorities have to be agreed, not imposed.
• Limited capacity. Performance work competes with feature work, so it has to be ranked by user and business impact, not by technical interest.

My role

The method I use when leading cross-team performance and resilience work. The specific systems, findings and results from that work are confidential and are not described here.

Technical challenge

Performance problems in a shared platform are rarely one bug. They are interacting workloads: a batch job, a reporting query and an interactive request contending for the same table, lock or connection pool. Fixing the loudest symptom usually just moves the bottleneck somewhere quieter.

Architecture and decisions

1. 01Observe. Start from symptoms users and operators can see, not from a favourite theory.
2. 02Measure. Profiling, distributed tracing and query plans establish where time and resources actually go.
3. 03Identify contention. Find the shared resource that different workloads are fighting over: a table, a connection pool, a lock, a queue.
4. 04Optimise. Fix the fundamentals on the hottest paths: query shape, indexing, batching and pagination.
5. 05Isolate workloads. Move heavy work off the request path, and give query-heavy features data shaped for the question being asked.
6. 06Validate. Compare before and after under realistic load; keep the evidence with the change.
7. 07Model capacity. Turn what you learned into a model of how load grows, so the next bottleneck is anticipated.

The order matters more than any single technique. Optimising before measuring tends to fix the wrong thing; isolating workloads before optimising tends to move a problem rather than solve it.

When the work spans teams, coordination becomes part of the method: one shared, ranked list of bottlenecks, a named owner for each, and before-and-after evidence with every change so progress is visible to the people who prioritise it.

Alternatives considered

• Scale the infrastructure. Fastest to try and sometimes right, but it hides contention, and cost grows with load.
• Cache aggressively. Very effective on read paths, but it adds invalidation risk and can mask a slow uncached path that still matters.
• Rewrite the slow service. Occasionally necessary, rarely the first move: most gains come from query shape, indexing and moving heavy work off the request path.

Trade-offs

• Read models trade freshness for speed. Be explicit about how stale each view is allowed to be.
• Caching hides problems as well as solving them. Measure the uncached path too.
• Asynchronous work needs a user experience. Progress, resumability and clear failure states become part of the design.

Execution

Running performance work across several teams:

• One ranked list of bottlenecks, agreed with the teams that own the services, ordered by user and business impact.
• A named owner for each item, so nothing is everybody's and nobody's.
• Small changes, shipped incrementally, each with before-and-after measurements attached.
• Short, regular updates for the people prioritising feature work, so they can see what the trade is buying.

Results

The specific findings and numbers from my own performance work are confidential and aren't published here.

Followed consistently, the loop leaves behind hot paths that are designed rather than accidental, heavy work separated from interactive work, and a shared, evidence-based picture of where the next limits are. That makes future scaling a planning exercise rather than an incident.

Reflection

• Rank by impact, not interest. The most technically satisfying fix is rarely the most valuable one.
• Evidence is the currency. Before-and-after measurements make prioritisation conversations short.
• Resilience is part of performance. A service that degrades gracefully under load is worth more than one that's fast until it falls over.

Technologies

• Query profiling
• Indexing
• Keyset pagination
• Read models
• Asynchronous processing
• Capacity modelling

Deep dive

Reference pattern: isolating heavy work

A common shape for separating interactive traffic from heavy work: writes go to the primary store, query-heavy views read from a purpose-built read model, and batch jobs are queued and processed in resumable chunks.

Example: keyset pagination for large result sets

Offset pagination gets slower with every page because the database still has to walk past every skipped row. Keyset pagination keeps each page cheap, which matters when reading large datasets in chunks.

-- Offset: cost grows with every page
SELECT id, created_at, payload
FROM records
ORDER BY created_at, id
LIMIT 1000 OFFSET 250000;

-- Keyset: each page is an index range scan
SELECT id, created_at, payload
FROM records
WHERE (created_at, id) > (:last_created_at, :last_id)
ORDER BY created_at, id
LIMIT 1000;