Building Reliable Event-Driven Financial Systems

Executive summary

Financial workflows have to be correct under retries, redelivery and partial failure, and every item has to be traceable. These are the event-driven patterns I rely on, drawn from a decade of Java, Kafka and PostgreSQL systems in financial services.

Context and constraints

High-volume processing is rarely hard because of the happy path. It's hard because data arrives in bursts, downstream dependencies slow down, messages are delivered more than once, and a single stuck step can hold up everything queued behind it.

In financial systems the bar is higher: a duplicated instruction or a silently dropped record is a client-facing incident, and someone will eventually ask exactly what happened to a single item. The design goal is to accept work quickly, process it correctly, and keep every item traceable from arrival to completion, while keeping the system understandable for the engineers who operate it.

Constraints

• Correctness over latency. A fast wrong answer is worse than a slow right one.
• At-least-once delivery. The broker will redeliver, so duplicates are normal, not exceptional.
• Auditability. Every item must be traceable end to end, after the fact.
• Sensitive data. Financial and personal data must be protected at rest without making it impossible to operate on.
• Operable by the team. On-call engineers need to be able to see, pause and replay work without bespoke scripts.

My role

Patterns I've applied as a backend engineer and technical owner on Java, Kafka and PostgreSQL services in financial services. The architecture is illustrative and does not describe a specific system.

Technical challenge

The hard part is correctness under failure, not throughput. A message can be delivered twice, arrive out of step with a related message, or fail halfway through a multi-step update. Each of those is rare for any single message and certain at volume, so the design has to make them harmless rather than hope they don't happen.

Architecture and decisions

• Decouple acceptance from processing. The entry point validates and publishes an event, so producers are never blocked by slow consumers.
• Design for at-least-once delivery. Every consumer is idempotent, keyed on a stable identifier, so retries and replays are always safe.
• Partition for parallelism, not chaos. Partitioning by entity key lets throughput scale horizontally while preserving ordering where it matters.
• Make failure a normal state. Bounded retries, dead-letter handling and replay tooling make failures visible and recoverable rather than silent.
• Observe the flow, not just the services. A correlation identifier travels with each message, so a single item can be traced across every hop.

In this shape the entry point does as little as possible: validate, assign identity, publish. Everything expensive happens asynchronously, where it can be retried, scaled and observed independently.

Sensitive data needs the same deliberate design: decide early which fields must be protected at rest and which of them still need to be queried, because retrofitting that later touches every read path.

Alternatives considered

• Synchronous request chains. Simpler to reason about, but they couple every caller to the slowest dependency and turn a downstream outage into an upstream one.
• Relying on exactly-once delivery. Attractive on paper, but the guarantee stops at the broker; the database and external calls still see duplicates. Idempotent consumers are simpler and hold end to end.
• A single global order. Easy to explain, but it caps throughput at one consumer. Ordering per entity keeps the guarantee where the business actually needs it.

Trade-offs

• Eventual consistency. Decoupling acceptance from processing means clients need a way to learn when work has completed, and operators need a view of in-flight work.
• Per-key ordering only. Partitioning buys throughput by giving up global ordering; choose the key that carries the real business ordering.
• Protecting data constrains querying. Every protection scheme limits which queries remain possible, so decide which queries matter before choosing one.

Execution

How I introduce this into a live system:

• One flow at a time, behind the existing interface, so callers don't change.
• Run old and new paths side by side and compare results before switching traffic.
• Ship replay and dead-letter tooling in the first release, not as a follow-up: recovery has to exist before it's needed.
• Agree with the people on call what ‘stuck’ looks like, and alert on it.

Results

This is an engineering pattern rather than a write-up of a specific system, so there are no figures here.

Designed this way, a processing path scales horizontally by adding partitions and workers, recovers from failure by replaying events rather than repairing data by hand, and can be reasoned about in production by the people who run it.

Reflection

• Ordering is a per-key promise, not a global one. Decide which entity needs ordering, and partition by it.
• Retries need a ceiling. Unbounded retries turn one bad message into a stalled partition.
• Replay is a feature. If you can't safely replay, you can't safely recover.

Technologies

• Event-driven architecture
• Kafka
• Java
• Spring Boot
• PostgreSQL
• Idempotency
• Dead-letter handling
• Distributed tracing

Deep dive

Pattern: the idempotent consumer

The single most important property in an at-least-once system is that processing the same message twice has the same effect as processing it once. A simplified version of the pattern:

@Transactional
public void handle(InboundEvent event) {
    // Claim the event inside the same transaction as the side effects.
    boolean firstTime = processedEvents.tryInsert(event.id());
    if (!firstTime) {
        metrics.increment("events.duplicate");
        return; // Already applied: safe to acknowledge.
    }

    var record = mapper.toRecord(event);
    repository.upsert(record);
}

Because the deduplication record and the side effects commit together, a crash at any point leaves either both or neither, and redelivery does the right thing.

Pattern: bounded retries, then dead letters

Transient failures deserve a retry; poison messages deserve to be moved aside so they stop blocking the work behind them. Separating the two keeps a partition moving:

void process(Message msg) {
    try {
        handler.handle(msg);
    } catch (TransientException e) {
        if (msg.attempts() < MAX_ATTEMPTS) {
            retries.schedule(msg.withAttempt(), backoff(msg.attempts()));
        } else {
            deadLetters.publish(msg, e); // Inspect, fix, replay.
        }
    } catch (PermanentException e) {
        deadLetters.publish(msg, e);     // Retrying won't help.
    }
}