The bug that never shows up when you’re testing it

Saving something to the database, then telling everyone else about it, is the most ordinary thing a service does — and also one of the least atomic operations you can perform. Two completely separate systems, and nothing built-in that makes “both succeed” or “both fail” a single step.

Here’s the version that works — right up until the moment it doesn’t:

public void createOrder(Order order) {
    orderRepository.save(order);                       // 1. write to the database
    kafkaProducer.send(new ProducerRecord<>(            // 2. tell Kafka about it
        "orders", order.getId().toString(), toJson(order)));
}

Run this on your laptop, it works every time. Run it in staging, still fine. Then it goes to production, and one day the pod gets rescheduled, or the network to Kafka hiccups for two seconds, right between line 1 and line 2. The event never goes out. Nothing crashes. Nothing logs an error.

I ran into a version of this while wiring up my order-processing system, right after writing about how Kafka handles — and sometimes doesn’t handle — message order. If you haven’t read that one, it’s worth a look before this post, because everything below assumes you already know Kafka only guarantees order per partition, and that a consumer has to be idempotent to handle a message safely more than once. This post picks up right where that one left off: not “will my messages arrive in order,” but “will my messages arrive at all.”

The root problem here is that saving to a database and publishing to Kafka are two completely separate systems. There’s no built-in way to make “both succeed” or “both fail” a single atomic thing. And once you actually sit with that, the usual instinctive fixes stop looking like fixes.

Here’s the shortest version of the answer before the full walk-through: the outbox pattern writes the event into the same database transaction as the business data, then hands the actual publishing to a background process that can retry forever without blocking anyone. Over the next few minutes — why every obvious fix fails, how a plain table sidesteps all of it, the two ways rows actually get to Kafka, and the one tradeoff almost every writeup leaves out: what to do about the duplicates it can’t avoid.

Why the obvious fixes don’t actually work

The first thing most people try is flipping the order: publish to Kafka first, then save to the database. That just moves the failure to the other side — now you can publish an event for an order that never actually got saved, which is arguably worse, because every downstream service now believes something exists that doesn’t.

The second thing people try is wrapping the Kafka call in a retry loop. That genuinely helps with transient failures — a broker hiccup, a brief timeout — but it does nothing for the case where the whole service crashes between the database write and the Kafka call. There’s no code left running at that point to retry anything.

The third thing, and the one that sounds the most “correct” on paper, is a distributed transaction spanning both systems — two-phase commit. Kafka simply doesn’t support being a participant in a 2PC transaction, so this isn’t even on the table. And even in ecosystems where XA transactions technically exist across other systems, most teams avoid them anyway, because they’re slow, they hold locks longer than you’d like, and they turn a subtle bug into a subtle performance problem instead.

None of these fixes actually solve the underlying issue, because they’re all still trying to make two independent systems agree with each other in the moment. The outbox pattern takes a completely different approach: it doesn’t try to make the database and Kafka agree. It makes the event part of the database write itself, and moves the actual “talk to Kafka” step somewhere else entirely, where it can be retried safely, for as long as it takes, without anyone waiting on it.

The actual idea: write the event where you know it’s safe

Instead of calling Kafka directly inside your request, you write the event as a row into a plain table — the “outbox” — inside the exact same database, in the exact same transaction as your real business data.

CREATE TABLE outbox (
    id            UUID PRIMARY KEY,
    aggregate_id  VARCHAR(255)   NOT NULL,   -- e.g. the order_id
    event_type    VARCHAR(255)   NOT NULL,   -- e.g. "OrderCreated"
    payload       JSONB          NOT NULL,
    created_at    TIMESTAMP      NOT NULL DEFAULT now(),
    published     BOOLEAN        NOT NULL DEFAULT false
);
@Transactional
public void createOrder(Order order) {
    orderRepository.save(order);

    OutboxEvent event = new OutboxEvent(
        UUID.randomUUID(),
        order.getId().toString(),
        "OrderCreated",
        toJson(order)
    );
    outboxRepository.save(event);   // same transaction as the order save above
}

Because it’s one database and one transaction, this part is genuinely, boringly atomic — the same guarantee your database has always given you for every other write you’ve ever made. Either the order row and the outbox row both commit, or neither does. There’s no in-between moment where one exists without the other, and no crash window that can leave you with half of it. You’re not building a new kind of reliability here. You’re just reusing the one your database already gives you for free, instead of trying to invent a new one across two systems that were never designed to share a transaction.

Actually getting that row to Kafka becomes a separate job entirely, one that runs after the fact and doesn’t have to happen inside the same request at all.

Two ways to move rows out of the outbox

The simple one: a poller. A background job wakes up on a timer, looks for rows that haven’t been published yet, and sends them.

@Scheduled(fixedDelay = 500)
public void publishPendingEvents() {
    List<OutboxEvent> events = outboxRepository.findByPublishedFalseOrderByCreatedAt();

    for (OutboxEvent event : events) {
        try {
            ProducerRecord<String, String> record = new ProducerRecord<>(
                "orders", event.getAggregateId(), event.getPayload());
            record.headers().add(
                "event_id", event.getId().toString().getBytes(StandardCharsets.UTF_8));

            kafkaProducer.send(record).get();  // wait for the broker to actually acknowledge it

            event.setPublished(true);
            outboxRepository.save(event);
        } catch (Exception e) {
            log.warn("Publish failed for event {}, will retry next cycle", event.getId(), e);
            // leave published = false so the next poll picks it up again
        }
    }
}

This is maybe thirty lines of code, it works with whatever database you’re already using, and it’s easy to reason about at 2am when something’s gone wrong. The tradeoff is that events sit for up to one poll interval before they go out, and you’re adding a repeated query to your database. For most systems, especially anything I’d build as a side project or early-stage product, that tradeoff is completely fine.

The scaled-up one: Change Data Capture. A tool like Debezium reads your database’s write-ahead log directly — the same internal log the database uses for its own crash recovery — and streams new outbox rows to Kafka the moment they’re written, with no polling and no extra query load on your app. This is the real production answer once you’re operating at a scale where poll latency or database load actually matters. But it’s genuinely another piece of infrastructure to run, monitor, and understand when it breaks, not just a method with a @Scheduled annotation on it. I’d start with the poller, measure whether it’s actually a problem, and only reach for CDC once you have a number that says it is.

Who actually runs this in production

None of this is a textbook pattern that looks good on a whiteboard but never leaves it. The outbox is a documented, standardized part of how distributed systems are expected to work. A few places you can read it straight from the source:

  • Chris Richardson’s microservices.io catalogs the “Transactional Outbox” as a first-class pattern in his reference architecture, alongside the saga and CQRS patterns — and his polling publisher pattern is exactly the poller in this post, codified.
  • AWS Prescriptive Guidance documents the pattern with a worked implementation (Lambda + RDS + SQS), and its event-driven reference architecture makes an outbox table part of the recommended backbone — including the same “make the consuming service idempotent” caveat this post covers in the next section.
  • Debezium’s docs ship the CDC version of exactly this pattern, and the outbox example pairs an order-service with a shipment-service that deduplicates by “comparing incoming event ids” — the same event-id dedup we used above.

And the scale is worth knowing: the Apache Kafka project itself reports that more than 80% of the Fortune 100 use Kafka. Every one of those deployments has to answer this same write-then-publish question — there’s no version of event-driven architecture that gets a free pass on the gap between “saved in the database” and “accepted by the broker”.

To put it in numbers closer to home: with the poller above running every 500ms, the worst case for an event sitting in the outbox is about half a second plus one Kafka round-trip before it’s out. That delay is the entire price of the guarantee that the event eventually arrives, no matter how many times the service restarts in between.

The part almost every writeup on this skips

Look closely at the poller code above. It sends the message to Kafka, and only after that succeeds does it mark the row as published. So what happens if the send genuinely succeeds, but the app crashes a split second later, before the published = true write makes it to the database?

The row is still sitting there marked unpublished. The next time the poller runs, it sends that same event again. Congratulations, Kafka now has the same event twice.

This is not a flaw you can code your way out of — it’s a fundamental tradeoff, and once you see it, you realize there are only two options. You could flip it: mark the row published first, then send. But now a crash between those two steps means the event is marked done and never actually goes out at all — silently missing, which is exactly the bug this whole pattern exists to prevent. Or you keep it the way it is above — send first, then mark — and accept that very occasionally, under exactly the wrong timing of a crash, the same event might get sent twice.

Sending twice is a far safer failure than never sending at all, which is why every real outbox implementation I’ve seen picks that order. What this actually gives you is called at-least-once delivery, not exactly-once, and that’s not a limitation of this particular implementation — it’s the honest ceiling of what this pattern can promise. The only way to live with at-least-once safely is to make sure whatever reads these events downstream can handle getting the same one twice without anything bad happening. That’s the idempotent consumer pattern again, the same one from the Kafka ordering post — check whether you’ve already handled a given event before you act on it.

public void handleOrderCreated(ConsumerRecord<String, String> record, Connection db) throws SQLException {
    String eventId = new String(
        record.headers().lastHeader("event_id").value(), StandardCharsets.UTF_8);

    db.setAutoCommit(false);
    try {
        if (alreadyProcessed(db, eventId)) {
            db.rollback();
            return;
        }
        applyEvent(db, record.value());
        markProcessed(db, eventId);
        db.commit();
    } catch (SQLException e) {
        db.rollback();
        throw e;
    }
}

One detail in that snippet deserves its own warning: dedupe on the event’s unique id, never on the Kafka key. The key is what you’re ordering by — here the order id — and every event for that order (OrderCreated, then OrderPaid, then OrderShipped) shares it. If the key were the idempotency check, the second event would be silently discarded as “already seen” the moment the first one was processed. The event_id header is unique per outbox row, which is what makes the check safe.

I want to be clear about something here, because it’s easy to read the outbox pattern as “the fix” on its own — it isn’t, by itself. The outbox guarantees the event definitely leaves your service, eventually, no matter what crashes in between. The idempotent consumer guarantees that if it leaves twice, nothing bad happens on the receiving end. You genuinely need both halves. Ship the outbox without the idempotent consumer, and you’ve traded “sometimes silently loses an event” for “sometimes silently double-processes one” — which is progress, but it’s not actually done.

What I’d actually check before putting this in production

  • Write the outbox row in the exact same transaction as the business data, every single time. The moment it becomes a second, separate call, you’ve quietly rebuilt the original bug in a new location.
  • Index the table the way your poller actually queries it. An index on (published, created_at) keeps that scan cheap even as the table grows into the millions of rows.
  • Clean the table up. An outbox that only ever grows becomes its own performance problem eventually. A scheduled delete for rows that are already published and a few days old keeps it bounded, once you’re confident nothing still needs to replay them.
DELETE FROM outbox WHERE published = true AND created_at < now() - interval '3 days';
  • Make the downstream consumer idempotent, not just the outbox itself. This pattern solves reliable publishing. It does not remove duplicates — it makes them safe to receive, and only if the other side is actually built to handle that. Dedupe on the event’s unique id (a header on the message, mirroring the outbox row’s id), never on the Kafka key — multiple events for the same aggregate share the key and would cancel each other out.
  • Alert on the count of unpublished rows, not just on errors. A steadily climbing number of unpublished events is the clearest sign your publisher has quietly stopped working, and it might never throw an exception that tells you so.
  • Keep your ordering guarantees. Poll and send in created_at order per aggregate_id, and key your Kafka messages the same way you already would without an outbox in the picture. Adding this pattern doesn’t replace the partition-key rules from the ordering post — it sits on top of them.

Conclusion

Saving to a database and publishing to Kafka are two separate systems, and there was never a clean way to make both of them succeed or fail together directly — every obvious fix, from reordering the two calls to wrapping them in a distributed transaction, either doesn’t work or brings its own new problems. The outbox pattern sidesteps the whole issue by writing the event as a plain row, in the same transaction as the real data, in a database that already knows how to be atomic. Getting that row to Kafka becomes someone else’s problem entirely — a poller or a CDC process that runs after the fact and can retry for as long as it needs to, without anyone waiting on it. That buys you at-least-once delivery: never a silently lost event, at the cost of an occasional duplicate. Which is exactly why the consumer on the other end still needs to check whether it’s already seen a message before it acts on it. Neither half fixes this alone. Together, they actually do.

Prakash Raj