Samuel J Mathew
Samuel J Mathew
Distributed Systems••8 min read

Designing Resilient Event-Driven Microservices with Kafka & Spring Boot 3

Practical patterns for transactional outboxes, idempotency keys, and consumer lag mitigation in high-throughput enterprise systems.

In distributed architectures, asynchronous communication through event streaming platforms like Apache Kafka offers decoupling and high throughput. However, naive implementations often break under high load or temporary network partitions.

The Dual-Write Problem

A common architectural trap occurs when a service must simultaneously update its local SQL database and emit an event to Kafka. If the database commit succeeds but the message broker publish fails, the system enters an inconsistent state.

Excalidraw Diagram • The Dual-Write Failure Mode
Excalidraw Diagram showing Dual-Write Failure Mode
Figure 1: The Dual-Write Problem • Why direct broker publishing leads to silent data divergence

Architecture Sequence Flow

Mermaid Architecture Flow
sequenceDiagram
  autonumber
  actor Client
  participant OrderSvc as Order Service
  participant DB as Database (SQL + Outbox)
  participant CDC as Debezium / CDC Poller
  participant Kafka as Apache Kafka
  participant PaymentSvc as Payment Consumer

  Client->>OrderSvc: POST /orders
  critical Atomic Local Transaction
    OrderSvc->>DB: INSERT into orders
    OrderSvc->>DB: INSERT into outbox_table
  end
  OrderSvc-->>Client: 201 Created (Instant ACK)
  
  loop Continuous Stream
    CDC->>DB: Read Outbox ChangeLog
    CDC->>Kafka: Stream OrderCreated Event
  end

  Kafka->>PaymentSvc: Consume Message
  PaymentSvc->>PaymentSvc: Idempotency Key Validation
  PaymentSvc->>PaymentSvc: Settle Ledger & ACK
Transactional Outbox Pattern and Resilient Event Streaming Architecture
Figure 2: Transactional Outbox Component Blueprint • Atomic Persistence and Reliable Event Streaming

The Transactional Outbox Pattern

To guarantee consistency without heavy two-phase commit (2PC) protocols, record state changes and outbound messages within the same local database transaction using an Outbox table. A separate publisher thread or Change Data Capture (CDC) mechanism then streams events reliably to Kafka.

Implementation: Transactional Outbox in Spring Boot 3

Here is the production implementation of the transactional outbox pattern ensuring atomic persistence of the business entity alongside the outbox event payload:

OrderService.java • Transactional Outbox Pattern
View Gist on GitHub →
// Transactional Outbox Pattern: Atomic Order Persistence & Outbox Event Insertion
@Service
@Transactional
public class OrderService {
private final OrderRepository orderRepository;
private final OutboxEventRepository outboxRepository;
private final ObjectMapper objectMapper;
public OrderService(OrderRepository orderRepository,
OutboxEventRepository outboxRepository,
ObjectMapper objectMapper) {
this.orderRepository = orderRepository;
this.outboxRepository = outboxRepository;
this.objectMapper = objectMapper;
}
public Order createOrder(OrderRequest request) {
// 1. Commit Domain State Locally
Order order = new Order(UUID.randomUUID(), request.getCustomerId(), request.getTotalAmount());
orderRepository.save(order);
// 2. Insert Outbox Event in the Same Database Transaction
OutboxEvent event = new OutboxEvent(
UUID.randomUUID(),
"ORDER_CREATED",
order.getId().toString(),
objectMapper.writeValueAsString(order),
Instant.now()
);
outboxRepository.save(event);
return order;
}
}

Idempotent Consumers

Network re-deliveries are guaranteed in distributed networks. Every consumer must implement deterministic deduplication using idempotent keys stored with short-lived distributed caches or database unique constraints.

Implementation: Idempotent Kafka Consumer with Redis

PaymentEventListener.java • Idempotent Consumer
View Gist on GitHub →
// Idempotent Kafka Consumer with Distributed Cache Deduplication
@Component
public class PaymentEventListener {
private final RedisTemplate<String, String> redisTemplate;
private final PaymentLedgerService ledgerService;
private static final Duration DEDUPLICATION_TTL = Duration.ofDays(7);
public PaymentEventListener(RedisTemplate<String, String> redisTemplate,
PaymentLedgerService ledgerService) {
this.redisTemplate = redisTemplate;
this.ledgerService = ledgerService;
}
@KafkaListener(topics = "order-events", groupId = "payment-service-group")
public void onOrderEvent(@Payload OrderEvent event, Acknowledgment ack) {
String idempotencyKey = "processed-event:" + event.getEventId();
// Atomic Check-and-Set: Process only if key does not exist
Boolean isFirstDelivery = redisTemplate.opsForValue()
.setIfAbsent(idempotencyKey, "PROCESSING", DEDUPLICATION_TTL);
if (Boolean.TRUE.equals(isFirstDelivery)) {
try {
ledgerService.processPayment(event);
redisTemplate.opsForValue().set(idempotencyKey, "COMPLETED", DEDUPLICATION_TTL);
ack.acknowledge();
} catch (Exception ex) {
redisTemplate.delete(idempotencyKey); // release key on failure for retry
throw ex;
}
} else {
// Duplicate delivery detected: acknowledge and discard safely
ack.acknowledge();
}
}
}
← Back to All ArticlesDiscuss Product →