Event-Driven Ledger Reconciliation with Go & Kafka

Relational database systems struggle to reconcile concurrent payouts under strict ACID boundaries at high transaction volumes. We decoupled write operations from audits by introducing an event-sourcing ledger pattern via Apache Kafka and a high-performance Go reconciliation daemon.

Log-Structured Ledger Design

Instead of performing direct resource updates, mutations are recorded as immutable event sequences in Kafka partitions. A consumer daemon reads these logs sequentially, resolving account balances in transaction batches.

[Transaction Request] --> [Kafka Event Log (Partitioned)]
                                   |
                                   +--> [Go Daemon (Batch Audits)]
                                              |
                                              +--> [PostgreSQL (Index States)]

Transaction Processing Loop (Go)

func ProcessEvents(ctx context.Context, reader *kafka.Reader) {
    for {
        msg, err := reader.ReadMessage(ctx)
        if err != nil {
            log.Fatalf("failed reading ledger batch: %v", err)
        }
        ReconcileTx(msg.Value)
    }
}