Event Sourcing

Problem Statement — what are we building and why?

In many domains (orders, payments, inventory, tickets, logistics), the hardest problems aren’t CRUD—they’re correctness over time, auditability, and scaling independent read models without turning the primary database into a bottleneck.

We’re designing an Event Sourcing Platform (ESP): a shared set of services and storage primitives that let product teams build event-sourced systems consistently.

Instead of persisting only “current state” rows (e.g., orders.status = SHIPPED), the platform persists the source of truth as an append-only sequence of immutable events:

• OrderPlaced
• PaymentAuthorized
• ItemAllocated
• OrderShipped

Current state is derived by replaying events or (more commonly) by reading from materialized projections (CQRS read models). The platform also reliably publishes committed events to a message bus so downstream consumers can build projections, trigger workflows (sagas), and integrate other services.

Why build this?

• Auditability & debugging: you can answer “what happened?” precisely.
• Rebuildability: you can rebuild projections after bugs or new product requirements.
• Scalable reads: projections are optimized per query pattern.
• Integration: event streams become the contract between services.

This mirrors how real interviews go: we’ll clarify requirements, estimate scale, propose architecture, dive into data modeling, and handle failure modes.

Requirements

Functional Requirements

1. Append domain events per aggregate (e.g., orderId) with optimistic concurrency (expected version).
2. Publish committed events to an event bus for consumers (projection builders, saga orchestrators, integrations).
3. Serve queries from read-optimized projections (CQRS) with low latency.
4. Replay events to rebuild projections deterministically (with snapshots to speed up rehydration).
5. Provide operational controls: consumer lag tracking, DLQ, backfills, schema evolution support.

Non-Functional Requirements

• Latency:
  • Command append (p50): < 30 ms in-region
  • Command append (p99): < 150 ms in-region
  • Query reads (p99): < 50 ms from projection store/cache
• Availability:
  • Command path: 99.99% (multi-AZ)
  • Query path: 99.99%
• Consistency model:
  • Strong consistency per aggregate on the write model (event store ordering + expected version).
  • Eventual consistency for projections (milliseconds to seconds lag).
  • Optional read-your-writes mode for UX-sensitive endpoints.
• Durability:
  • Events are durable once acknowledged; replication across AZs.
  • Long-term retention with tiering (hot storage + cold archive).
• Security & compliance:
  • Encryption in transit and at rest.
  • PII handling strategy compatible with “immutable log” (crypto-shredding or PII references).

Capacity Estimation (back-of-envelope calculations)

Because event sourcing is a storage + processing strategy, capacity is best expressed as:

• commands/sec
• events/sec (commands often generate multiple events)
• event log growth (GB/day)
• projection consumer throughput and replay time

Assume we’re building a platform used by multiple product domains (orders, payments, support cases). Interviewers like concrete numbers, so we’ll pick a “large but plausible” target.

Traffic assumptions

• DAU: 50M
• Commands per DAU per day: 2 (create/update/cancel/etc.)
• Writes/day: 50M × 2 = 100M commands/day
• Average commands/sec: 100M / 86,400 ≈ 1,157 commands/sec
• Peak (3× avg): ≈ 3,500 commands/sec

Events per command:

• Average 3 events/command (e.g., OrderPlaced, InventoryReserved, PaymentInitiated)
• Average events/sec: 1,157 × 3 ≈ 3,470 events/sec
• Peak events/sec: 3,500 × 3 ≈ 10,500 events/sec

This is a comfortable scale for Postgres + Kafka in a single region, but we’ll design with headroom (and discuss Spanner/Cassandra options).

Event size and storage

Typical event payloads are small; assume:

• Event payload: 1 KB (including metadata, JSON/Protobuf)
• Index overhead / storage overhead: ~0.5 KB
• Total per event: ~1.5 KB

Storage/day:

• 3,470 events/sec × 86,400 sec/day ≈ 300M events/day
• 300M × 1.5 KB ≈ 450,000,000 KB/day ≈ 450 GB/day

Storage/year:

• 450 GB/day × 365 ≈ 164 TB/year (hot if you keep everything hot)

In practice:

• compress events (often 2–5×)
• tier older events to object storage
• keep snapshots and recent events hot

Bandwidth

Peak ingestion to event store:

• 10,500 events/sec × 1.5 KB ≈ 15,750 KB/sec ≈ 15.8 MB/sec write throughput

Peak publish to bus is similar (plus replication factor overhead).

Cache sizing (query side)

Assume query traffic is much higher than commands:

• Reads per DAU per day: 20
• Reads/day: 50M × 20 = 1B reads/day
• Average read QPS: 1B / 86,400 ≈ 11,600 QPS
• Peak (3×): 35,000 QPS

If 20% of reads are for “hot” entities and we cache those:

• Cached objects: 10M entities
• Avg cached projection: 2 KB
• Cache: 10M × 2 KB = 20 GB (plus overhead; provision ~50–100 GB Redis cluster)

High-Level Design with architecture diagram (mermaid)

flowchart LR
  U[Client] --> DNS[DNS/CDN]
  DNS --> AGW[API Gateway + Auth + Rate Limit]

  AGW --> CMD[Command Service]
  AGW --> QRY[Query Service]

  CMD -->|Validate + expectedVersion| ES[(Event Store)]
  CMD -->|TX write| OB[(Outbox Table)]

  OB --> RELAY[Outbox Relay / CDC]
  RELAY --> BUS[(Kafka / PubSub)]

  BUS --> PROJ1[Projection Builder: Entity Views]
  BUS --> PROJ2[Projection Builder: Search Index]
  BUS --> SAGA[Saga Orchestrator]

  PROJ1 --> RDB[(Read DB / KV Store)]
  PROJ2 --> ESCH[(Elasticsearch/OpenSearch)]

  QRY --> CACHE[(Redis/Memcached)]
  QRY --> RDB
  QRY --> ESCH

  SAGA --> CMD

  ES --> SNAP[Snapshot Store]
  ES --> ARCH[(Cold Archive: Object Storage)]

  subgraph Observability
    MON[Metrics/Tracing]
    LAG[Consumer Lag Monitor]
    DLQ[Dead Letter Queue]
  end

  CMD --> MON
  QRY --> MON
  RELAY --> MON
  PROJ1 --> MON
  PROJ2 --> MON
  BUS --> LAG
  BUS --> DLQ

Key idea: persist events first, then publish them reliably (outbox/CDC), then build projections asynchronously.

Detailed Design for each component

1) API Gateway

Responsibilities:

• AuthN/Z (JWT/OAuth)
• Rate limiting (token bucket) per user/app/IP
• Request validation and routing
• Idempotency key enforcement at edge (optional but helpful)

Rate limiting is important because event sourcing is append-heavy and “hot aggregates” can melt partitions.

2) Command Service (Write Model)

The Command Service implements domain rules and aggregate consistency.

Flow:

1. Receive command (e.g., PlaceOrder).
2. Load aggregate state (rehydrate from snapshot + events) or use a cached aggregate state.
3. Validate invariants (e.g., cannot ship before payment).
4. Produce one or more events.
5. Append events to the event store with expected version.
6. Write outbox record(s) in the same transaction.
7. Return success.

Expected version enforces optimistic concurrency:

• Client sends expectedVersion (or the service looks up last known version).
• Event store append succeeds only if currentVersion == expectedVersion.

This ensures “strong consistency per aggregate” without global locks.

3) Event Store

Core responsibilities:

• Append-only persistence
• Ordering per aggregate stream
• Concurrency control (expected version)
• Efficient reads by aggregate (load stream)
• Retention/tiering + snapshots

Two practical implementations in interviews:

• Postgres for single-region / moderate scale
• Spanner (or CockroachDB) for multi-region strong consistency

We’ll detail schema later.

4) Outbox Relay / CDC

Problem: dual writes.

If you append to event store and then publish to Kafka in a separate step, you can get:

• DB commit succeeds, publish fails → consumers never see the event.

Solution: transactional outbox.

• In the same DB transaction as the event append, insert an outbox row.
• A relay process reads outbox rows and publishes to Kafka.
• After publish, mark outbox row as dispatched.

Implementation options:

• Polling relay (simple; good for interviews)
• CDC (Debezium) streaming changes from outbox table (more real-time)

5) Event Bus (Kafka / Pub/Sub)

Responsibilities:

• Durable fanout to many consumers
• Replay capability (offset-based)
• Partitioned ordering (per key)

Partition key should usually be aggregateId, so all events for an aggregate are ordered.

6) Projection Builders (CQRS Read Models)

Consumers read events and update read-optimized stores.

Examples:

• “Order summary view” in a key-value store for GET /orders/{id}
• “Orders by user” table for GET /users/{id}/orders
• Search index in Elasticsearch for GET /orders/search?q=...

Projection builders must be:

• idempotent (at-least-once delivery)
• deterministic (replay yields same result)
• version-aware (schema evolution)

7) Query Service + Cache

Query service reads projections and serves APIs.

Caching is typically cache-aside:

• check Redis
• on miss, read from projection store
• populate cache with TTL

Invalidation is “best effort” because projections are eventually consistent anyway. For stricter semantics, use versioned keys.

8) Saga Orchestrator

Coordinates multi-step workflows across services without 2PC.

Example “order placement” saga:

• On OrderPlaced → command AuthorizePayment
• On PaymentAuthorized → command ReserveInventory
• On InventoryReserved → command ShipOrder
• On failure → compensating actions (e.g., ReleaseInventory, RefundPayment)

Sagas are where interviewers probe your understanding of distributed transactions.

Database Design with schema

Choice of SQL vs NoSQL (with justification)

Default interview choice: Postgres as the event store

Why Postgres works well initially:

• Strong transactional semantics for append + outbox in one transaction
• Simple optimistic concurrency via unique constraints / version checks
• Mature tooling for backups, replication, observability
• CDC integration (logical replication / Debezium)

When you would pick something else:

• Spanner/CockroachDB if you need multi-region active-active writes with strong consistency and global transactions.
• Cassandra/Scylla if you need extremely high write throughput and can accept more complex modeling and eventual consistency trade-offs.
• Purpose-built event store (EventStoreDB) if you want native stream semantics and subscription features and can operate it reliably.

In this case study, we’ll implement:

• Event store + outbox in Postgres (multi-AZ)
• Kafka as bus
• Read models in a mix of Postgres read DB / key-value store + Elasticsearch

Event Store schema (Postgres)

We need:

• events stored per aggregate
• strict ordering per aggregate
• concurrency control
• ability to fetch events after a sequence number

Table: events

CREATE TABLE events (
  event_id        UUID PRIMARY KEY,
  aggregate_type  TEXT NOT NULL,
  aggregate_id    TEXT NOT NULL,
  seq            BIGINT NOT NULL,        -- monotonically increasing per aggregate
  event_type     TEXT NOT NULL,
  event_version  INT NOT NULL,
  occurred_at    TIMESTAMPTZ NOT NULL,
  actor_id       TEXT NULL,
  trace_id       TEXT NULL,
  payload        JSONB NOT NULL
);

-- Enforce per-aggregate ordering + optimistic concurrency
CREATE UNIQUE INDEX ux_events_agg_seq
  ON events (aggregate_type, aggregate_id, seq);

-- Fast load of an aggregate stream
CREATE INDEX ix_events_agg
  ON events (aggregate_type, aggregate_id, seq);

-- Optional: time-based queries for ops/backfills
CREATE INDEX ix_events_time
  ON events (occurred_at);

How we assign seq:

• Read current max seq for aggregate, then append with expected seq+1.
• Better: maintain an aggregates table with current version and update it transactionally.

Table: aggregates

CREATE TABLE aggregates (
  aggregate_type TEXT NOT NULL,
  aggregate_id   TEXT NOT NULL,
  current_seq    BIGINT NOT NULL,
  PRIMARY KEY (aggregate_type, aggregate_id)
);

Append algorithm:

• SELECT current_seq FROM aggregates ... FOR UPDATE
• ensure current_seq == expected_seq
• insert N events with seq = current_seq+1..current_seq+N
• update aggregates.current_seq

This avoids scanning events for max(seq).

Outbox schema

Table: outbox

CREATE TABLE outbox (
  outbox_id      UUID PRIMARY KEY,
  event_id       UUID NOT NULL,
  topic          TEXT NOT NULL,
  key            TEXT NOT NULL,           -- usually aggregate_id
  payload        JSONB NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL,
  dispatched_at  TIMESTAMPTZ NULL,
  dispatch_attempts INT NOT NULL DEFAULT 0
);

CREATE INDEX ix_outbox_undispatched
  ON outbox (dispatched_at, created_at)
  WHERE dispatched_at IS NULL;

Relay behavior:

• Poll undispatched rows in created_at order
• Publish to Kafka
• Mark dispatched_at
• Retry with exponential backoff; after N attempts send to DLQ/alert

Snapshot schema

Snapshots speed up rehydration.

Table: snapshots

CREATE TABLE snapshots (
  aggregate_type TEXT NOT NULL,
  aggregate_id   TEXT NOT NULL,
  seq            BIGINT NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL,
  state          JSONB NOT NULL,
  PRIMARY KEY (aggregate_type, aggregate_id)
);

Snapshot strategy:

• Snapshot every K events (e.g., 100) or based on size/time.
• On load: fetch snapshot, then fetch events with seq > snapshot.seq.

Shard / partition key selection and why

In Postgres you’ll likely use:

• Partitioning by time (monthly partitions) for events if the table grows huge.
• Or partition by hash of aggregate_id if you need parallelism.

In Kafka:

• partition key = aggregate_id to guarantee ordering per aggregate.

Why it matters:

• Ordering per aggregate is the core correctness requirement.
• Global ordering doesn’t scale; you avoid it.

Indexing strategy

• UNIQUE(aggregate_type, aggregate_id, seq) is mandatory.
• INDEX(aggregate_type, aggregate_id, seq) for stream reads.
• Optional occurred_at index for ops.
• Avoid indexing payload fields on the event store; query via projections.

API Design with REST or gRPC endpoints

We’ll expose REST at the edge; internally we may use gRPC.

Command APIs (REST)

Place order

POST /v1/orders

Request:

{
  "idempotencyKey": "c0b9f7d0-...",
  "userId": "u_123",
  "items": [{"sku": "sku_1", "qty": 2}],
  "paymentMethodId": "pm_456"
}

Response (async acceptance):

{
  "orderId": "o_789",
  "commandId": "cmd_111",
  "status": "ACCEPTED",
  "expectedReadVersion": 12
}

Notes:

• Returning 202 Accepted is common when downstream saga steps (payment/inventory) are async.
• expectedReadVersion can be used for read-your-writes waiting.

Cancel order

POST /v1/orders/{orderId}:cancel

Request:

{
  "idempotencyKey": "...",
  "expectedVersion": 12,
  "reason": "USER_REQUEST"
}

Response:

{
  "orderId": "o_789",
  "status": "ACCEPTED"
}

Query APIs (REST)

Get order

GET /v1/orders/{orderId}

Response:

{
  "orderId": "o_789",
  "userId": "u_123",
  "status": "SHIPPED",
  "items": [{"sku": "sku_1", "qty": 2}],
  "version": 15,
  "updatedAt": "2026-09-09T12:34:56Z"
}

List user orders (cursor pagination)

GET /v1/users/{userId}/orders?cursor=...&limit=50

Response:

{
  "orders": [ ... ],
  "nextCursor": "..."
}

Internal gRPC (optional)

• AppendEvents(aggregateId, expectedVersion, events[]) -> newVersion
• Subscribe(topic, group) -> stream EventEnvelope

gRPC is especially useful for high-throughput internal traffic and typed contracts.

Deep Dive: interesting challenges

1) Hot partitions / “celebrity problem”

Problem: a single aggregate receives disproportionate writes.

Examples:

• A “celebrity user” aggregate with millions of followers generating constant updates
• A flash sale “inventory” aggregate
• A giant group chat

Why it breaks:

• Event store row lock contention on aggregates row
• Kafka partition hot-spot if aggregateId is the partition key
• Projection consumers can’t keep up for that key

Mitigations (choose based on domain semantics):

1. Split the aggregate (preferred)

   • Instead of one Inventory(sku) aggregate, use InventoryBucket(sku, bucketId).
   • Commands route to a bucket; you trade strict single-entity invariants for scalable throughput.

2. Bucketed streams

   • For high-volume entities, write to multiple streams: aggregateId#0..N.
   • Requires careful read-side merging and may weaken ordering guarantees.

3. Admission control / rate limiting

   • Apply per-aggregate rate limits at command service.
   • Backpressure returns 429/503 to callers.

4. Redesign invariants

   • If you truly need single-aggregate strict ordering at massive QPS, you may need to rethink the model (e.g., pre-allocations, escrow, or CRDT-like approaches).

Interview tip: explicitly say global ordering is avoided; ordering is per aggregate.

2) Consistency vs availability trade-offs (specific to event sourcing)

Event-sourced systems naturally split into:

• Write model: strong per-aggregate consistency
• Read model: eventually consistent projections

Interviewers will ask: “Do users see their updates immediately?”

Options:

1. Pure eventual consistency (simplest)

   • After command returns, reads may lag.
   • Acceptable for many back-office flows.

2. Read-your-writes via sync wait

   • Command returns newVersion.
   • Query endpoint supports ?minVersion=....
   • Query service waits (with timeout) until projection version ≥ minVersion.

3. Fallback read from write model

   • For a short window, read state by rehydrating from event store.
   • Costs more CPU/latency; use selectively.

4. Stronger read store

   • Use a strongly consistent projection store (often expensive and can reduce decoupling).

A real-world lesson: CQRS + caches can create propagation delays that hurt UX; some teams move to different read model technology (e.g., in-memory object stores) to meet preview/freshness needs.

3) Caching strategy + invalidation

Because projections are the read source, caching sits in front of the query service.

Default: cache-aside

• Pros: simple, resilient to cache failures
• Cons: invalidation complexity, thundering herds

Invalidation approaches:

• TTL only: simplest; accept staleness.
• Best-effort delete-on-update: projection builder publishes cache invalidation events.
• Versioned keys: order:{id}:v{version}
  • Query service knows current version from projection store; cache naturally rolls forward.

Thundering herd protection:

• request coalescing / singleflight
• stale-while-revalidate
• small randomized TTL jitter

Scalability & Performance

Scaling the command path

• Stateless command service scales horizontally behind L7 LB.
• Event store bottlenecks:
  • write IOPS
  • row-lock contention on hot aggregates
  • index maintenance

Practical improvements:

• Batch append multiple events in one transaction.
• Keep event payloads compact (prefer Protobuf/Avro over verbose JSON internally).
• Partition events table by time to keep indexes manageable.
• Use connection pooling (PgBouncer).

If Postgres becomes the limiter:

• Move to distributed SQL (Spanner/CockroachDB) for horizontal scaling and multi-region.
• Or adopt Cassandra/Scylla with careful partitioning (accepting more complexity).

Scaling Kafka / PubSub

• Increase partitions to scale consumer parallelism.
• Keep ordering constraints: partition by aggregateId.
• Watch for hot partitions; apply the hot-key mitigations above.

Scaling projections

Projection builders scale horizontally by consumer group.

Key practices:

• Idempotent writes: store lastProcessedOffset or lastEventSeq per aggregate/view.
• Use upserts with version checks:
  • only apply event if eventSeq > view.version.

Replay and rebuild

Rebuild is a first-class requirement.

Strategies:

• Parallel replay by partitioning event streams.
• Backfill pipelines that write into new tables/indexes.
• Cutover by swapping aliases (e.g., Elasticsearch index alias) or renaming tables.

Define an SLO for rebuild:

• “We can rebuild the primary projections for last 90 days within 6 hours.”

Observability

You must monitor:

• command latency and error rates
• event store write throughput
• outbox lag (rows undispatched)
• Kafka consumer lag per group/partition
• DLQ rate
• projection freshness (time since last event applied)

Trade-offs and Alternatives — what would you do differently for different constraints?

If you need global strong consistency and multi-region writes

• Use Spanner (or similar distributed SQL) as event store.
• Publish to Pub/Sub after commit (or via change streams).
• Trade-off: higher write latency due to synchronous replication.

If you need extremely high write throughput and can accept eventual consistency

• Use Cassandra/Scylla event store with partition key = aggregateId and clustering by seq/time.
• Trade-off: more careful modeling, tombstones/compaction tuning, and consistency semantics.

If you want fewer moving parts for a small team

• Use Postgres for event store and projections (read replicas), skip Kafka initially.
• Use CDC later when you outgrow.
• Trade-off: less decoupling; harder to scale consumers independently.

If you need very fresh reads (read-after-write UX)

• Provide sync-wait read-your-writes.
• Or serve certain reads from write model temporarily.
• Or use a specialized in-memory read store.
• Trade-off: complexity and cost.

If GDPR/PII deletion is mandatory

Because events are immutable, you can’t “delete history” easily.

Approaches:

• Crypto-shredding: encrypt PII fields with per-user keys; delete keys to render data unreadable.
• PII reference: store PII in a mutable store; events contain references/ids.

Trade-off: operational complexity vs compliance.

Closing interview checklist (what interviewers look for)

• You clearly separate event store (truth) vs bus (distribution) vs projections (reads).
• You state the consistency model: strong per aggregate, eventual for reads.
• You handle reliability: outbox/CDC, at-least-once + idempotency.
• You address hot keys, replay, schema evolution, and projection rebuild.
• You present concrete numbers and partitioning keys.

If you want a more concrete “product” version (payments ledger, ticketing, inventory, chat), share the domain and we can pin down exact event schemas, projection tables, and sagas end-to-end.