Horizontal Scaling Blueprint (Multi-Tenant Activity Feed)

Problem Statement — what are we building and why?

You searched for “horizontal scaling”, but in interviews you’re rarely asked to “design horizontal scaling” in the abstract. Instead, the interviewer expects you to design a real system in a way that scales horizontally: add more instances for compute, add more shards/nodes for storage, and keep the system stable under spikes.

This case study designs a Multi-Tenant Activity Feed (think: “home timeline” for a social product or product updates feed for SaaS). It’s a great vehicle for horizontal scaling because it naturally forces you to address:

• Read-heavy traffic (feeds are fetched often)
• Fanout (one write may affect many followers)
• Hot partitions (celebrity accounts)
• CQRS (write model vs read-optimized model)
• Async processing (queue/stream for fanout and projections)
• Caching (low latency + cost control)
• Consistency trade-offs (freshness vs availability)

We’ll design it like a real interview: clarify requirements, estimate capacity, propose a high-level architecture, then deep dive into the scaling pain points.

Requirements

Functional Requirements (user-facing features, 3-5 key ones)

1. Publish activity: a user/service can publish an activity item (e.g., “Alice posted”, “Build finished”, “Order shipped”).
2. Read home feed: a user can fetch a personalized feed of activities from accounts/projects they follow.
3. Follow/unfollow: users can follow other entities (users, projects, orgs).
4. Pagination: feed supports cursor-based pagination.
5. Basic moderation/deletes: publisher can delete an activity; it should disappear from feeds eventually (within minutes).

Non-Functional Requirements (latency <Xms, availability 99.99%, consistency model, durability)

• Availability: 99.99% for reads; 99.9% for writes (reads dominate UX).
• Latency:
  • GET /feed: p50 < 50 ms, p99 < 200 ms (same region)
  • POST /activities: p99 < 300 ms (acknowledge quickly; heavy work async)
• Durability: activities must be durable once acknowledged.
• Consistency:
  • Write path (creating activity): strong consistency within the source-of-truth store.
  • Read path (home feed): eventual consistency acceptable (seconds to a minute), but must be monotonic per user session where possible.
• Scalability goal: support growth by adding nodes (web/app servers, consumers, DB shards, cache nodes).
• Multi-region: start single-region; support active-active reads later.

Capacity Estimation (back-of-envelope calculations)

We’ll choose concrete “interview-friendly” numbers.

Users and traffic assumptions

• DAU: 500M (internet scale example)
• Avg sessions/user/day: 2
• Feed fetches/session: 5

Feed reads/day:

• 500M DAU × 2 sessions/day × 5 reads/session = 5B reads/day

Convert to QPS (1 day ≈ 86,400 sec ≈ 10^5 sec):

• Average read QPS ≈ 5B / 100k = 50k QPS
• Peak read QPS (3× avg) = 150k QPS

Writes (activity publishes): assume 20% of DAU publishes 2 activities/day:

• 500M × 0.2 × 2 = 200M writes/day
• Average write QPS ≈ 200M / 100k = 2k QPS
• Peak write QPS (3× avg) = 6k QPS

Follow graph updates: smaller; assume 50M follow/unfollow/day:

• 50M/day → 500 QPS avg, 1.5k QPS peak

Storage estimates

Activity record (metadata only; media stored elsewhere):

• activity_id (16B), actor_id (8B), verb (16B), object_id (16B), timestamp (8B), tenant_id (8B), misc JSON/attributes (~200B)
• Total ≈ 300B per activity (round up with overhead)

Daily activity storage:

• 200M activities/day × 300B ≈ 60,000,000,000B ≈ 60 GB/day
• Yearly ≈ 60 GB/day × 365 ≈ 22 TB/year (metadata only)

Home feed materialization (if we precompute inbox entries):

• Each activity may be copied to many followers. This is the real scaling challenge.
• Suppose average fanout = 50 followers (varies wildly).

Inbox entries/day:

• 200M activities/day × 50 = 10B inbox entries/day

Inbox entry size (activity_id + score + small flags) ~ 40B.

• 10B × 40B = 400B GB? Actually 10B * 40B = 400B bytes = 400 GB/day
• Yearly ≈ 146 TB/year

This is why feeds often use hybrid fanout and careful retention.

Bandwidth

Assume feed response returns 20 items, each ~200B of payload (IDs + minimal fields) = 4KB plus overhead; round to 10KB.

• Peak 150k QPS × 10KB ≈ 1.5GB/s ≈ 12 Gbps egress (single region)

Cache sizing

If 20% of feed reads hit a “first page” cache and each cached page is ~10KB:

• Working set: 100M active users/hour × 1 page/user = 100M pages × 10KB = 1,000,000,000KB = 1 TB

In practice you’ll use a mix of:

• Distributed cache (Redis/Memcached) for hot keys
• In-process cache for micro-hot keys
• CDN caching only for public/non-personalized content (feeds are personalized, so limited CDN benefit)

High-Level Design with architecture diagram (mermaid)

flowchart LR
  U[Clients] --> DNS[DNS]
  DNS --> CDN[CDN/WAF]
  CDN --> GW[API Gateway + Rate Limiter]
  GW --> LB[Load Balancer]
  LB --> APP[Stateless Feed API Pods]

  APP --> RC[(Redis Cluster
Feed Cache)]
  APP --> READ[(Read Store
Feed Materialized Views)]
  APP --> AUTH[Auth Service]

  APP -->|POST /activities| ACTSVC[Activity Write Service]
  ACTSVC --> SQL[(SQL Source of Truth
Activities + Follows)]
  ACTSVC --> OUTBOX[(Outbox Table)]

  OUTBOX --> OBP[Outbox Publisher]
  OBP --> BUS[(Kafka/PubSub
Activities Topic)]

  BUS --> FAN[Fanout Workers
Consumer Group]
  FAN --> READ
  FAN --> RC

  BUS --> IDX[Search/Index Workers]
  IDX --> ES[(Search Index)]

  BUS --> AN[Analytics Stream]
  AN --> DL[(Data Lake/Warehouse)]

  APP --> OBS[Metrics/Logs/Traces]

Horizontal scaling is enabled by:

• Stateless API pods behind load balancers
• Partitioned state (SQL sharded; read store partitioned; Redis clustered)
• Async fanout via Kafka/PubSub consumer groups
• CQRS: write model (SQL) vs read model (materialized feed)

Detailed Design for each component

1) API Gateway + Load Balancing

Responsibilities:

• TLS termination, auth pre-checks
• Rate limiting (token bucket) per user/tenant/IP
• Request routing, request size limits
• Backpressure signals: return 429 + Retry-After when downstream is saturated

Scaling:

• Multiple gateway instances; stateless
• Use L7 routing and canary rollouts

2) Stateless Feed API Service (Read Path)

Responsibilities:

• Authenticate user
• Fetch feed page (cursor)
• Hydrate minimal item details (optionally via batch calls)
• Apply filtering (muted users, deleted items tombstones)

Read strategy (fast path):

1. Try Redis cache for feed:first_page:{user_id}
2. On miss, query read store for top N items
3. Populate cache with TTL + stampede protection

Scaling:

• Horizontal pod autoscaling (HPA) based on p95 latency, CPU, and request rate
• Keep instances stateless; store session/state in JWT or external store

3) Activity Write Service (Write Path)

Responsibilities:

• Validate and persist new activity
• Enforce invariants (tenant ownership, permissions)
• Produce an event for downstream fanout/indexing

Key horizontal scaling move: acknowledge quickly and push expensive work async.

Write flow:

• Write activities row in SQL
• Write corresponding outbox row in the same SQL transaction
• Return 201 to client

Then an outbox publisher reads the outbox table and publishes to Kafka/PubSub.

Why outbox pattern?

• Avoids “DB write succeeded but event publish failed” inconsistency.
• Guarantees eventual publication without distributed transactions.

4) SQL Source of Truth (Activities + Follows)

We keep a strongly consistent system of record for:

• Activities
• Follow edges
• Deletions/tombstones

Horizontal scaling techniques:

• Read replicas for read-heavy admin queries
• Sharding by tenant/user for write scale
• Partitioning tables by time for retention

5) Event Bus (Kafka/PubSub)

Topic: activity-events

• Partitions keyed by actor_id or tenant_id depending on ordering needs.
• Delivery semantics: typically at-least-once.

Why it helps horizontal scaling:

• Producers and consumers scale independently
• Spikes buffer in the log
• Multiple consumer groups (fanout, indexing, analytics) can be added without changing producers

Operational realities:

• Consumers must be idempotent (duplicates happen)
• Ordering is typically per-partition/per-key, not global

6) Fanout Workers + Read Store (CQRS)

Fanout workers consume activity events and update the read model.

Two common feed models:

• Fanout-on-write: push activity into each follower’s inbox (fast reads, expensive writes)
• Fanout-on-read: store activities per actor; assemble feed at read time (cheaper writes, heavier reads)

At our scale, we use hybrid fanout:

• Normal users: fanout-on-write to followers’ inboxes
• Celebrities: fanout-on-read (do not push to millions of followers)

This hybrid approach is one of the most important “horizontal scaling” interview answers.

7) Redis Cluster (Caching)

Use Redis (clustered) for:

• First-page feed cache
• Hot follower lists (for fanout)
• Dedup/idempotency keys for consumers

Scaling:

• Redis Cluster with hash slots
• Replicas for read scaling
• Use consistent hashing for client-side routing

8) Search/Index + Analytics

Not required for the core feed, but common in real systems:

• Index workers update Elasticsearch/OpenSearch for activity search
• Analytics consumers write to data lake/warehouse

This is horizontal scaling via pub/sub fanout: add consumers without touching the write path.

9) Observability + Automation

To operate a horizontally scaled system, you need:

• Metrics: QPS, error rate, p95/p99 latency, queue lag, DB/Redis saturation
• Tracing: end-to-end request traces and consumer processing spans
• Logging: structured logs with request_id/event_id
• Autoscaling: HPA for API pods; consumer autoscaling based on Kafka lag
• Safe deploys: canary + rollback; schema migrations with backward compatibility

Database Design with schema

Choice of SQL vs NoSQL with JUSTIFICATION

Source of truth: SQL (e.g., PostgreSQL / MySQL, or Spanner-like if global strong consistency is required)

Justification:

• We need strong correctness for core entities: activity creation authorization, deletes, and follow edges.
• SQL makes it easier to enforce constraints and run transactional updates.
• Horizontal scaling is achieved via sharding + replicas.

Read model: NoSQL / wide-column / key-value

Justification:

• Feed reads are simple key-based access (get inbox page by user_id + cursor).
• We want predictable low-latency reads and easy partitioning.
• A wide-column store (Cassandra/Bigtable) or a sharded KV store works well because it’s designed for horizontal scale and high throughput, and we can tolerate eventual consistency on derived views.

In an interview, it’s acceptable to say:

• SQL for correctness (writes)
• NoSQL for scale (reads)
• Redis for caching

SQL schema (source of truth)

-- Activities are append-only; deletions are tombstones.
CREATE TABLE activities (
  tenant_id      BIGINT NOT NULL,
  activity_id    UUID PRIMARY KEY,
  actor_id       BIGINT NOT NULL,
  verb           TEXT NOT NULL,
  object_type    TEXT NOT NULL,
  object_id      TEXT NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL,
  deleted_at     TIMESTAMPTZ NULL,
  attributes     JSONB
);

-- Follow graph (who receives whose activities)
CREATE TABLE follows (
  tenant_id      BIGINT NOT NULL,
  follower_id    BIGINT NOT NULL,
  followee_id    BIGINT NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (tenant_id, follower_id, followee_id)
);

-- Transactional outbox
CREATE TABLE outbox (
  tenant_id      BIGINT NOT NULL,
  event_id       UUID PRIMARY KEY,
  event_type     TEXT NOT NULL,
  aggregate_id   TEXT NOT NULL,
  payload        JSONB NOT NULL,
  created_at     TIMESTAMPTZ NOT NULL,
  published_at   TIMESTAMPTZ NULL
);

CREATE INDEX outbox_unpublished_idx ON outbox (published_at) WHERE published_at IS NULL;

Partition key / shard key selection and why

For horizontal scaling, shard keys must:

• Distribute load evenly
• Match access patterns
• Avoid hotspots

Recommended:

• Shard SQL by tenant_id first (multi-tenant isolation), then by user_id for large tenants.
• For follows, most queries are “get followees for follower” or “get followers for followee”. You may need two tables (or a secondary index) to support both directions at scale.

Example dual-write approach:

• follows_by_follower(tenant_id, follower_id, followee_id)
• followers_by_followee(tenant_id, followee_id, follower_id)

This is a deliberate denormalization for scale.

Indexing strategy

• outbox_unpublished_idx to efficiently scan unpublished events
• follows primary key supports “list followees”
• Secondary structure (or separate table) for “list followers”
• Time-based partitioning on activities.created_at for retention and faster pruning

Read store schema (NoSQL example: Cassandra/Bigtable style)

We store a per-user inbox (materialized feed) as time-ordered entries.

Table: user_inbox

• Partition key: (tenant_id, user_id_bucketed)
• Clustering key: created_at DESC, activity_id

Why bucketing?

• Prevent a single user with extremely high activity from creating a hot partition.
• Also helps with parallelism for reads/writes.

Example (conceptual):

• user_id_bucketed = hash(user_id) % 64 for distributing across partitions while still allowing efficient retrieval.
• Alternatively, keep partition key (tenant_id, user_id) but add time bucketing (e.g., (user_id, yyyy_mm_dd)), which is common for time-series style access.

API Design with REST endpoints, request/response schemas

External APIs: REST over HTTP. Internal service-to-service: gRPC is a good choice for lower overhead at high QPS, but REST is acceptable if you standardize retries/timeouts.

1) Publish activity

POST /v1/tenants/{tenantId}/activities

Headers:

• Idempotency-Key: <uuid> (critical for retry safety)

Request:

{
  "actor_id": "12345",
  "verb": "posted",
  "object": {"type": "post", "id": "p_999"},
  "attributes": {"text": "hello"}
}

Response (201):

{
  "activity_id": "550e8400-e29b-41d4-a716-446655440000",
  "created_at": "2026-08-14T12:00:00Z"
}

Notes:

• If the same Idempotency-Key is retried, return the original activity_id.

2) Read home feed

GET /v1/tenants/{tenantId}/users/{userId}/feed?limit=20&cursor=...

Response (200):

{
  "items": [
    {
      "activity_id": "...",
      "actor_id": "12345",
      "verb": "posted",
      "object": {"type": "post", "id": "p_999"},
      "created_at": "2026-08-14T12:00:00Z"
    }
  ],
  "next_cursor": "opaque_cursor"
}

Cursor design:

• Opaque cursor encodes (created_at, activity_id) to continue from last item.

3) Follow/unfollow

POST /v1/tenants/{tenantId}/users/{userId}/follows

{ "followee_id": "777" }

DELETE /v1/tenants/{tenantId}/users/{userId}/follows/{followeeId}

4) Delete activity

DELETE /v1/tenants/{tenantId}/activities/{activityId}

Semantics:

• Mark deleted in SQL (tombstone)
• Emit delete event; read model removes/filters eventually

Deep Dive sections (2-3 interesting challenges)

Deep Dive 1: How to handle hot partitions / celebrity problem

Problem: One actor has 100M followers. If we do fanout-on-write, one activity triggers 100M inbox writes.

Symptoms:

• Fanout workers fall behind (queue lag grows)
• Read store partitions for those followers get hammered
• Tail latency spikes due to compaction/flush pressure

Mitigations (hybrid fanout):

1. Classify celebrities

   • Maintain a threshold: if follower_count > X (e.g., 1M), treat as celebrity.

2. For celebrities: fanout-on-read

   • Store celebrity activities in an actor_activity table keyed by actor_id.
   • When building a user’s feed, merge:
     • precomputed inbox items (normal followees)
     • recent activities from celebrity followees (fetched on demand)

3. Bound the merge cost

   • Only merge top K celebrity sources per request
   • Use caching for celebrity recent lists

4. Shard/bucket hot keys

   • For counters (follower_count, like_count), use sharded counters (N-way) to avoid single hot row.

5. Adaptive partition management

   • If using Bigtable-like systems, rely on tablet splits/moves.
   • If self-managed, implement rebalancing: split partitions when QPS or size crosses threshold.

Interview framing:

• Call out that “choose a shard key” is not enough—you must plan for skew.

Deep Dive 2: Consistency vs availability trade-offs specific to this system

We intentionally choose a hybrid consistency model:

• Strong consistency for:

  • creating an activity (don’t lose writes)
  • follow/unfollow edges (so permissions/fanout correctness is stable)
  • delete/tombstone (source of truth)

• Eventual consistency for:

  • home feed materialization
  • cache contents
  • search index

Why this is the right trade:

• Users tolerate slight staleness in feeds.
• Strong consistency everywhere would increase write latency and couple services.

Edge cases and how we handle them:

• User posts and immediately refreshes feed: may not see the post for a few seconds.

  • UX mitigation: client can optimistically insert the new activity at top.

• Unfollow then still sees a few items from that followee:

  • Acceptable within a short window; enforce server-side filtering using the latest follow graph if required.

• Deletes:

  • For sensitive deletes, filter at read time by checking a small “tombstone cache” (Redis) to hide quickly.
  • Full cleanup happens async.

Deep Dive 3: Caching strategy (cache-aside, write-through, write-behind) with invalidation approach

We use cache-aside for feed pages.

Why cache-aside:

• Keeps write path simple and durable
• Cache failures degrade gracefully (fallback to read store)

Cache keys:

• feed:first_page:{tenant}:{user} → first page JSON
• feed:page:{tenant}:{user}:{cursor_hash} → optional, usually not worth caching deep pages

TTL strategy:

• First page TTL: 5–30 seconds (short TTL reduces staleness and invalidation complexity)
• Soft TTL + hard TTL:
  • Serve slightly stale data briefly while a background refresh recomputes

Invalidation:

• On fanout worker writing new inbox entries for user U, publish an invalidation event or directly delete feed:first_page key.
• To avoid excessive invalidations, batch them (e.g., invalidate at most once per user per 2 seconds).

Stampede protection:

• Use request coalescing (singleflight): only one recompute per key; others wait or serve stale.
• Probabilistic early refresh to spread recompute load.

Negative caching:

• Cache “empty feed” for new users for a short TTL to avoid repeated DB hits.

Scalability & Performance section

Horizontal scaling levers (what you add when load increases)

1. API layer: add more stateless pods/instances behind LB.
2. Kafka/PubSub consumers: increase consumer group size to raise fanout throughput.
3. Read store: add nodes/shards; rebalance partitions.
4. Redis: add shards/replicas; tune eviction policy.
5. SQL:
   • add read replicas
   • shard by tenant/user
   • partition tables by time

Backpressure and graceful degradation

When downstream is slow (read store or Redis saturated):

• API returns cached/stale feed (soft TTL) rather than recomputing.
• If cache miss and read store is failing, return:
  • partial results (smaller limit)
  • or a friendly error with retry-after

When fanout lags (Kafka backlog):

• Reads still work but show older content.
• Monitor consumer lag and autoscale workers.
• If lag exceeds threshold, temporarily switch more actors to fanout-on-read mode.

Tail latency control

• Timeouts on all downstream calls (Redis ~5-10ms budget, read store ~20-50ms)
• Retries with exponential backoff + jitter (careful: retries can amplify load)
• Circuit breakers around failing dependencies
• Bulkheads: separate thread pools for cache vs DB calls

Trade-offs and Alternatives — what would you do differently for different constraints?

Alternative 1: Pure fanout-on-read (simpler writes)

Pros:

• Write path is very cheap
• No massive inbox storage

Cons:

• Reads become expensive: must merge many followees’ recent activities
• Harder to hit p99 latency at 150k QPS

Use when:

• Follow graph is small per user (e.g., enterprise project updates)
• Read QPS is moderate

Alternative 2: Pure fanout-on-write (fast reads)

Pros:

• Reads are extremely fast and simple

Cons:

• Explodes write amplification; celebrity problem severe
• Storage costs huge

Use when:

• Follow graph is bounded (e.g., max 1k followers)
• You can afford heavy async infra

Alternative 3: Globally consistent database (Spanner-style)

Pros:

• Strong consistency across regions; simpler correctness story

Cons:

• Higher write latency due to synchronous replication
• Cost/complexity

Use when:

• Regulatory or product requirements demand strong cross-region invariants

Alternative 4: Microservices vs modular monolith

A modular monolith can still scale horizontally:

• Deploy many identical stateless instances
• Keep internal boundaries as modules

Microservices add independent scaling per domain but increase:

• distributed failures
• operational overhead
• need for sagas and idempotency everywhere

A common real-world approach: start modular monolith + event bus, then split services with a strangler fig migration.

Closing: what interviewers look for when you say “horizontal scaling”

In interviews, “horizontal scaling” isn’t a single feature—it’s a set of design commitments:

• Stateless compute so you can add instances freely
• Partitioning/sharding for stateful stores with a plan for skew
• Replication + load balancing for availability and throughput
• Async messaging to decouple and buffer spikes
• Idempotency + dedupe because at-least-once delivery is normal
• Caching with explicit TTL/invalidation and stampede protection
• Backpressure + graceful degradation to avoid cascading failures
• Observability + automation (autoscaling, safe rollouts)

This activity feed blueprint demonstrates those commitments end-to-end, with concrete shard keys, CQRS read models, and operational mitigations for hotspots and fanout.