System Design Interview: How to Design a Notification System

System Design Interview: How to Design a Notification System

Notification systems are among the most commonly asked system design questions at FAANG and tier-1 companies. They look simple on the surface — "just send a message" — but at 100M+ users they expose every weak point in your architecture: throughput, latency, consistency, fault tolerance, and operational complexity. This guide walks you through the entire design in interview format, with numbered questions an interviewer can ask and the detailed answers you need to give.

Part 1: Requirements and Scope

1. What are the functional requirements of a notification system?

A notification system must:

• Support multiple channels: push (mobile + web), email, SMS, in-app
• Allow user preference management (opt-in/opt-out per channel, per notification type)
• Deliver notifications reliably — no drops, minimal duplicates
• Handle priority tiers — OTP and security alerts must arrive in seconds; marketing digests can wait minutes
• Support templating — the same event renders differently per locale and channel
• Provide delivery tracking and analytics — sent, delivered, opened, failed
• Enforce rate limits per user to prevent spam
• Deduplicate — the same notification must not be sent twice even if upstream publishes it twice

2. What are the non-functional requirements?

• Scale: 100M daily active users (DAU). Assume 5 notifications/user/day on average → ~500M notifications/day → ~5,800/second average, peaks at 3–5× (Black Friday, sports events, breaking news).
• Latency: Push and SMS for transactional events < 2 seconds end-to-end. Email can tolerate up to 30 seconds.
• Availability: 99.99% for the ingestion path. Delivery itself is best-effort because third-party providers add their own SLAs.
• Durability: No notification should be silently dropped. If delivery fails, it must be retried or dead-lettered.
• Idempotency: The system must be idempotent end-to-end; producers can safely retry publishing.

3. What are out of scope for a first iteration?

• Real-time two-way messaging (that is a chat system)
• In-house SMS carrier integration (Twilio handles SMPP)
• A/B testing of notification content (you can mention it as a future extension)
• GDPR data deletion (mention you would add a delete-by-user pipeline)

Part 2: High-Level Architecture

4. Draw the high-level architecture of the notification system.

At the top level there are four layers:

[Producers / Internal Services]
         |
   [Notification Service API]
         |
   [Message Queue / Topic Layer]
      /       |      \
[Priority  [Default] [Batch/Low]
  Queue]    Queue    Queue
      \       |      /
      [Dispatcher Workers]
       /    |     \    \
  [Push] [Email] [SMS] [In-App]
 Handler Handler Handler Handler
      \       |      /      |
  [3rd-party providers]  [DB Write]
  FCM, APNs, SendGrid,
  Twilio, Vonage
         |
  [Delivery Tracker]
         |
  [Analytics / Warehouse]

Producers are any internal microservice (order service, auth service, social graph). They call the Notification Service API with a structured event. The API validates, enriches, de-duplicates, and publishes to the appropriate queue tier. Workers pull from queues, resolve recipients, expand templates, and call third-party providers.

5. Why use a message queue instead of calling providers directly from the API?

Direct calls create tight coupling and fragile synchronous chains. A queue provides:

• Decoupling: The order service does not care about FCM being down.
• Backpressure: Workers consume at a sustainable rate; the queue absorbs spikes.
• Retry semantics: If a worker crashes after pulling a message, the message reappears (visibility timeout) and another worker picks it up.
• Observability: Queue depth is a leading indicator of problems.

At 100M users, even a 100ms added latency from a direct call multiplied across millions of concurrent requests makes synchronous delivery untenable.

6. What message queue technology would you use?

Apache Kafka is the default choice for this scale:

• Partitioned topics give horizontal throughput scaling (add partitions = add parallelism).
• Persistent log allows replay — critical if a downstream bug caused silent drops.
• Consumer groups let you add workers without changing producers.
• Retention allows auditing: "did we send this notification?"

Alternatively, Amazon SQS (or RabbitMQ for smaller scale) works well if you need simpler operational overhead and are AWS-native. SQS FIFO queues add deduplication IDs natively.

For the priority separation, Kafka makes this easy: use separate topics (notifications.critical, notifications.default, notifications.batch) with separate consumer groups and worker pools.

Part 3: Notification Channels Deep-Dive

7. How do you implement mobile push notifications (iOS and Android)?

You never send directly to devices. You send to Apple Push Notification service (APNs) for iOS and Firebase Cloud Messaging (FCM) for Android.

Flow:

1. 1App registers with APNs/FCM at install time and receives a device token.
2. 2App sends token to your backend, which stores it in a device_tokens table: (user_id, token, platform, created_at, last_seen_at).
3. 3When you want to push, your worker sends an HTTP/2 request to APNs or an HTTP request to FCM with the token and payload.

import httpx

async def send_fcm(device_token: str, title: str, body: str, data: dict) -> dict:
    headers = {
        "Authorization": f"Bearer {await get_fcm_access_token()}",
        "Content-Type": "application/json",
    }
    payload = {
        "message": {
            "token": device_token,
            "notification": {"title": title, "body": body},
            "data": data,
            "android": {"priority": "high"},
        }
    }
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "https://fcm.googleapis.com/v1/projects/YOUR_PROJECT/messages:send",
            json=payload,
            headers=headers,
            timeout=5.0,
        )
    resp.raise_for_status()
    return resp.json()

Token management is critical: APNs returns error code 410 Gone when a token is stale (user uninstalled). Your worker must detect this and delete the token from the DB immediately, or you waste calls and hit rate limits.

8. How do you handle web push notifications?

Web push uses the Web Push Protocol (RFC 8030). The browser generates a subscription object containing an endpoint URL (hosted by the browser vendor — Chrome uses FCM, Firefox uses Mozilla's push service), a public key, and an auth secret. You store this subscription and use the webpush library to send VAPID-authenticated requests.

const webpush = require('web-push');

webpush.setVapidDetails(
  'mailto:ops@yourcompany.com',
  process.env.VAPID_PUBLIC_KEY,
  process.env.VAPID_PRIVATE_KEY
);

async function sendWebPush(subscription, payload) {
  try {
    await webpush.sendNotification(subscription, JSON.stringify(payload));
  } catch (err) {
    if (err.statusCode === 410 || err.statusCode === 404) {
      // Subscription expired — remove from DB
      await deleteWebPushSubscription(subscription.endpoint);
    } else {
      throw err;
    }
  }
}

9. How does the email delivery pipeline work?

Email uses SMTP relay providers like SendGrid, AWS SES, Mailgun, or Postmark. Your worker calls their API (HTTP), which handles the actual SMTP delivery, ISP reputation, bounce handling, and compliance.

Key considerations:

• Deliverability: Warm up sending IPs gradually. Never blast a cold IP with 10M emails.
• Bounce handling: Hard bounces (invalid address) must be suppressed immediately. Soft bounces (mailbox full) can be retried.
• Webhooks: Providers send delivery events back to your system via webhook. Store these in your delivery tracking table.
• Unsubscribe links: Legally required (CAN-SPAM, GDPR). The link must invalidate a signed JWT tied to the user and notification type.

import sendgrid
from sendgrid.helpers.mail import Mail

def send_email(to_email: str, subject: str, html_content: str, 
               template_id: str = None, dynamic_data: dict = None):
    sg = sendgrid.SendGridAPIClient(api_key=os.environ['SENDGRID_API_KEY'])
    
    if template_id:
        message = Mail(from_email='noreply@yourapp.com', to_emails=to_email)
        message.template_id = template_id
        message.dynamic_template_data = dynamic_data or {}
    else:
        message = Mail(
            from_email='noreply@yourapp.com',
            to_emails=to_email,
            subject=subject,
            html_content=html_content,
        )
    
    # Custom arg for tracking
    message.custom_arg = {"notification_id": dynamic_data.get("notification_id")}
    
    response = sg.client.mail.send.post(request_body=message.get())
    return response.status_code

10. How does SMS delivery work?

SMS goes through carriers via aggregators. Twilio and Vonage are the two most common. You POST to their HTTP API with the recipient number, sender ID, and message body.

from twilio.rest import Client

def send_sms(to_number: str, body: str, notification_id: str) -> str:
    client = Client(
        os.environ['TWILIO_ACCOUNT_SID'],
        os.environ['TWILIO_AUTH_TOKEN'],
    )
    message = client.messages.create(
        body=body,
        from_=os.environ['TWILIO_FROM_NUMBER'],
        to=to_number,
        status_callback=f"https://api.yourapp.com/webhooks/twilio/{notification_id}",
    )
    return message.sid

SMS is expensive (~$0.0075/message in the US). At 100M users you use it only for transactional/OTP messages. Have a fallback (e.g., send push if user has no verified phone number).

Twilio sends status callbacks (queued, sent, delivered, failed) that you process via webhook to update your delivery tracking.

11. How do in-app notifications work?

In-app notifications are stored in a notifications table and surfaced to the client on demand (pull) or via a real-time channel (push over WebSocket or SSE).

The table:

CREATE TABLE notifications (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    user_id       UUID NOT NULL REFERENCES users(id),
    type          TEXT NOT NULL,            -- 'message', 'alert', 'promo'
    title         TEXT NOT NULL,
    body          TEXT,
    action_url    TEXT,
    metadata      JSONB,
    read_at       TIMESTAMPTZ,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at    TIMESTAMPTZ
);

CREATE INDEX idx_notifications_user_unread
    ON notifications(user_id, created_at DESC)
    WHERE read_at IS NULL;

The client polls or subscribes via WebSocket. When a worker writes the notification row, it also publishes to a Redis pub/sub channel (user:{user_id}:notifications). If the user is connected, they get the notification immediately. If not, they get it on next app open.

Part 4: Fan-Out Patterns

12. What is the fan-out problem in a notification system?

Fan-out happens when one event (e.g., a celebrity posts on a social network) must generate notifications for many recipients (e.g., 10M followers). Naively, this is an O(N) operation at send time. The question is: *when* do you perform that work?

13. What is fan-out on write (push model)?

On every write event, immediately push the notification to each recipient's notification queue or inbox. When the user opens the app, their notifications are already pre-computed.

Pros: Fast read — the client just fetches a pre-built list.

Cons: Write amplification. A celebrity with 10M followers triggers 10M queue inserts synchronously. This can overwhelm the system for large accounts.

14. What is fan-out on read (pull model)?

Store the event once. When a user opens the app, compute their notification list by pulling from a central feed and merging with their follows/subscriptions.

Pros: Write is O(1). No write amplification.

Cons: Read is expensive. Every app open triggers a fan-out computation. Can be slow for users with many follows.

15. Which model should you use and when?

The correct answer is hybrid:

• Fan-out on write for users with < N followers (e.g., < 1M). This covers 99.9% of users.
• Fan-out on read for celebrity/high-follower accounts. Detect these at write time and skip the pre-computation; instead, merge their events at read time.

CELEBRITY_THRESHOLD = 1_000_000

async def handle_new_post(author_id: str, post_id: str):
    follower_count = await get_follower_count(author_id)
    
    if follower_count < CELEBRITY_THRESHOLD:
        # Fan-out on write: enqueue one task per follower chunk
        await enqueue_fanout_job(author_id, post_id, strategy="write")
    else:
        # Fan-out on read: just store the event; merge at read time
        await store_celebrity_event(author_id, post_id)

At read time, the client fetches their pre-built notification list PLUS a merge of any celebrity events from accounts they follow.

16. How do you handle fan-out for a live event with 100M simultaneous users?

This is a spike scenario (World Cup final, product launch). Key tactics:

• Throttle the fan-out workers to avoid overwhelming downstream providers.
• Pre-warm the notification queue hours before the event.
• Batch API calls — FCM supports sending to up to 500 tokens in a single BatchMessage request.
• Segment delivery — deliver in cohorts (10% of users every 2 minutes) to stay within provider rate limits.
• Circuit break — if a provider's error rate spikes above threshold, back off and accumulate in a retry queue.

Part 5: Priority Queues

17. Why do you need priority queues for notifications?

Not all notifications are equal:

| Priority | Example | Latency SLA |

|----------|---------|-------------|

| Critical | OTP, security alert, payment failure | < 2 seconds |

| High | Direct message, order shipped | < 10 seconds |

| Default | Social mention, comment reply | < 60 seconds |

| Low/Batch | Weekly digest, marketing promo | < 5 minutes |

If you use a single queue, a batch of 1M marketing emails can block an OTP for a minute — catastrophic for user trust.

18. How do you implement priority queues in Kafka?

Kafka does not have native message priority. The canonical solution is separate topics per priority tier:

notifications.critical   → 10 partitions, dedicated consumer group (10 workers)
notifications.high       → 20 partitions, dedicated consumer group (20 workers)
notifications.default    → 50 partitions, shared consumer group (50 workers)
notifications.batch      → 20 partitions, shared consumer group (20 workers)

Workers for notifications.critical are always running and lightly loaded. Workers for notifications.batch can be scaled down during off-peak hours.

Alternatively, use SQS with multiple queues and run more EC2/Lambda capacity against the critical queue.

19. How does the Notification Service decide which priority tier to use?

The caller specifies a priority field, or the service derives it from the notification type:

PRIORITY_MAP = {
    "otp":              "critical",
    "security_alert":   "critical",
    "payment_failed":   "high",
    "order_shipped":    "high",
    "new_message":      "high",
    "social_mention":   "default",
    "comment_reply":    "default",
    "weekly_digest":    "batch",
    "promo":            "batch",
}

def resolve_priority(notification_type: str, caller_priority: str = None) -> str:
    # Callers can never escalate above their type's natural priority
    natural = PRIORITY_MAP.get(notification_type, "default")
    if caller_priority and PRIORITY_LEVELS[caller_priority] < PRIORITY_LEVELS[natural]:
        return natural
    return natural

Part 6: Deduplication

20. Why is deduplication necessary?

Producers retry. Networks duplicate packets. At-least-once delivery semantics mean a notification can enter the pipeline multiple times. Without deduplication, a user gets the same OTP 5 times, which is confusing and erodes trust.

21. How do you implement idempotency at the ingestion layer?

Require producers to pass an idempotency_key with every request (typically a UUID or a deterministic hash of the event). Store seen keys in Redis with a TTL:

import redis
import hashlib

r = redis.Redis()
DEDUP_TTL_SECONDS = 86_400  # 24 hours

def is_duplicate(idempotency_key: str) -> bool:
    key = f"notif:dedup:{idempotency_key}"
    # SET NX (only if not exists) returns True if key was newly set
    was_new = r.set(key, "1", nx=True, ex=DEDUP_TTL_SECONDS)
    return was_new is None  # None means key already existed

def generate_idempotency_key(user_id: str, event_type: str, 
                              event_id: str) -> str:
    raw = f"{user_id}:{event_type}:{event_id}"
    return hashlib.sha256(raw.encode()).hexdigest()

If is_duplicate returns True, the API returns 200 OK (not 409) to the caller — idempotent behavior — and discards the event.

22. How do you handle deduplication at the delivery layer?

Even if ingestion is idempotent, worker crashes can cause a message to be consumed twice. Add a second dedup check before calling the provider:

-- Delivery log table
CREATE TABLE notification_delivery_log (
    notification_id UUID NOT NULL,
    channel         TEXT NOT NULL,  -- 'push', 'email', 'sms'
    provider_msg_id TEXT,
    status          TEXT NOT NULL,  -- 'pending', 'sent', 'failed'
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (notification_id, channel)
);

Before calling FCM/SendGrid/Twilio, INSERT with ON CONFLICT DO NOTHING and check the affected row count. If 0 rows inserted, the delivery was already attempted — skip.

INSERT INTO notification_delivery_log (notification_id, channel, status)
VALUES ($1, $2, 'pending')
ON CONFLICT (notification_id, channel) DO NOTHING
RETURNING id;

Part 7: Retry Logic and Delivery Guarantees

23. What delivery guarantee should a notification system provide?

At-least-once delivery is the standard for notifications. Exactly-once is theoretically achievable but operationally very expensive and unnecessary when you have good deduplication (which converts at-least-once into effectively-once from the user's perspective).

At-least-once means: a notification will be delivered eventually, and may be delivered more than once. With the dedup layer in place, the user sees it only once.

24. How do you implement retry logic with exponential backoff?

import asyncio
import random
from dataclasses import dataclass
from typing import Callable, Any

@dataclass
class RetryConfig:
    max_attempts: int = 5
    base_delay_seconds: float = 1.0
    max_delay_seconds: float = 60.0
    jitter: bool = True

async def retry_with_backoff(
    fn: Callable,
    config: RetryConfig = RetryConfig(),
    retryable_exceptions: tuple = (Exception,),
) -> Any:
    last_exception = None
    for attempt in range(config.max_attempts):
        try:
            return await fn()
        except retryable_exceptions as e:
            last_exception = e
            if attempt == config.max_attempts - 1:
                break
            
            delay = min(
                config.base_delay_seconds * (2 ** attempt),
                config.max_delay_seconds,
            )
            if config.jitter:
                delay *= (0.5 + random.random())  # ±50% jitter
            
            await asyncio.sleep(delay)
    
    raise last_exception

Delays with 5 attempts and base 1s: 1s, 2s, 4s, 8s, 16s (plus jitter). Total max wait: ~31 seconds before giving up.

25. What happens when all retries are exhausted?

The message goes to a Dead Letter Queue (DLQ). The DLQ is monitored by an alerting system. An on-call engineer can investigate and manually replay from the DLQ after fixing the underlying issue.

async def process_notification(msg: dict):
    try:
        await retry_with_backoff(
            lambda: deliver_notification(msg),
            config=RetryConfig(max_attempts=5),
            retryable_exceptions=(ProviderTimeoutError, ProviderRateLimitError),
        )
    except Exception as e:
        # Non-retryable (e.g., invalid token) or exhausted retries
        await publish_to_dlq(msg, error=str(e))
        await update_delivery_status(
            msg['notification_id'], msg['channel'], 'dead_lettered'
        )

Non-retryable errors (invalid token, invalid email address) should go to the DLQ immediately without burning retry attempts.

26. How do you distinguish retryable from non-retryable errors?

| Error | Retryable? |

|-------|-----------|

| Provider timeout (5xx) | Yes |

| Rate limit (429) | Yes, with longer backoff |

| Invalid token (FCM 404) | No — delete token |

| Invalid email (400 bad request) | No — suppress address |

| Unsubscribed (SendGrid 400) | No — update preferences |

| Network error | Yes |

| Internal server error | Yes |

class ProviderError(Exception):
    def __init__(self, message: str, status_code: int, retryable: bool):
        super().__init__(message)
        self.status_code = status_code
        self.retryable = retryable

def parse_fcm_error(response: dict) -> ProviderError:
    error_code = response.get('error', {}).get('code', 0)
    details = response.get('error', {}).get('details', [])
    
    for detail in details:
        if detail.get('errorCode') == 'UNREGISTERED':
            return ProviderError("Token unregistered", 404, retryable=False)
        if detail.get('errorCode') == 'QUOTA_EXCEEDED':
            return ProviderError("Quota exceeded", 429, retryable=True)
    
    if error_code >= 500:
        return ProviderError("FCM server error", error_code, retryable=True)
    return ProviderError("Unknown FCM error", error_code, retryable=False)

Part 8: Rate Limiting

27. Why do you need per-user rate limiting?

Without it, a bug in a producer service could spam a user with thousands of notifications in minutes. This destroys trust, gets your app uninstalled, and gets your sending domain/IP flagged by providers.

28. How do you implement per-user rate limiting with a sliding window?

Use Redis with the sliding window log algorithm:

import time
import redis

r = redis.Redis()

def check_rate_limit(
    user_id: str,
    channel: str,
    notification_type: str,
    limit: int,
    window_seconds: int,
) -> bool:
    """Returns True if allowed, False if rate limited."""
    now = time.time()
    window_start = now - window_seconds
    key = f"rate:{user_id}:{channel}:{notification_type}"
    
    pipe = r.pipeline()
    pipe.zremrangebyscore(key, 0, window_start)     # remove old entries
    pipe.zcard(key)                                   # count in window
    pipe.zadd(key, {str(now): now})                  # add current
    pipe.expire(key, window_seconds + 1)
    results = pipe.execute()
    
    count = results[1]
    return count < limit

Define limits per channel and notification type:

RATE_LIMITS = {
    ("push",  "promo"):          (3,  86400),   # 3/day
    ("push",  "social_mention"): (20, 3600),    # 20/hour
    ("sms",   "otp"):            (5,  600),     # 5 per 10 min
    ("email", "promo"):          (1,  86400),   # 1/day
    ("email", "transactional"):  (50, 3600),    # 50/hour
}

29. Should rate limits be enforced before or after the queue?

Before — at ingestion, when the API receives the request. This prevents the queue from filling with messages that will be dropped anyway. You can also enforce a softer check at the worker level as a safety net.

Part 9: User Preferences and Unsubscribe

30. How do you model user notification preferences?

A preferences table with one row per user per notification category per channel:

CREATE TABLE notification_preferences (
    user_id           UUID NOT NULL REFERENCES users(id),
    notification_type TEXT NOT NULL,
    channel           TEXT NOT NULL,  -- 'push', 'email', 'sms', 'in_app'
    enabled           BOOLEAN NOT NULL DEFAULT TRUE,
    updated_at        TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (user_id, notification_type, channel)
);

Before dispatching, the worker queries this table. Cache the result in Redis with a short TTL (60 seconds) to avoid DB hammering during fan-out.

async def get_user_preferences(user_id: str) -> dict:
    cache_key = f"prefs:{user_id}"
    cached = await redis.get(cache_key)
    if cached:
        return json.loads(cached)
    
    rows = await db.fetch(
        "SELECT notification_type, channel, enabled "
        "FROM notification_preferences WHERE user_id = $1",
        user_id
    )
    prefs = {(r['notification_type'], r['channel']): r['enabled'] for r in rows}
    await redis.setex(cache_key, 60, json.dumps(prefs))
    return prefs

async def should_send(user_id: str, notif_type: str, channel: str) -> bool:
    prefs = await get_user_preferences(user_id)
    return prefs.get((notif_type, channel), True)  # default: enabled

31. How do you handle one-click unsubscribe?

One-click unsubscribe (required by Gmail/Yahoo for bulk senders since 2024) means a single GET or POST to a URL immediately unsubscribes the user from that notification type:

from fastapi import FastAPI, HTTPException
import jwt

app = FastAPI()

def generate_unsub_token(user_id: str, notif_type: str, channel: str) -> str:
    payload = {
        "user_id": user_id,
        "notif_type": notif_type,
        "channel": channel,
        "exp": time.time() + 365 * 86400,  # 1 year
    }
    return jwt.encode(payload, os.environ['UNSUB_SECRET'], algorithm='HS256')

@app.post("/unsubscribe")
async def unsubscribe(token: str):
    try:
        payload = jwt.decode(token, os.environ['UNSUB_SECRET'], algorithms=['HS256'])
    except jwt.ExpiredSignatureError:
        raise HTTPException(400, "Token expired")
    except jwt.InvalidTokenError:
        raise HTTPException(400, "Invalid token")
    
    await db.execute(
        """INSERT INTO notification_preferences (user_id, notification_type, channel, enabled)
           VALUES ($1, $2, $3, FALSE)
           ON CONFLICT (user_id, notification_type, channel)
           DO UPDATE SET enabled = FALSE, updated_at = NOW()""",
        payload['user_id'], payload['notif_type'], payload['channel']
    )
    # Invalidate cache
    await redis.delete(f"prefs:{payload['user_id']}")
    return {"status": "unsubscribed"}

Part 10: Notification Templates

32. How do you manage notification templates?

Templates decouple content from code. Store templates in a DB or CMS, render at dispatch time:

CREATE TABLE notification_templates (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    notification_type TEXT NOT NULL,
    channel         TEXT NOT NULL,
    locale          TEXT NOT NULL DEFAULT 'en',
    subject         TEXT,             -- email subject
    title_template  TEXT NOT NULL,    -- Handlebars/Jinja
    body_template   TEXT NOT NULL,
    action_url_template TEXT,
    version         INT NOT NULL DEFAULT 1,
    active          BOOLEAN NOT NULL DEFAULT TRUE,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (notification_type, channel, locale, version)
);

Render at worker time using Jinja2 or Handlebars:

from jinja2 import Template

async def render_notification(
    notif_type: str,
    channel: str,
    locale: str,
    context: dict,
) -> dict:
    template_row = await get_template(notif_type, channel, locale)
    
    title = Template(template_row['title_template']).render(**context)
    body = Template(template_row['body_template']).render(**context)
    action_url = None
    if template_row.get('action_url_template'):
        action_url = Template(template_row['action_url_template']).render(**context)
    
    return {"title": title, "body": body, "action_url": action_url}

Template example for an order shipped email (body_template):

Hi {{ first_name }},

Your order #{{ order_id }} has shipped! 
Estimated delivery: {{ estimated_date }}.

Track your package: {{ tracking_url }}

— The {{ brand_name }} Team

33. How do you handle locale/language selection for templates?

Fall back gracefully:

async def get_template(notif_type: str, channel: str, locale: str) -> dict:
    # Try exact locale (e.g., 'es-AR')
    row = await db.fetchrow(
        "SELECT * FROM notification_templates "
        "WHERE notification_type=$1 AND channel=$2 AND locale=$3 AND active=TRUE "
        "ORDER BY version DESC LIMIT 1",
        notif_type, channel, locale
    )
    if row:
        return row
    
    # Fall back to language only (e.g., 'es')
    lang = locale.split('-')[0]
    row = await db.fetchrow(
        "SELECT * FROM notification_templates "
        "WHERE notification_type=$1 AND channel=$2 AND locale=$3 AND active=TRUE "
        "ORDER BY version DESC LIMIT 1",
        notif_type, channel, lang
    )
    if row:
        return row
    
    # Fall back to English
    return await db.fetchrow(
        "SELECT * FROM notification_templates "
        "WHERE notification_type=$1 AND channel=$2 AND locale='en' AND active=TRUE "
        "ORDER BY version DESC LIMIT 1",
        notif_type, channel
    )

Part 11: Analytics and Delivery Tracking

34. What delivery events should you track?

For every notification and every channel:

| Event | When |

|-------|------|

| created | Notification persisted |

| queued | Published to message queue |

| dispatched | Worker sent to provider |

| delivered | Provider confirmed delivery (webhook) |

| failed | Provider returned non-retryable error |

| dead_lettered | Exhausted retries |

| opened | User tapped notification / opened email |

| clicked | User tapped CTA link |

| unsubscribed | User opted out |

Store these as an append-only event log:

CREATE TABLE notification_events (
    id              BIGSERIAL PRIMARY KEY,
    notification_id UUID NOT NULL,
    user_id         UUID NOT NULL,
    channel         TEXT NOT NULL,
    event_type      TEXT NOT NULL,
    provider        TEXT,            -- 'fcm', 'apns', 'sendgrid', 'twilio'
    provider_msg_id TEXT,
    metadata        JSONB,
    occurred_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_notif_events_notification ON notification_events(notification_id);
CREATE INDEX idx_notif_events_user_time ON notification_events(user_id, occurred_at DESC);

35. How do you track email opens and clicks?

• Opens: Embed a 1×1 transparent tracking pixel with a unique URL: https://t.yourapp.com/open/{notification_id}/{user_id_hash}.png. When the email client loads the image, your server logs the event.
• Clicks: Replace all links with redirect URLs: https://t.yourapp.com/click/{notification_id}/{link_id}. Your redirect service logs the click and forwards the user to the original URL.

Note: iOS Mail Privacy Protection (MPP) pre-fetches all images, inflating open rates. Flag events from known MPP user-agent strings as open_machine vs open_human.

36. What metrics do you expose to the business?

Aggregate in a data warehouse (Redshift, BigQuery, ClickHouse) from the event log:

• Delivery rate: delivered / dispatched per channel per day
• Open rate: opened / delivered
• Click-through rate (CTR): clicked / delivered
• Unsubscribe rate: unsubscribed / delivered
• Provider error rate: failed / dispatched — alerts if above 1%
• p50/p95/p99 latency: from created to delivered
• DLQ volume: number of dead-lettered notifications per hour

Part 12: Common Mistakes and What Interviewers Look For

37. What are the most common mistakes candidates make in this design?

1. 1Single queue for all notification types: Leads to priority inversion. Marketing emails block OTPs.
2. 2Calling providers synchronously from the API: No retry, tight coupling, poor scalability.
3. 3Not handling stale tokens: FCM and APNs return clear signals when tokens expire; ignoring them wastes calls and burns rate limits.
4. 4Forgetting deduplication: At 100M scale, retries and network issues mean you will send duplicates without a dedup layer.
5. 5No rate limiting: A single bug can spam all users. Interviewers specifically probe for this.
6. 6Over-engineering fan-out: Applying fan-out on read universally is as bad as write universally. Hybrid is the answer.
7. 7No DLQ: "What happens when all retries fail?" must have a concrete answer.
8. 8Ignoring compliance: CAN-SPAM requires unsubscribe links. GDPR requires data deletion. Senior candidates mention these.

38. What does a strong candidate do that a mediocre one doesn't?

• Starts with requirements and estimation, not the architecture diagram.
• Quantifies everything: "500M notifications/day → 5,800/sec average → need N partitions to sustain P×5,800 at peak."
• Discusses trade-offs explicitly: "I chose fan-out on write for regular users because reads are more frequent than writes, and writes are cheap at < 1M followers."
• Handles failure modes proactively: token expiry, provider downtime, DLQ draining strategy.
• Knows the third-party APIs: mentioning APNs HTTP/2, FCM batching, SendGrid dynamic templates, Twilio status callbacks signals real-world experience.
• Mentions observability: metrics, alerts, on-call runbooks — this distinguishes senior from mid-level.

39. How would you scale the dispatch workers?

Workers are stateless, so horizontal scaling is straightforward. Scale based on queue depth:

# Kubernetes HPA custom metric via KEDA (Kubernetes Event-Driven Autoscaling)
# keda-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: notification-worker-scaler
spec:
  scaleTargetRef:
    name: notification-worker
  minReplicaCount: 5
  maxReplicaCount: 500
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: kafka:9092
      consumerGroup: notification-workers
      topic: notifications.default
      lagThreshold: "1000"  # scale up if lag > 1000 messages

For notifications.critical, set minReplicaCount high enough to always clear the queue within SLA, regardless of load.

40. How do you handle a provider outage?

If FCM is down:

1. 1Workers receive 503 responses → detected as retryable errors.
2. 2Messages accumulate in the queue (Kafka consumer lag grows).
3. 3KEDA scales up workers (more consumers don't help while FCM is down, but they're ready to burst when it recovers).
4. 4Retries use exponential backoff to avoid hammering the recovering provider.
5. 5Alerting fires: "FCM error rate > 10% for 2 minutes."
6. 6On-call engineer sees the alert. If FCM is degraded long-term, consider falling back to a secondary provider (e.g., direct APNs for iOS while FCM recovers for Android).

41. How do you handle the thundering herd when a provider recovers after outage?

After a provider recovers, all queued messages retry simultaneously. This can overwhelm the provider again. Apply staggered retry jitter (already covered in the backoff code) and add a global rate limiter per provider:

from asyncio import Semaphore

# Per-process semaphore limiting concurrent FCM calls
FCM_CONCURRENCY_LIMIT = 100
fcm_semaphore = Semaphore(FCM_CONCURRENCY_LIMIT)

async def send_fcm_with_limit(token: str, payload: dict):
    async with fcm_semaphore:
        return await send_fcm(token, payload)

Coordinate across worker instances using a Redis-based distributed rate limiter (token bucket or leaky bucket).

42. How would you add multi-tenancy (SaaS model — your service sends notifications on behalf of other companies)?

• Each tenant gets their own API key and sending limits.
• Templates are tenant-scoped.
• Tenants configure their own provider credentials (their own FCM project, SendGrid sender, Twilio number) or use your shared pool.
• Delivery events are scoped by tenant.
• Rate limits are applied both per-user and per-tenant to prevent one noisy tenant from degrading others.
• Separate Kafka topics per tenant tier if strict isolation is needed.

43. How do you ensure data privacy?

• Notification bodies may contain PII. Store only in encrypted columns or use tokenization.
• Logs redact sensitive fields (body, to_address) after a retention period.
• GDPR right to erasure: a delete_user_data(user_id) pipeline removes rows from notifications, notification_events, device_tokens, notification_preferences, and purges Redis caches.
• Audit log of preference changes (who changed what, when).

44. How would you add scheduled notifications?

A scheduler service reads from a scheduled_notifications table and publishes to the notification queue at the right time:

CREATE TABLE scheduled_notifications (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    notification_type TEXT NOT NULL,
    recipient_query JSONB,            -- e.g., {"segment": "free_users"}
    template_context JSONB,
    scheduled_for   TIMESTAMPTZ NOT NULL,
    status          TEXT NOT NULL DEFAULT 'pending',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_scheduled_pending ON scheduled_notifications(scheduled_for)
    WHERE status = 'pending';

The scheduler runs every minute, picks up scheduled_for <= NOW() AND status = 'pending', updates status to processing, resolves the recipients, and fans out to the notification queue.

Use a distributed lock (Redlock) to ensure only one scheduler instance processes each row.

45. What monitoring and alerting would you set up?

Metrics (Prometheus + Grafana):

• notification_queue_lag per topic — alert if lag > threshold for > 2 min
• notification_dispatch_rate per channel
• notification_delivery_rate per provider
• notification_error_rate per provider — alert if > 1%
• notification_dlq_size — alert if > 0

Distributed tracing (OpenTelemetry):

• Trace from API ingestion → queue publish → worker consume → provider call → webhook receipt
• Identify latency hotspots (template rendering? DB query? provider call?)

Alerting (PagerDuty):

• Critical queue lag > 500 messages for > 30 seconds → wake on-call
• Provider error rate > 5% → high-severity alert
• DLQ growth rate > 10/min → investigate

Appendix: Quick Reference Architecture Decisions

| Decision | Choice | Rationale |

|----------|--------|-----------|

| Queue | Kafka | Throughput, replay, partitioning |

| Priority | Separate topics | Kafka lacks native priority |

| Push provider | FCM + APNs | Industry standard |

| Email provider | SendGrid | Deliverability, webhooks, templates |

| SMS provider | Twilio | Coverage, status callbacks |

| Fan-out | Hybrid (< 1M followers: write, >= 1M: read) | Balances write amplification and read latency |

| Dedup | Redis NX + DB INSERT ON CONFLICT | Two-layer protection |

| Retry | Exponential backoff + jitter, 5 attempts | Standard resilience pattern |

| Rate limiting | Redis sliding window | Accurate, low latency |

| Storage | PostgreSQL for preferences/templates, ClickHouse for events | OLTP vs OLAP |

Summary

A production-grade notification system for 100M users requires deliberate decisions at every layer: channel abstraction over FCM/APNs/SendGrid/Twilio, tiered Kafka topics for priority, hybrid fan-out for scalability, a two-layer dedup strategy, exponential backoff with DLQ, Redis-based per-user rate limits, and a full delivery event pipeline for analytics. In an interview, the strongest signal is not memorizing all of this — it is walking through requirements → estimation → design → failure modes → trade-offs in a structured way, showing you understand *why* each piece exists.