System Design Interview: How to Design a Payment System

System Design Interview: How to Design a Payment System

Designing a payment system is one of the most technically demanding prompts in a senior or staff-level system design interview. It touches almost every hard problem in distributed systems simultaneously: consistency, idempotency, fault tolerance, compliance, fraud, and scale. A weak answer gets you a polite rejection. A strong answer gets you the offer — and earns you real respect from the panel.

This guide walks through the entire architecture from first clarifying question to monitoring in production. Every section maps to what an interviewer is actually evaluating. No padding — if a sentence is here, it earns its place.

Part 1 — Clarifying Requirements (Questions 1–9)

Before drawing a single box, you need to understand the system you are building. Interviewers reward candidates who ask sharp, specific questions rather than launching straight into a diagram.

1. What is the primary use case — consumer payments, B2B invoicing, or marketplace payouts?

This shapes the entire model. Consumer payments (think Venmo, PayPal, or a checkout flow) are high-volume, low-value, latency-sensitive, and fraud-prone. B2B invoicing is low-volume, high-value, and reconciliation-heavy. Marketplace payouts add a split-settlement layer (platform fee + seller payout) that requires a ledger with multiple parties per transaction. State this distinction out loud — it shows you understand that "payment system" is a category, not a single design.

2. What is the expected transaction volume? Peak TPS?

Get a number. A system handling 100 TPS has very different infrastructure than one handling 50,000 TPS. Push the interviewer: "Are we designing for something like a mid-sized e-commerce checkout, or are we designing for a Black Friday scenario at Shopify scale?" At 50K TPS, your database write path needs sharding, your Kafka partitions need careful sizing, and your idempotency key store needs to be distributed. At 100 TPS, a single Postgres instance with proper indexing is fine. Know which problem you are solving.

3. What are the latency requirements for the checkout path?

Synchronous checkout flows (card present, online checkout) have a user waiting for a confirmation screen. Every 100ms of added latency hurts conversion. Internal reconciliation jobs can be async and can tolerate minutes of delay. Separate these two SLAs explicitly: P99 < 300ms for the payment initiation API; reconciliation jobs can run in a nightly batch window.

4. Which currencies and geographies must we support?

Multi-currency introduces foreign exchange (FX) rate storage, conversion logic, and regulatory complexity. Storing amounts as integers in the minor currency unit (cents for USD, pence for GBP, sen for JPY — but note JPY has no minor unit, so 1 JPY = 1 unit) avoids floating-point rounding bugs. If the system spans the EU, you need PSD2 Strong Customer Authentication (SCA). If it spans India, you need UPI integration. If it spans Brazil, you need Pix. Knowing this early changes your PSP selection.

5. Who are the actors? Buyers, sellers, platform, banks?

Draw the trust boundary now. A simple checkout has two actors: payer and merchant. A marketplace has three: buyer, seller, and platform. A payroll system has employer and employees. The actor model determines your account structure, your ledger schema, and your payout logic.

6. What consistency guarantee do we need — strong or eventual?

Money is one domain where eventual consistency in the wrong place causes real-world harm (double charges, missing funds). State this explicitly: the core debit/credit operation must be strongly consistent — we cannot debit an account and fail to credit the destination, or vice versa. Reporting, analytics, and reconciliation views can be eventually consistent.

7. Do we need to support refunds, partial captures, and chargebacks?

Each adds state to the transaction lifecycle. A simple system has states: INITIATED → PROCESSING → SETTLED | FAILED. Adding refunds introduces REFUND_REQUESTED → REFUND_PROCESSING → REFUNDED. Partial captures (common in travel — authorize on booking, capture on check-in) require an AUTHORIZED state separate from CAPTURED. Chargebacks introduce a dispute workflow with its own states. Every new state is a new code path and a new source of bugs.

8. What fraud controls are required?

Minimal answer: velocity rules (rate limiting per card, per user, per merchant). Better answer: risk scoring via a rules engine or ML model that gates transactions above a threshold. Best answer: async fraud scoring on the hot path with a risk-based step-up to 3DS2 for high-risk transactions, plus offline model retraining.

9. Do we handle PCI DSS scope, or do we minimize it via tokenization?

Raw card data (PAN, CVV, expiry) is in PCI DSS scope. If your system ever touches raw card numbers, every component in the data flow must be PCI-compliant. The far better answer: tokenize at the browser/mobile layer using a PSP-hosted field (Stripe Elements, Braintree Drop-in), so your backend never sees raw card data. This reduces PCI scope to SAQ A (simplest level). Say this in the interview — it shows security maturity.

Part 2 — High-Level Architecture

After requirements, sketch the macro architecture before zooming in. A payment system at this level has the following components:

Browser / Mobile App
        │
        ▼
   API Gateway  ──── Auth / Rate Limiting
        │
        ▼
 Payment Service  ──── Idempotency Store (Redis)
        │                      │
        ├──► PSP Adapter  (Stripe / Braintree)
        │
        ▼
 Kafka Topic: payment.events
        │
   ┌────┴────────────────────┐
   ▼                         ▼
Ledger Service          Fraud Service
   │                         │
   ▼                         ▼
Postgres (sharded)     Risk DB (Cassandra)
        │
        ▼
Reconciliation Job (nightly)
        │
        ▼
Finance Reporting DB (read replica / data warehouse)

This diagram communicates several design decisions at once: the PSP handles PCI-sensitive card processing; Kafka decouples the synchronous checkout path from async downstream processing; the Ledger Service owns the source of truth for balances; and reconciliation is a separate offline process. Keep this map in your head as you go deeper.

Part 3 — Data Model (Questions 10–16)

10. How do you model a transaction?

The transaction table is the core of the system. Every payment attempt creates one row.

CREATE TABLE transactions (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key VARCHAR(255) NOT NULL UNIQUE,
    payer_account_id UUID NOT NULL REFERENCES accounts(id),
    payee_account_id UUID NOT NULL REFERENCES accounts(id),
    amount          BIGINT NOT NULL,        -- minor currency unit (cents)
    currency        CHAR(3) NOT NULL,       -- ISO 4217
    status          VARCHAR(32) NOT NULL,   -- INITIATED, PROCESSING, SETTLED, FAILED, REFUNDED
    psp_reference   VARCHAR(255),           -- PSP transaction ID
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    metadata        JSONB                   -- merchant order ID, customer IP, etc.
);

CREATE INDEX idx_transactions_payer    ON transactions(payer_account_id, created_at DESC);
CREATE INDEX idx_transactions_status   ON transactions(status) WHERE status NOT IN ('SETTLED','FAILED');
CREATE INDEX idx_transactions_idempotency ON transactions(idempotency_key);

Note the use of BIGINT for amount. Never use FLOAT or DECIMAL for money in application code — floating-point math will eventually produce rounding errors. Store cents as integers.

11. How do you model accounts?

CREATE TABLE accounts (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    owner_id        UUID NOT NULL,          -- user or merchant ID
    account_type    VARCHAR(32) NOT NULL,   -- CONSUMER, MERCHANT, PLATFORM, ESCROW
    currency        CHAR(3) NOT NULL,
    balance         BIGINT NOT NULL DEFAULT 0,
    available_balance BIGINT NOT NULL DEFAULT 0,  -- balance minus holds
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

available_balance is important: when a payment is authorized, you place a hold (reduce available_balance without touching balance). When it settles, you reduce balance. When it's released or voided, you restore available_balance. This prevents users from spending funds that are in-flight.

12. What is double-entry bookkeeping and why does it matter?

Double-entry bookkeeping is the accounting principle that every financial transaction has equal debits and credits. In a payment system, every transfer from account A to account B generates two ledger entries: a debit on A and a credit on B. The sum of all ledger entries across all accounts must always equal zero. This is your audit trail and your consistency check.

CREATE TABLE ledger_entries (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    transaction_id  UUID NOT NULL REFERENCES transactions(id),
    account_id      UUID NOT NULL REFERENCES accounts(id),
    entry_type      VARCHAR(8) NOT NULL,   -- DEBIT or CREDIT
    amount          BIGINT NOT NULL,       -- always positive
    currency        CHAR(3) NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

A $10.00 payment creates exactly two rows:

• DEBIT $10.00 from payer account
• CREDIT $10.00 to payee account

A platform charging a 2.9% fee creates four rows (debit payer, credit payee, debit platform fee from payee, credit platform revenue account). Every dollar is always accounted for.

13. How do you enforce that ledger entries always balance?

Use a database check constraint or a trigger. More robustly, wrap the entire ledger write in a database transaction and assert the invariant in application code before committing:

def post_payment(session, transaction_id, payer_id, payee_id, amount, currency):
    entries = [
        LedgerEntry(transaction_id=transaction_id, account_id=payer_id,
                    entry_type='DEBIT', amount=amount, currency=currency),
        LedgerEntry(transaction_id=transaction_id, account_id=payee_id,
                    entry_type='CREDIT', amount=amount, currency=currency),
    ]
    total = sum(e.amount if e.entry_type == 'CREDIT' else -e.amount for e in entries)
    assert total == 0, f"Ledger imbalance: {total}"  # hard fail before commit
    session.add_all(entries)
    session.commit()

Run a nightly job that queries SUM(amount) WHERE entry_type='DEBIT' and SUM(amount) WHERE entry_type='CREDIT' across all entries for the day and alerts on any discrepancy. This is reconciliation at the ledger level.

14. How do you model the transaction state machine?

INITIATED
    │
    ├──► PSP call fails immediately ──► FAILED
    │
    ▼
PROCESSING (PSP call in-flight or awaiting webhook)
    │
    ├──► PSP declines ──► FAILED
    ├──► PSP timeout  ──► PENDING_VERIFICATION (query PSP for status)
    │
    ▼
AUTHORIZED (for card-present or hotel-style flows)
    │
    ├──► Void ──► VOIDED
    ▼
CAPTURED
    │
    ▼
SETTLED
    │
    ├──► Refund requested ──► REFUND_PROCESSING ──► REFUNDED
    └──► Dispute ──► CHARGEBACK_REVIEW ──► CHARGEBACK_LOST | CHARGEBACK_WON

Every state transition writes a row to an audit_log table with (transaction_id, from_state, to_state, actor, timestamp, reason). This is mandatory for PCI DSS and for debugging production incidents.

15. How do you handle multi-currency?

Store the original currency and amount from the customer's perspective, the settlement currency and amount, and the FX rate used at the time of conversion. Never recompute FX rates retroactively — they change, and retroactive recomputation will break reconciliation.

ALTER TABLE transactions ADD COLUMN
    settlement_currency  CHAR(3),
    settlement_amount    BIGINT,
    fx_rate              NUMERIC(18,8),  -- rate at transaction time
    fx_rate_source       VARCHAR(64);    -- e.g. 'ECB_2024_01_15'

16. How do you index for common query patterns?

The most frequent queries in a payment system are:

1. 1Fetch transaction by ID (primary key — trivial)
2. 2Fetch all transactions for a user in a date range (index on payer_account_id, created_at DESC)
3. 3Find all PROCESSING transactions older than 5 minutes (partial index on status for non-terminal states)
4. 4Reconciliation: sum of settlements in a date range by currency (partial index + materialized view for reporting)

Avoid full table scans in reconciliation jobs by always bounding by created_at and partitioning the table by month.

Part 4 — Idempotency (Questions 17–20)

17. What is idempotency and why is it non-negotiable in payments?

Idempotency means that performing the same operation multiple times has the same effect as performing it once. In payments, without idempotency, a network timeout on the client side causes a retry, and you charge the customer twice. This is catastrophic. Every payment initiation endpoint must be idempotent.

The mechanism: the client generates a unique idempotency_key (a UUID or ULID) and sends it as a request header. The server stores the key with its response. If the same key arrives again (within a TTL, typically 24 hours), the server returns the stored response without re-executing the operation.

18. How do you implement idempotency keys correctly?

import redis
import json
from uuid import UUID

IDEMPOTENCY_TTL_SECONDS = 86400  # 24 hours

def process_payment_idempotent(idempotency_key: str, payment_request: dict) -> dict:
    r = redis.Redis()
    cache_key = f"idempotency:{idempotency_key}"

    # Check if we already processed this request
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    # Use a distributed lock to prevent concurrent execution of the same key
    lock_key = f"idempotency_lock:{idempotency_key}"
    with r.lock(lock_key, timeout=10, blocking_timeout=5):
        # Re-check after acquiring lock (double-checked locking)
        cached = r.get(cache_key)
        if cached:
            return json.loads(cached)

        # Execute the payment
        result = execute_payment(payment_request)

        # Store result with TTL
        r.setex(cache_key, IDEMPOTENCY_TTL_SECONDS, json.dumps(result))
        return result

The double-checked locking pattern prevents two concurrent requests with the same idempotency key from both proceeding past the initial cache check.

19. What happens when the same idempotency key is sent with different request parameters?

Return a 422 Unprocessable Entity with a clear error: "An idempotency key cannot be reused with different request parameters." Log the attempt — it is either a client bug or a potential attack. Do not silently ignore the mismatch.

20. Should idempotency keys live in Redis or in the database?

Both, at different layers. Redis gives you fast, low-latency deduplication on the hot path. The database gives you durability. The pattern: check Redis first (fast path); if the key is not in Redis but is in the database (key expired from Redis but we still have the record), reconstruct the response from the database and re-populate Redis.

CREATE TABLE idempotency_keys (
    key             VARCHAR(255) PRIMARY KEY,
    response_status INT NOT NULL,
    response_body   JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at      TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '24 hours'
);
CREATE INDEX idx_idempotency_keys_expires ON idempotency_keys(expires_at);
-- Run a nightly DELETE WHERE expires_at < now()

Part 5 — PSP Integration (Questions 21–25)

21. How do you integrate with Stripe or Braintree without taking on PCI scope?

Use the PSP's client-side tokenization SDK. The flow:

1. 1Your frontend loads Stripe.js or Braintree's Drop-in UI.
2. 2The user enters card details into an iframe hosted by Stripe/Braintree — your JavaScript never touches raw card data.
3. 3The SDK returns a single-use token (e.g., tok_xxx for Stripe, nonce_xxx for Braintree).
4. 4Your frontend sends this token to your backend.
5. 5Your backend calls the PSP API with the token to create a charge or payment method.

Your backend never sees the card number. You are outside PCI DSS scope for card data storage and transmission.

22. How do you handle PSP webhooks reliably?

PSPs send webhook events for async operations (payment settled, refund processed, dispute opened). Your webhook handler must be:

1. 1Idempotent: PSPs guarantee at-least-once delivery. You will receive the same event multiple times.
2. 2Fast: Acknowledge with 200 OK immediately. Process asynchronously.
3. 3Verified: Check the webhook signature (Stripe uses Stripe-Signature header with HMAC-SHA256) before processing.

import stripe
from flask import request, abort

@app.route('/webhooks/stripe', methods=['POST'])
def stripe_webhook():
    payload = request.data
    sig_header = request.headers.get('Stripe-Signature')
    
    try:
        event = stripe.Webhook.construct_event(
            payload, sig_header, STRIPE_WEBHOOK_SECRET
        )
    except (ValueError, stripe.error.SignatureVerificationError):
        abort(400)
    
    # Acknowledge immediately
    # Enqueue for processing — do NOT process synchronously here
    kafka_producer.produce('stripe.events', key=event['id'], value=json.dumps(event))
    return '', 200

The Kafka enqueue is the critical part. If you process the webhook synchronously and your database is slow or down, you will return a non-200 and Stripe will retry — causing duplicates. By acknowledging immediately and processing async, you decouple receipt from processing.

23. How do you handle PSP-specific error codes?

Map PSP error codes to your internal domain errors. Do not leak PSP error strings to end users.

STRIPE_DECLINE_CODES = {
    'card_declined':          ('CARD_DECLINED', 'Your card was declined.', True),  # (internal_code, user_message, retriable)
    'insufficient_funds':     ('INSUFFICIENT_FUNDS', 'Your card has insufficient funds.', True),
    'expired_card':           ('EXPIRED_CARD', 'Your card has expired.', False),
    'incorrect_cvc':          ('INVALID_CVC', 'Your card security code is incorrect.', False),
    'card_velocity_exceeded': ('VELOCITY_EXCEEDED', 'Too many attempts. Try again later.', True),
    'do_not_honor':           ('DO_NOT_HONOR', 'Your card was declined.', True),
}

def map_stripe_error(err) -> PaymentError:
    code, message, retriable = STRIPE_DECLINE_CODES.get(
        err.decline_code or err.code,
        ('UNKNOWN_ERROR', 'Payment failed. Please try another method.', False)
    )
    return PaymentError(code=code, user_message=message, retriable=retriable)

The retriable flag tells the frontend whether to offer a retry or prompt the user to try a different card.

24. How do you support multiple PSPs for redundancy or regional optimization?

Build a PSP Adapter interface and implement it for each provider. A router layer selects the PSP based on rules (currency, card BIN, geography, failover):

from abc import ABC, abstractmethod

class PSPAdapter(ABC):
    @abstractmethod
    def charge(self, amount: int, currency: str, token: str, idempotency_key: str) -> ChargeResult:
        pass

    @abstractmethod
    def refund(self, psp_reference: str, amount: int) -> RefundResult:
        pass

class StripeAdapter(PSPAdapter):
    def charge(self, amount, currency, token, idempotency_key):
        return stripe.PaymentIntent.create(
            amount=amount,
            currency=currency,
            payment_method=token,
            confirm=True,
            idempotency_key=idempotency_key,
        )

class BraintreeAdapter(PSPAdapter):
    def charge(self, amount, currency, token, idempotency_key):
        result = self.gateway.transaction.sale({
            'amount': str(amount / 100),
            'payment_method_nonce': token,
            'options': {'submit_for_settlement': True},
        })
        return result

class PSPRouter:
    def select(self, currency: str, amount: int, country: str) -> PSPAdapter:
        if currency == 'BRL':
            return self.braintree  # better Brazil coverage
        if amount > 100_000_00:  # >$100K: use primary PSP with better limits
            return self.stripe
        return self.stripe  # default

If your primary PSP returns a 5xx, your router can fall back to the secondary PSP. The idempotency key prevents double-charging if you retry across PSPs.

25. How do you handle PSP downtime?

Two strategies, not mutually exclusive:

1. 1Fast-fail with retry later: Return an error to the user, persist the payment in INITIATED state, and retry via a background job when the PSP recovers. Works for non-urgent flows.
2. 2PSP failover: Route to a secondary PSP immediately. Requires pre-established contracts with multiple PSPs and careful testing to ensure the fallback works end-to-end.

For consumer checkout, fast-fail hurts conversion. For subscription billing, retrying later is acceptable. Know which flow you are designing for.

Part 6 — Async Processing with Kafka (Questions 26–29)

26. Why use Kafka for payment event processing?

The synchronous checkout path must be fast and reliable. Downstream operations — fraud scoring, ledger updates, notifications, analytics, reconciliation data ingestion — are secondary and should not block the user's response. Kafka decouples these concerns.

Kafka also gives you:

• Replayability: If your fraud model is updated, you can replay historical payment events.
• Fan-out: Multiple consumers (ledger service, notification service, analytics) can consume the same event independently.
• Backpressure absorption: A spike in payments does not crash your downstream services — it builds queue depth, and services drain at their own pace.

27. How do you design your Kafka topics for payments?

Use separate topics by domain, not one mega-topic:

| Topic | Producers | Consumers |

|---|---|---|

| payment.initiated | Payment Service | Fraud Service, Ledger Service |

| payment.psp_result | PSP Webhook Handler | Payment Service (state machine) |

| payment.settled | Payment Service | Notification Service, Analytics |

| payment.failed | Payment Service | Notification Service, Retry Scheduler |

| ledger.entries | Ledger Service | Reconciliation Job, Finance Reporting |

Partition payment.initiated by payer_account_id to ensure ordering for a given user (prevents race conditions when a user submits two payments simultaneously).

28. How do you guarantee exactly-once processing in Kafka consumers?

Kafka's consumer model gives you at-least-once delivery by default — your consumer may process the same message twice if it commits the offset before processing completes (or if the consumer crashes between processing and committing).

For payments, you cannot process the same payment event twice. Solution: idempotent consumers.

def consume_payment_settled(message):
    event = json.loads(message.value())
    transaction_id = event['transaction_id']
    
    with db.transaction():
        # Check if we already processed this event
        already_processed = db.query(
            "SELECT 1 FROM processed_events WHERE event_id = %s",
            (message.key(),)
        ).fetchone()
        
        if already_processed:
            return  # idempotent: skip
        
        # Perform the ledger write
        post_payment_to_ledger(transaction_id, event['amount'], event['currency'])
        
        # Mark as processed within the same transaction
        db.execute(
            "INSERT INTO processed_events(event_id, processed_at) VALUES (%s, now())",
            (message.key(),)
        )
    
    # Commit Kafka offset only after successful DB commit
    consumer.commit()

The processed_events check inside the same database transaction as the ledger write makes the entire operation idempotent and atomic.

29. How do you handle consumer lag — what if Kafka consumers fall behind?

Monitor consumer group lag as a primary SLI. If lag grows:

1. 1Scale out consumer instances (Kafka consumers within a group scale horizontally up to the number of partitions).
2. 2Increase partition count (requires careful rebalancing and offset management).
3. 3Identify slow consumers — profile the DB write path, look for N+1 queries, add connection pooling.

Alert on consumer lag > 10,000 messages. Page if lag exceeds 5 minutes of throughput.

Part 7 — Failure Handling and Retries (Questions 30–33)

30. How do you implement exponential backoff with jitter for PSP retries?

Naive retry with fixed intervals creates thundering herd problems — all retried requests hit the PSP at the same instant after a timeout. Exponential backoff spreads them out. Jitter randomizes them further.

import random
import time

def retry_with_backoff(func, max_attempts=5, base_delay=1.0, max_delay=60.0):
    for attempt in range(max_attempts):
        try:
            return func()
        except RetriableError as e:
            if attempt == max_attempts - 1:
                raise  # exhausted retries
            
            # Full jitter: random value between 0 and the capped exponential
            delay = min(max_delay, base_delay * (2 ** attempt))
            jitter = random.uniform(0, delay)
            
            print(f"Attempt {attempt + 1} failed: {e}. Retrying in {jitter:.1f}s")
            time.sleep(jitter)

Retry only on retriable errors (network timeouts, 5xx from PSP). Never retry on 4xx errors like card_declined or invalid_card_number — those will not succeed on retry.

31. What is the dead-letter queue pattern and when do you use it?

A dead-letter queue (DLQ) is a Kafka topic or queue where messages are sent after exhausting all retry attempts. It gives you:

• A persistent record of failed operations for investigation
• A mechanism to replay the message after the underlying issue is fixed
• Alerting: high DLQ depth signals a systemic problem

def process_with_dlq(message, processor, dlq_producer, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            processor(message)
            return
        except Exception as e:
            if attempt < max_attempts - 1:
                time.sleep(2 ** attempt)
            else:
                # Send to DLQ with error context
                dlq_producer.produce(
                    topic='payment.dlq',
                    key=message.key(),
                    value=json.dumps({
                        'original_message': message.value(),
                        'error': str(e),
                        'failed_at': datetime.utcnow().isoformat(),
                        'attempts': max_attempts,
                    })
                )

Operationally, have a runbook for DLQ messages. They need human investigation — automated replay without understanding the failure mode can make things worse.

32. How do you handle a split-brain scenario — PSP says success, your DB write fails?

This is the canonical distributed systems failure in payments. The PSP charged the card, but your transaction record did not get written (DB crash, network partition). The user now has money taken but no record in your system.

Solution: the transaction must exist in your database (in PROCESSING state) before you call the PSP. Write the record first, then call the PSP, then update the status. This is the "write-ahead log" pattern.

If the DB write succeeds but the status update fails after the PSP confirms, your reconciliation job (which queries the PSP for all settled transactions) will detect the discrepancy and update the status retroactively.

def initiate_payment(request):
    # 1. Write transaction in PROCESSING state FIRST
    txn = Transaction(
        idempotency_key=request.idempotency_key,
        amount=request.amount,
        status='PROCESSING',
    )
    db.save(txn)
    db.commit()  # committed before PSP call
    
    try:
        # 2. Call PSP
        psp_result = psp_adapter.charge(
            amount=request.amount,
            token=request.payment_token,
            idempotency_key=request.idempotency_key,
        )
        # 3. Update status
        txn.status = 'SETTLED'
        txn.psp_reference = psp_result.id
        db.commit()
        return txn
    except PSPError as e:
        txn.status = 'FAILED'
        txn.failure_reason = str(e)
        db.commit()
        raise

If the process crashes between step 2 and step 3, the reconciliation job finds a PROCESSING transaction older than 10 minutes and queries the PSP: did this idempotency key succeed? If yes, mark it SETTLED. If no, mark it FAILED.

33. How do you handle timeout — you sent a request to the PSP but never got a response?

Never interpret a timeout as a failure. The PSP may have processed the charge and the response was lost. The correct state is PENDING_VERIFICATION. A background job queries the PSP's retrieval API using your idempotency key to determine the true outcome.

This is why PSP idempotency keys are not just a nice-to-have — they are the recovery mechanism for timeouts.

Part 8 — Reconciliation (Questions 34–36)

34. What is reconciliation and how do you implement it?

Reconciliation is the process of verifying that your internal records match the PSP's records match your bank records. It runs daily (or more frequently for high-volume systems) and catches:

• Transactions settled by the PSP but not updated in your system
• Refunds processed by the PSP but not recorded internally
• Fees charged by the PSP not accounted for in your ledger

The process:

1. 1Download the PSP settlement report for the previous day (Stripe and Braintree both expose this via API).
2. 2For each PSP transaction, find the matching internal transaction by psp_reference.
3. 3Assert that amounts, currencies, and statuses match.
4. 4For discrepancies, create a reconciliation_discrepancy record and alert the finance team.

def run_reconciliation(date: date):
    psp_settlements = stripe.BalanceTransaction.list(
        created={'gte': date.timestamp(), 'lt': (date + timedelta(days=1)).timestamp()},
        type='charge',
        limit=100,
    ).auto_paging_iter()
    
    discrepancies = []
    for psp_txn in psp_settlements:
        internal_txn = db.query(Transaction).filter_by(
            psp_reference=psp_txn.id
        ).first()
        
        if not internal_txn:
            discrepancies.append({
                'type': 'MISSING_INTERNAL',
                'psp_reference': psp_txn.id,
                'amount': psp_txn.amount,
            })
            continue
        
        if internal_txn.amount != psp_txn.amount:
            discrepancies.append({
                'type': 'AMOUNT_MISMATCH',
                'psp_reference': psp_txn.id,
                'internal_amount': internal_txn.amount,
                'psp_amount': psp_txn.amount,
            })
    
    if discrepancies:
        alert_finance_team(discrepancies)
    
    return discrepancies

35. How do you reconcile across time zones?

PSPs settle in UTC. Your customers may be in Tokyo or São Paulo. Ensure your settlement date logic normalizes to UTC before comparing. Store all timestamps in UTC (TIMESTAMPTZ in Postgres, not TIMESTAMP). Never store local time without offset.

36. What is the difference between settlement and payout?

Settlement is when the PSP confirms that funds have transferred from the card network to the PSP. Payout is when the PSP transfers funds from their account to your bank account. These happen on different schedules (Stripe settles in 2 days for US card payments, then batches payouts weekly or daily depending on your configuration). Your reconciliation must account for both steps — many systems only reconcile to settlement and miss the payout layer.

Part 9 — PCI DSS and Compliance (Questions 37–39)

37. What are the key PCI DSS requirements an architect must know?

PCI DSS has 12 requirements. The ones most relevant to an architect:

• Req 1-2: Network segmentation — cardholder data environment (CDE) must be isolated from general corporate network.
• Req 3: Do not store sensitive authentication data (CVV, full magnetic stripe). You may store the truncated PAN (last 4 digits) and expiry date.
• Req 4: Encrypt cardholder data in transit (TLS 1.2 minimum, TLS 1.3 preferred).
• Req 6: Patch management and secure SDLC — no known vulnerabilities in the payment code path.
• Req 7-8: Access control and MFA — no shared credentials, least privilege, MFA for all administrative access.
• Req 10: Audit logging of all access to cardholder data — write-once logs (append-only S3 bucket with object lock).
• Req 12: Incident response plan documented and tested.

The practical win: use PSP-hosted fields (Stripe Elements, Braintree Drop-in). Your backend never touches raw card data. Your PCI scope drops from SAQ D (most complex, requires annual audit) to SAQ A (self-assessment, no auditor needed).

38. What other compliance requirements matter beyond PCI DSS?

• GDPR / CCPA: Customer payment records are personal data. They must be deletable on request (or anonymized — you cannot fully delete transactions for accounting law, but you can anonymize the cardholder's PII while retaining the transaction amount and date).
• AML (Anti-Money Laundering): For amounts above $10,000 (US) or equivalent, you must file a Currency Transaction Report. Flag high-value transactions for compliance review.
• KYC (Know Your Customer): For wallet-style products or marketplace payouts, you must verify the identity of senders and recipients. Integrate with a KYC provider (Stripe Identity, Jumio, Persona).
• SOX (Sarbanes-Oxley): For public companies, your audit trail and reconciliation records must be retained for 7 years with tamper-proof logging.

39. How do you design tamper-proof audit logs?

Write audit events to a separate append-only data store:

• An S3 bucket with Object Lock in Compliance Mode (not even the root account can delete objects before the retention period expires)
• A separate Postgres schema with a trigger that prevents UPDATE and DELETE on the audit table
• A blockchain-style hash chain: each audit record contains the SHA-256 hash of the previous record, so tampering with any historical record breaks the chain

CREATE TABLE payment_audit_log (
    id              BIGSERIAL PRIMARY KEY,
    occurred_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    actor_type      VARCHAR(32) NOT NULL,  -- USER, SYSTEM, ADMIN
    actor_id        UUID NOT NULL,
    transaction_id  UUID REFERENCES transactions(id),
    action          VARCHAR(64) NOT NULL,
    old_state       JSONB,
    new_state       JSONB,
    ip_address      INET,
    prev_hash       CHAR(64),  -- SHA-256 of previous row
    this_hash       CHAR(64)   -- SHA-256 of this row's content
);

-- Trigger prevents modification of existing rows
CREATE RULE no_update_audit AS ON UPDATE TO payment_audit_log DO INSTEAD NOTHING;
CREATE RULE no_delete_audit AS ON DELETE TO payment_audit_log DO INSTEAD NOTHING;

Part 10 — Fraud Detection (Questions 40–42)

40. How do you design a fraud scoring system?

Fraud detection has two components: a rules engine (fast, deterministic, runs synchronously on the hot path) and an ML model (accurate, async, informs future decisions):

Rules engine (synchronous, < 5ms):

• Velocity rules: > 3 payment attempts per card per hour → block
• Amount threshold: > $5,000 for a new account → flag for review
• Geographic mismatch: billing address country != IP country → increase risk score
• Card BIN checks: prepaid cards or known fraud BIN ranges → flag

ML model (asynchronous, scores retroactively or gates high-risk transactions):

• Features: user tenure, transaction history, device fingerprint, behavioral biometrics
• Score threshold: if risk score > 0.7, step up to 3DS2 authentication; if > 0.9, auto-decline
• Model retrained weekly on labeled data (chargebacks = fraud labels)

41. What is 3DS2 and when do you trigger it?

3DS2 (3D Secure version 2) is an authentication protocol that shifts fraud liability from the merchant to the card issuer when used. The user sees an additional verification step (OTP, biometric, bank app notification). It adds ~10-15% drop-off in checkout conversion, so you do not want to trigger it for every transaction.

Trigger 3DS2 when:

• Risk score exceeds threshold
• Transaction above a configurable amount threshold (e.g., > $500)
• New device or new billing address
• EU regulation requires it (PSD2 SCA mandate)

Stripe Radar and Braintree's Advanced Fraud Tools handle this automatically if you configure them correctly.

42. How do you handle chargebacks?

A chargeback is when a customer disputes a charge with their bank and the bank reverses it. Your flow:

1. 1Receive chargeback webhook from PSP
2. 2Update transaction status to CHARGEBACK_REVIEW
3. 3Notify the merchant (they need to provide evidence)
4. 4Debit the merchant's account for the chargeback amount + fee (typically $15–$25 per dispute)
5. 5Submit evidence (order records, delivery confirmation, IP logs) to PSP within the deadline (usually 7–10 days)
6. 6Receive PSP verdict: CHARGEBACK_WON (funds returned) or CHARGEBACK_LOST (debit stands)

Chargeback rate > 1% of transactions puts you in Visa/Mastercard's monitoring programs. Track it as a primary business metric.

Part 11 — Monitoring and Scaling (Questions 43–47)

43. What are the key metrics to monitor for a payment system?

Business metrics (alert on thresholds):

• Authorization rate: successful charges / total charge attempts — baseline ~85-92%, drops signal PSP issues or fraud spike
• Chargeback rate: chargebacks / settled transactions — target < 0.5%
• Refund rate: refunds / settled — tracks product quality
• Payment success latency P50/P95/P99

Infrastructure metrics:

• Kafka consumer lag per topic
• Postgres replication lag (if using read replicas for reconciliation)
• Redis hit rate for idempotency store
• PSP API error rate by error code
• DLQ depth

# Prometheus alert rules (pseudocode)
- alert: PaymentAuthRateDrop
  expr: rate(payments_authorized_total[5m]) / rate(payments_attempted_total[5m]) < 0.75
  for: 2m
  annotations:
    summary: "Payment authorization rate below 75% for 2 minutes"

- alert: KafkaConsumerLagHigh
  expr: kafka_consumer_group_lag{topic="payment.settled"} > 10000
  for: 5m
  annotations:
    summary: "Ledger service falling behind on payment.settled topic"

44. How do you trace a payment across multiple services?

Use distributed tracing (OpenTelemetry with Jaeger or Datadog APM). Propagate a trace_id from the initial API request through Kafka messages and across service boundaries. Every log entry for a payment should include the transaction_id and trace_id.

from opentelemetry import trace
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

tracer = trace.get_tracer(__name__)

def initiate_payment(request):
    with tracer.start_as_current_span("payment.initiate") as span:
        span.set_attribute("payment.transaction_id", str(txn.id))
        span.set_attribute("payment.amount", request.amount)
        span.set_attribute("payment.currency", request.currency)
        
        # Propagate trace context into Kafka message headers
        headers = {}
        TraceContextTextMapPropagator().inject(headers)
        kafka_producer.produce('payment.initiated', headers=headers, value=payload)

This lets you reconstruct the complete journey of a payment — from API call to PSP response to ledger write to notification — in a single trace view.

45. How do you scale the database horizontally?

The transactions table is the write bottleneck. Strategies in order of complexity:

1. 1Read replicas: Move reconciliation queries and reporting to read replicas. The primary handles writes only.
2. 2Table partitioning by date: Partition the transactions table by month. Queries bounded by date stay within a single partition. Old partitions can be archived to cheaper storage.
3. 3Sharding by account ID: Shard the transactions table across multiple database instances. Route queries using a consistent hash on payer_account_id. Complex: cross-shard queries (e.g., reconciliation across all accounts) require a scatter-gather pattern.
4. 4CQRS (Command Query Responsibility Segregation): Separate write model (Postgres, strongly consistent) from read model (Elasticsearch or Redshift for analytics). The Ledger Service writes to Postgres; a CDC pipeline (Debezium) streams changes to the analytics store.

For most systems, read replicas + table partitioning handles 10K TPS comfortably. Sharding is only necessary beyond that scale.

46. How do you design for zero-downtime deployments in a payment system?

Payment systems cannot tolerate downtime. Key practices:

• Database migrations must be backward-compatible: Never drop a column or rename a table in a single deployment. Use the expand-contract pattern: add new column (expand), deploy code that reads both old and new columns, backfill data, deploy code that reads only new column, drop old column (contract).
• Feature flags: Gate new payment flows behind feature flags. Roll out to 1% of traffic, then 10%, then 100%. Roll back instantly by toggling the flag.
• Blue-green deployments: Run two identical production environments. Route traffic to the new version, keep the old running for instant rollback.
• Graceful shutdown: When a pod receives a SIGTERM, stop accepting new requests but finish processing in-flight payments before exiting. Set Kubernetes terminationGracePeriodSeconds to 60–120 seconds for payment services.

47. What are the most common mistakes candidates make in this interview?

1. 1Jumping to solutions before clarifying requirements. The interviewer may be testing a B2B invoicing system, not Stripe-scale consumer payments. Ask first.
2. 2Using floating-point for money. Always BIGINT for minor currency units.
3. 3Forgetting idempotency. Every payment API must be idempotent. State this proactively.
4. 4Synchronous PSP calls in the critical path without timeout handling. Always set a timeout on PSP calls and handle the timeout as PENDING_VERIFICATION, not as failure.
5. 5Ignoring reconciliation. Candidates design the write path beautifully and forget that payments need to be verified against PSP and bank records.
6. 6Treating consistency as binary. Not all data in a payment system needs to be strongly consistent. Analytics views, notification delivery, and reporting can be eventually consistent. Knowing where to draw this line shows architecture maturity.
7. 7No mention of PCI DSS. For a senior interview, not mentioning PCI DSS signals that you have not worked in payments. Mention tokenization and PCI scope reduction proactively.

Architecture Summary Diagram

Here is the complete architecture in ASCII, suitable for drawing on a whiteboard:

┌─────────────────────────────────────────────────────────────┐
│                     CLIENT LAYER                            │
│  Browser (Stripe.js) │ Mobile SDK │ Server-to-Server API   │
└────────────┬────────────────────────────────────────────────┘
             │ HTTPS / TLS 1.3
             ▼
┌────────────────────────────────────────────────────────────┐
│  API GATEWAY  (Auth, Rate Limiting, WAF)                   │
└──────────────────────┬─────────────────────────────────────┘
                       │
             ┌─────────▼──────────┐
             │  PAYMENT SERVICE   │◄─── Redis (idempotency keys)
             │  (idempotent API)  │
             └─────────┬──────────┘
                       │                   ┌──────────────┐
              ┌────────▼────────┐          │  PSP ADAPTER │
              │  Transaction DB │          │  (Stripe /   │
              │  (Postgres)     │◄────────►│  Braintree)  │
              └────────┬────────┘          └──────┬───────┘
                       │                          │ webhooks
             ┌─────────▼──────────────────────────▼────────┐
             │            KAFKA CLUSTER                     │
             │  payment.initiated │ payment.psp_result      │
             │  payment.settled   │ payment.failed          │
             └──────┬──────┬──────┬────────────────────────┘
                    │      │      │
        ┌───────────▼┐  ┌──▼───┐  ▼─────────────────┐
        │  LEDGER    │  │FRAUD │  NOTIFICATION       │
        │  SERVICE   │  │SCORE │  SERVICE            │
        └───────────┬┘  └──────┘  └────────────────  ┘
                    │
          ┌─────────▼──────────┐
          │  LEDGER DB         │
          │  (double-entry)    │──► Reconciliation Job (nightly)
          └────────────────────┘         │
                                         ▼
                                   Finance Warehouse
                                   (Redshift / BigQuery)

What Interviewers Are Actually Evaluating

At senior level, the interviewer is not checking whether you memorized a payment architecture. They are evaluating:

• Prioritization: Do you know which problems are load-bearing versus nice-to-have?
• Failure thinking: Do you ask "what happens when X fails?" unprompted?
• Compliance awareness: Do you know that card data has regulatory requirements?
• Communication: Can you explain a complex distributed system clearly, without jargon soup?
• Trade-off articulation: When you choose eventual consistency for analytics, do you explain why?

The best candidates treat the interview as a design collaboration, not a performance. They use phrases like "this approach has a downside — we would need to..." and "in practice I would validate this assumption with the product team before committing." That is senior-level thinking.

Quick Reference: Key Numbers to Know

| Metric | Typical value |

|---|---|

| Stripe settlement time (US) | T+2 business days |

| PCI DSS SAQ A eligibility | No raw card data on your servers |

| Chargeback rate warning threshold | > 0.9% (Visa Early Warning) |

| Idempotency key TTL | 24 hours |

| Exponential backoff max delay | 60 seconds |

| P99 latency target for checkout API | < 500ms |

| Kafka message retention (payment events) | 7 days minimum |

| Audit log retention | 7 years (SOX), 5 years (PCI DSS) |

Preparing for the Interview: Practice Drills

1. 1Whiteboard the state machine from memory: every state, every transition, every error path. Practice until you can draw it in under 90 seconds.
2. 2Explain double-entry bookkeeping to a non-technical colleague. If they understand it, your explanation is clear enough for an interview.
3. 3Write the idempotency handler without looking at notes. If you cannot produce the double-checked locking pattern from memory, review it.
4. 4Diagram the PSP webhook flow: client → PSP → webhook → Kafka → consumer → ledger. Include the failure case where your webhook handler is down.
5. 5Practice the reconciliation explanation in two minutes: what it is, why it exists, how it works, what discrepancies it catches.

Payments is one of the most unforgiving system design topics — every mistake costs real money. But that is also why interviewers respect candidates who know it deeply. Master this material and you will stand out.