DevOps Engineer Interview Questions and How to Answer Them (45+ Questions)

DevOps Engineer Interview Questions and How to Answer Them (45+ Questions)

You've got the interview. Now you need to prove you can actually do the job.

This guide covers 45+ DevOps interview questions asked at companies like Google, Amazon, Stripe, and Cloudflare — with complete answers, real code, and the context interviewers actually want to hear. No vague theory. No recycled answers from 2018.

Questions are organized by topic so you can drill the areas where you're weakest.

Table of Contents

1. 1[CI/CD and Pipelines](#cicd)
2. 2[Containers and Kubernetes](#containers)
3. 3[Infrastructure as Code](#iac)
4. 4[Monitoring, Observability, and Reliability](#observability)
5. 5[Networking and Security](#networking)
6. 6[Linux and Systems](#linux)
7. 7[Cloud Platforms](#cloud)
8. 8[Databases and Storage](#databases)
9. 9[Incident Management and On-Call](#incidents)
10. 10[Culture and Process](#culture)

CI/CD and Pipelines {#cicd}

Q1. Walk me through your ideal CI/CD pipeline for a microservice.

What they're testing: Whether you've actually built pipelines end to end, not just pressed buttons in Jenkins.

Answer:

A solid pipeline for a microservice has five stages:

1. 1Lint + test — Unit tests and static analysis run on every commit. Fast. If this fails, nothing proceeds.
2. 2Build — Docker image built and tagged with the commit SHA. Not latest. Never latest in production.
3. 3Integration/contract tests — Spin up dependent services (or mocks), run integration tests against the built image.
4. 4Push + sign — Image pushed to registry, signed with Cosign or Notary.
5. 5Deploy — Progressive delivery: canary or blue/green. Automated rollback on error rate spike.

Here's a GitHub Actions example for stages 1–4:

name: CI

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: "1.22"
      - run: go test ./... -race -coverprofile=coverage.out
      - run: go vet ./...

  build-and-push:
    needs: test
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    steps:
      - uses: actions/checkout@v4
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3
      - name: Sign image
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.build-and-push.outputs.digest }}

Follow-up the interviewer will ask: "How do you handle secrets in the pipeline?" — Use OIDC-based auth to cloud providers (no static credentials), and for app secrets use Vault or AWS Secrets Manager with short-lived tokens. Never put secrets in env vars baked into the image.

Q2. What's the difference between blue/green and canary deployments? When do you use each?

Answer:

Blue/green: You maintain two identical production environments. One serves live traffic (blue). You deploy to the idle one (green), run smoke tests, then flip the load balancer. Rollback is instant — just flip back.

Canary: You route a small percentage of traffic (say 5%) to the new version. Monitor error rates, latency, and business metrics. Gradually increase traffic if metrics look good. Roll back by routing 100% back to the old version.

When to use which:

• Blue/green is better when: rollback speed is critical, you can afford double infrastructure, and the deployment is atomic (schema migrations are done separately).
• Canary is better when: you want real traffic validation before full rollout, you have feature flags or user segmentation, and you can tolerate a small blast radius.

In practice, many teams combine both: blue/green for the infrastructure layer, canary logic at the application layer via a feature flag system.

Q3. How do you prevent a bad deployment from taking down production?

Answer:

Defense in depth — multiple gates, each catching different failure modes:

Before deployment:

• All tests pass (unit, integration, contract)
• Dependency vulnerability scan passes
• Image signed and provenance verified
• Change request approved (for regulated environments)

During deployment:

• Progressive traffic rollout (canary)
• Automated rollback triggers: error rate > X%, p99 latency > Y ms, custom business metric drops
• Deployment timeout: if rollout doesn't complete in N minutes, roll back

After deployment:

• Synthetic monitoring (probes hitting key user flows)
• Real User Monitoring (RUM) for actual user experience
• On-call alert if anomalies detected within the bake window

The key is automating the rollback trigger, not relying on a human to notice and act fast enough.

Q4. You're getting flaky tests in CI. How do you diagnose and fix them?

Answer:

Flaky tests are a reliability tax — they train the team to ignore failures, which is dangerous.

Diagnosis:

# Run the test 50 times and collect failures
for i in $(seq 1 50); do
  go test ./... -run TestFlaky -count=1 2>&1 | grep -E "FAIL|PASS" >> results.txt
done
grep "FAIL" results.txt | wc -l

Common root causes and fixes:

| Cause | Fix |

|-------|-----|

| Time-dependent logic | Mock time; don't use time.Sleep |

| Shared global state | Reset state in setUp/tearDown |

| Race conditions | Run with -race flag; fix data races |

| External service calls | Mock or stub external dependencies |

| Port conflicts in parallel tests | Use random ports or test-specific namespaces |

| Order-dependent tests | Make each test independent |

Systemic fix: Add a flaky test quarantine. Tag flaky tests, move them to a separate suite that runs but doesn't block merges, file a ticket for each one, and track the count in your metrics dashboard. Set a policy: flaky tests must be fixed within N days or deleted.

Q5. How do you manage database migrations in a CI/CD pipeline without downtime?

Answer:

Schema migrations are the hardest part of zero-downtime deployments. The rule: every migration must be backward compatible with the previous version of the application.

This forces a multi-phase approach:

Phase 1 — Expand: Add new column/table/index without removing old ones. Deploy this migration. Both old and new app versions work.

-- Safe: adding a nullable column
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);

Phase 2 — Migrate: Deploy new app version that writes to both old and new structure. Backfill existing data.

UPDATE users SET display_name = name WHERE display_name IS NULL;

Phase 3 — Contract: Remove the old column/table once all traffic is on the new version and you're satisfied.

ALTER TABLE users DROP COLUMN name;

Tool recommendation: [Flyway](https://flywaydb.org/) or [golang-migrate](https://github.com/golang-migrate/migrate) in the deployment pipeline, but with the migration run as a pre-deployment step, not inside the application startup. This gives you control over failure handling.

Q6. What is GitOps and how does it differ from traditional CI/CD?

Answer:

GitOps treats Git as the single source of truth for both application code and infrastructure state. Any desired state of your system is expressed as files in Git. A controller (ArgoCD, Flux) continuously reconciles actual cluster state against the desired state in Git.

Traditional CI/CD push model:

Code push → CI builds → CI pushes to cluster (imperative)

GitOps pull model:

Code push → CI builds image → CI updates manifest in Git →
ArgoCD detects diff → ArgoCD pulls and applies to cluster (declarative)

Why it matters:

• Audit trail: every change is a Git commit with author and timestamp
• Rollback = git revert
• Cluster credentials never leave the cluster (agent pulls, no external push access)
• Drift detection: if someone kubectl applys manually, ArgoCD will flag and revert it

ArgoCD application manifest example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-service
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/k8s-manifests
    targetRevision: HEAD
    path: apps/my-service
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Containers and Kubernetes {#containers}

Q7. How do you write a production-ready Dockerfile?

Answer:

Most Dockerfiles you'll see in the wild have serious problems: bloated images, root user, secrets baked in, no health checks. Here's a production-ready example for a Go service:

# syntax=docker/dockerfile:1.6

# Build stage
FROM golang:1.22-alpine AS builder

# Install ca-certificates for TLS, timezone data
RUN apk add --no-cache ca-certificates tzdata

WORKDIR /app

# Layer cache: copy deps first, code second
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download

COPY . .

# Build statically linked binary
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=linux go build \
    -ldflags="-s -w -X main.version=$(git describe --tags --always)" \
    -o /bin/service ./cmd/service

# Final stage: distroless for minimal attack surface
FROM gcr.io/distroless/static-debian12:nonroot

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /bin/service /bin/service

# Run as non-root (nonroot = uid 65532)
USER nonroot:nonroot

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD ["/bin/service", "healthcheck"]

ENTRYPOINT ["/bin/service"]

Key decisions:

• Multi-stage build — builder image never ships
• distroless/static-debian12:nonroot — no shell, no package manager, non-root
• Build cache mounts — fast rebuilds in CI
• Version embedded at build time via ldflags

Q8. A pod is stuck in CrashLoopBackOff. How do you debug it?

Answer:

This is a classic on-call scenario. Work through it systematically:

# Step 1: What's the current state?
kubectl get pod <pod-name> -n <namespace> -o wide

# Step 2: Look at events — often tells you exactly what's wrong
kubectl describe pod <pod-name> -n <namespace>

# Step 3: Get logs from the failing container
kubectl logs <pod-name> -n <namespace> --previous

# Step 4: Get logs from init containers if present
kubectl logs <pod-name> -n <namespace> -c <init-container-name> --previous

# Step 5: If the container exits too fast to inspect, override the entrypoint
kubectl debug <pod-name> -it --copy-to=debug-pod \
  --container=<container> -- /bin/sh

Common causes and what to look for:

| Symptom | Likely cause |

|---------|-------------|

| OOMKilled in describe | Memory limit too low, or memory leak |

| Error: failed to create containerd task | Image pull issue or corrupt image layer |

| Exit code 1 immediately | Application misconfiguration, bad env var |

| Exit code 137 | OOM kill or SIGKILL |

| Readiness probe failed | App starts but isn't ready yet; adjust probe timing |

| Back-off pulling image | Image doesn't exist, bad tag, missing imagePullSecret |

For OOMKilled pods, set proper resource requests/limits:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    # No CPU limit — CPU throttling is worse than OOM in most cases

Q9. Explain the difference between resource requests and limits in Kubernetes. Why does it matter?

Answer:

Requests: The amount of CPU/memory the scheduler uses to place the pod. Kubernetes guarantees this much is available on the node.

Limits: The maximum the container can use. Exceeding the memory limit kills the container (OOM). Exceeding the CPU limit throttles it (no kill).

Why CPU limits are often harmful:

CPU throttling via cgroups is coarse — your container can get throttled even when the node has idle CPU, simply because the accounting period boundary was hit. This causes mysterious latency spikes.

Recommendation from the Kubernetes community (and validated by Netflix, Uber, others): set CPU requests but not CPU limits. Let the kernel's CFS scheduler handle it naturally.

resources:
  requests:
    cpu: "500m"      # scheduler uses this
    memory: "256Mi"
  limits:
    # cpu: no limit — avoids CFS throttling
    memory: "512Mi"  # hard limit; OOM if exceeded

QoS classes (important for eviction):

• Guaranteed — requests == limits for all containers. Last to be evicted.
• Burstable — requests < limits. Middle priority.
• BestEffort — no requests or limits. Evicted first.

For production workloads, aim for Guaranteed on critical services and Burstable on everything else.

Q10. How does Kubernetes handle service discovery and load balancing?

Answer:

Three layers:

1. DNS (kube-dns / CoreDNS)

Every Service gets a DNS record: ..svc.cluster.local. Pods use this to find services by name, not IP.

2. ClusterIP Service

kube-proxy (or eBPF with Cilium) programs iptables/IPVS rules that translate a stable virtual IP (ClusterIP) to the actual pod IPs behind it. The kernel handles the load balancing at L4.

3. Endpoints / EndpointSlices

The controller watches pods matching the Service's selector. When pods come and go, it updates the endpoint list. kube-proxy watches these and reprograms routing rules.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP  # internal only

For L7 load balancing (path-based routing, header manipulation, TLS termination), you need an Ingress controller (nginx, Traefik) or a service mesh (Istio, Linkerd).

Follow-up: "What happens during a pod rolling update — do requests get dropped?" — Yes, they can. Fix: set terminationGracePeriodSeconds properly, add preStop sleep to allow iptables to update, and configure readiness probes so traffic only routes to ready pods.

Q11. What is a Kubernetes Operator and when would you write one?

Answer:

An Operator extends Kubernetes with custom domain knowledge. It combines a Custom Resource Definition (CRD) — which defines a new API type — with a controller that watches those resources and takes action to reconcile desired state.

When to write an Operator:

• You have stateful, complex operational logic that humans do manually today (backups, failover, resizing)
• You want to encode runbooks as code
• The application has lifecycle semantics Kubernetes doesn't understand natively

Classic example: A PostgreSQL Operator that handles cluster setup, primary election, streaming replication, automated failover, and scheduled backups.

A minimal controller reconcile loop in Go:

func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // Fetch the custom resource
    myApp := &appsv1alpha1.MyApp{}
    if err := r.Get(ctx, req.NamespacedName, myApp); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // Check if desired state matches actual state
    deploy := &appsv1.Deployment{}
    err := r.Get(ctx, req.NamespacedName, deploy)
    if errors.IsNotFound(err) {
        // Create the deployment
        newDeploy := r.buildDeployment(myApp)
        if err := r.Create(ctx, newDeploy); err != nil {
            return ctrl.Result{}, err
        }
        log.Info("Created Deployment", "name", newDeploy.Name)
        return ctrl.Result{}, nil
    }

    // Reconcile differences
    if deploy.Spec.Replicas != myApp.Spec.Replicas {
        deploy.Spec.Replicas = myApp.Spec.Replicas
        if err := r.Update(ctx, deploy); err != nil {
            return ctrl.Result{}, err
        }
    }

    return ctrl.Result{}, nil
}

Use [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) and [Operator SDK](https://sdk.operatorframework.io/) rather than building from scratch.

Q12. How do you secure container workloads in Kubernetes?

Answer:

Security is layered. Cover all four layers:

1. Image security

# Admission webhook (Kyverno or OPA Gatekeeper) policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-image-signature
      match:
        resources:
          kinds: [Pod]
      verifyImages:
        - imageReferences: ["ghcr.io/org/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/org/*"
                    issuer: "https://token.actions.githubusercontent.com"

2. Pod security

securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: [ALL]

3. Network policies — default deny, explicit allow:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

4. RBAC — least privilege. Audit with kubectl auth can-i --list --as system:serviceaccount:namespace:sa-name.

Infrastructure as Code {#iac}

Q13. How do you structure a Terraform project for multiple environments?

Answer:

The most common mistake: putting dev, staging, and prod in separate directories with copied code. That's maintenance hell.

Better structure: workspaces + environment-specific tfvars, OR separate state per env with shared modules.

Recommended layout:

infra/
├── modules/
│   ├── vpc/
│   ├── eks/
│   └── rds/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   │   └── ...
│   └── prod/
│       └── ...
└── shared/
    └── state-backend.tf

Module example:

# modules/eks/main.tf
variable "cluster_name" { type = string }
variable "node_instance_type" { type = string }
variable "min_nodes" { type = number }
variable "max_nodes" { type = number }

resource "aws_eks_cluster" "this" {
  name     = var.cluster_name
  role_arn = aws_iam_role.cluster.arn

  vpc_config {
    subnet_ids = var.subnet_ids
  }
}

Environment-specific override:

# environments/prod/main.tf
module "eks" {
  source             = "../../modules/eks"
  cluster_name       = "prod-eks"
  node_instance_type = "m5.xlarge"
  min_nodes          = 3
  max_nodes          = 20
}

State management: Separate S3 bucket/prefix per environment. Never share state between prod and non-prod.

terraform {
  backend "s3" {
    bucket  = "my-tf-state"
    key     = "prod/eks/terraform.tfstate"
    region  = "us-east-1"
    encrypt = true
    dynamodb_table = "terraform-locks"
  }
}

Q14. What is `terraform plan` and how do you prevent dangerous changes from being applied?

Answer:

terraform plan computes the diff between current state and desired state. It shows you exactly what will be created, modified, or destroyed before you apply.

Policy enforcement — three layers:

1. Sentinel (Terraform Enterprise) or OPA for policy as code:

# opa/policies/no-public-s3.rego
package terraform.analysis

deny[msg] {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  resource.change.after.acl == "public-read"
  msg := sprintf("S3 bucket '%v' must not be public", [resource.address])
}

2. lifecycle block to protect critical resources:

resource "aws_rds_instance" "prod" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

3. CI pipeline review gate:

# .github/workflows/terraform.yml
- name: Terraform Plan
  run: terraform plan -out=tfplan -detailed-exitcode

- name: Check for destructive changes
  run: |
    terraform show -json tfplan | \
      jq '.resource_changes[] | select(.change.actions[] | contains("delete"))' | \
      tee destructive.json
    if [ -s destructive.json ]; then
      echo "WARNING: Destructive changes detected. Manual approval required."
      exit 1
    fi

Q15. Terraform vs Pulumi vs CDK — how do you choose?

Answer:

This is an opinion question, but have a defensible position:

Terraform (HCL):

• Pros: Mature ecosystem, huge module registry, state management is well understood, most teams already know it
• Cons: HCL is not a real programming language — loops and conditionals are awkward, testing is harder

Pulumi:

• Pros: Real programming language (TypeScript, Python, Go) — you get loops, functions, type safety, unit tests
• Cons: Smaller ecosystem, debugging state issues is harder, requires more discipline to avoid imperative anti-patterns

AWS CDK:

• Pros: First-class AWS types, synths to CloudFormation so you can use existing CF workflows
• Cons: AWS-only, the synth layer adds complexity, and CloudFormation stacks have limits

My take (be direct in the interview): For most teams, Terraform with strong module structure and Terratest for testing wins on practical grounds — existing knowledge, ecosystem, and support. Switch to Pulumi if your team has strong software engineering chops and you're writing complex conditional logic that HCL can't express cleanly.

Q16. How do you handle Terraform state corruption or drift?

Answer:

State corruption (file is damaged/inconsistent):

# Always back up first
aws s3 cp s3://my-tf-state/prod/terraform.tfstate terraform.tfstate.backup

# If a resource exists in state but not in real infra:
terraform state rm aws_instance.example

# If a resource exists in real infra but not in state:
terraform import aws_instance.example i-0abc12345def67890

# Nuclear option: recreate state from scratch
terraform state pull > old-state.json
# Manually edit, then:
terraform state push new-state.json

Drift detection (state diverged from reality due to manual changes):

# terraform plan will show drift
terraform plan -refresh-only

# Apply the refresh to sync state with reality (doesn't change infra)
terraform apply -refresh-only

Prevention:

• Enable S3 versioning on state bucket
• DynamoDB state locking (prevents concurrent applies)
• Ban manual changes via SCPs (AWS Service Control Policies) or IAM
• Scheduled drift detection in CI: terraform plan daily, alert on non-zero exit code

Monitoring, Observability, and Reliability {#observability}

Q17. What's the difference between monitoring and observability?

Answer:

Monitoring tells you that something is wrong. You define the metrics and thresholds upfront. It works well for known failure modes.

Observability tells you why it's wrong. It's the property of a system that lets you ask arbitrary questions about its internal state from the outside — without deploying new code. It works for unknown unknowns.

The three pillars: metrics (what), logs (what happened), traces (where).

Metrics alone aren't enough. A p99 latency spike on checkout-service tells you there's a problem. Without distributed traces, you don't know if it's the DB, a downstream service, or a GC pause in your JVM.

Practical example:

Metric alert: p99 checkout latency > 2s → PagerDuty fires
→ Open Grafana: see checkout service latency spiking
→ Click trace ID → open Jaeger/Tempo
→ Trace shows: 1.8s spent in payment-service waiting for redis-cache
→ Open logs for redis-cache → connection pool exhausted
→ Root cause: deployment of payment-service increased connection concurrency without adjusting pool size

This full journey requires: metrics (Prometheus), logs (Loki/ELK), traces (Tempo/Jaeger), all with a shared trace ID propagated via headers (W3C TraceContext).

Q18. How do you write a good Prometheus alert?

Answer:

Bad alerts wake people up unnecessarily. Good alerts are:

• Actionable — there's something a human can and should do
• Signal over noise — don't fire for blips; fire for sustained problems
• Symptom-based — alert on user impact, not internal causes

# Bad: too noisy, not actionable
- alert: HighCPU
  expr: cpu_usage > 80
  for: 1m

# Good: symptom-based, with context
groups:
  - name: api-slos
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
          ) > 0.01
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: >
            {{ $labels.service }} has a {{ $value | humanizePercentage }} error rate
            (threshold: 1%) for the past 5 minutes.
          runbook_url: "https://runbooks.internal/high-error-rate"
          dashboard_url: "https://grafana.internal/d/api-slos"

Alert fatigue is a culture problem. Every alert that fires should have a ticket created or a human taking action. If alerts fire and get silenced or ignored, they degrade the system. Prune aggressively.

Q19. What is SLO/SLI/SLA and how do you implement error budgets?

Answer:

• SLI (Service Level Indicator): A quantitative measure of behavior. E.g., "the proportion of HTTP requests that return a non-5xx response within 500ms."
• SLO (Service Level Objective): A target for your SLI. E.g., "99.9% of requests succeed within 500ms, measured over 30 days."
• SLA (Service Level Agreement): A contract with customers. If you miss it, there are business consequences (credits, penalties). SLOs should be more aggressive than SLAs.

Error budget: The allowed downtime/errors under the SLO. A 99.9% SLO gives you 43.2 minutes/month of budget. If you spend it, you stop feature work and focus on reliability.

Prometheus recording rule for error budget:

groups:
  - name: slo-calculations
    interval: 1m
    rules:
      # 30-day success rate
      - record: slo:success_rate:30d
        expr: |
          sum(increase(http_requests_total{status!~"5.."}[30d]))
          /
          sum(increase(http_requests_total[30d]))

      # Error budget remaining (fraction)
      - record: slo:error_budget_remaining
        expr: |
          (slo:success_rate:30d - 0.999) / (1 - 0.999)

Error budget policy:

• Budget > 50%: Normal feature velocity
• Budget 10–50%: No new risky deployments; invest in reliability
• Budget < 10%: Feature freeze, all hands on reliability
• Budget exhausted: Post-mortem required, executive visibility

Q20. How do you implement distributed tracing in a microservices system?

Answer:

Three parts: instrumentation, propagation, collection.

Instrumentation (OpenTelemetry — the standard):

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

# Setup
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

# Auto-instrument FastAPI
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

# Manual span for business logic
tracer = trace.get_tracer(__name__)

async def process_order(order_id: str):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        span.set_attribute("order.source", "web")
        try:
            result = await charge_payment(order_id)
            span.set_attribute("payment.status", "success")
            return result
        except PaymentError as e:
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise

Propagation: Use W3C TraceContext headers (traceparent, tracestate). OpenTelemetry SDK handles this automatically for HTTP and gRPC.

Collection stack:

• OTel Collector (receives, processes, exports)
• Tempo or Jaeger (storage and querying)
• Grafana (visualization, linked from metrics and logs)

Key integration: same trace_id in logs so you can pivot from a log line to the full trace.

Q21. What is the RED method? How does it compare to USE?

Answer:

Two complementary frameworks for what to measure:

RED (for services — request-driven):

• Rate — requests per second
• Errors — error rate
• Duration — latency distribution (p50, p95, p99)

USE (for resources — capacity-driven):

• Utilization — % of time resource is busy
• Saturation — work queued, waiting
• Errors — error events

Use RED to understand user-facing behavior. Use USE to understand infrastructure bottlenecks (CPU, memory, disk, network).

In practice: start with RED alerts (those are what users experience), then use USE to diagnose why the RED metrics are bad.

Networking and Security {#networking}

Q22. Explain how TLS works. What happens in a TLS handshake?

Answer:

TLS provides encryption (nobody can read the traffic), authentication (you're talking to who you think you are), and integrity (data wasn't tampered with).

TLS 1.3 handshake (simplified):

Client                                          Server
  |                                               |
  |--- ClientHello (supported ciphers, key_share) -->|
  |                                               |
  |<-- ServerHello (chosen cipher, key_share) ---|
  |<-- Certificate (server's cert chain) --------|
  |<-- CertificateVerify (signature) ------------|
  |<-- Finished (HMAC) --------------------------|
  |                                               |
  | [Client verifies cert against trusted CAs]    |
  |                                               |
  |--- Finished (HMAC) ------------------------->|
  |                                               |
  |=== Encrypted application data ===============|

Key improvements in TLS 1.3 vs 1.2:

• 1-RTT handshake (vs 2-RTT)
• 0-RTT session resumption (with replay attack caveats)
• Removed weak cipher suites (RSA key exchange, RC4, DES, 3DES)
• Forward secrecy mandatory (ephemeral Diffie-Hellman)

What to say when asked "how do you validate a cert is legit": The client checks the server's certificate is signed by a CA it trusts (via cert chain), the domain matches the SNI, the cert isn't expired, and it isn't on the CRL or OCSP says it's not revoked.

Q23. What is mTLS and when do you use it?

Answer:

Mutual TLS — both client and server authenticate with certificates. In regular TLS, only the server proves its identity. In mTLS, both sides do.

When to use it:

• Service-to-service communication within a zero-trust network
• Replacing API keys for machine-to-machine auth
• Meeting compliance requirements (PCI-DSS, HIPAA)

Service meshes (Istio, Linkerd) automate mTLS between pods transparently — you don't change application code. Istio's PeerAuthentication:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT  # Reject all non-mTLS traffic

For manual mTLS in Go:

cert, err := tls.LoadX509KeyPair("client.crt", "client.key")
caCert, _ := os.ReadFile("ca.crt")
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)

client := &http.Client{
    Transport: &http.Transport{
        TLSClientConfig: &tls.Config{
            Certificates: []tls.Certificate{cert},
            RootCAs:      caCertPool,
            MinVersion:   tls.VersionTLS13,
        },
    },
}

Q24. How do you manage secrets in a production environment?

Answer:

What not to do:

• Secrets in environment variables baked into container images
• Secrets in Git (even if deleted — they're in history)
• Secrets in ConfigMaps (they're base64, not encrypted)

Proper approaches:

1. HashiCorp Vault — full secrets lifecycle management:

# AppRole auth for services
vault auth enable approle
vault write auth/approle/role/my-service \
  token_policies="my-service-policy" \
  token_ttl=1h

# Service fetches a short-lived token at startup
ROLE_ID=$(vault read -field=role_id auth/approle/role/my-service/role-id)
SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/my-service/secret-id)
VAULT_TOKEN=$(vault write -field=token auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID)

2. AWS Secrets Manager + IRSA (IAM Roles for Service Accounts):

# Pod gets an IAM role via annotation — no static credentials
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-service-role

import boto3
import json

def get_secret(secret_name: str) -> dict:
    client = boto3.client("secretsmanager", region_name="us-east-1")
    response = client.get_secret_value(SecretId=secret_name)
    return json.loads(response["SecretString"])

3. Kubernetes External Secrets Operator — syncs secrets from Vault/AWS/GCP into Kubernetes Secrets, with automatic rotation.

Q25. What is a VPC and how do you design one for production?

Answer:

A VPC (Virtual Private Cloud) is your isolated network in the cloud. Production VPC design principles:

Subnet strategy:

• Public subnets: only load balancers and NAT gateways
• Private subnets: application workloads (EKS nodes, EC2)
• Isolated subnets: databases, caches (no outbound internet)

CIDR planning: Use /16 for the VPC, /24 per subnet. Leave room to expand. Don't overlap with on-prem ranges if you'll connect via Direct Connect or VPN.

resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
}

resource "aws_subnet" "public" {
  count             = 3
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 8, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  map_public_ip_on_launch = true
  tags = { Name = "public-${count.index + 1}" }
}

resource "aws_subnet" "private" {
  count             = 3
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 8, count.index + 10)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = { Name = "private-${count.index + 1}" }
}

Security groups (stateful): Allow only what's needed. Default: deny all inbound.

NACLs (stateless): Use as a backstop for subnet-level blocking, not for fine-grained application rules.

VPC Flow Logs: Always enable. Cheap, essential for security forensics.

Linux and Systems {#linux}

Q26. How do you debug high CPU usage on a Linux server?

Answer:

Work top-down:

# Step 1: Is it one process or system-wide?
top -bn1 | head -20
# or for better output:
htop

# Step 2: Which process is consuming CPU?
ps aux --sort=-%cpu | head -20

# Step 3: What is that process doing? (perf is the gold standard)
sudo perf top -p <PID>
# or for a flame graph:
sudo perf record -g -p <PID> -- sleep 30
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg

# Step 4: Is it CPU or I/O wait?
# wa% in top = I/O wait. High wa% = disk or network bottleneck, not CPU
iostat -x 1 5

# Step 5: Check for kernel-level issues
dmesg | tail -50
sar -u 1 10  # historical CPU from sysstat

For containerized workloads, cgroup accounting:

# CPU usage per cgroup (Kubernetes pod)
cat /sys/fs/cgroup/cpu/kubepods/pod<pod-uid>/cpu.stat
# or
systemd-cgtop

Common causes: runaway process, GC thrash in JVM, busy poll loop, unindexed database query called in a tight loop.

Q27. Explain Linux file descriptor limits and how to tune them.

Answer:

Every open file, socket, and pipe consumes a file descriptor. Two limits:

• System-wide: fs.file-max — total FDs the kernel allows
• Per-process (soft/hard limits): configured via ulimit and /etc/security/limits.conf

# Check current limits for a process
cat /proc/<PID>/limits | grep "open files"

# Check current usage
ls /proc/<PID>/fd | wc -l

# Temporary change for current shell
ulimit -n 65536

# Permanent change (requires re-login)
# /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

# For systemd services:
# /etc/systemd/system/myservice.service
[Service]
LimitNOFILE=65536

# System-wide kernel limit
sysctl -w fs.file-max=2097152
# Make permanent:
echo "fs.file-max = 2097152" >> /etc/sysctl.conf

Why it matters: High-traffic services (nginx, node.js, Kafka) will hit the default 1024 limit and start failing with EMFILE: too many open files. This is a common incident root cause.

Q28. How does the Linux kernel OOM killer work? How do you tune it?

Answer:

When the system runs out of memory and can't reclaim enough via swap or page cache eviction, the OOM killer selects a process to kill.

Selection algorithm: Each process has an oom_score (0–1000). The killer picks the process with the highest score. Score is based on: RSS memory usage, swap usage, how long the process has been running, and oom_score_adj.

# Check what got killed
dmesg | grep -i "oom\|killed process"
journalctl -k | grep -i oom

# See OOM score for a process
cat /proc/<PID>/oom_score

# Tune OOM behavior
# oom_score_adj: -1000 (never kill) to +1000 (kill first)
echo -1000 > /proc/<PID>/oom_score_adj  # protect a process
echo 500   > /proc/<PID>/oom_score_adj  # prefer to kill

# vm.overcommit_memory settings:
# 0 = heuristic (default) — usually allow overcommit
# 1 = always overcommit — more memory available but OOM risk
# 2 = never overcommit — fail allocations that exceed RAM+swap
sysctl -w vm.overcommit_memory=2

In Kubernetes, set memory limits and the QoS class will influence OOM priority. Guaranteed pods have oom_score_adj = -997 — very unlikely to be killed.

Q29. What is the difference between a process and a thread? How do containers use namespaces?

Answer:

Process vs thread:

• A process has its own virtual address space, file descriptors, PID, and resources. Isolated from other processes.
• A thread shares the address space and file descriptors of its parent process but has its own stack and CPU registers. Cheaper to create, faster to context-switch.
• In Linux, both are implemented as tasks via clone() — the difference is which resources are shared via flags.

Linux namespaces (what containers are built on):

| Namespace | Isolates |

|-----------|----------|

| pid | Process IDs — processes see different PID 1 |

| net | Network interfaces, routes, firewall rules |

| mnt | Mount points — each container has its own filesystem view |

| uts | Hostname and domain name |

| ipc | Shared memory, message queues |

| user | UID/GID mappings — rootless containers |

| cgroup | cgroup hierarchy visibility |

A container is just a process (or process tree) running in its own set of namespaces, with resource limits enforced by cgroups. There's no VM or hypervisor — it shares the host kernel.

# See what namespaces a process is in
ls -la /proc/<PID>/ns/

# Run a command in a container's namespace
nsenter -t <PID> --net --pid -- ip addr

Cloud Platforms {#cloud}

Q30. How do you design for high availability in AWS?

Answer:

High availability requires designing for failure at every layer:

Regions and Availability Zones:

• Multi-AZ within a region for most workloads (3 AZs minimum)
• Multi-region for disaster recovery and ultra-low latency (significantly more complex and expensive)

Compute (EKS):

# EKS node group across 3 AZs
resource "aws_eks_node_group" "main" {
  scaling_config {
    desired_size = 6
    min_size     = 3
    max_size     = 12
  }
  subnet_ids = [aws_subnet.private_a.id, aws_subnet.private_b.id, aws_subnet.private_c.id]
}

Database (RDS):

• Multi-AZ = synchronous standby replica in another AZ. Auto-failover in 1–2 min.
• Read replicas for read scaling — async replication.
• Aurora: storage automatically replicates across 3 AZs; 6 copies of data.

Load balancing:

• ALB spans multiple AZs automatically.
• Enable cross-zone load balancing.
• Use health checks — remove unhealthy targets before they cause user impact.

Checklist:

• No single-AZ resources
• S3 for stateless assets (99.999999999% durability)
• Auto Scaling Groups with health check replacement
• Route 53 health checks with failover routing
• Runbook for: AZ failure, region failure, database failover

Q31. What is AWS IAM and how do you implement least privilege?

Answer:

IAM controls who can do what to which AWS resources.

Identity types: Users (humans), Groups (collections), Roles (assumed by services/users temporarily), Service principals.

Least privilege in practice:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSpecificS3Actions",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/uploads/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "us-east-1"
        }
      }
    }
  ]
}

Key practices:

• Never use root account. Enable MFA on it, then lock it in a drawer.
• Use IAM roles, not users, for applications (rotate keys via STS).
• Use permission boundaries to limit what developers can grant themselves.
• Use AWS Access Analyzer to find overly permissive policies.
• Use SCPs (Service Control Policies) at the org level to enforce guardrails.

# Audit what a role can actually do
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/my-role \
  --action-names s3:DeleteObject \
  --resource-arns arn:aws:s3:::production-bucket/*

Q32. How do you reduce AWS costs in a production environment?

Answer:

Cost optimization is an ongoing practice, not a one-time project.

Compute:

• Savings Plans or Reserved Instances for stable baseline (up to 72% discount)
• Spot Instances for stateless, fault-tolerant workloads (up to 90% discount)
• Right-size instances — use AWS Compute Optimizer
• Use Graviton (ARM) instances — same performance, 20% cheaper

Kubernetes-specific:

• Cluster autoscaler + Karpenter: scale down unused nodes
• Spot node groups for batch/non-critical workloads
• Set proper requests (not artificially low) — poor bin-packing wastes money

Storage:

• S3 Intelligent-Tiering for data with unknown access patterns
• EBS gp3 vs gp2: gp3 is cheaper and provisioned separately
• Delete unattached EBS volumes (common waste)

Data transfer:

• Data transfer within a region across AZs costs money (~$0.01/GB)
• Use S3 endpoint (no NAT gateway) for S3 traffic from VPC
• Use Gateway Load Balancer or Transit Gateway pricing analysis

Tooling: AWS Cost Explorer, Infracost (Terraform pre-plan cost estimates), CloudHealth or Apptio for large orgs.

Databases and Storage {#databases}

Q33. How do you handle database connection pooling? What can go wrong?

Answer:

Databases have a limited number of connections. Each connection consumes ~5–10MB RAM on PostgreSQL. A 64GB RDS instance might support 500–1000 connections practically before performance degrades.

In a Kubernetes environment with 100 pods each opening 10 connections = 1000 connections before any query runs. This kills databases.

Solution: PgBouncer (or RDS Proxy for AWS):

# pgbouncer.ini
[databases]
mydb = host=postgres.internal port=5432 dbname=mydb

[pgbouncer]
listen_port = 5432
listen_addr = *
auth_type = scram-sha-256
auth_file = /etc/pgbouncer/users.txt
pool_mode = transaction  # recommended for most apps
max_client_conn = 1000   # max connections from apps
default_pool_size = 20   # connections to actual postgres
server_idle_timeout = 600

Pool modes:

• session — client holds server connection for entire session. Least efficient.
• transaction — server connection released after each transaction. Best for most apps.
• statement — released after each statement. Can't use SET, advisory locks, etc.

What goes wrong:

• Pool exhaustion: all connections in use, new queries queue or timeout. Set proper pool_size based on DB capacity.
• Connection leaks: code opens connections but doesn't close them on error path. Use connection defer close() or context managers.
• Thundering herd: all pods restart simultaneously, all try to connect at once. Add jitter to connection retry.

Q34. You notice a Postgres query is slow. How do you diagnose and fix it?

Answer:

-- Step 1: Find slow queries
SELECT query,
       calls,
       total_exec_time / calls AS avg_ms,
       rows / calls AS avg_rows,
       100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0) AS cache_hit_pct
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;

-- Step 2: EXPLAIN ANALYZE the slow query
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT u.*, o.total
FROM users u
JOIN orders o ON o.user_id = u.id
WHERE u.status = 'active'
  AND o.created_at > NOW() - INTERVAL '30 days';

What to look for in EXPLAIN output:

• Seq Scan on a large table = missing index
• Hash Join with large hash = might need index-based join
• Rows=10000 but actual rows=1 = stale statistics, run ANALYZE
• Buffers: hit=100 read=50000 = data not in cache, needs more shared_buffers or smaller working set

-- Fix: Add index on the filtered/joined columns
CREATE INDEX CONCURRENTLY idx_orders_user_created
  ON orders (user_id, created_at DESC)
  WHERE created_at > NOW() - INTERVAL '90 days';  -- partial index

-- Update statistics if stale
ANALYZE users;
ANALYZE orders;

-- Check for bloat (update-heavy tables)
SELECT relname, n_dead_tup, n_live_tup,
       100 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0) AS dead_pct
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC;

Incident Management and On-Call {#incidents}

Q35. Walk me through how you'd run a major incident.

Answer:

A well-run incident has clear roles, fast communication, and no ego.

Roles:

• Incident Commander (IC): Owns the incident. Coordinates, delegates, communicates to stakeholders. Does NOT debug.
• Technical Lead: Leads the technical investigation.
• Scribe: Documents the timeline in real time.
• Comms: Updates status page and stakeholders.

Timeline for a severity-1 incident:

T+0:00  Alert fires. IC declares incident, opens war room (Slack channel #incident-2024-1234).
T+0:05  Scribe starts timeline. IC assesses impact: "checkout is down, ~500 orders/min affected."
T+0:10  Technical Lead forms hypothesis: "Recent deploy 30min ago. Rolling back."
T+0:15  Rollback initiated.
T+0:25  Traffic restored. Monitoring metrics confirm. IC: "Green. Incident resolved."
T+0:30  IC posts preliminary summary to #incidents-summary.
T+2:00  Status page updated. Customer email drafted.
T+48:00 Blameless post-mortem published.

What to say in the interview: The most important things are: establish an IC quickly (avoids "too many cooks"), communicate clearly to stakeholders so they stop pinging engineers mid-incident, and document in real time so the post-mortem is accurate.

Q36. How do you write a good post-mortem?

Answer:

A post-mortem's job is to prevent the same failure from happening again, not to assign blame.

Structure:

# Post-Mortem: Checkout Service Outage (2024-03-15)

## Impact
- Duration: 47 minutes (14:32–15:19 UTC)
- Affected: ~100% of checkout requests returned 503
- Revenue impact: ~$47,000 based on historical throughput
- Customers affected: ~23,000

## Timeline
- 14:28: Deploy of checkout-service v2.4.1 completed
- 14:32: PagerDuty alert: error rate > 1% [T+4min from deploy]
- 14:35: IC declared, war room opened
- 14:41: Hypothesis formed: new connection pool config
- 14:45: Rollback initiated
- 15:19: Error rate returned to baseline

## Root Cause
v2.4.1 changed connection pool max_size from 20 to 5 as part of a
"memory optimization." Under production load, all 5 connections were
exhausted within seconds, causing request queuing and eventual timeout.

## Contributing Factors
1. The config change was not part of the diff review (separate config file)
2. Load testing environment uses 10% of production traffic; exhaustion not triggered
3. No alert on connection pool saturation

## Action Items
| Action | Owner | Due |
|--------|-------|-----|
| Add pool saturation alert (>80%) | Platform | 2024-03-22 |
| Add config changes to PR diff review checklist | Process | 2024-03-22 |
| Scale load test environment to 50% of prod traffic | Platform | 2024-04-15 |
| Document connection pool sizing guidelines | Docs | 2024-03-29 |

## What Went Well
- Rollback completed in under 10 minutes
- IC was paged and declared incident quickly
- Clear communication to stakeholders during incident

Key principle: "What Went Well" is not optional. Identifying what worked reinforces good practices and keeps post-mortems from being purely negative.

Q37. How do you set up effective on-call rotations?

Answer:

On-call is sustainable only if the on-call engineer can sleep. High-alert-volume rotations cause burnout, turnover, and more incidents (tired engineers make mistakes).

Principles:

• Actionable alerts only. Every page should require immediate action. If you silence it without doing anything, it's noise.
• Alert on symptoms, not causes. "High 500 rate on checkout" wakes a human. "High DB query time" is a cause — might self-resolve, might be monitored and escalated automatically.
• Team rotation, not hero engineers. Distribute on-call so no one person carries the team.
• Compensation. On-call is work. It should be compensated (extra pay, time off) or factored into headcount.

Metrics to track:

- MTTA (Mean Time to Acknowledge): target < 5 min
- MTTR (Mean Time to Resolve): track by service, trending over time
- Alert volume per week: target < 10 pages/week per on-call
- % of pages that required action: target > 90%

Runbooks: Every alert should link to a runbook. Runbooks should be tested and updated during post-mortems. A runbook that hasn't been updated in 6 months is probably wrong.

Culture and Process {#culture}

Q38. What is the "shift left" principle in DevOps?

Answer:

Shift left means moving quality, security, and operational concerns earlier in the development lifecycle — closer to when code is written, not after it's deployed.

Traditional (shift right):

Code → Build → Test → Staging → Security scan → Deploy → Monitor
                                       ↑
                          Problems found here are expensive

Shift left:

Code ← Static analysis, linting, unit tests run in editor
  ↓
PR ← Security scanning (SAST), dependency vuln check, contract tests
  ↓
Build ← Integration tests, DAST against ephemeral environment
  ↓
Deploy ← Already high confidence; monitoring catches the rest

In practice:

• Pre-commit hooks: golangci-lint, trivy for image scanning
• PR gates: SAST with Semgrep or CodeQL, Dependabot alerts
• Ephemeral preview environments per PR for manual QA
• Security and reliability review part of design, not post-launch

The earlier a bug is found, the cheaper it is to fix. A bug found in development costs 1x. Found in staging: 10x. Found in production: 100x.

Q39. How do you manage technical debt in infrastructure?

Answer:

Infrastructure technical debt includes: undocumented manual changes, deprecated dependencies, overly permissive IAM policies, single points of failure, and runbooks that no longer work.

Making it visible:

• Tag tech debt as issues in your issue tracker with a tech-debt label
• Add it to sprint planning — dedicate 20% of each sprint to debt reduction
• Track a "health score" metric for infrastructure: % of services with current base images, % with runbooks updated in last 90 days, % with DR tested

Prioritization framework:

Score = Impact × Probability × (1 / Cost to Fix)

High impact + likely to bite + cheap to fix = fix now
High impact + unlikely + expensive = document and monitor
Low impact + unlikely + expensive = accept the risk

Avoid: rewrites. The temptation to rewrite everything is strong but usually wrong. Iterative improvement of existing systems with clear before/after metrics is more effective.

Q40. What's your approach to capacity planning?

Answer:

Capacity planning prevents surprise outages caused by growth, not bugs.

Data-driven approach:

import pandas as pd
from sklearn.linear_model import LinearRegression
import numpy as np

# Load historical resource metrics from Prometheus/CloudWatch
df = pd.read_csv("cpu_usage_30d.csv")  # timestamp, cpu_percent, requests_per_second

# Build a simple model: CPU usage as function of RPS
X = df[["requests_per_second"]].values
y = df["cpu_percent"].values

model = LinearRegression()
model.fit(X, y)

# Forecast: given projected growth of 2x traffic in 90 days
current_rps = 1000
projected_rps = 2000

predicted_cpu = model.predict([[projected_rps]])[0]
print(f"Projected CPU at {projected_rps} RPS: {predicted_cpu:.1f}%")

# Determine how many nodes you need
current_node_count = 6
cpu_per_node = 100  # representing 100% of one node's capacity
current_total_capacity = current_node_count * cpu_per_node
utilization_target = 0.7  # keep at 70% max to have headroom

required_capacity = predicted_cpu * current_node_count / utilization_target
required_nodes = int(np.ceil(required_capacity / cpu_per_node))
print(f"Nodes needed: {required_nodes}")

Beyond math: capacity planning also means knowing your hard limits (DB max connections, service mesh control plane limits, Kubernetes API server throughput) and testing them in load tests before hitting them in production.

Q41. How do you approach building a DevOps culture in an org that doesn't have one?

Answer:

This question tests whether you understand that DevOps is primarily a culture change, not a tool change.

The actual problem: In orgs without DevOps culture, Dev and Ops have different incentives. Dev is rewarded for shipping features fast. Ops is rewarded for keeping production stable. These conflict.

What works:

1. 1Shared on-call. Developers who are on-call for their own code write more reliable code. No single change does more to align incentives.

1. 2Blameless post-mortems. If engineers fear blame for outages, they'll hide problems. Psychological safety = earlier escalation = shorter incidents.

1. 3Embed SREs in product teams. Not a separate "ops team" that other teams throw work over the wall to. SREs work alongside product engineers.

1. 4Measure and share DORA metrics. Deployment frequency, lead time, MTTR, change failure rate. Make them visible to leadership. Don't use them to blame teams.

1. 5Small wins first. Don't try to transform the org. Pick one team, improve their pipeline, show the before/after metrics. Let results spread organically.

What doesn't work: Buying a tool and declaring "we're doing DevOps now."

Q42. What are DORA metrics and what do good numbers look like?

Answer:

DORA (DevOps Research and Assessment) metrics are the industry-standard measure of software delivery performance, derived from research across thousands of teams.

| Metric | Elite | High | Medium | Low |

|--------|-------|------|--------|-----|

| Deployment Frequency | Multiple/day | Weekly | Monthly | < Monthly |

| Lead Time for Changes | < 1 hour | < 1 week | < 1 month | > 6 months |

| MTTR | < 1 hour | < 1 day | < 1 week | > 1 week |

| Change Failure Rate | 0–5% | 5–10% | 10–15% | > 15% |

How to measure them:

# Deployment frequency: count production deployments
# Query your CD system (ArgoCD, Spinnaker) or git tags

# Lead time: time from first commit to production
git log --format="%H %ai" v1.2.3..v1.2.4 | head -1  # first commit
# Compare to deploy timestamp

# MTTR: time from alert to resolution
# From your incident management tool (PagerDuty, Opsgenie)

# Change failure rate: deploys that caused incidents / total deploys

Elite teams don't trade off speed for stability — they achieve both. High deployment frequency correlates with lower change failure rate (smaller batches = easier to isolate issues).

Q43. How do you evaluate and adopt new tools without creating chaos?

Answer:

New tool adoption is a vector for both improvement and instability. Apply rigor:

Evaluation process:

1. 1Define the problem. Don't adopt a tool because it's new. "We have X problem, current solution Y is inadequate because Z" is the threshold.

1. 2Evaluate alternatives. Usually 2–3 candidates. Time-box the POC to 2–4 weeks.

1. 3POC criteria: Does it solve the problem? What's the operational burden (upgrades, on-call, expertise required)? What's the migration cost? Is it actively maintained?

1. 4Pilot in non-production. Run on one service or team before org-wide adoption.

1. 5Sunset plan. What does the old tool do that you need to migrate off? Who owns the migration?

Governance for the platform team:

• Maintain a "blessed stack" — tools that are supported, documented, and the platform team knows well
• Tools outside the blessed stack: teams can use them, but the platform team won't support them
• Annual review of the stack — retire things that are not used or not providing value

The trap: Platform teams that adopt every interesting new tool end up with an unmaintainable Frankenstein stack that nobody fully understands.

Q44. How do you think about toil reduction in SRE?

Answer:

Toil is manual, repetitive, automatable work that scales linearly with service growth and doesn't provide lasting value. The SRE principle: keep toil below 50% of time.

Identifying toil:

Is it manual? (vs automated)
Is it repetitive? (vs novel problem-solving)
Is it automatable? (vs requires human judgment)
Does it scale with traffic? (vs fixed cost)
Does it produce lasting value? (if yes, it might be worth it)

Examples of toil: manually provisioning test environments, responding to the same alert with the same fix, manually cutting releases, running cron job outputs through a checklist.

Eliminating toil — three levels:

1. 1Automate it: Write a script, runbook automation, or self-healing system.
2. 2Eliminate the need: If you're constantly restarting a service, fix the underlying bug.
3. 3Delegate/prevent: If a team keeps creating toil for your team, fix the process upstream.

Tracking:

# Simple toil tracking in your incident management tool
# Tag incidents/tickets as "toil" and measure weekly

weekly_toil_hours = sum(
    ticket.time_spent for ticket in tickets
    if ticket.tag == "toil"
    and ticket.created_at >= last_week
)
total_hours = sum(ticket.time_spent for ticket in tickets if ticket.created_at >= last_week)
toil_pct = weekly_toil_hours / total_hours * 100
print(f"Toil: {toil_pct:.0f}% of engineering time")

Q45. What's your experience with multi-cloud or hybrid cloud? What are the tradeoffs?

Answer:

Multi-cloud and hybrid cloud solve real problems but create real complexity. Be honest about the tradeoffs.

Legitimate reasons for multi-cloud:

• Regulatory requirements (data residency in specific countries)
• Vendor lock-in risk mitigation for critical dependencies
• Best-of-breed: GCP for ML/analytics, AWS for everything else
• DR strategy: active-active across clouds for extreme resilience requirements

Hybrid cloud:

• Existing on-prem investment not fully depreciated
• Latency requirements (processing near edge hardware)
• Compliance: some data can never leave the data center

Tradeoffs to be honest about:

| Benefit | Cost |

|---------|------|

| Vendor independence | 2x operational complexity |

| Resilience | 2x networking costs (egress) |

| Best of breed | Teams need expertise in multiple platforms |

| Negotiating leverage | No economies of scale with either vendor |

Practical note: Most companies that say they're multi-cloud actually mean "we have some legacy stuff in Azure and everything new in AWS." True active-active multi-cloud is rare because the complexity cost is enormous. Kubernetes helps abstract the compute layer, but storage, databases, IAM, and networking are cloud-specific.

My recommendation: Unless you have a specific, justified requirement, be single-cloud with strong disaster recovery practices. You'll move faster and your team will be more expert.

Q46. How do you approach performance testing before a major launch?

Answer:

Performance testing prevents launch-day disasters. The goal is to find the bottleneck before your users do.

Types of tests:

• Load test: Ramp to expected peak traffic. Does it hold?
• Stress test: Push past expected peak. Where does it break? How does it fail?
• Soak test: Run at sustained load for hours. Memory leaks? Connection exhaustion over time?
• Spike test: Sudden 10x traffic increase. Do auto-scaling policies respond fast enough?

Tooling: k6 (recommended for most teams):

// k6 load test script
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp up
    { duration: '5m', target: 100 },   // hold
    { duration: '2m', target: 500 },   // spike
    { duration: '5m', target: 500 },   // hold
    { duration: '2m', target: 0 },     // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95% of requests under 500ms
    http_req_failed: ['rate<0.01'],    // <1% error rate
  },
};

export default function () {
  const res = http.post('https://api.staging.example.com/checkout', JSON.stringify({
    cart_id: 'test-cart-123',
  }), {
    headers: { 'Content-Type': 'application/json' },
  });

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time OK': (r) => r.timings.duration < 500,
  });

  sleep(1);
}

After running: Correlate k6 results with Grafana dashboards. What saturated first? DB connections? CPU? A specific downstream service? Fix that, rerun. Iterate until all thresholds pass at 2x expected peak.

Q47. Describe how you'd build a self-service infrastructure platform for developers.

Answer:

Internal developer platforms (IDPs) reduce cognitive load on developers and toil on platform teams. The goal: developers should be able to provision what they need without filing tickets.

What to build (start small):

1. 1Service catalog — "I need a new microservice." Developer picks a template (Go API, Python worker, etc.), fills in name + config, and gets a repo pre-configured with CI/CD, observability, and a staging deployment.

1. 2Environment provisioning — "I need a staging environment for my feature." Ephemeral namespaces in Kubernetes, spun up from a PR, torn down when merged.

1. 3Database provisioning — "I need a Postgres instance for my service." Self-service via Crossplane or AWS Service Catalog. Developer gets a connection string in Vault.

Crossplane for database self-service:

# Developer creates this manifest
apiVersion: database.example.com/v1alpha1
kind: PostgreSQLInstance
metadata:
  name: my-service-db
  namespace: my-team
spec:
  storageGB: 20
  version: "15"
  environment: staging

Crossplane Composition translates this to an actual RDS instance, security groups, parameter group, and Vault secret — without the developer knowing any of that.

Key principle: The platform team's job is to make developers productive, not to gatekeep. Measure success by developer NPS on the platform, not by tickets closed.

Quick-Hit Questions

These often come up as rapid-fire questions or follow-ups. Know them cold.

Q48. What is idempotency and why does it matter in infrastructure automation?

An operation is idempotent if running it multiple times produces the same result as running it once. Critical in IaC: terraform apply twice should produce no second change. Critical in CI: deploying the same version twice should be a no-op. Design automation to be safe to retry.

Q49. What is a service mesh and when is it worth the complexity?

A service mesh (Istio, Linkerd) adds a sidecar proxy to every pod, enabling mTLS, observability, traffic management, and fault injection transparently. Worth it when: you have 10+ services, you need zero-trust networking, or you need fine-grained traffic control (canary, circuit breaking). Not worth it for small systems — the operational complexity is real.

Q50. What's the difference between horizontal and vertical scaling?

Horizontal: add more instances. Scales indefinitely (in theory), handles failure gracefully (stateless services). Vertical: add more CPU/RAM to existing instance. Simpler, but has a ceiling and usually requires downtime. For stateless services, always prefer horizontal. For stateful services (databases), vertical scaling is often first, then sharding for horizontal.

How to Prepare for the Interview

Two weeks before: Work through every section above. Don't memorize — understand the reasoning. Interviewers can tell the difference.

One week before: Set up a home lab or use free tier cloud accounts. Build the things described here: a CI/CD pipeline, a Kubernetes deployment, a Terraform module. Hands-on beats reading every time.

Day before: Review your past incidents. What happened? What did you learn? The best interview answers come from real experience. "I once had a production incident where..." is always more compelling than "you should probably..."

In the interview: Think out loud. DevOps interviewers want to see how you reason about tradeoffs, not whether you've memorized the right answer. When you don't know something, say: "I haven't worked with X directly, but based on what I know about Y, I'd approach it by..."

The companies hiring senior DevOps engineers are not looking for people who've memorized configuration syntax. They're looking for engineers who can design resilient systems, debug novel problems under pressure, and make their teams more effective. That's what every answer above is trying to demonstrate.