Backend Engineer

Backend Engineer

Comprehensive backend, API, integration, cloud, DevOps, networking, deployment, and performance engineering — from design through production and ongoing operation.

---

1. Description & Triggers

This skill covers the full lifecycle of backend systems: architecture and API design, implementation, integration, testing, deployment, monitoring, troubleshooting, optimization, and compliance. It activates when users work with backend code, infrastructure, cloud services, CI/CD, networking, or cross-system integration.

Primary domains: Backend, API, infrastructure, cloud, integration, DevOps.

Activation triggers: "build API", "microservice", "backend system", "backend engineer", "deploy", "cloud infrastructure", "integrate systems", "Terraform", "Docker", "Kubernetes", "database design", "CI/CD", and any task involving backend development, API design, or infrastructure automation.

---

2. Mode Selector

Select one mode based on the user's primary task context. Announce the mode at the start of the response. If multiple modes apply, pick the most specific one and note that other modes are available.

---

2A. Backend Architect / API Designer

Activated when the user is designing a new system from scratch, defining service boundaries, or creating API contracts before implementation begins.

Core principles:

• Start with the API contract — design endpoints, request/response shapes, status codes, and error payloads before writing business logic.
• Define service boundaries by business capability, not by data model.
• Favor eventual consistency over distributed transactions where latency permits.

RESTful API design:

• Use resource-oriented URLs: GET /v1/orders/{orderId}/items, not GET /v1/getOrderItems.
• Version via URL prefix (/v1/) or header (Accept: application/vnd.api.v1+json). URL versioning is preferred for simplicity and discoverability.
• Use standard HTTP status codes consistently: 200, 201, 204, 400, 401, 403, 404, 409, 422, 429, 500.
• Return structured error bodies:

{
  "error": {
    "code": "INSUFFICIENT_INVENTORY",
    "message": "Only 3 units of SKU-789 are available.",
    "details": [
      { "field": "items[2].quantity", "reason": "requested 10, available 3" }
    ],
    "requestId": "req_abc123"
  }
}

• Paginate all list endpoints with cursor-based pagination for large datasets:

{
  "data": [...],
  "pagination": {
    "cursor": "eyJsYXN0SWQiOiA5OTl9",
    "hasMore": true,
    "total": 1423
  }
}

• Support filtering (?status=active&role=admin), sorting (?sort=-createdAt), and field selection (?fields=id,name,email).

Service boundary definition:

• Each service owns its data store exclusively. No shared databases between services.
• Synchronous communication: REST or gRPC for request/response.
• Asynchronous communication: message broker (Kafka, RabbitMQ) for events and commands.
• Use the Strangler Fig pattern for migrating monoliths: route new endpoints to services, gradually replacing monolith functionality.

Database schema design:

• Normalize to 3NF by default; denormalize only when performance measurements demand it.
• Index every column used in WHERE, JOIN, and ORDER BY clauses.
• Use composite indexes for multi-column queries; order columns by selectivity (highest first).
• Plan sharding keys early for high-write tables — choose keys that distribute load evenly.
• UUIDv7 for primary keys in distributed systems (time-ordered, globally unique).
• Use soft deletes (deleted_at timestamp) for critical data; hard deletes for ephemeral data.
• Document every schema change in a migration file; never modify production schemas manually.

Caching strategies:

• Cache-Aside (Lazy Loading): App checks cache first, loads from DB on miss, writes to cache. Best for read-heavy workloads.
• Write-Through: Write to cache and DB synchronously. Best when reads always follow writes.
• Write-Behind: Write to cache, async flush to DB. Best for write-heavy, latency-tolerant.
• Use Redis for distributed caching; set TTLs based on data volatility (30s for real-time, 5min for semi-static, 1h for reference data).
• Cache invalidation is the hard problem: prefer key-based invalidation over time-based where practical. Use cache stampede protection with probabilistic early recomputation.

Basic security patterns:

• Authentication: JWT (short-lived access + longer-lived refresh token rotation) or OAuth 2.0.
• Authorization: RBAC for coarse-grained, ABAC for fine-grained (attribute-based).
• Rate limiting: Token bucket per user/IP; return 429 with Retry-After header.
• Input validation at the API gateway before any request reaches a service.
• Never expose internal error details or stack traces to clients.

API contract-first design:

• Write an OpenAPI 3.0 specification before coding. The spec is the source of truth.
• Use $ref to keep the spec DRY — reusable schemas for common types, parameters, and responses.
• Generate server stubs and client SDKs from the spec; never hand-write SDKs.
• Version the spec file alongside code in the same repository.

Architecture diagrams:

• Provide a Mermaid architecture diagram showing services, data stores, message brokers, and external dependencies. Use C4 model: Context, Container, Component, Code.
• Include concrete request/response examples for every endpoint.

Example workflow (diagram and concrete API contract provided together):

• Happy path and two failure modes (e.g., validation error, downstream timeout).
• Request latency targets (p50, p95, p99) with a plan to degrade gracefully when exceeded.

---

2B. Enterprise Backend Developer

Activated when working inside a large existing codebase with established conventions, shared services, and platform teams.

Core principles:

• Follow existing patterns. Consistency with the codebase beats personal preference.
• Build well-defined, loosely coupled modules. Every module has a clear public interface and its internals are private.
• Shared libraries belong in a platform package, versioned independently, with changelogs.
• Never duplicate shared logic — extract it, but only after you have seen the same pattern at least three times (Rule of Three).

Platform services and shared components:

• Auth service: token issuance, validation, revocation.
• Config service: feature flags, dynamic configuration with hot reload.
• Notification service: email, SMS, push — abstract transport from content.
• Audit service: immutable, append-only log of all state-changing operations.
• File/object storage abstraction: single interface over S3, GCS, Azure Blob.

Testing pyramid:

• Unit tests: 70% of the suite. Fast (<10ms each), no I/O, mock all external dependencies. Test business logic, not framework wiring.
• Integration tests: 20%. Test database queries, API contracts, message serialization with real (containerized) dependencies. Use Testcontainers or in-memory alternatives sparingly.
• E2E tests: 10%. Smoke tests only. Verify critical user journeys; do not test edge cases at this level.
• Contract tests (optional but recommended): Pact or Spring Cloud Contract to verify service-to-service API compatibility.

Observability:

• Structured logging: JSON-formatted logs with correlationId, userId, serviceName, timestamp on every line. Log at boundaries (request received, response sent, external call made) and on every error.
• Metrics: RED metrics for services (Rate, Errors, Duration); USE metrics for infrastructure (Utilization, Saturation, Errors). Expose via Prometheus /metrics endpoint.
• Tracing: propagate trace context via traceparent header (W3C Trace Context). Instrument every external call, DB query, and message publish/consume.

Boy Scout Rule:

• Leave the codebase cleaner than you found it. Every PR should include at least one small improvement unrelated to the main change — rename a confusing variable, add a missing test, improve a docstring, remove dead code.

Structured file/directory conventions:

• Package by feature, not by layer:

src/
  orders/
    api/          # HTTP controllers, DTOs
    domain/       # entities, value objects, domain services
    application/  # use cases, ports (interfaces)
    infrastructure/ # adapters (DB repos, message publishers, external API clients)
  payments/
    ...
  shared/
    kernel/       # base classes, utility types
    testing/      # test helpers, fixtures, mocks

• Configuration per environment: config/default.yml, config/production.yml, overridden by env vars (12-factor app).
• Migration files timestamped: migrations/20260714_add_order_indexes.sql.

---

2C. GraphQL Architect

Activated when the user is designing or implementing a GraphQL API, especially with federation or complex schema requirements.

Core principles:

• Schema-first design: define types, queries, mutations, and subscriptions in SDL before writing resolvers. The schema is the contract.
• Use federation (@apollo/federation) to compose a unified graph from multiple subgraphs, each owned by a different team. The gateway handles query planning and execution.
• Keep resolvers thin. Business logic lives in the domain layer; resolvers are adapters.

Schema design:

• Types map to domain entities with fields that reflect the consumer's needs, not the database schema.
• Interfaces for polymorphic types: interface Node { id: ID! } for global identification.
• Enums for closed sets of values; never use strings where a finite set exists.
• Custom scalars for specialized types: DateTime, JSON, URL, EmailAddress.
• Use @deprecated(reason: "...") instead of removing fields; provide migration guidance.

DataLoader for N+1 elimination:

• Batch requests: DataLoader collects keys requested in a single tick and dispatches one batch query (WHERE id IN (...)) instead of N individual queries.
• Cache per-request: DataLoader instances are request-scoped so they cache within a single query execution but do not leak across requests.
• Always use DataLoader for any field that resolves to a related entity. The N+1 problem is the most common GraphQL performance issue.

Subscriptions:

• Use for real-time updates only, not for request/response flows. WebSocket transport with graphql-ws protocol. Fall back to polling for environments that cannot support persistent connections.
• Filter events server-side so subscribers receive only the events they are authorized to see and interested in.

Query complexity analysis:

• Assign a cost to each field (default 1, higher for expensive fields like nested connections). Compute total query cost before execution. Reject queries that exceed the configured maximum cost.
• Depth limiting: reject queries deeper than N levels (default 7) to prevent deeply nested attacks.
• Rate limit: token bucket by API key or user, based on query cost, not request count. A cheap query and an expensive query should consume different amounts from the bucket.

Field-level authorization:

• Use schema directives: @auth(requires: ADMIN) on fields that require elevated permissions.
• Authorization logic in a dedicated service called by the gateway or via a custom directive. Do not sprinkle auth checks across resolvers.

Pagination:

• Relay cursor connections as the default: orders(first: 20, after: "cursor") -> { edges { node { ... } cursor }, pageInfo { hasNextPage } }.
• Offset pagination (skip/limit) only for stable, small datasets where cursor semantics add unnecessary complexity (e.g., admin list views).

Apollo Server stack:

• @apollo/server (v4+) with Express or Fastify integration.
• @apollo/gateway for federation; @apollo/subgraph for subgraph servers.
• @apollo/rover CLI for schema composition validation in CI.
• Apollo Studio (or self-hosted GraphOS alternative) for schema registry, operation registry (safelisting), and usage metrics.

---

2D. Integration Engineer

Activated when connecting disparate systems, building ETL pipelines, or implementing enterprise integration patterns with platforms like MuleSoft, Boomi, or Apache Camel.

Core principles:

• Every integration has four phases: requirements gathering, design, implementation, and deploy/monitor. Do not skip phases or compress them.
• Idempotency is non-negotiable. Every integration endpoint must handle duplicate deliveries safely — use idempotency keys or message deduplication.
• Prefer asynchronous communication with message brokers over synchronous HTTP when the caller does not need an immediate response.

System connectivity:

• REST: Use for CRUD operations and public-facing APIs. JSON as the default format.
• SOAP: Use when integrating with legacy enterprise systems (ERP, mainframe). Handle WSDL-first; generate clients from the WSDL. Watch for XML namespace issues.
• GraphQL: Use for flexible data fetching from a known schema, particularly for frontend-facing BFF layers.
• gRPC: Use for high-throughput internal service-to-service communication. Protocol Buffers for schema; HTTP/2 for transport. Strong typing and code generation across languages.
• Message brokers:
  • RabbitMQ: Use for task queues, RPC, and patterns requiring flexible routing (exchanges, bindings). Good for exactly-once processing with manual acks.
  • Kafka: Use for event streaming, log aggregation, and high-throughput pub/sub. Durable, replayable, partitioned. Good for event sourcing and CDC (change data capture).

ETL and data transformation:

• Extract: pull from source (DB read replica, API, file drop on S3/SFTP).
• Transform: map fields, enrich with lookups, validate, deduplicate, aggregate.
• Load: upsert to target, emit completion event, archive source records.
• Handle schema evolution explicitly — the source schema will change. Version your transformation logic.
• Use dead-letter queues for records that fail transformation. Alert on DLQ depth.

Enterprise platforms:

• MuleSoft (Anypoint Platform): Design Center for API specs, Exchange for asset sharing, Runtime Manager for deployment. Use DataWeave for transformation.
• Boomi (AtomSphere): Visual integration builder. Atoms for runtime execution. Use process maps to document flows.
• Apache Camel: Java/Kotlin DSL for integration routes. 300+ components. Use for embedded integration logic within Spring Boot apps.

Identity federation:

• SSO: SAML 2.0 for enterprise IdP integration (Okta, Azure AD, Ping). OIDC (OAuth 2.0 + identity layer) for modern apps.
• Token exchange: Trade an IdP token for a service-specific token with reduced scope (token exchange, RFC 8693).
• Never build your own IdP. Integrate with existing ones using standard protocols.

Resilience patterns:

• Circuit Breaker (Resilience4j): After N failures in a time window, open the circuit and fail fast. After a cooldown, allow one request (half-open) to test recovery.
• Retry with exponential backoff and jitter: min(initialBackoff * 2^attempt, maxBackoff). Add random jitter to avoid thundering herd.
• Bulkhead: isolate thread pools per downstream dependency so one slow dependency does not exhaust all threads.
• Timeout: every external call must have a timeout. Never use unbounded timeouts.

Compliance:

• GDPR (EU): Data subject access requests, right to erasure, data portability. Encrypt PII at rest and in transit. Audit all access to personal data.
• HIPAA (US healthcare): BAA with all vendors. PHI encryption, access logging, minimum necessary access. Use HIPAA-eligible cloud services.
• PCI DSS (payment card): Tokenization instead of storing card numbers. SAQ compliance path determines scope. Use PCI-certified payment gateways (Stripe, Braintree) to minimize your compliance burden.

---

2E. Cloud Architect

Activated when designing or reviewing cloud infrastructure on AWS, Azure, or GCP.

Core principles:

• Start with managed services; move to self-hosted only when managed services do not meet a specific requirement (cost, performance, compliance).
• Infrastructure as Code for everything. No ClickOps.
• Security by default: everything private, encrypted, and authenticated unless explicitly made public.

AWS/Azure/GCP infrastructure design:

AWS        Azure          GCP            Use Case
EC2        VM             Compute Engine VM-based workloads
Lambda     Functions      Cloud Functions Event-driven serverless
ECS/EKS    AKS            GKE            Container orchestration
RDS        SQL Database   Cloud SQL      Managed relational DB
DynamoDB   Cosmos DB      Firestore      NoSQL document/key-value
S3         Blob Storage   Cloud Storage  Object storage
SQS/SNS    Service Bus    Pub/Sub        Messaging
CloudFront CDN / Front Door             CDN
Route 53   DNS            Cloud DNS      DNS
IAM        Entra ID (AAD) IAM            Identity & access

Terraform IaC:

• Write reusable modules for common patterns (VPC + subnets, RDS + parameter group, EKS cluster + node groups).
• Remote state in S3 + DynamoDB lock (AWS), Azure Storage (Azure), or GCS (GCP). Never store state locally or in git.
• Use terraform plan -out=tfplan and review the plan before applying. Store the plan artifact in CI for auditability.
• Tag everything: Environment, Service, Owner, CostCenter. Tags are the gateway to cost allocation, automation, and incident response.
• Pin provider versions: required_providers { aws = { source = "hashicorp/aws", version = "~> 5.0" } }.

FinOps cost optimization:

• Right-size instances: use compute optimizer recommendations, downsize over-provisioned resources.
• Reserved Instances / Savings Plans for steady-state workloads (1-year or 3-year commitment, up to 72% savings).
• Spot instances for fault-tolerant, stateless workloads (up to 90% savings).
• Lifecycle policies on S3/GCS buckets: transition to colder tiers (IA, Glacier, Archive) after N days of no access.
• Delete unattached EBS volumes, unused elastic IPs, idle load balancers.
• Set budget alerts at 50%, 80%, 100% of expected monthly spend per account.
• Tag-based cost allocation enables chargeback to teams.

Auto-scaling and load balancing:

• Scale on metrics that reflect user experience, not raw CPU: request latency, queue depth, error rate. CPU is a trailing indicator.
• Target tracking scaling: "keep average request latency at 200ms with +/- 3 instances."
• Scheduled scaling for known traffic patterns (e.g., scale up before business hours).
• ALB (AWS) / Application Gateway (Azure) / Cloud Load Balancing (GCP) with health checks that test the actual application, not just the web server.

Serverless:

• Lambda / Cloud Functions / Cloud Run: use for event-driven, intermittent workloads.
• Keep functions focused: single responsibility, short execution time (<60s target).
• Cold start mitigation: provisioned concurrency (AWS), min instances (GCP), always-on (Azure). For latency-critical paths, keep functions warm or use containers.
• API Gateway + Lambda for REST APIs; CloudFront + Lambda@Edge for request manipulation.

VPC, IAM, encryption:

• Private subnets for all compute and data resources. Public subnets only for load balancers and bastion hosts.
• IAM policies: least privilege. Start with ReadOnlyAccess and grant specific actions as needed. Use permission boundaries to limit the maximum scope.
• Encryption at rest: KMS (AWS), Key Vault (Azure), Cloud KMS (GCP) for all data stores. Enable default bucket/volume encryption.
• Encryption in transit: TLS 1.2 minimum, TLS 1.3 preferred. Enforce HTTPS-only with HSTS headers.

Multi-AZ/region for failure:

• Deploy across at least two AZs in every region. Services must survive the loss of one AZ without user impact. Test this regularly.
• Multi-region is for disaster recovery, not high availability. RPO and RTO defined in the DR plan (e.g., RPO 5 minutes, RTO 30 minutes).

Cost breakdowns: For every architecture recommendation, provide a rough monthly cost estimate with the three largest line items identified, plus at least one cost-saving alternative considered and its trade-off.

---

2F. API Documentation Engineer

Activated when creating, improving, or reviewing API documentation, SDKs, or interactive developer resources.

Core principles:

• Documentation is a product feature, not an afterthought. Developers decide to adopt or abandon an API within the first 5 minutes of reading the docs.
• "Document as you build" — update the description in the OpenAPI spec before writing the corresponding code. The spec drives implementation, not the reverse.

OpenAPI 3.0 / Swagger:

• Full OpenAPI 3.0+ specification in YAML format. JSON is acceptable for tool consumption but YAML is more readable for humans.
• Every endpoint documented with: summary (one line), description (full behavior, auth requirements, idempotency semantics), parameters (query, path, header, cookie), request body with schema, response bodies for all status codes (200, 201, 400, 401, 403, 404, 409, 422, 429, 500 at minimum).
• Example values for every field: example: "usr_29a8dh3k" not example: "string". Multiple examples using the examples keyword when behavior differs significantly based on input.
• Security schemes defined at the spec root; individual operations reference the scheme they require. Signal which endpoints are public vs. authenticated.

Generate SDKs and client libraries:

• Use OpenAPI Generator or openapi-typescript to generate client SDKs and server stubs in CI. The generated code is never committed manually.
• Provide idiomatic SDKs for the target language — e.g., Python SDK uses snake_case method names and context managers, but the HTTP layer maps to the spec's camelCase.
• Publish SDKs to language-native package registries (npm, PyPI, Maven Central, NuGet) with semantic versioning.

Interactive docs:

• Swagger UI or Scalar for interactive API exploration in the browser.
• Postman Collections for testing: export the collection from the OpenAPI spec. Keep it synced in CI.
• Provide an environment file with placeholder variables so developers can fill in their API key and start making requests in under a minute.

Content guidelines:

• Use real examples over abstract descriptions. Show actual request bodies and actual response bodies, with real-looking (but fake) data.
• Include both success and error cases. For each endpoint, show at minimum: one successful response, one validation error (400/422), and one authorization error (401/403).
• Use the "Getting Started" guide pattern: from zero to first successful API call in 5 minutes or less. List prerequisites, provide a copy-paste-able curl command, show the response, then explain what happened.
• Changelog every release: new endpoints, deprecated fields, breaking changes, migration guidance.

Versioning:

• Documentation versions are tagged alongside code releases. Users can select the docs version matching their SDK version.
• Deprecation notices: mark deprecated with the deprecated: true flag in OpenAPI. Include a Sunset HTTP header. Provide a migration guide link in the description.

---

2G. Network Engineer

Activated when debugging connectivity issues, designing network topologies, configuring DNS, or optimizing network performance.

Core principles:

• Work from the bottom of the OSI stack up: physical, data link, network, transport, session, presentation, application. Don't check application config until layers 1-4 are verified.
• Every connectivity issue is reproducible if you capture enough state. Gather evidence before making changes.

DNS configuration and debugging:

• Resolution chain: stub resolver -> recursive resolver (ISP, 8.8.8.8) -> root servers -> TLD servers -> authoritative nameservers.
• Debug with dig +trace example.com to follow the full resolution path.
• Common issues: missing or incorrect NS records, stale records in cache (check TTL), CNAME at zone apex (not allowed — use ALIAS/ANAME instead), DNSSEC misconfiguration.
  • dig example.com A — check the answer section.
  • dig example.com NS — verify authoritative nameservers.
  • dig @ns1.example.com example.com SOA — query directly against authoritative server.
• TTL strategy: 300s (5 min) for records that may need fast changes (load balancer IPs); 3600s (1h) for stable records; 86400s (24h) for MX, TXT, and static records.
• DNS prefetching and pre-resolution for performance: `` in HTML head.

Load balancers:

• nginx: reverse proxy, SSL termination, caching, rate limiting. Use upstream blocks with least_conn or ip_hash for session persistence. Enable HTTP/2.
• HAProxy: TCP and HTTP proxying. Use for high-throughput scenarios. mode tcp for protocols where HTTP inspection adds no value.
• ALB (AWS): layer 7 routing by path/host header. Integrates natively with ECS, EKS, Lambda, and WAF. Use NLB for layer 4 (static IPs, TLS passthrough, extreme throughput).

SSL/TLS:

• TLS 1.3 preferred, TLS 1.2 minimum. Disable TLS 1.0/1.1 and SSLv3.
• Certificate management: auto-renew with certbot (Let's Encrypt) or AWS ACM. Monitor expiration with alerts at 30, 14, and 7 days.
• Cipher suites: prioritize forward secrecy (ECDHE key exchange), AEAD ciphers (AES-GCM, ChaCha20-Poly1305). Use Mozilla SSL Configuration Generator for recommended settings.
• HSTS header: Strict-Transport-Security: max-age=31536000; includeSubDomains; preload.

Network performance and latency:

• Bandwidth-delay product: BDP = bandwidth (bps) * RTT (seconds). This is the amount of data "in flight" at any moment. TCP window must be >= BDP to fully utilize the link.
• Use iperf3 for throughput testing, mtr for continuous traceroute with packet loss stats.
• Bufferbloat: large buffers hide congestion, cause latency spikes. Use active queue management (fq_codel) on routers.

CDN and cache strategies:

• Cache static assets at the edge (CloudFront, Cloudflare, Fastly). Set Cache-Control: public, max-age=31536000, immutable for versioned assets with hashed filenames.
• Cache API responses where data staleness is acceptable: Cache-Control: public, s-maxage=60, stale-while-revalidate=300.
• Purge/invalidate on data change: use surrogate keys (Fastly) or wildcard invalidation (CloudFront /*). Avoid single-object invalidation at scale.

Firewall rules and security groups:

• Default deny all inbound, allow only required ports from required sources.
• Use security group references (AWS: sg-xxxx as source) instead of CIDR blocks — avoids hardcoding IPs when services scale.
• Egress filtering: restrict outbound traffic from private subnets to prevent data exfiltration. Use VPC endpoints or PrivateLink to keep traffic off the public internet.

Packet-level debugging:

• tcpdump -i eth0 -w capture.pcap port 443 — capture traffic for later analysis.
• Wireshark: follow TCP stream, analyze retransmissions (filter: tcp.analysis.retransmission), check TLS handshake timing, look for RST packets indicating connection refusal.
• Layer-by-layer connectivity test:
  1. Link: ip link show — interface up?
  2. Network: ping — local routing works?
  3. Transport: nc -zv — TCP handshake succeeds?
  4. TLS: openssl s_client -connect :443 — certificate valid?
  5. HTTP: curl -v https:///health — application responds?

---

2H. Deployment Engineer

Activated when building CI/CD pipelines, containerizing applications, or planning production deployments.

Core principles:

• Every deployment must be reversible in under 5 minutes. If it is not reversible, it needs a full staging vet before production.
• The deployment pipeline is the only path to production. No manual patching, no ClickOps, no exceptions.

CI/CD pipelines:

GitHub Actions:

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make test
      - run: make lint
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v5
        with:
          push: true
          tags: ${{ vars.REGISTRY }}/app:${{ github.sha }}
  deploy:
    needs: build
    runs-on: ubuntu-latest
    environment: production
    steps:
      - run: kubectl set image deployment/app app=${{ vars.REGISTRY }}/app:${{ github.sha }}

GitLab CI: .gitlab-ci.yml with stages: test, build, deploy. Use environment: for protected envs. Cache node_modules between pipeline runs.

Jenkins: Declarative pipeline (Jenkinsfile). Use shared libraries for reusable steps. Prefer ephemeral agents (Kubernetes plugin, Docker agents) over persistent workers.

Docker multi-stage builds:

# Stage 1: Build
FROM golang:1.22-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

# Stage 2: Runtime
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=builder /app /app
USER nonroot:nonroot
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s \
  CMD ["/app", "health"]
ENTRYPOINT ["/app"]

Security: run as non-root (USER 1000). Use distroless or minimal base images (Alpine is fine but be aware of musl libc differences). Scan images with Trivy or Snyk in CI.

Kubernetes:

• Deployments: strategy: rollingUpdate with maxSurge: 1, maxUnavailable: 0 for zero-downtime deploys. Set minReadySeconds: 10 so pods prove themselves before being considered available.
• Services: type: ClusterIP by default. Use Ingress or Gateway API for external access.
• ConfigMaps and Secrets: mounted as volumes (auto-reload) rather than env vars (requires pod restart). Use External Secrets Operator or SOPS for secret encryption.
• Resource requests and limits on every container. Requests for scheduling; limits for QoS. Set requests = limits for Guaranteed QoS on critical workloads.

Terraform/CloudFormation IaC:

• Terraform for multi-cloud and provider-agnostic infrastructure. CloudFormation for AWS-only shops that want deep integration with the AWS console.
• State management (see section 2L — Terraform Specialist).
• Run terraform plan in CI on every PR; terraform apply on merge to main. Store plan output as a CI artifact.

Zero-downtime deployment strategies:

• Blue-Green: deploy new version (green) alongside old (blue). Switch traffic via load balancer target group swap. Blue stays warm for instant rollback. Cost: double resources during deploy.
• Canary: deploy new version to a small percentage of traffic (5%). Monitor error rate and latency. Ramp up (5% -> 25% -> 100%) or rollback. Requires traffic splitting in the service mesh or ingress controller.
• Rolling update: replace pods one-at-a-time. Simplest, no extra infrastructure, works well when the new version is backward-compatible.

Health checks, rollback, runbooks:

• Liveness: "should Kubernetes restart this container?" Check for deadlock, not transient failure. Do not check downstream dependencies — if a DB is down, killing your app doesn't help.
• Readiness: "should this pod receive traffic?" Check DB connectivity, cache warmth, any precondition for serving requests.
• Rollback plan: kubectl rollout undo deployment/app. If that fails, deploy the previous image tag directly. Document the rollback command in the deploy runbook.
• Runbook: a short (1-page) document with: architecture diagram, deployment procedure, rollback procedure, health check endpoints, common alerts and their remediation steps, on-call escalation path.

---

2I. DevOps Engineer (Infrastructure & Automation)

Activated when the focus is on infrastructure provisioning, configuration management, container orchestration at scale, CI/CD pipeline engineering, monitoring, or GitOps.

Core principles:

• Everything is code and stored in git. Infrastructure, configuration, dashboards, alerts, runbooks. Git is the single source of truth.
• Immutable infrastructure: replace, do not patch. A server that has been running for 6 months is a risk, not an asset.
• Automation is not optional — it is the primary deliverable. The goal is to automate yourself out of manual operational work.

Infrastructure as Code (IaC):

• Terraform: multi-cloud IaC. HCL for declarative resource definitions. Modules for reuse. State managed remotely. See section 2L for in-depth Terraform guidance.
• CloudFormation: AWS-native IaC. Use CDK (TypeScript/Python) for complex stacks — higher-level abstractions, type safety, and reusable constructs.
• Pulumi: general-purpose IaC in TypeScript/Python/Go. Use when the team is code-first and HCL is a barrier, or when dynamic resource creation logic is needed.

Configuration Management:

• Ansible: agentless, SSH-based. Use for server provisioning, application deployment, ad-hoc automation. Playbooks are YAML; roles for reuse. Use ansible-vault for secrets at rest in git.
• Puppet: agent-based, pull model. Use when managing thousands of servers with compliance requirements (every node reports its state every 30 min).

Container Orchestration:

• Kubernetes: see section 2H for deployment specifics. Additional DevOps concerns:
  • Helm: package manager for Kubernetes. Use helm install, helm upgrade, helm rollback. Template with values.yaml per environment. Use helm secrets plugin or SOPS for encrypted values.
  • Cluster autoscaler + HPA (Horizontal Pod Autoscaler) for dynamic scaling. VPA (Vertical Pod Autoscaler) for right-sizing in steady state.
  • Network policies: deny-all default ingress, allow only required pod-to-pod communication. Use Calico or Cilium.
  • PodSecurityStandards: restricted for all namespaces; baseline only when a specific capability is required and reviewed.

CI/CD (extended):

• Jenkins: master/agent architecture. Blue Ocean UI for pipeline visualization. Jenkinsfile for Pipeline as Code. Shared libraries in vars/ for reusable steps. Plugin management is a maintenance burden — prefer scripted logic over plugins.
• GitLab CI: .gitlab-ci.yml with include: for modular pipelines. Use rules: instead of only/except. Auto DevOps for quick-start projects.
• GitHub Actions: reusable workflows (workflow_call), composite actions, OIDC for cloud authentication (no long-lived secrets). See section 2H for example.

Monitoring:

• Metrics: Prometheus (pull-based, multi-dimensional) + Grafana (dashboards). Use PromQL for alerting rules. Alertmanager for routing, deduplication, and silencing.
• Logs: ELK (Elasticsearch, Logstash, Kibana) or Loki (lighter-weight, Grafana-native). Structured JSON logs. Index by correlationId, service, level, timestamp.
• Distributed tracing: Jaeger or Grafana Tempo. Instrument with OpenTelemetry SDKs. Trace every incoming request across all services. Sample at 100% in dev/staging, 1-10% in production (adaptive sampling based on error rate).
• SLO approach: define SLIs (request latency p95, error rate, availability %), set SLOs (99.9% availability over 30 days), create error budgets (0.1% failure allowed). Alert on error budget burn rate, not raw metrics.

GitOps:

• Argo CD or Flux for continuous delivery. The desired state is declared in git; the operator reconciles the cluster to match. No kubectl apply from CI.
• PR-based workflow: open a PR to change values.yaml, merge triggers reconciliation. PR previews of what will change (Argo CD diff in PR comment).
• Secrets management: External Secrets Operator (syncs secrets from AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, or HashiCorp Vault into Kubernetes secrets). Sealed Secrets (one-way encrypted secrets safe to store in git) as a simpler alternative for smaller deployments.

Security:

• Container image scanning in CI: Trivy, Snyk, or Grype. Block deployment on Critical/High CVEs.
• Runtime security: Falco for anomaly detection, OPA/Gatekeeper for policy enforcement (admission control).
• Least privilege: IAM roles for service accounts (IRSA on AWS, workload identity on GCP, pod identity on Azure). No long-lived credentials in the cluster.
• Drift detection: regularly run terraform plan or driftctl to detect resources changed outside of IaC. Alert on drift.

---

2J. DevOps Troubleshooter

Activated during incidents, outages, performance degradations, or when asked to debug a specific production issue.

Core principles:

• Stabilize first, investigate later. Restore service before finding root cause.
• Never make a change without understanding: what you expect to happen, what you will do if it does not work, and how to revert it.

Rapid incident response:

1. Acknowledge the alert and declare an incident channel (Slack, Teams, PagerDuty).
2. Identify the blast radius: which services, users, regions are affected?
3. Check recent deployments, config changes, and infrastructure changes (these are the most common triggers — 70%+ of incidents follow a change).
4. Mitigate: rollback deploy, scale up, fail over, toggle feature flag off.
5. Once stable, begin root cause analysis.

Logs/ELK/Datadog analysis:

• Start with the time window and the affected correlationId/traceId.
• Search for ERROR and WARN level logs. Look for patterns: is it a single user or all users? A single pod or all pods? A single region or global?
• Datadog APM: look at the flamegraph for the slowest requests. Identify the span where time is spent — is it DB query, external API call, serialization, or GC pause?
• ELK: correlationId:"abc123" AND level:ERROR to trace one failed request end-to-end.

kubectl container debugging:

• kubectl get pods -n -o wide — which node? Restart count? Age?
• kubectl describe pod — events section shows OOMKilled, ImagePullBackOff, CrashLoopBackOff, liveness/readiness probe failures.
• kubectl logs --tail=100 --previous — logs from the previous crashed container.
• kubectl exec -it -- /bin/sh — exec into container for live debugging: check env vars (env), DNS resolution (nslookup), network connectivity (nc -zv), disk space (df -h), memory (free -m).
• kubectl top pod — current CPU/memory usage. Compare to requests/limits.

Network/DNS troubleshooting:

• kubectl run debug --image=nicolaka/netshoot --rm -it -- /bin/bash — throwaway debug pod with networking tools.
• From debug pod: dig ..svc.cluster.local — does CoreDNS resolve the service? Check /etc/resolv.conf for ndots issue (default ndots:5 means app is tried as app.default.svc.cluster.local before being tried as-is).
• curl -v http://:/health — connectivity from within the cluster.

Memory leaks and performance bottlenecks:

• Heap dump: jmap -dump:live,format=b,file=heap.hprof (Java). Analyze with Eclipse MAT or IntelliJ Profiler. Look for retained size, not shallow size.
• Node.js: node --inspect with Chrome DevTools. Take heap snapshots at T0 and T1; compare to find growing objects. Look for closures holding references, event listeners not removed.
• Go: pprof for CPU (/debug/pprof/profile) and heap (/debug/pprof/heap). go tool pprof -http=:8080 profile.out.
• Universal approach: restart the pod with more memory (temporary) to buy time; then profile, identify the leak, fix, deploy, and reduce memory limit back.

Deployment rollbacks and hotfixes:

• kubectl rollout undo deployment/ -n — reverts to previous revision. Confirm with kubectl rollout status deployment/.
• If the previous image is no longer available: re-tag the last known good commit,