Amanyi
Daniel

Managed Kubernetes services (EKS, GKE) abstract too much. When production incidents happen, I need to understand what the control plane is actually doing — not just what the cloud vendor reports. TITAN exists to study cluster behavior under real failure conditions: node loss, network partition, etcd degradation, and resource exhaustion.

Three KVM guests on a single physical host. One control plane node, two workers. Libvirt manages VM lifecycle; networking is intentionally constrained to simulate real inter-node latency. The point is not to replicate production — it's to create a controlled environment where I can inject failures and observe cascading behavior without paging anyone at 2am.

• Single-host colocation — failure domains overlap (intentional for this phase)
• No cloud networking — must simulate latency and partition manually
• Runbook-first discipline: every failure injection has a pre-defined runbook before execution
• Phase (-1) is purely a fluency gate — no research claims until baseline competency is verified

The brief was to ingest high-velocity telemetry events, fan them out across three storage backends (hot, cold, and search), and do it without managing servers. The real constraint: the cost model had to stay predictable — no provisioned capacity bills appearing at end of month.

Kinesis Data Streams as the ingest buffer gives replay capability and decouples producers from consumers. Lambda shards off the stream — one function per concern. DynamoDB for hot read access (single-digit ms latency SLA), S3 for cold archival, OpenSearch for full-text query. The fan-out happens inside Lambda, not via SNS — this reduces hop count and keeps the critical path tight.

• Lambda cold start latency on the OpenSearch write path needed a keep-warm strategy
• DynamoDB write amplification from hot partition keys — resolved with composite key design
• IAM scope creep: initial role was too permissive; hardened to resource-level least privilege
• OpenSearch cluster sizing: started too small, index rotation policy required mid-flight

Client had 14 microservices communicating over plain HTTP inside a Kubernetes cluster. Compliance requirements mandated in-transit encryption with mutual authentication — not just TLS, but provable service identity. The team had no PKI experience and needed an operational solution, not a proof of concept.

Internal CA provisioned via cert-manager on the cluster. Certificates issued per-service using workload identity (SPIFFE). mTLS enforced at the sidecar proxy layer — no application code changes required. Certificate rotation automated with a 24-hour TTL; rotation failures trigger PagerDuty before expiry. Root CA stored offline; intermediate CA rotated annually.

• Sidecar proxy adds ~15ms p99 latency per hop — acceptable given compliance requirement
• Vault PKI backend chosen over self-managed CA for audit trail and secrets governance
• Short-lived certs (24h) increase rotation ops burden — mitigated by full automation
• SPIFFE over manual CN — harder to implement, but eliminates human error in cert naming

A 21-chapter practitioner-focused manual covering infrastructure resilience from first principles. Five main parts plus a philosophy chapter and bridge chapter — written from operational experience, not theory. Covers failure modes, recovery design, observability, runbook discipline, and the human factors in incident response.

Most resilience content is either too abstract (academic papers) or too narrow (vendor documentation). SRIM is the resource I wished existed when I started dealing with production incidents — opinionated, practitioner-grade, and grounded in systems that actually failed. Writing it forced me to articulate assumptions I'd been operating on implicitly for years.

Invoice processing is a high-volume, error-prone manual operation in most finance workflows. The goal was to build a production-ready serverless pipeline that handles raw document uploads, extracts structured fields using real AI services, and persists results — all with pay-per-use cost characteristics and zero server management.

Two Lambda functions — one for upload handling (API Gateway → S3), one for inference — decoupled via S3 event trigger. Textract handles OCR and form extraction; Bedrock (Claude 3 Haiku) runs intelligent field normalization for vendor names, amounts, and dates that rule-based extraction misses. Results land in DynamoDB for downstream consumption. All infra is modular Terraform — six discrete modules covering compute, storage, IAM, API Gateway, and both Lambda functions.

• Hybrid extraction strategy — rule-based for common patterns, AI for edge cases — keeps Bedrock costs to ~$0.0005/invoice
• S3 event trigger over SNS: fewer hops, tighter coupling acceptable here since upload and inference are always paired
• DynamoDB as output store: optimized for downstream single-key reads, not analytics — a warehouse would be added for reporting use
• Modular Terraform over a monolith: each AWS resource category is independently versioned and reusable across projects
• Total per-invoice cost: ~$0.002 — Textract dominates at $0.0015/page