Evals & Production

What you'll learn

1. 1

   Evals: Testing LLM Systems

   If you can't measure it, you can't safely change it.

   • Unit tests for fuzzy outputs
   • Watch: build a mini eval suite
   • Score a classifier
   • Judge biases
   • Pick the cheapest reliable grader
   • The chatbot that swore

   Take-home template: Eval Harness + LLM-Judge Rubric

2. 2

   Prompt Injection & Guardrails

   The #1 security risk in LLM apps, and how to reduce it.

   • When data becomes instructions
   • Find the injection
   • Spot the attack
   • Wrap untrusted content
   • Defense in depth
   • A Chevy Tahoe for $1

   Take-home template: Prompt-Injection Defense Checklist

3. 3

   Reliability, Cost & Observability

   Retries, caching, model routing and the metrics that matter in production.

   • Production is a different sport
   • Exponential backoff
   • Make caching work
   • What to log

   Take-home template: Resilient LLM Call: Backoff, Jitter, Timeouts

4. 4

   Observability: Traces, Latency & Cost

   See inside every request: spans, p95 latency and cost per feature.

   • You can't fix what you can't see
   • Compute p95 latency
   • Find the bottleneck
   • Cost per feature

   Take-home template: Minimal LLM Tracer

5. 5

   Cutting LLM Costs: Caching, Batching & Routing

   The levers teams use to cut AI bills by half or more, with real numbers.

   • Four big levers
   • What does caching save?
   • Is a cascade worth it?
   • Real-time or batch?

   Take-home template: LLM Cost-Cutting Playbook