Eval Harness: measure before you ship

# Eval Harness — measure your AI before you ship it

A tiny but real evaluation harness. It loads cases from JSONL, runs your model, checks required and forbidden text, prints failures and exits non-zero in CI.

## Run it

python3 evals.py
python3 -m pytest -q     # 5 passed

Try breaking the demo model and watch the score drop. That is the point: a prompt change is only an improvement if the eval score says so.

## Add cases

Append one JSON object per line to data/cases.jsonl:

{"name": "refund window", "input": "How long do refunds take?", "expected": "14 days", "forbidden": "30 days"}

Use real user questions (with PII removed). Start with 30, grow to hundreds as you find failures in production.

## Put it in CI

- name: Evals
  run: python evals.py

A failed eval fails the build, so a prompt regression cannot reach users silently.

MIT licensed.