How to Test LLMs, AI Agents and Generative AI Applications
Somatirtha
Promptfoo: Open-source tool that runs prompts against test cases, compares models side by side, and flags regressions before changes reach production.
DeepEval: Python framework that unit-tests LLM outputs for answer relevancy, faithfulness, hallucination and bias, and plugs straight into your CI pipelines.
Ragas: Evaluation library for retrieval-augmented generation that scores context precision, context recall, faithfulness and answer relevance without needing large hand-labeled datasets.
LangSmith: Tracing and evaluation platform that records every agent step, lets teams build test datasets from real traces, and scores runs.
Arize Phoenix: Open-source observability tool that traces LLM calls, surfaces retrieval problems, and runs evaluations to find where applications drift or fail.
Giskard: Testing framework that scans models for hallucinations, prompt injection, bias and sensitive data leaks, then generates reusable regression test suites.
Garak: Open-source vulnerability scanner that probes language models with adversarial prompts to expose jailbreaks, toxic output, data leakage and safety weaknesses.
OpenAI Evals: Framework and registry for writing custom benchmarks, grading model answers against references, and tracking quality changes across different model versions.
Human Review: Domain experts rate sampled outputs for tone, accuracy and usefulness, catching subtle failures that automated metrics and LLM judges miss.