AgentTest Documentation¶

A pytest-like testing framework for AI agents and prompts

AgentTest is a comprehensive testing framework designed specifically for AI agents, providing evaluation, logging, and regression tracking capabilities similar to pytest but tailored for AI/ML applications.

📚 Documentation Index¶

Getting Started¶

• Installation & Setup - How to install and configure AgentTest
• Quick Start Guide - Your first test in 5 minutes
• Configuration - Complete configuration reference

Core Concepts¶

• Writing Tests - How to structure and write agent tests
• Evaluators - Understanding evaluation metrics and criteria

Advanced Features¶

• Git Integration - Version tracking and comparison features
• CLI Commands - Complete command-line interface reference

Evaluator Reference¶

• String Similarity - Text similarity evaluation
• LLM Judge - AI-powered evaluation
• NLP Metrics - ROUGE, BLEU, METEOR metrics
• Pattern Matching - Regex and contains evaluators

Examples & Tutorials¶

• Basic Examples - Simple test cases

API Reference¶

• Core API - Core classes and functions

🚀 Key Features¶

• 🤖 Intelligent Auto Test Generation: Automatically analyze your code and generate comprehensive test cases with proper imports and function calls
• 🧪 Pytest-like Interface: Familiar CLI and decorator-based testing
• 🧠 Smart Code Analysis: Understands project structure, classes, functions, and generates realistic test data
• Multiple Evaluators: String similarity, LLM judges, NLP metrics, pattern matching
• Git Integration: Track performance across commits with detailed comparisons
• Rich CLI: Comprehensive command-line interface with filtering and export options
• Enhanced Logging: Detailed debugging and structured logging
• Flexible Configuration: YAML-based configuration with environment variable support
• Multiple LLM Providers: OpenAI, Anthropic, Google Gemini support

🏃‍♂️ Quick Example¶

from agent_test import agent_test

@agent_test(criteria=['similarity', 'llm_judge'])
def test_summarization_quality():
    """Test if the agent can summarize text effectively."""
    return {
        "input": "Write a summary of the latest AI developments...",
        "actual": agent_response,
        "expected": "A concise summary highlighting key points...",
        "evaluation_criteria": ["accuracy", "conciseness", "relevance"]
    }

# Run tests with enhanced output
agenttest run --verbose --detailed

# Compare performance across git commits
agenttest compare abc123 HEAD --metric similarity --detailed

# Generate test reports
agenttest run --export results.json --log-output debug.log

🛠️ Installation¶

# Basic installation
pip install agenttest

# With optional dependencies
pip install agenttest[langchain,llamaindex]

# Development installation
pip install agenttest[dev,all]

🔗 Links¶

• Repository: GitHub
• Issues: Bug Reports & Feature Requests
• Examples: Sample Projects

_Need help? Check out our Quick Start Guide or browse the Examples!