The LLM Evaluation Framework

Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪

Open-source platform & SDK for testing LLM and agentic apps. Define expected behavior, generate and run test scenarios, and review failures collaboratively.

The official evaluation suite and dynamic data release for MixEval.

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

Develop reliable AI apps

Benchmarking Large Language Models for FHIR

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

Open source framework for evaluating AI Agents

FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts

Realign is a testing and simulation framework for AI applications.

Evaluation Framework for LLM applications in Java and Kotlin

Code for "Prediction-Powered Ranking of Large Language Models", NeurIPS 2024.

Measure of estimated confidence for non-hallucinative nature of outputs generated by Transformer-based Language Models.

Create an evaluation framework for your LLM based app. Incorporate it into your test suite. Lay the monitoring foundation.

MindTrial: Evaluate and compare AI language models (LLMs) on text-based tasks with optional file/image attachments and tool use. Supports multiple providers (OpenAI, Google, Anthropic, DeepSeek, Mistral AI, xAI, Alibaba, Moonshot AI, OpenRouter), custom tasks in YAML, and HTML/CSV reports.