Llm Evaluation Framework Collection

◆

◆

◆

◆

💎Aqua

★★★★

“Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.”

◆

◆

◆

◆

🔮Psychic

★★★

“The LLM Evaluation Framework”

◆

◆

◆

◆

🔮Psychic

★★

“Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪”

◆

◆

◆

◆

🔮Psychic

★

“The testing platform for AI teams. Bring engineers, PMs, and domain experts together to generate tests, simulate (adversarial) conversations, and trace every failure to its root cause.”

◆

◆

◆

◆

🔮Psychic

★

“LangFair is a Python library for conducting use-case level LLM bias and fairness assessments”

◆

◆

◆

◆

🔮Psychic

★

“The official evaluation suite and dynamic data release for MixEval.”

◆

◆

◆

◆

🔮Psychic

★

“ MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.”

◆

◆

◆

◆

🔮Psychic

★

“Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)”

◆

◆

◆

◆

💎Aqua

★

“Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs”

◆

◆

◆

◆

🔮Psychic

★

“An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.”