Llm Eval Collection

◆

◆

◆

◆

💎Aqua

★★★★

“Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.”

◆

◆

◆

◆

🔮Psychic

★★★

“AI Observability & Evaluation”

◆

◆

◆

◆

🔮Psychic

★★★

“🐢 Open-Source Evaluation & Testing library for LLM Agents”

◆

◆

◆

◆

🔮Psychic

★★

“Evaluation and Tracking for LLM Experiments and AI Agents”

◆

◆

◆

◆

🔮Psychic

★★

“UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.”

◆

◆

◆

◆

💎Aqua

★★

“A desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.”

◆

◆

◆

◆

💎Aqua

★

“Complete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR”

◆

◆

◆

◆

🔮Psychic

★

“Python SDK for running evaluations on LLM generated responses”

◆

◆

◆

◆

💻Normal

★

“An open-source benchmark for evaluating AI agents' ability to find backdoors hidden in compiled binaries.”

◆

◆

◆

◆

🔮Psychic

★

“A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.”

◆

◆

◆

◆

🔮Psychic

★

“Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)”

◆

◆

◆

◆

🔮Psychic

★

“llm-eval-simple is a simple LLM evaluation framework with intermediate actions and prompt pattern selection”

◆

◆

◆

◆

🔮Psychic

★

“First-of-its-kind AI benchmark for evaluating the protection capabilities of large language model (LLM) guard systems (guardrails and safeguards)”

◆

◆

◆

◆

💎Aqua

★

“Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs”

◆

◆

◆

◆

📦Normal

★

“Бенчмарк сравнивает русские аналоги ChatGPT: Saiga, YandexGPT, Gigachat”