Model Serving Collection

◆

◆

◆

◆

🔮Psychic

★★★★★

“A high-throughput and memory-efficient inference and serving engine for LLMs”

◆

◆

◆

◆

🔮Psychic

★★★

“The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!”

◆

◆

◆

◆

🌊Water

★★★

“Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes”

◆

◆

◆

◆

🔮Psychic

★★★

“A framework for efficient model inference with omni-modality models”

◆

◆

◆

◆

🌊Water

★★

“Olares: An Open-Source Personal Cloud to Reclaim Your Data”

◆

◆

◆

◆

📦Normal

★★

“In this repository, I will share some useful notes and references about deploying deep learning-based models in production.”

◆

◆

◆

◆

🔮Psychic

★★

“LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.”

◆

◆

◆

◆

📦Normal

★★

“🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials. ”

◆

◆

◆

◆

🔮Psychic

★★

“FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.”

◆

◆

◆

◆

🔮Psychic

★★

“Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs”

◆

◆

◆

◆

🔮Psychic

★★

“High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.”

◆

◆

◆

◆

🔥Fire

★★

“Community maintained hardware plugin for vLLM on Ascend”

◆

◆

◆

◆

🌊Water

★★

“🏕️ Reproducible development environment for humans and agents”

◆

◆

◆

◆

⚙️Steel

★★

“AICI: Prompts as (Wasm) Programs”

◆

◆

◆

◆

🔮Psychic

★★

“MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.”

◆

◆

◆

◆

⚙️Steel

★★

“OpenLake is a high performance storage engine for efficient LLM inference and GPU Training”

◆

◆

◆

◆

🌊Water

★★

“An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.”

◆

◆

◆

◆

⚡Thunder

★★

“Hopsworks - Data-Intensive AI platform with a Feature Store”