Multimodal AI

huggingface/transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

sgl-project/sglang

SGLang is a high-performance serving framework for large language models and multimodal models.

google-ai-edge/mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

ultralytics/yolov5

Ultralytics YOLOv5 in PyTorch for object detection, instance segmentation, classification, training, and export.

NVIDIA-NeMo/Speech

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

huggingface/datasets

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

lancedb/lancedb

Developer-friendly OSS embedded retrieval library for multimodal AI. Search More; Manage Less.

multimodal-art-projection/YuE

YuE2: frontier music generation with symbolic planning, zero-shot covers, and agentic music editing.

bytedance/UI-TARS-desktop

The Open-Source Multimodal AI Agent Stack: Connecting Cutting-Edge AI Models and Agent Infra

apache/seatunnel

SeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.

Neywa-Labs/mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

lance-format/lance

Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and

BradyFU/Awesome-Multimodal-Large-Language-Models

:sparkles::sparkles:Latest Advances on Multimodal Large Language Models

OpenRLHF/OpenRLHF

An Easy-to-use, Scalable and High-performance Agentic RL Framework based on Ray (PPO & DAPO & REINFORCE++ & VLM & TIS & vLLM & Ray & Async RL)

xorbitsai/inference

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready

Blaizzy/mlx-vlm

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

xlang-ai/OSWorld

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Eventual-Inc/Daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

roboflow/inference

Turn any computer or edge device into a command center for your computer vision projects.

roboflow/notebooks

A collection of tutorials on state-of-the-art computer vision models and techniques. Explore everything from foundational architectures like ResNet to cutting-edge models like RF-DETR, YOLO11, SAM 3,

google-research/scenic

Scenic: A Jax Library for Computer Vision Research and Beyond

xLLM-AI/xllm

A high-performance inference engine for LLM, VLM, DiT and REC models, optimized for diverse AI accelerators. It is hosted in OpenAtom Foundation.

open-edge-platform/geti

Build, train, optimize, and run computer vision models locally, from raw images to live inference. Open source, optimized for Intel® XPU (CUDA also supported).

DEEIX-AI/DEEIX-Chat

An enterprise AI workspace for model routing, multimodal chat, files, tools, billing, identity, and operations.

roboflow/maestro

streamline the fine-tuning process for multimodal models: PaliGemma 2, Florence-2, and Qwen2.5-VL

pixeltable/pixeltable

Unified multimodal backend for AI data apps

yanshengjia/ml-road

Machine Learning and Agentic AI Resources, Practice and Research

om-ai-lab/VLM-R1

Solve Visual Understanding with Reinforced VLMs

xlite-dev/Awesome-LLM-Inference

📚A curated list of Awesome LLM/VLM Inference Papers with Codes: Flash-Attention, Paged-Attention, WINT8/4, Parallelism, etc.🎉

aiming-lab/SimpleMem

[ICML'26] SimpleMem: Efficient Lifelong Memory for LLM Agents — Text & Multimodal

yzhao062/anomaly-detection-resources

Anomaly detection related books, papers, videos, and toolboxes. Last update late 2025 for LLM and VLM works!

activeloopai/deeplake

Deeplake is AI Data Runtime for Agents. It provides serverless postgres with a multimodal datalake, enabling scalable retrieval and training.

tong-io/tongflow

TongFlow — Multimodal GenAI Studio

fikrikarim/parlor

On-device, real-time multimodal AI with features similar to GPT-Live

IDEA-CCNL/Fengshenbang-LM

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。

OFA-Sys/Chinese-CLIP

Chinese version of CLIP which achieves Chinese cross-modal retrieval and representation generation.

go-kratos/blades

Blades is a Go-based multimodal AI Agent framework.

kha-white/manga-ocr

Optical character recognition for Japanese text, with the main focus being Japanese manga

langfengQ/verl-agent

verl-agent is an extension of veRL, designed for training LLM/VLM agents via RL. verl-agent is also the official code for paper "Group-in-Group Policy Optimization for LLM Agent Training"

docarray/docarray

Represent, send, store and search multimodal data

InternLM/InternLM-XComposer

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

TeleAI-UAGI/telemem

TeleMem is a high-performance drop-in replacement for Mem0, featuring semantic deduplication, long-term dialogue memory, and multimodal video reasoning.

Dicklesworthstone/franken_ocr

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.

modelstudioai/cli

Official Model Studio CLI（阿里云百炼 CLI）built for AI Agent frameworks, exposing models, search, multimodal, and workflow capabilities as structured tool calls.

autogluon/autogluon-assistant

Multi-Agent System Powered by LLMs for End-to-end Multimodal ML Automation

duckyshell/ComfyUI-MiniMaxH3-Prompt-Writer

Multimodal MiniMax H3 prompt writer for ComfyUI with local and API providers.

ByteDance-Seed/Seed1.5-VL

Seed1.5-VL, a vision-language foundation model designed to advance general-purpose multimodal understanding and reasoning, achieving state-of-the-art performance on 38 out of 60 public benchmarks.

eggbrid2/mobileClaw

Open Android AI agent runtime for phone control, app automation, VLM screen reading, skill routing, mini apps, and Mihomo VPN workflows.

gokayfem/ComfyUI_VLM_nodes

ComfyUI nodes for vision-language models: Qwen3-VL, Moondream 3, Florence-2, SmolVLM2, InternVL, Gemma 3, MiniCPM-V. Plus open-vocabulary detection, SAM2/SAM3 segmentation, video temporal reasoning, G

unum-cloud/UForm

Pocket-Sized Multimodal AI for content understanding and generation across multilingual texts, images, and 🔜 video, up to 5x faster than OpenAI CLIP and LLaVA 🖼️ & 🖋️

modelscope/mcore-bridge

MCore-Bridge: Providing Megatron-Core model definitions for state-of-the-art large models and making Megatron training as simple as Transformers — with support for 300+ large language models (Qwen3-Ne

pipeless-ai/pipeless

An open-source computer vision framework to build and deploy apps in minutes

yycyyv/M-Cube

M-Cube (M³) — Multi-thinking, Multimodal, Multi-verification Patent Drafting Assistant

shure-dev/small-vlm-sop-check

Catch procedure errors before they become incidents. Train and evaluate small VLMs for temporal grounding in first-person factory video.

zjysteven/lmms-finetune

A minimal codebase for finetuning large multimodal models, supporting llava-1.5/1.6, llava-interleave, llava-next-video, llava-onevision, llama-3.2-vision, qwen-vl, qwen2-vl, phi3-v etc.

Coobiw/MPP-LLaVA

Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train

yuhangzang/ContextDET

Contextual Object Detection with Multimodal Large Language Models

Genesis1231/Eva01

Eva01 is NOT an assistant. She is an AI being with her own mind, feelings, and intrinsic drives. Multimodal, Modular design. Built-in voice & face recognition. Plug'n play tools. Compatible with ChatG

john-rocky/swift-litert-lm

Gemma 4 on iPhone as a Swift package: one dependency line, five lines of code, ~50 tok/s on iPhone 17 Pro. Metal GPU, text + image + audio, in-app download. Also an Apple Foundation Models backend (iO

westlake-repl/IDvs.MoRec

End-to-end Training for Multimodal Recommendation Systems

yassa9/dvlt.cu

Suckless no dependencies CUDA/C++ port of NVIDIA's DVLT, feed it images, get a 3D point cloud + camera poses. NO python. One fast 5MB binary.

cobusgreyling/Muse-Glimmer

Introducing Muse Glimmer: open-weight 30B agentic multimodal model that runs on your device (Meta). Interactive local agent lab + guide. Apache 2.0 · on-device AI · function calling

westlake-repl/PixelRec

A Large-scale Multimodal Dataset for recommender System

Hai-chao-Zhang/ThinkJEPA

[NeurIPS 2026] ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

GameTechDev/IntelAIGamingAssistantLibrary

Intel AI Gaming Assistant Library is an on-device AI backend for in-game assistance, enabling screenshot understanding, multimodal retrieval, knowledge-base Q&A, and memory management with Intel GPU/N

RainBowLuoCS/OpenOmni

(NIPS 2025) OpenOmni: Official implementation of Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis

cnzzx/VSA

Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

LAMDA-CL/Prism

Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning

MIT-RLX/rlx-models

ML model ports (ASR · TTS · LLM · VLM · vision · audio codecs) on the rlx multi-backend compiler + runtime — CPU/Metal/MLX/CUDA/wgpu

JaeBinCHA7/Nested-U-Net-based-Real-time-Speech-Enhancement-Mobile-App

Real-time speech enhancement mobile app using Nested U-Net

TouHouQing/alicization

Build your own local-first digital lifeform: an AI companion with memory, soul, multimodal perception, proactive dialogue, and auditable execution.

Apageoflove/VLM-Driving-Scene-Understanding

基于Qwen2.5-VL-3B + QLoRA的智能驾驶场景结构化理解系统

jaeseok614/llm-gpu-checker-ko

Find local AI models for your GPU. Open-source VRAM, quantization and speed estimates in your browser — no install or sign-up. 한국어·English.

MiG-NJU/PersonaVLM

[CVPR 2026 Highlight] PersonaVLM: Long-Term Personalized Multimodal LLMs

yhzhu99/EMERGE

[CIKM 2024] EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation

cosmo-wander-ai/cosmo-edge

Production-grade C++ edge AI engine for video analytics and on-device VLM across Sophon, Rockchip RKNN, and x86, with visual orchestration, real-time OSD, events, and reproducible benchmarks.

yuezhiai/jonex

All-in-One Multimodal Parsing Engine + Ontology-Powered, LLM Wiki-Driven AI-Ready Knowledge Engine

NPC-Worldwide/npcpy

The python library for research and development in NLP, multimodal LLMs, Agents, ML, Knowledge Graphs, and more.

XiaomiMiMo/MiMo-Audio

MiMo-Audio: Audio Language Models are Few-Shot Learners

Tencent-Hunyuan/UniRL

UniRL is a Framework for Unified Multimodal Model Reinforcement Learning

whitecircle/halo

Halo is an open-source framework built by White Circle for training large language and multimodal models

Haervwe/open-webui-tools

Open‑WebUI Tools is a modular toolkit designed to extend and enrich your Open WebUI instance, turning it into a powerful AI workstation. With a suite of over 15 specialized tools, function pipelines,

Yangyi-Chen/Multimodal-AND-Large-Language-Models

Paper list about multimodal and large language models, only used to record papers I read in the daily arxiv for personal needs.

OpenMOSS/MOSS-VL

An open-weight 11B model series for long-form and real-time video understanding

Code-Amadeus/Amadeus

Real-time multimodal desktop agent evolving toward a persistent AI OS interface (0.17 α).

lablup/mlxcel

High-performance LLM, VLM, embedding, reranking, and audio inference for Apple Silicon, NVIDIA CUDA and AMD ROCm systems.

ZhanYang-nwpu/Awesome-Remote-Sensing-Multimodal-Large-Language-Model

Multimodal Large Language Models for Remote Sensing (RS-MLLMs): A Survey

PipeNetwork/kimi-k3-mlx

MLX port of moonshotai/Kimi-K3 (2.78T multimodal MoE): streaming converter, REAP expert pruning, and per-language expert-overlap analysis

OmniJev/OneJev

🚀🚀 A multimodal System One decision model that gives calibrated answers to typed questions about screens, photos, video and text in one forward pass.

ForeverBlue816/GRACE

[ICML 2026] GRACE: deployable INT4 VLM via quantization-aware distillation.

zqc3117/Spotter

Official codebase for Spotter: Let the Embodied Model Lead, and the VLM Reflect for It

HsienW/chat-gun-react-agent

🤖 A full-stack Agent Chat Runtime for pre-production validation, built with act, TypeScript, and LangGraph JS combining streaming, multimodal, HITL, and tools with durable Task/Step execution, retry b

1115714829/llama.cpp-v100

llama.cpp optimized for NVIDIA Tesla V100 (SM70). Qwen3.8-Flash-Next (MoE) with MTP on 1/2/3/6x V100-16GB at 262K context (6 GPUs: prefill 3402 tok/s at 200K; 2 GPUs on one socket: 2152 tok/s). Qwen3.

ProjectArgus-cc/libargus.cc

Unified native AI runtime for Java 17+, Java 22+ (Panama FFM), Android & C++. Run GGUF LLMs, Vision, Video, Whisper ASR, TTS & Embeddings in-process.