The open-source llama.cpp project has significantly expanded its capabilities by adding comprehensive multimodal vision support to both its server and command-line interface tools. This integration allows users to run vision-language models locally on their own hardware, enabling image description and analysis without relying on cloud services.
Unified Multimodal Implementation
The new implementation consolidates previously separate vision functionalities under a unified framework. According to community discussions, developer ngxson played a key role in this effort, first adding support for various vision models with individual CLI programs, then unifying them under a single command-line tool called llama-mtmd-cli, and finally bringing this capability to the server component. The multimodal support works through a library called libmtmd that handles the image-to-embedding preprocessing separately from the main language model.
This architectural approach mirrors how text preprocessing evolved in the transformer ecosystem, with specialized libraries handling tokenization separately from the core models. The separation allows for optimization specific to image processing while maintaining compatibility with the broader llama.cpp framework.
Supported Models and Performance
The implementation supports an impressive range of multimodal models, including Gemma 3 (in 4B, 12B, and 27B variants), SmolVLM models, Pixtral 12B, Qwen2 VL, Qwen2.5 VL, and Mistral Small 3.1. Users have reported particularly good experiences with the Gemma 3 4B model, which despite its relatively small size delivers impressive image descriptions.
Performance reports from the community indicate that on an M1 MacBook Pro with 64GB RAM, the Gemma 3 4B model processes prompts at about 25 tokens per second and generates tokens at 63 per second. Image processing takes approximately 15 seconds regardless of image size. This level of performance makes these models practical for real-world applications on consumer hardware.
Supported Multimodal Models
-
Gemma 3 Series
- ggml-org/gemma-3-4b-it-GGUF
- ggml-org/gemma-3-12b-it-GGUF
- ggml-org/gemma-3-27b-it-GGUF
-
SmolVLM Series
- ggml-org/SmolVLM-Instruct-GGUF
- ggml-org/SmolVLM-256M-Instruct-GGUF
- ggml-org/SmolVLM-500M-Instruct-GGUF
- ggml-org/SmolVLM2-2.2B-Instruct-GGUF
- ggml-org/SmolVLM2-256M-Video-Instruct-GGUF
- ggml-org/SmolVLM2-500M-Video-Instruct-GGUF
-
Pixtral
- ggml-org/pixtral-12b-GGUF
-
Qwen 2 VL
- ggml-org/Qwen2-VL-2B-Instruct-GGUF
- ggml-org/Qwen2-VL-7B-Instruct-GGUF
-
Qwen 2.5 VL
- ggml-org/Qwen2.5-VL-3B-Instruct-GGUF
- ggml-org/Qwen2.5-VL-7B-Instruct-GGUF
- ggml-org/Qwen2.5-VL-32B-Instruct-GGUF
- ggml-org/Qwen2.5-VL-72B-Instruct-GGUF
-
Mistral Small
- ggml-org/Mistral-Small-3.1-24B-Instruct-2503-GGUF
Real-World Applications
Community members have already begun putting these capabilities to practical use. One user described creating a system to generate keywords and descriptions for vacation photos, noting that the Gemma 3 4B model was able to extract meaningful information including basic OCR (optical character recognition) from images containing text and identifying contextual location information.
The SmolVLM series of models has been highlighted as particularly suitable for real-time applications like home video surveillance due to their small size and fast response times. These models range from just 256MB to 2.2GB, making them accessible even on devices with limited resources.
Performance Metrics (Gemma 3 4B on M1 MacBook Pro 64GB)
- Prompt processing: 25 tokens/second
- Token generation: 63 tokens/second
- Image processing time: ~15 seconds per image (regardless of size)
Installation and Usage
Getting started with the multimodal capabilities is straightforward. Users can download pre-compiled binaries from the llama.cpp GitHub releases page or install via package managers like Homebrew. The tools can be run with simple commands that specify the model to use, with options to control GPU offloading for improved performance.
For those on macOS using Homebrew, the package will be updated to include these new capabilities, allowing users to simply run brew upgrade llama.cpp to get the latest features. The implementation automatically leverages GPU acceleration where available, with Metal backend users benefiting from automatic layer offloading.
This development represents a significant step forward for edge AI capabilities, bringing powerful vision-language models to local devices without requiring cloud connectivity or subscription services. As these tools continue to mature, we can expect to see an increasing number of applications that leverage multimodal AI for personal and professional use cases.
Reference: Multimodal