Host Requirements
Docker: Installation Guide
Docker Compose: Installation Guide
Compatible with Linux and Windows Host
Ensure port 8501 and 11434 are not already in use
At least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models. Source
Project can be run on either CPU or GPU
Running on GPU
NVIDIA Container Toolkit (Linux) Installation Guide
NVIDIA CUDA Toolkit (Windows) Installation
WSL (Windows) Installation
Tested Model(s)
Model Name |
Size |
Link |
|---|---|---|
llava:7b |
4.7GB |
|
llava:34b |
20GB |
Llava is pulled and loaded by default, other models from Ollama can be added into ollama/ollama-build.sh.