This page is an effort to collate and capture the state of various local & community LLM hosting projects, in an effort to help guide refugees from the large centralized players. The initial collection of projects was initially generated with an LLM (Claude Sonnet high thinking) so please consider adding any personal experiences, advocacy, warnings, etc.
If your nontechy neighbour asks about AI, what would you recommend?
Local LLM Tools Comparison
| Project | Category | Mission / Positioning | Platform(s) | Engine/Backend | Model Formats | Interface | API Compatibility | License |
|---|---|---|---|---|---|---|---|---|
| Ollama | Inference server | Simple, single-command local model management — the de facto default backend many other tools build on | macOS, Windows, Linux, Docker | Own engine (llama.cpp-derived) | GGUF, Modelfile | CLI-first; minimal official GUI, large third-party UI ecosystem | OpenAI-compatible + native API | Open source (MIT) |
| LM Studio | Desktop chat app | Polished, beginner-friendly app for discovering, downloading, and running local models | macOS, Windows, Linux | llama.cpp, MLX (Apple Silicon) | GGUF, MLX | Full native chat GUI with model browser | Local OpenAI-compatible server; MCP tool support | Free, proprietary |
| Jan | Desktop chat app | Open-source, privacy-first “ChatGPT alternative” that runs entirely on your own machine | macOS, Windows, Linux | llama.cpp | GGUF | Native chat app with assistants/threads | Local OpenAI-compatible server; optional cloud provider passthrough | Open source (AGPLv3) |
| Osaurus | Inference server / edge runtime | Lightweight, native Apple Silicon–only LLM server, expanding into a full agentic “AI edge runtime” | macOS (Apple Silicon only) | MLX, Apple Foundation Models (macOS 26+) | MLX | Built-in chat UI; server-first | OpenAI, Anthropic, and Ollama-compatible endpoints | Open source (MIT) |
| Tiles | Desktop chat app | Private, collaborative AI assistant built on AT Protocol, emphasizing user-owned identity and encrypted peer-to-peer sync | macOS (Apple Silicon), Linux (NVIDIA) — public alpha | llama.cpp, embedded “Pi” agent harness | GGUF | Native app; browser WebUI planned | — | Open source (MIT / Apache 2.0) |
| GPT4All | Desktop chat app | Fully offline, private AI that runs on ordinary consumer hardware, including CPU-only | macOS, Windows, Linux | llama.cpp-based, Vulkan (CPU) | GGUF | Native chat app with built-in document (LocalDocs) RAG | Local API server | Open source (MIT) |
| Msty Studio | Desktop chat app | Polished multi-model desktop client with split-view chat and local + cloud model switching | macOS, Windows, Linux | llama.cpp, Ollama | GGUF | Native chat GUI | Local API server | Freemium (paid “Aurum” tier) |
| text-generation-webui (oobabooga) | Desktop chat app | Maximally configurable local chat interface for sampler-level and format tuning | macOS, Windows, Linux | llama.cpp, Transformers, ExLlama, others | GGUF, GPTQ, EXL2, AWQ | Web-based local GUI | OpenAI-compatible server | Open source (AGPLv3) |
| Open WebUI | Self-hosted web platform | Feature-rich, self-hosted chat interface for teams, built primarily for Ollama and OpenAI-compatible backends | Docker, self-hosted (any OS) | Backend-agnostic (Ollama, OpenAI-compatible APIs) | Depends on backend | Browser-based, multi-user | Consumes OpenAI-compatible APIs | Open source (MIT) |
| AnythingLLM | Self-hosted web platform | All-in-one document/RAG workspace with zero-config retrieval and built-in agents (web search, SQL) | Desktop (macOS/Windows/Linux) + Docker | Backend-agnostic (Ollama, LocalAI, OpenAI-compatible) | Depends on backend | Native app or browser (Docker mode) | Consumes and exposes OpenAI-compatible APIs | Open source (MIT) |
| LibreChat | Self-hosted web platform | Enterprise-ready, open-source multi-model chat platform with agents, code execution, and persistent memory | Docker, self-hosted | Backend-agnostic | Depends on backend | Browser-based, multi-user | Consumes multiple provider APIs, including local | Open source (MIT) |
| PrivateGPT | Self-hosted web platform | Framework for fully private, offline document interaction, built for strict air-gapped deployments | Local, Docker, cloud | llama.cpp, LlamaIndex | GGUF | API-first; minimal bundled UI | REST API | Open source (Apache 2.0) |
| LocalAI | Inference server | Drop-in, OpenAI-API-compatible REST server for local inference; infrastructure component rather than a chat app | Docker, Linux, macOS, Windows | llama.cpp and others (ggml-based) | GGUF and others | None (API only) | OpenAI-compatible REST API | Open source (MIT) |
| llama.cpp | Inference engine | The foundational C/C++ inference engine underlying most GGUF-based local LLM tools | Cross-platform, including low-resource hardware | Native | GGUF | Minimal CLI/server; bundled web UI | OpenAI-compatible server mode | Open source (MIT) |
| vLLM | Inference engine | High-throughput serving engine optimized for concurrent, production-scale inference rather than single-user chat | Linux, Docker | Native (PagedAttention) | Safetensors, GGUF (limited) | None (API only) | OpenAI-compatible REST API | Open source (Apache 2.0) |
| koboldcpp | Inference engine / chat app | Single-binary llama.cpp fork popular for story-writing and roleplay use cases | macOS, Windows, Linux | llama.cpp fork | GGUF | Bundled lightweight web UI | OpenAI-compatible + native API | Open source (AGPLv3) |
Notes
- “Engine/Backend” refers to the underlying inference technology; several front-end tools (Open WebUI, AnythingLLM, LibreChat) are backend-agnostic and typically run against Ollama, LocalAI, or another OpenAI-compatible server rather than performing inference themselves.
- Many details - especially license and platform details are subject to change; verify against each project’s official repository before citing as current.