OwnersClub.ai

Knowledge

The local-first AI stack

How runtime, inference, agents, and interfaces fit together when you run AI on your own hardware.

local-firstarchitectureconcepts

Draft: not yet reviewed/published by a human.

The local-first AI stack

"Local AI" is not one program. It is a stack of layers, and almost every confusing failure happens at a boundary between two of them.

The layers

Layer Job Examples
Container runtime Provides isolation and repeatable environments Docker Desktop, Podman
Inference engine Loads model weights and generates tokens llama.cpp inside Unsloth Desktop and Jan
Model The weights themselves, usually quantized GGUF Qwen, Gemma, Llama, DeepSeek variants
Serving Exposes the model as an API Unsloth's OpenAI-compatible endpoint, Jan's local API server
Agent / app Consumes the API and does work Agent Zero, Hermes Agent
Interface What you actually look at Desktop chat UI, web UI, Telegram bot

The pattern that matters

Once a model is served on an OpenAI-compatible endpoint, everything above the serving layer stops caring which app produced it. That single idea is why we recommend proving a local endpoint early:

  1. Start a model in Unsloth Desktop or Jan.
  2. Note the endpoint (host, port) and confirm it answers.
  3. Point every other tool — agents, editors, scripts — at that endpoint.

Swapping the model or the engine then becomes a configuration change, not a migration.

The boundary that breaks

The most common failure is container-to-host networking. Inside a container, localhost is the container itself. A model server running on your host is not at 127.0.0.1 from inside the agent's container. Use host.docker.internal (Docker) or host.containers.internal (Podman Desktop), and make sure the server is bound to an interface the container can reach — not just loopback.

That is documented per-app in Agent Zero and the Hermes devcontainer, because it is the reason local agent setups "do not work" for an evening.

Where the data actually is

  • Model weights live on disk, often tens of gigabytes. Keep them outside directories you sync to git or to a cloud drive that will re-upload them forever.
  • Chats and settings live in each app's own data folder or volume. Containerised agents keep theirs in a named volume; delete the volume and the state is gone.
  • Secrets belong in an environment file or keychain, never in a repository.

Choosing layers

  • You rarely need every layer on day one. Interface + inference is enough to start.
  • Add the container runtime when you add an agent — that is when containment starts to matter.
  • Add serving (the API) before you add a second consumer of the model.

See Choosing local AI tools for the goal-first version of this decision.

Related

The networking behaviour above is taken from Agent Zero's official installation documentation, which documents the container-internal localhost caveat explicitly.