Your Own Private AI Assistant: Building a Local LLM with AnythingLLM and Ollama

A private AI assistant that answers questions from your own documents no longer needs a cloud subscription or a data-sharing agreement. A recent desktop PC with a mid-range graphics card can run a capable open model, read your procedures, reports and contracts, and answer with citations, without a byte leaving the building.

This guide builds one on a single Windows PC with two free tools: Ollama, which runs the models, and AnythingLLM Desktop, which handles the documents, the chat and the agents. It covers each setting that decides how well the assistant works: the AI provider, the embedding model, the maximum embedding chunk length, the vector database, the text splitter, agent skills, and finally accuracy and security.

Written against AnythingLLM Desktop v1.19 and Ollama 0.40 in November 2026. Menu names move between releases; the settings themselves have been stable.

What you are building

AnythingLLM is the front end. It stores your documents, splits them into chunks, keeps their vectors in a built-in database and runs the chat. It does not run the model itself in this setup: it sends requests to Ollama on the same PC, which loads the models onto the graphics card.

The example PC throughout is a Windows 11 desktop with an Intel Core Ultra 7, 32 GB of RAM and an NVIDIA RTX 5070 with 12 GB of video memory (VRAM). VRAM is the number that matters most: the model has to fit in it to run at full speed.

"Training" the model: what actually happens

People ask how to train their local model on their documents. In almost every case the answer is: you do not. AnythingLLM uses retrieval-augmented generation (RAG). The model's weights never change. Instead, for each question the most relevant passages of your documents are found and placed in front of the model, which writes its answer from them.

Start with RAG and a good system prompt. Fine-tune only when you can show, with examples, a failure that better retrieval and instructions cannot fix. A fine-tuned model can be loaded into Ollama and selected in AnythingLLM like any other.

Step 1: Install Ollama and pull two models

Install Ollama for Windows from ollama.com. It runs in the background and listens on http://127.0.0.1:11434. Then, in a terminal, download a chat model and an embedding model:

ollama pull gemma4:12b
ollama pull embeddinggemma
ollama run gemma4:12b "Reply with OK"
ollama ps

ollama ps lists the loaded models. The PROCESSOR column should read 100% GPU. If it shows a CPU share, the model plus its context window does not fit in VRAM: choose a smaller model or a shorter context, or answers will be several times slower.

Rule of thumb only: a 4-bit model needs roughly 0.6 GB per billion parameters, plus room for the context window.

Step 2: Install AnythingLLM and choose the AI provider

Download AnythingLLM Desktop from anythingllm.com and run the installer. No account is needed. The first-run wizard asks for an LLM provider; you can change every choice later under Settings > AI Providers.

AnythingLLM can call many providers. The choice decides where your questions and document snippets are sent.

Select Ollama, keep the base URL, and pick gemma4:12b. Set the context window (Max Tokens) to the size Ollama loads the model with; a mismatch either wastes the window or lets text be cut without warning.

Step 3: The embedding model and its maximum chunk length

The embedding model turns text into a vector: a list of numbers that places similar meanings close together. Every chunk of every document is embedded once; every question is embedded when it is asked, and the nearest chunks are retrieved. Retrieval quality depends more on the embedder than most people expect.

Under Settings > AI Providers > Embedder, choose Ollama and select embeddinggemma. Ollama lists chat and embedding models in the same dropdown, so make sure you pick the embedder.

Max Embedding Chunk Length tells AnythingLLM the largest piece of text, in characters, that the embedding model accepts. It is a ceiling, not the chunk size. If the text splitter's chunk size is set above it, AnythingLLM quietly uses this limit instead and logs a warning. Set it from the model's token limit at about four characters per token: embeddinggemma reads 2,048 tokens, or roughly 8,000 characters of English. With 1,000-character chunks you are far below it either way.

Changing the embedding model makes every stored vector meaningless, because the old and new models place text in different spaces. Pick the embedder before you load your library, and re-embed everything if you change it.

Step 4: The vector database

The vector database stores each chunk's vector next to its original text and finds the nearest ones to a question. AnythingLLM ships with LanceDB built in. It needs no configuration, keeps its files inside AnythingLLM's storage folder, and handles libraries of many thousands of documents on one PC.

Like the embedder, a change of vector database starts with an empty store, and every document has to be embedded again.

Step 5: Text splitting and chunking

Before a document is embedded it is cut into chunks. Each chunk becomes one vector, and chunks, not whole documents, are what the model sees when it answers. AnythingLLM uses a recursive splitter: it tries to cut at a paragraph break, then at a line break, then at a space, and splits a word only as a last resort. Two settings control it, under Settings > AI Providers > Text Splitter & Chunking.

• Text Chunk Size: the maximum length of a chunk, in characters (not tokens). The default is 1,000, about 250 tokens.
• Text Chunk Overlap: how many characters each chunk repeats from the end of the one before. The default is 20.

The chunk size is a trade-off. Small chunks match precisely but may lose the sentence that gives them meaning. Large chunks keep context together but bring irrelevant text with them, and fewer fit in the model's context window. The table works through one example: a 100-page manual of about 300,000 characters, with four chunks retrieved per question.

Three practical points. Change one setting at a time and test with the same questions. New settings apply only to documents embedded afterwards, so remove and re-add documents to compare. And raise the overlap to 10% to 20% of the chunk size (100 to 200 characters at the default) for procedures and contracts, where one sentence often depends on the one before it.

Clean input matters as much as the numbers. Scanned PDFs without a text layer, multi-column layouts, and tables flattened into a stream of numbers all produce poor chunks. Converting important documents to clean text or Markdown before uploading usually helps more than any slider.

Step 6: Create a workspace and tune retrieval

A workspace is a separate assistant with its own documents, system prompt and settings. The model can only see documents embedded in that workspace. Create one per subject (for example "Plant procedures" and "Client contracts") rather than one for everything; a smaller pool of chunks gives cleaner matches.

Drag documents into the workspace and choose Move to Workspace to embed them. Then open the workspace settings:

Two more tools help. Pinning a document sends its full text with every question instead of a few chunks, which suits one short, critical document if it fits the context window. Attaching a file to a single chat does the same for a one-off question without adding it to the library.

Step 7: Agent skills

A normal chat only answers. An agent can act: search the documents again, summarize a whole file, browse the web, save a file, query a database or call another program. Start one by typing @agent in a chat. Under Settings > Agent Skills each skill has its own switch.

Beyond the built-in skills, AnythingLLM can load custom skills (small JavaScript programs placed in the plugins/agent-skills folder of its storage directory), connect to MCP servers that expose other tools, and run Agent Flows, fixed step-by-step workflows built without code. Its intelligent tool selection sends only the tools relevant to each request to the model, which keeps prompts short for local models.

Agents ask more of the model than plain Q&A: it has to decide which tool to call and format the call correctly. A 12-billion-parameter model handles simple, single-tool requests well and becomes unreliable on long multi-step tasks. If the agent ignores its tools, try a model trained for tool use or a larger one before blaming the settings.

Accuracy: testing and fixing answers

Do not judge a setup on three questions. Write 20 to 30 questions whose answers you know and whose source page you can name. Run them after every change and record, for each one, whether the right chunk was retrieved (shown in the citations) and whether the answer was right. Those are two different failures with different fixes.

A local assistant can still be wrong, fluently. Keep a person accountable for any answer that feeds a decision, and make the assistant show its sources so that person can check them in seconds.

For a deeper treatment of retrieval design and evaluation, see the journal paper Retrieval-Augmented Generation: Design, Evaluation and Failure Analysis.

Security: what stays private, and what you must still do

With Ollama, the built-in embedder or an Ollama embedder, and LanceDB, your documents, vectors, chats and model inference all stay on the PC. That removes the largest risk of cloud AI. It does not make the setup secure by itself.

The checklist

1. Install Ollama, pull a chat model that fits your VRAM and an embedding model, and confirm 100% GPU with ollama ps.
2. Install AnythingLLM Desktop. Set the LLM provider to Ollama and match the context window.
3. Set the embedder to the Ollama embedding model and its maximum chunk length to the model's limit, before loading documents.
4. Keep LanceDB and the default chunking (1,000 characters) to start; raise the overlap for procedures.
5. Create one workspace per subject, use Query mode, a sources-only system prompt and a low temperature.
6. Switch on only the agent skills you need.
7. Build a 20-question test set and rerun it after every change.
8. Encrypt the disk, keep Ollama on localhost, review telemetry and back up the storage folder.

A private assistant like this is often the first building block of a company's own AI operating system. The journal paper AI Operating Systems: A Reference Architecture and Governance Model describes how the pieces fit together at company scale, and Score Versus Cost Per Task covers when a cloud model is worth paying for instead.

Settings screens in this article are simplified redrawings, not screenshots, and some field names vary by version. Hardware guidance is a rule of thumb; check the model's page for its actual memory use.