Private and Local AI for SMEs

A practical course for owners, managers and technically minded staff who want to use AI with sensitive information. You will learn the spectrum from consumer cloud AI to fully local models, when local AI makes sense, how open-weight models and quantization work, what hardware you need, and how to set up tools such as Ollama, LM Studio, Open WebUI and AnythingLLM. You will build accurate document Q&A with retrieval-augmented generation, secure the system, meet privacy obligations, compare costs with cloud services, connect tools safely and run a 90-day pilot.

Course structure

• Module 1: Why Private AI (4 lessons). Cloud and private options, when local AI makes sense, open-weight models, and hardware and quantization.
• Module 2: Setting Up (4 lessons). Tools for running local AI, choosing models, retrieval-augmented generation and making document answers accurate.
• Module 3: Running It Safely (4 lessons). Security, privacy and compliance, desktop versus team server, and limits and the hybrid approach.
• Module 4: Value and Roadmap (4 lessons). SME use cases, cost comparison with cloud, local agents and tools, and a 90-day pilot.

Interactive labs

• Where should this work run?: Decide where twelve AI tasks should run: any approved cloud AI, business cloud AI with data protections, or local / private AI only. Then check your answers.
• Picking a first local AI use case: Place seven possible use cases on a grid of business value against data sensitivity to find the best candidates for a local AI pilot.
• Local vs cloud AI: three-year cost: Enter users, the cloud subscription price, local hardware and setup costs and monthly upkeep to compare totals and see when local breaks even.

How it works

• Work through 16 short lessons at your own pace. Each has key takeaways, an exercise for your own workplace and a quick knowledge check.
• Try the three interactive labs as you reach them in the course.
• Pass the 20-question final exam (70% or higher, unlimited retakes) to unlock your Chmltech certificate of completion.
• Free, with no sign-up. Your progress is saved in this browser.

Most people meet AI through cloud services such as ChatGPT, Claude, Gemini or Microsoft Copilot. These are powerful and easy to use, but your prompts and files are processed on the provider's servers. For many tasks that is fine. For some, such as confidential contracts, customer records, health information or proprietary designs, businesses want more control.

A spectrum of options

What "local AI" means

Local AI runs an open-weight language model, one whose model files can be downloaded, on hardware you control: a desktop with a capable graphics card, a laptop, or a small server for a team. Prompts and documents never leave your premises unless you send them elsewhere.

Not all or nothing

Most organizations use a hybrid approach: approved business cloud AI for everyday work, and local AI for the most sensitive tasks. This course helps you decide where local AI fits, set it up, connect it to your documents, run it safely and measure whether it is worth it.

Local AI brings control, but also responsibility and some limits. It makes most sense when one or more of these apply.

Reasons to consider local AI

• Sensitive data: legal, financial, health, personnel or proprietary technical information that your policy or clients don't allow in third-party services.
• Client or contract requirements: customers who require that their data stays on your systems.
• Data residency: a need to keep data in Canada or on-premises.
• Offline or remote work: field sites, plants or vessels with poor connectivity.
• Cost at volume: heavy, repetitive use, such as processing thousands of documents, where per-use cloud fees add up.
• Control: a fixed model version that won't change behaviour without notice.

Reasons to stay with business cloud AI

• You need the most capable models for complex reasoning, writing or coding.
• You have no one to set up and maintain local systems.
• Usage is light, so subscriptions cost less than hardware and time.
• Enterprise plans already meet your data protection needs.

Be honest about trade-offs

Open-weight models that run on modest hardware are capable for many tasks, such as summarizing, drafting, extraction and document Q&A, but typically less capable than the largest frontier cloud models on difficult reasoning. Local AI also needs setup, updates, security and support. The lab after this lesson asks you to decide where different data belongs.

Open-weight models are language models whose trained parameters (weights) are published for download. Several AI labs and research groups release them in families of different sizes, such as Meta's Llama, Mistral, Alibaba's Qwen, Google's Gemma and OpenAI's open-weight models, among others. New versions appear frequently.

Size and capability

Model size is measured in parameters, usually billions (B). Larger models are generally more capable but need more memory and run more slowly. Small models (a few billion parameters) run on laptops and handle simple tasks; medium models (roughly 7 to 15B) run on a single good graphics card and handle most office tasks well; larger models (30B and up) need high-end hardware but come closer to cloud quality.

Specialized models

• Chat or instruct models: tuned to follow instructions and converse.
• Coding models: tuned for writing and explaining code.
• Embedding models: turn text into numerical vectors for search; essential for chatting with documents.
• Vision models: can read images and scanned documents.

Licences matter

"Open-weight" doesn't always mean "open source" or "free for any use". Licences vary: some allow broad commercial use, others restrict use above a certain number of users or for certain purposes. Read the licence before using a model in your business.

Source and integrity

Download models from the publisher or reputable repositories through established tools. Model files from unknown sources can be tampered with. Record which model and version you use, so results can be reproduced and issues traced.

The main hardware limit for local AI is memory: the model must fit in the memory of the graphics card (GPU) or, more slowly, in system memory.

Quantization

Models are usually run in quantized form, storing each parameter with fewer bits, commonly 4 to 8 bits instead of 16. This cuts memory needs several-fold with a modest loss of quality. As a rough guide, a 4-bit quantized model needs about 0.5 to 0.7 GB of memory per billion parameters, plus extra for the conversation context.

These are approximate; actual needs vary with the model, quantization method and context length.

Other options

• Unified memory computers, such as Apple silicon Macs, share memory between processor and graphics and can run fairly large models.
• CPU only: works for small models but is much slower.
• Small servers with one or more GPUs can serve a team.

Speed

Speed is measured in tokens per second. For interactive chat, a reply that streams at reading speed or faster feels comfortable. Models that spill out of GPU memory into system memory slow down sharply, so choose a model that fits.

You don't need to write code to run local AI. Several free tools make it straightforward.

A simple starting setup

1. Install Ollama on a computer with a suitable graphics card.
2. Download a medium-sized instruct model and an embedding model.
3. Install a front end such as AnythingLLM or Open WebUI and point it at Ollama.
4. Create a workspace, upload a few non-critical documents and test questions.

The Chmltech Insights article Private AI assistant on your own PC walks through this setup step by step, including choosing models, embeddings, chunking and security settings.

Keep it up to date

These tools change quickly. Update them regularly for security fixes and improvements, and check release notes before updating a system the team depends on.

No single model is best at everything. Choose by task, hardware and licence, then test with your own examples.

Match model to task

Build a small test set

Collect 10 to 20 realistic tasks from your work, with what a good answer looks like. Run each candidate model on the set and score the results for accuracy, completeness, format and speed. This takes an afternoon and is far more reliable than general rankings, because it reflects your documents and your questions.

Settings that matter

• Context length: how much text the model can consider at once; longer contexts use more memory.
• Temperature: lower values give more consistent, focused answers, which suit business tasks.
• System prompt: standing instructions such as "answer only from the provided documents and say when you don't know".

Revisit regularly

New open-weight models are released often, and a newer model of the same size may be much better. Re-run your test set when a promising model appears.

Language models don't know your company's documents. Retrieval-augmented generation (RAG) lets a model answer questions using your files, such as procedures, contracts, manuals and reports, without retraining it.

How RAG works

1. Ingest: documents are loaded and converted to text.
2. Chunk: text is split into passages, for example a few hundred words each, often with some overlap.
3. Embed: an embedding model converts each chunk into a vector that represents its meaning.
4. Store: vectors are saved in a vector database.
5. Retrieve: when a user asks a question, the question is embedded and the most similar chunks are found.
6. Generate: the model receives the question plus the retrieved chunks and writes an answer, ideally citing the sources.

Why it works well locally

All steps can run on your own hardware: the embedding model, the vector database and the language model. Documents never leave your systems. Tools such as AnythingLLM and Open WebUI handle these steps for you.

Good uses

• "What is our procedure for returning a damaged shipment?" from the quality manual.
• "What are the termination terms in the Acme contract?" from a contract library.
• "What torque does the maintenance manual specify for this pump?"

Limits

RAG answers are only as good as the documents and the retrieval. If the right passage isn't retrieved, the model may answer from general knowledge or guess. Always show sources so users can check the original. The next lesson covers making RAG more accurate.

Most RAG accuracy problems come from documents and settings, not the model. A few practices make a large difference.

Prepare documents

• Use clean, text-based files; run OCR on scanned PDFs and check the result.
• Remove outdated versions and duplicates so old answers don't appear.
• Keep headings and structure; they help chunking and retrieval.
• Organize documents into separate workspaces by topic or department so retrieval searches the right set.

Tune retrieval

• Chunk size: too small loses context, too large dilutes relevance. A few hundred words with some overlap is a common start.
• Number of chunks retrieved: enough to cover the answer without flooding the model; test a few values.
• Embedding model: a better embedding model often improves accuracy more than a bigger language model.

Instruct the model

Use a system prompt such as: "Answer only from the provided documents. Quote or cite the source. If the answer isn't in the documents, say so." This reduces made-up answers.

Evaluate

Create 20 to 30 test questions with known answers and their source documents. Run them after each change and track how many are answered correctly with the right source. Ask users to flag wrong answers, and add those questions to the test set.

Document Q&A supports people; for decisions with legal, safety or financial consequences, users should read the cited source.

Local AI keeps data in-house, but the system itself must be secured like any other server or application.

Key practices

• Don't expose it to the internet. Local model servers and chat interfaces should be reachable only on your internal network or through a VPN, never directly from the internet.
• Require logins with strong passwords and multi-factor authentication where the tool supports it; give each person their own account.
• Limit access by workspace: HR documents should only be searchable by HR, contracts by those who need them.
• Keep software updated: the model engine, front end, operating system and drivers.
• Protect the machine: disk encryption, endpoint protection and physical security for the computer or server.
• Back up configurations and the document store, and keep the original documents in their normal managed location.

Model and tool provenance

Download models and plug-ins only from reputable sources. Prefer widely used tools with active maintenance. Be careful with community add-ons that run code.

Prompt injection

Documents can contain hidden or malicious instructions, such as text telling the AI to ignore its rules. This matters most when the AI can take actions, such as sending email or running tools. Limit what local agents can do, require human approval for actions and treat content from outside sources with caution.

The Cybersecurity and Ransomware Basics for SMEs course covers broader security practices that apply here.

Running AI locally helps with privacy, but it doesn't remove legal duties for the personal information you process.

Privacy principles still apply

Canadian privacy law, including PIPEDA, applies to personal information wherever it is processed. Key principles:

• Purpose: use personal information only for the purposes it was collected for, or with consent.
• Minimization: load only the documents and information the use case needs.
• Safeguards: protect it with security appropriate to its sensitivity.
• Access: limit who can query workspaces containing personal information.
• Retention: remove documents and chat logs when no longer needed.

Logs and chat history

Local tools often store every prompt and answer. Those logs can contain sensitive information. Decide whether to keep them, for how long and who can see them, and configure the tool accordingly.

Client and contract obligations

If clients have shared confidential information under agreements, check whether using it in any AI system, even a local one, is permitted, and whether you must tell them.

Document your approach

Write a short record of each local AI use case: purpose, data used, who has access, retention and safeguards. It helps answer client questions, supports privacy obligations and makes reviews easier. Get privacy or legal advice for high-risk uses, such as health information or decisions about individuals.

Local AI can start on one person's computer and grow into a shared team service. Each has its place.

Start small

Many SMEs begin with a pilot on one capable desktop used by a few people for a defined use case, such as searching the quality manual or summarizing contracts. If it proves useful, they move to a small server with proper accounts and backups.

Operating a team server

• Name an owner responsible for updates, access and support.
• Monitor usage and response times to know when hardware needs upgrading.
• Plan for power, cooling and noise: GPUs under load use significant power.
• Keep a simple runbook: how to restart services, update models and restore from backup.

Managed private options

Some providers offer private AI hosted in Canadian data centres or in your own cloud account, managed for you. These can give much of the control of local AI without running hardware yourself. Compare them on data location, contract terms and cost.

Local AI is a valuable tool, not a replacement for every cloud service. Knowing its limits avoids disappointment.

Common limits

• Capability: models that fit on modest hardware typically trail the largest frontier models on complex reasoning, nuanced writing and advanced coding.
• Hallucination: local models can make things up, like any model; RAG with citations helps but doesn't eliminate it.
• Speed: large models on limited hardware can be slow.
• Features: cloud services often include web search, advanced file handling, voice and integrations out of the box.
• Maintenance: someone must update, secure and support the system.

A sensible hybrid

Watch the trend

Open-weight models have improved quickly, and hardware keeps getting more capable. Tasks that needed cloud models a year ago can often run locally today. Revisit your split every six to twelve months using your test set.

The best local AI use cases combine sensitive or proprietary information with repeated, text-heavy work.

Choosing the first use case

• Clear value: time saved or faster answers for a defined group.
• Sensitive data that justifies local processing.
• A manageable, well-organized document set.
• Easy to check answers, at least during the pilot.
• An enthusiastic owner in the business.

An internal knowledge assistant over a quality manual or procedures library is often an ideal first project: high value, well-defined documents and answers that can be checked. The lab after this lesson asks you to rank use cases by value and sensitivity.

Local AI swaps per-user or per-use fees for upfront hardware and ongoing upkeep. Compare them honestly over a realistic period.

Local costs

• Hardware: a capable GPU workstation or small server; possibly upgrades later.
• Setup time: installation, model testing, document preparation, security configuration.
• Running costs: electricity, maintenance, updates, support time and backups.

Cloud costs

• Per-user subscriptions for business AI plans, or per-use fees for API-based tools.
• Possible add-ons for document features, higher usage or admin controls.

Example

Fifteen users on a business AI plan at $40 a month each cost $7,200 a year, or $21,600 over three years. A local setup with $4,000 of hardware, 40 hours of setup at $90 an hour and $150 a month of upkeep costs $7,600 upfront and $5,400 in upkeep over three years: $13,000 in total. In this example local breaks even in about 17 months. With fewer users or more upkeep, cloud wins.

Not just cost

The comparison isn't only financial. Cloud plans often give access to more capable models and features; local gives control and privacy. Many businesses keep both, using local only for sensitive work, which changes the numbers: compare the cost of the local system with the value of being able to use AI on documents you otherwise couldn't use it on at all.

The lab after this lesson compares the two for your own numbers.

Beyond chat and document Q&A, local models can power agents: AI that uses tools to take steps, such as searching files, querying a database, filling a form or drafting an email for approval.

How local agents connect to tools

• Function or tool calling: many models can request a specific tool with structured inputs, which the application then runs.
• Model Context Protocol (MCP): an open standard for connecting AI applications to tools and data sources. Some local front ends support MCP servers.
• Built-in skills: tools such as AnythingLLM offer agent skills like web search, file saving and summarization.

Start with read-only tools

The safest first agents only read: search documents, look up an order status in a read-only database view, or summarize files. Tools that change data, send messages or spend money carry more risk.

Safeguards

• Give agents the minimum permissions they need, with separate service accounts.
• Require human approval for any action with external, financial or irreversible effects.
• Log every tool call and review the logs during pilots.
• Test with tricky inputs, including documents containing misleading instructions.

Capability matters

Reliable multi-step tool use is demanding, and smaller local models are often less reliable at it than large cloud models. Keep local agents to simple, well-defined tasks and test them thoroughly. The AI Agents for Business Operations course covers agent design in depth.

A structured pilot proves whether local AI delivers value before you invest further.

Success measures

• Share of test questions answered correctly with the right source.
• Time saved per week for pilot users.
• User satisfaction and frequency of use.
• No security or privacy incidents.

Where to go next

The LLMs for Business Leaders course covers model choice and governance; AI Agents for Business Operations covers agent design; Building Your AI Operating System shows how local and cloud AI fit into a company-wide AI layer; and Cybersecurity and Ransomware Basics covers protecting the systems that run it.