How to Train an LLM on Company Data: RAG vs Fine-Tuning

When a business asks how to train an LLM on company data, the honest answer is that it almost never needs training in the strict sense. What it needs is for the model to look things up: search your documents for each question, read the relevant passages and answer from them with a citation. That is retrieval-augmented generation (RAG), and for policies, product details, procedures and anything else that changes, it beats retraining the model on accuracy, cost and freshness.

People who search "train LLM on company data" usually want one of three things: an assistant that knows their facts, a model that writes or classifies the way their team does, or both without the data leaving their control. Only the second one is a job for fine-tuning, which changes the model's weights with examples. Even then, the facts still come from retrieval.

This guide explains the difference in plain English, shows what the research says about teaching models new facts, prices each option at published November 2026 rates, and ends with a decision tree you can use in a meeting. It also covers what changed this year: OpenAI is winding down self-serve fine-tuning, and ChatGPT and Claude both now ship built-in retrieval over company files.

What "train LLM on company data" actually means

The phrase covers three different problems, and each has a different fix. Name yours before you talk to a vendor, because the word "train" makes every problem sound like the most expensive one.

1. "It doesn't know our stuff." The model gives generic answers about your refund policy, your product line or your onboarding process because it has never seen them. This is a knowledge gap. The fix is retrieval: give the model the right documents at the moment it answers.

2. "It knows enough, but answers the wrong way." The model writes in the wrong tone, ignores your template, labels tickets with the wrong categories or calls a tool with malformed arguments. This is a behavior gap. The fix starts with better instructions and examples in the prompt, and only sometimes ends in fine-tuning.

3. "We can't send our data to a public chatbot." This is a control requirement, not a training requirement. The fix is choosing where the model and the index run (a business plan with contractual data terms, your own cloud account, or your own hardware) and keeping access permissions intact. You can meet it without training anything.

Most real requests are problem 1 with a dose of problem 3. A sales team that wants "ChatGPT trained on our product catalog" needs retrieval over the catalog, with the catalog's access rules respected. Nobody in that sentence needs new model weights.

Key takeaway: Ask "is the model missing facts or getting the behavior wrong?" before anyone says the word "training." Missing facts are solved by retrieval, which is cheaper, faster to update and easier to audit.

RAG vs fine-tuning explained in plain English

RAG gives the model an open-book exam; fine-tuning sends it back to school. With RAG, every question triggers a search of your documents, and the best passages are pasted into the prompt along with an instruction to answer only from them. With fine-tuning, you show the model hundreds or thousands of example inputs and correct outputs, and the training process adjusts the model's internal weights so it produces similar outputs on its own.

That one difference drives everything else:

Two rows deserve emphasis for regulated or privacy-conscious companies. Access control: a fine-tuned model has no concept of who is asking, so salary bands or legal memos trained into it are available to anyone who can prompt it. Deletion: if a customer asks you to delete their data, removing a document from an index is a routine operation; removing it from a model's weights means retraining.

Microsoft's own fine-tuning documentation makes the same distinction. Its Foundry overview lists what fine-tuning is good for (accuracy on a specific task, consistent style and structure, better tool use, specializing a smaller model) and then states that fine-tuning "doesn't replace retrieval for current information or application-level safety controls" (Microsoft Learn, updated October 2026).

Why fine-tuning is the wrong tool for teaching facts

Fine-tuning is unreliable for teaching a model new facts, and published research shows it can make the model less truthful. This surprises many buyers, because "train it on our documents" sounds like exactly what training is for.

Two studies are worth knowing:

• Retrieval beat fine-tuning for knowledge. Researchers comparing the two methods for "knowledge injection" found that retrieval "consistently outperforms" unsupervised fine-tuning, both for knowledge the model had seen before and for entirely new knowledge, and that models "struggle to learn new factual information through unsupervised fine-tuning" (Ovadia et al., arXiv 2312.05934). They found that exposing the model to many paraphrases of the same fact helped, which in practice means generating and curating far more training data than the original documents.
• Learning new facts by fine-tuning raised hallucinations. A team of Google researchers, in a paper accepted at EMNLP 2024, found that training examples introducing new knowledge are learned more slowly than examples consistent with what the model already knows, and that once learned, they "linearly increase the model's tendency to hallucinate" (Gekhman et al., arXiv 2405.05904). Their conclusion is that models acquire factual knowledge mostly in pre-training, and fine-tuning teaches them to use it more efficiently.

There are also three practical problems that no research paper is needed to see:

1. Your documents change. A price list updated on Tuesday is wrong in a model trained on Monday. With RAG the fix is re-indexing one file.
2. You cannot check the answer. A fine-tuned model cannot point to the page it learned something from, so reviewers cannot verify it. A RAG answer can show its source.
3. Fine-tuning does not respect permissions. Everything in the training set is available to every user of the model.

Key takeaway: Use fine-tuning to change how a model answers, not what it knows. Facts belong in an index you can update, permission and audit.

Five ways to give an AI your company knowledge

There are five practical options, from a few minutes of setup to a multi-month build. Most companies should start at the top and move down only when a specific requirement forces it.

1. Put the documents in the prompt

If the material is small, skip retrieval entirely. Anthropic's guidance is that a knowledge base under about 200,000 tokens, roughly 500 pages, can simply be included in the prompt, with prompt caching keeping repeated use affordable (Anthropic, "Introducing Contextual Retrieval"). An employee handbook, a product FAQ or a single contract set often fits. The trade-off is cost per question: every question pays to process the whole document (see the cost section below).

2. Use built-in workspace retrieval

Both major chat assistants now do retrieval over company files without a development project:

• ChatGPT Company Knowledge. Available on ChatGPT Business, Enterprise and Edu, it is a plugin that searches connected apps and custom apps built with the Model Context Protocol (MCP). OpenAI states it respects permissions in the connected source, so "ChatGPT can only access what each user is already allowed to view," and that it does not use Business, Enterprise or Edu workspace data to train its models by default (OpenAI Help Center, checked November 2026).
• Claude Projects. You upload files into a project; when the project approaches the context window limit, Claude switches automatically to retrieval, which Anthropic says expands a project's capacity "by up to 10x." The feature is listed for paid plans: Pro, Max, Ultra, Team and Enterprise (Claude Help Center, checked November 2026).

These are the right answer for a large share of internal "train it on our docs" requests. They fall short when the assistant must be customer-facing, live inside your own application, pull from systems the tool has no connector for, or produce an audit trail your regulators expect.

3. Use a managed RAG service in your cloud

The cloud providers sell retrieval as a component you call from your own application, so the documents and index stay in your account:

• Amazon Bedrock Knowledge Bases charges $5.00 per GB of raw data per month for index storage and $1.00 per 1,000 retrieval calls, with parsing, embeddings and managed reranking listed at $0 when you use the managed defaults (AWS Bedrock pricing, November 2026).
• Google Cloud's RAG Engine, now a component of Gemini Enterprise Agent Platform, ingests local files, Cloud Storage and Google Drive (Google Cloud documentation).
• OpenAI's file search tool in the Responses API costs $2.50 per 1,000 tool calls, with vector storage at $0.10 per GB per day after the first free GB (OpenAI pricing, November 2026).

4. Build custom RAG

A custom build is warranted when you need several source systems synchronized, per-user permission filtering that mirrors your source systems, hybrid keyword and semantic search, structured data lookups, a citation format your reviewers accept and an evaluation harness. Our RAG chatbot cost breakdown puts an internal pilot at roughly 15 to 33 engineer-days and a production assistant at roughly 65 to 150.

5. Fine-tune, usually on top of retrieval

Fine-tuning comes last, and it usually sits on top of one of the options above rather than replacing it. It earns its place when a behavior gap survives good prompting: a model that must produce a strict output schema, apply your classification labels, write in a regulated house style or run cheaper by specializing a smaller model.

Which approach fits: a decision tree

Start with the failure, not the technology. Run 20 real questions from the people who will use the system against a capable model with no customization, and sort each wrong answer into "didn't know our facts" or "knew enough but answered the wrong way." The pattern tells you which branch you are on.

On the knowledge branch, size and audience decide. Small, stable material goes in the prompt or a project. Staff-only use over data already sitting in a supported workspace app goes to built-in retrieval. Everything else (customer-facing assistants, many systems, permission rules more complex than the source apps enforce, an audit trail, or a requirement that data stays in your own cloud) points to managed or custom RAG.

On the behavior branch, cheap fixes come first. Clear instructions and five to twenty examples in the prompt fix a surprising share of format and tone problems. If they do not, fine-tuning is the next step, but only once you have the training examples and a test set to prove it helped. OpenAI's fine-tuning guide sets a minimum of 10 examples, recommends starting with 50 well-crafted demonstrations, and reports improvements from 50 to 100 (OpenAI). If you do not yet have 50 outputs your experts agree are correct, the project is not ready for training; it is ready for collecting examples.

Many real systems land on both branches. A claims assistant may need retrieval for policy wording and a fine-tuned output format for the adjuster's summary. That is fine, as long as each technique is doing the job it is good at.

What it costs: a worked example at November 2026 prices

The model and infrastructure bill for retrieval over company documents is small; the engineering is the real budget. Here is a worked example with every input stated, using list prices checked in November 2026.

Assumptions: 2,000 documents averaging five pages, about 10,000 pages in total. At roughly 500 tokens per page, that is about 5 million tokens of text and, for this example, about 0.5 GB of raw files. Staff ask 3,000 questions a month. Each question sends about 4,000 tokens of instructions and retrieved passages to the model and gets about 400 tokens back.

The arithmetic for the model rows: 3,000 questions times 4,000 tokens is 12 million input tokens, and 3,000 times 400 is 1.2 million output tokens. At gpt-6-luna prices that is $1.20 plus $0.60; at gpt-6.1-sol prices it is $24 plus $12. Prices come from OpenAI's pricing page and AWS's Bedrock pricing page.

Compare that with putting everything in the prompt. The 5-million-token library does not fit in a prompt, but suppose you trimmed it to a 200,000-token core. Sending that core with each of 3,000 questions is 600 million input tokens a month: about $60 at gpt-6-luna prices and about $1,200 at gpt-6.1-sol prices, before any prompt-caching discount. Long context is the fastest way to start and the most expensive way to scale.

Where the real money goes. None of the line items above is the budget. The budget is engineering: connecting sources, keeping the index in sync, filtering by permissions, testing answers and fixing what fails. Using the estimates in our RAG chatbot cost article, that is roughly 65 to 150 engineer-days for a production system, against an infrastructure bill measured in tens of dollars a month at this volume. Fine-tuning adds its own work on top: our custom AI model cost breakdown estimates 20 to 45 engineer-days for fine-tuning a hosted model, most of it spent on data and evaluation.

Key takeaway: If a quote for "training AI on your documents" is mostly compute, ask what it leaves out. The durable costs are data preparation, permissions and evaluation.

What changed in 2026 for companies that want to fine-tune

Three changes this year make retrieval the default even more clearly, and they are missing from most guides on this topic.

OpenAI is winding down self-serve fine-tuning. According to its deprecations page, organizations that had never fine-tuned lost the ability to create jobs on May 7, 2026; from July 2, 2026, job creation also ended for organizations that had not run inference on a fine-tuned model in the previous 60 days; and on January 6, 2027, active existing customers can no longer create new fine-tuning jobs. Models already trained keep serving until their base model is deprecated. If a vendor's plan for you depends on fine-tuning an OpenAI model directly, ask how it survives that date.

Managed fine-tuning moved to the clouds and to open weights. Microsoft Foundry's model table, updated in October 2026, lists managed supervised fine-tuning for gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4o and gpt-4o-mini, reinforcement fine-tuning for o4-mini, and open-weight models such as Llama-3.3-70B-Instruct, Qwen3-32B and gpt-oss-20b (Microsoft Learn). Amazon Bedrock lists reinforcement fine-tuning for gpt-oss-20b and Qwen3 32B at $80.00 per training hour in its US East (N. Virginia) and US West (Oregon) regions (AWS). Fine-tuning remains possible; it is just no longer a button inside the best-known API.

Built-in retrieval became a standard feature. ChatGPT's Company Knowledge plugin and Claude's automatic project retrieval mean that the most common request, "an assistant that knows our documents," is now often a configuration task rather than a development project. That changes the build-versus-buy math covered in our private LLM vs ChatGPT Enterprise comparison: custom work now has to justify itself with integration, control or customer-facing needs that the built-in tools do not meet.

How to train a local Llama on company data

The self-hosted version of this question has the same answer: run the model locally and give it a local index of your documents, rather than training the documents into it. That keeps everything on hardware you control while avoiding the knowledge problems described above.

A typical self-hosted setup has four parts:

1. An open-weight model served on your own GPU server or private cloud (Llama, Qwen and gpt-oss families are common choices; check each model's license for commercial use).
2. An embedding model and a vector index, often PostgreSQL with a vector extension for modest volumes, running alongside it.
3. An ingestion job that pulls documents from your file shares and systems, splits them into passages, records each passage's source and access rules, and re-indexes changes.
4. An application layer that checks who is asking, retrieves only passages that person may see, and asks the model to answer with citations.

Fine-tuning a local model is reasonable for the behavior gaps described earlier, and it is far cheaper than it used to be. LoRA (low-rank adaptation) trains a small adapter instead of the whole model; its authors reported that for GPT-3 175B it cut trainable parameters by 10,000 times and GPU memory needs by 3 times compared with full fine-tuning (Hu et al., arXiv 2106.09685). QLoRA went further and showed a 65-billion-parameter model could be fine-tuned on a single 48 GB GPU (Dettmers et al., arXiv 2305.14314). Cheap training does not change what training is good for, though: use the adapter for format and style, and keep facts in the index.

The real cost of the local route is serving, not training. A model on dedicated hardware bills whether anyone uses it or not, and someone has to patch, monitor and scale it. Our on-premise LLM deployment cost model walks through the five-year math.

Preparing company data: the checklist that decides accuracy

Data preparation decides whether the assistant is trusted, whichever approach you choose. Most "the AI is wrong" complaints trace back to the documents, not the model: two versions of the same policy, a superseded price sheet nobody archived, or a scanned PDF with no text layer.

Work through this list before connecting any source:

1. Pick authoritative sources. For each topic, name the system of record and its owner. If the HR policy lives in three places, choose one.
2. Remove superseded and duplicate versions. Retrieval will find the outdated version as readily as the current one. Archive what is out of date or tag it so it is excluded.
3. Record access rules with every passage. Store which users or groups may see each document, and filter at query time. OWASP's guidance on vector and embedding weaknesses calls for "permission-aware vector and embedding stores" and strict partitioning of datasets, because a shared index can leak context between users (OWASP LLM08:2025).
4. Fix unreadable formats. Scanned PDFs need text recognition; tables and slide decks often need special parsing to keep their meaning.
5. Keep metadata. Title, date, owner, product, region and document type let retrieval filter ("only current US policies") instead of guessing.
6. Exclude what should never be retrievable. Personal data, privileged legal material and secrets in old wiki pages should be filtered out or tightly permissioned before indexing, not discovered afterward.
7. Write the test set first. Collect 50 to 200 real questions with expert-approved answers and the document each should come from. This becomes the yardstick for every change; our guide to evaluating AI chatbot accuracy explains how to build and use one.
8. Plan the refresh. Decide how quickly changes in each source must reach the index (minutes, hours or a nightly job) and who is alerted when ingestion fails.

Retrieval quality itself is tunable. Anthropic reported that adding context to each chunk before embedding cut failed retrievals by 35%, combining that with keyword (BM25) search cut them by 49%, and adding reranking cut them by 67%, in its own benchmarks (Anthropic). Techniques like these are where engineering days go, and they pay off only if you have a test set to measure them.

Key takeaway: A clean, permissioned, well-labeled document set with a test set attached is the asset. Models and vendors can change; that asset carries over.

When fine-tuning is worth it, and how to avoid lock-in

Fine-tuning is worth it when four things are true: the gap is behavior rather than knowledge, prompting with examples has been tried and measured, you have at least 50 to 100 expert-approved examples, and the gain justifies maintaining a custom model through base-model retirements. Good candidates include:

• Strict output formats, such as a structured summary your downstream system parses.
• Classification with your own labels, such as routing tickets or coding documents into internal categories.
• Consistent tone in high volume, such as customer replies that must follow a style guide.
• Shrinking cost or latency by training a smaller model to do one task as well as a larger one.
• Reliable tool calls in an agent that must pass exact arguments to your systems, a case Microsoft lists among fine-tuning's uses.

To keep a fine-tune portable, own three things in your own version control: the training examples, the evaluation set and the scripts that build both. Then a platform change, such as OpenAI's January 2027 cutoff, costs a retraining run and a re-evaluation rather than a rebuild. Microsoft's documentation also warns that an adapter "isn't interchangeable across similarly named base models," so every base-model change means retraining and retesting.

Mistakes that sink these projects

The same handful of mistakes account for most failed "AI on our documents" projects:

• Starting with training. Teams spend weeks preparing a fine-tuning dataset for a knowledge problem, then discover the model still invents details and cannot cite anything.
• Indexing everything. Pointing retrieval at an entire shared drive pulls in drafts, duplicates and confidential files. Start with one curated source and grow.
• Ignoring permissions until launch. Retrofitting access control into an index is far harder than storing access rules from the first ingestion.
• No test set. Without expert-approved questions and answers, every change is judged by whoever tried it last, and quality drifts.
• Treating the demo as the product. A demo that answers ten friendly questions well is a few days of work; the remaining effort is in sync, permissions, edge cases and evaluation. Our article on moving AI pilots to production covers that gap.
• Building what you could configure. If ChatGPT Company Knowledge or a Claude Project meets the need for staff, a custom build has to earn its cost elsewhere.

Next steps

To give an LLM your company's knowledge, start with retrieval, not training. Try the cheapest option that meets your audience and control requirements: the prompt for small material, built-in workspace retrieval for staff, managed or custom RAG for customer-facing and regulated use. Add fine-tuning only for a behavior gap you have measured, with examples and a test set you own.

A practical sequence for the next month:

1. Write down 20 to 50 real questions and the correct answers.
2. Test them against a capable model with no customization, then with the relevant documents provided, and sort the failures into knowledge and behavior gaps.
3. Clean and permission the one source that answers most of the questions.
4. Pilot the simplest option on the decision tree and measure it against your test set.

If the answer points to a custom system, our custom AI development team builds retrieval and fine-tuned models with permissions, citations and evaluation designed in from the first week, and an AI agent can take actions on top of the same knowledge. Talk to a specialist for a free discovery call, or see everything we do on the Fleurant AI home page.