“Can we train the model on our documents?” is a fair question that usually stands in for a more specific one: can it answer questions about our material, correctly, in a way we can check? Training is one way to try. Retrieval is another. They change different things, and the choice is easier once you see what each one touches.
One changes the model, the other changes what it reads
Fine-tuning changes the weights. You show the model examples and it adjusts its parameters so that outputs like those become more likely. Whatever it picks up is spread across millions or billions of numbers. There is no record of which document taught it what, and no way to remove one fact without training again. LoRA, which I use for fine-tuning small models, makes this far cheaper, but it is still the same kind of learning.
Retrieval changes the input. The model stays exactly as it was. When a question arrives, you search your documents for the passages most likely to hold the answer and put them into the prompt next to the question. The model reads them and answers. Its knowledge of your material lasts exactly one request.
Fine-tuning changes how a model behaves. Retrieval changes what it is looking at when it answers.
Most of the decision follows from that sentence.
When retrieval wins
Facts that change. Picture a CA firm’s internal notes on tax rules, revised every time a new circular or notification arrives. With retrieval, you update the note and the next answer reflects it. With fine-tuning, the old rule is still somewhere in the weights beside the new one, and you cannot be sure which will surface.
Citations. Every retrieved passage has an address, a document and a position in it, so an answer can cite its sources and a reader can check them. My years in audit make me weigh this heavily. An answer you cannot trace back to a source is an opinion, however confident it sounds.
Permissions and deletion. Retrieval can search only the documents a given user is allowed to see, and a deleted document stops appearing in answers the moment it leaves the index. A fine-tuned model can repeat anything it was trained on to anyone who asks the right way.
Knowing when it does not know. If the search returns nothing close to the question, you can detect that and say so instead of letting the model improvise.
When fine-tuning wins
Style and voice. The way a firm writes to its clients is hard to describe in a prompt and easy to show in a few hundred examples.
Format. Always valid JSON in one schema, always the same headings in the same order. A prompt can ask for that. A fine-tune makes it a habit.
Narrow tasks done many times. Sorting documents into a fixed set of categories, or pulling the same ten fields from the same kind of form. A small fine-tuned model does this faster and more cheaply than a large general one that needs a long prompt with every request.
What fine-tuning is bad at is reliably adding facts. It learns the shape of an answer readily and the facts loosely, which produces the worst combination: a wrong answer in exactly the right tone.
When you need both
Many real systems end up here. Retrieval supplies the facts; fine-tuning teaches the model what to do with them: answer only from the passages provided, cite them in a house format, say plainly when they do not cover the question, and write the way the business writes. Train on examples that include retrieved passages, and include cases where the passages do not contain the answer and the correct output is to say so. Without those, the model learns that there is always an answer.
The order matters. Build retrieval first, with a clear prompt, and measure it. Fine-tune only for what is still wrong after that. Often the answer is nothing, and you have saved yourself a training pipeline to maintain.
Retrieval, without the tooling
Retrieval is less mysterious than the tooling around it suggests. Cut the documents into chunks, turn each chunk into a vector, and at question time find the chunks whose vectors sit closest to the question’s. Here is the whole idea in plain Python, with a stand-in embed so the sketch runs as it is.
import math
import re
import zlib
DIM = 1024
def chunk(text, size=200, overlap=40):
"""Split text into overlapping windows of words."""
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size])
for i in range(0, max(len(words) - overlap, 1), step)]
def embed(text):
"""Stand-in: hashed word counts scaled to unit length."""
vec = [0.0] * DIM
for word in re.findall(r"[a-z0-9]+", text.lower()):
vec[zlib.crc32(word.encode()) % DIM] += 1.0
norm = math.sqrt(sum(v * v for v in vec)) or 1.0
return [v / norm for v in vec]
def similarity(a, b):
return sum(x * y for x, y in zip(a, b)) # cosine, as both are unit length
def build_index(documents):
index = []
for doc_id, text in documents.items():
for n, piece in enumerate(chunk(text)):
index.append({"source": f"{doc_id}#{n}", "text": piece, "vec": embed(piece)})
return index
def search(index, question, k=3):
q = embed(question)
return sorted(index, key=lambda c: similarity(q, c["vec"]), reverse=True)[:k]
def build_prompt(question, hits):
sources = "\n\n".join(f"[{h['source']}]\n{h['text']}" for h in hits)
return ("Answer only from the sources below and cite the [source] tag for each claim. "
"If the sources do not contain the answer, say so.\n\n"
f"{sources}\n\nQuestion: {question}")In a real system, replace embed with an embedding model and leave the rest alone. The stand-in only matches words the question and the chunk share; a real model matches meaning, so a question about “fees” can find a passage about “charges”.
Chunk size and overlap matter more than they look. Chunks that are too small lose the sentence that gave them meaning; chunks that are too large dilute the match. The overlap keeps a point that straddles a boundary intact in at least one chunk. Sorting every chunk on every question is fine for a few thousand chunks. Beyond that, a vector index does the same job faster.
Test retrieval on its own, before you judge the answers. Write down twenty questions and the passage that holds each answer, then check whether that passage appears in the top results. If it does not, no model can answer from it.
Two questions decide it
Should the answer change when the document changes? Then retrieve. Should the answer sound a particular way, or come back in a particular shape, every single time? Then fine-tune. If both are true, build retrieval first and fine-tune for whatever is left. If you are weighing this for your own documents, tell me what they are and what people need to ask of them.
