RAG or Fine-Tuning for Internal Knowledge?

Use retrieval when an assistant needs to consult changing documents and show which source supports an answer. Consider model adaptation only for a different, well-defined behavior that prompt design and retrieval cannot achieve. Neither choice removes the need for access control and evaluation.

Ask what is changing

Policies, manuals, prices, and project files change often. A retrieval index can refresh those sources without changing model parameters, provided ingestion and versioning are managed.

Test the failure point

If the right passage is not retrieved, tuning the model will not fix the search problem. If the passage is present but the response mishandles it, improve prompts, context formatting, or evaluation before changing models.

Include governance

Track document permissions, answer citations, update cadence, retention, and the owner who resolves contradictory sources.

Separate knowledge from behavior

Retrieval-augmented generation is often the first test when answers depend on documents that change or require citations. Fine-tuning can help with a consistent task format or behavior when representative examples exist, but it does not turn outdated weights into a reliable knowledge source. List the desired answer, the source of truth, update frequency, and permission boundary before choosing either approach. If the answer requires a current policy or customer-specific record, the system needs a controlled way to fetch it. Prototype with real questions and measure evidence quality before investing in a larger training effort.

A decision table should separate source freshness, citation need, task format, permission scope, examples available, and update cadence. Fill it with three real questions from the intended users. Mark which information is in a document, which is in an application record, and which is simply a desired response style. Test one retrieval baseline before estimating a fine-tuning effort, then compare both against the same evaluation cases. The table may lead to retrieval, fine-tuning, both, or a simpler rules-based path. Recording why an option was selected makes later model and content changes easier to review.

Choose an evaluation that can fail

Build cases where the correct source is absent, two versions disagree, a user lacks permission, or the question invites an unsupported assumption. Score retrieval, answer support, refusal, latency, and operating cost separately. A fine-tuned response that sounds polished can still be wrong; a retrieved passage can still be irrelevant or misread. Keep the test set stable across model and document changes so the team can see regressions. Document the point at which a human must review or an answer must be withheld. The decision should follow observed failure patterns, not a label attached to the model.

Decision checklist

• Collect 30 representative questions and expected sources.
• Measure retrieval separately from answer quality.
• Check permission boundaries with multiple user roles.
• Choose the simplest approach that passes the task evaluation.

A small test before committing

Gather representative questions with approved answers and source documents. First score whether the correct passage is retrieved for each user role. Then score whether the answer uses that passage accurately and cites it. Change one source document and confirm when the answer updates. If retrieval misses the passage, improve source preparation and search. If retrieval succeeds but the answer still fails, inspect prompt, context, and evaluation before considering model adaptation.

Worked scenario

A hypothetical policy assistant must answer from the latest approved handbook and cite it. Retrieval addresses the changing source and review need; fine-tuning on last year's handbook would not establish current policy authority.

For a scoped application of this decision, see RAG & Knowledge Systems.