---
title: "Enterprise RAG: how to build an AI assistant that talks to company documents"
author: "Abdulaziz Akyol"
author_url: https://www.abdulazizakyol.com/en/about/
url: https://www.abdulazizakyol.com/en/blog/enterprise-rag-an-ai-assistant-that-talks-to-company-documents/
language: en
published: 2026-09-25
categories: ["Artificial intelligence", "Software development"]
tags: ["RAG", "LLM", "pgvector", "hybrid search", "embeddings", "data protection", "Python"]
translation: https://www.abdulazizakyol.com/blog/kurumsal-rag-sirket-dokumanlariyla-konusan-yapay-zeka-asistani/
description: "Build a RAG assistant that answers from company documents with citations: parsing, chunking, embeddings, pgvector, hybrid search, permissions, evaluation."
---

# Enterprise RAG: how to build an AI assistant that talks to company documents

By [Abdulaziz Akyol](https://www.abdulazizakyol.com/en/about/) · 2026-09-25

## Key takeaways

- Retrieval-augmented generation (RAG) is an architecture in which a language model searches company documents before answering and bases its reply on the passages it finds; it does not teach the model facts, it fetches them for every question.
- Because company knowledge changes often and answers need citations, RAG suits document questions better than fine-tuning; fine-tuning is mainly for teaching behaviour, tone and format.
- Most of the effort in enterprise RAG goes into data preparation: parsing PDF and Office files while keeping their structure, indexing only the current version and attaching source, page and permission data to every chunk.
- Hybrid search that merges keyword (BM25) and vector results with Reciprocal Rank Fusion is more reliable than vector search alone for questions that need exact matches, such as product codes or clause numbers.
- Document-level access control must be enforced in the retrieval layer, not by the model: a chunk the user is not allowed to see must never enter the model's context.

Retrieval-augmented generation (RAG) is an architecture in which a language model searches the company's own documents before answering a question and bases its reply on the passages it finds. An enterprise RAG assistant answers from regulations, procedures, contracts and technical documents, cites its sources and never goes beyond the user's permissions.

The term became widespread with the 2020 [paper by Lewis and colleagues](https://arxiv.org/abs/2005.11401). In my observation it is the AI application companies ask for most often, and also one of the most often left half-finished. The reason is almost never the model; it is data preparation, permissions and measurement. This article focuses on those three.

## What is the difference between RAG and fine-tuning?

Fine-tuning retrains the model's weights on your data and teaches it a behaviour, a tone or a format. RAG does not change the model; for every question it finds the relevant information and puts it into context.

| Criterion        | RAG                                      | Fine-tuning                                            |
| ---------------- | ---------------------------------------- | ------------------------------------------------------ |
| Freshness        | Update the index when a document changes | Requires retraining                                    |
| Citations        | Natural; the source passage is shown     | Hard; knowledge is spread across weights               |
| Access control   | Enforced per document at retrieval time  | Not possible; the model tells everyone what it learned |
| Erasure requests | Delete the chunk and its vector          | Making a model "forget" is hard in practice            |
| Best at          | Factual Q\&A, document search            | Fixed format, tone, domain jargon                      |

My rule: if the knowledge changes, use RAG; if the behaviour should change, fine-tune. The two can be combined, but the first project is almost always RAG.

## How does a RAG pipeline work?

There are two pipelines. The indexing pipeline runs in the background: documents are pulled from their source, parsed into text, split into chunks, an embedding (semantic vector) is computed for each chunk, and the chunk is stored with its text, vector and metadata. The query pipeline runs for each question: the user's permissions are resolved, the search runs only over permitted chunks, results are re-ranked, and the model writes an answer with citations based on those chunks.

## Step 1: Data preparation (parsing PDF and Office files)

This is the least discussed and most labour-intensive part of the project. The most common problems:

- **Scanned PDFs:** Without a text layer you need OCR first; OCR quality caps answer quality.
- **Tables and multi-column pages:** Converted to plain text, rows get mixed up. Tables need to be turned into text row by row, with their column headers.
- **Headers and footers:** Lines like "Confidential – Page 3/40" repeated on every page pollute search.
- **Word and PowerPoint structure:** Chunking is much better when the heading hierarchy is preserved. Converting documents to a Markdown-like intermediate format with headings works well.
- **Version sprawl:** If three versions of the same procedure sit in a shared folder, the assistant answers from the old one. Only the current version should be indexed, which means every document family needs an owner.

Each chunk gets metadata: document ID, title, version, effective date, page or section, source link and access groups. Citations and permissions work through these fields.

## Step 2: Chunking strategies

If chunks are too small, context is lost; if they are too large, retrieval precision drops and the model reads unnecessary text.

| Strategy              | How it works                                                    | When                                  |
| --------------------- | --------------------------------------------------------------- | ------------------------------------- |
| Fixed size + overlap  | Splits at a set length, overlapping the edges                   | Poorly structured text, first attempt |
| Heading/section-based | Splits at clause, heading and section boundaries                | Regulations, procedures, contracts    |
| Semantic              | Splits where sentence similarity drops                          | Long narrative text without headings  |
| Parent–child          | Searches small chunks, gives the model the whole parent section | Short clauses that need wider context |

For corporate documents, heading-based chunking is usually the best starting point. Prefix each chunk with its heading path ("Travel Policy > 4. Expenses > 4.2 Accommodation"); a clause on its own is often meaningless. Set size and overlap with the evaluation set described below, not by guesswork.

## Step 3: The embedding model

An embedding model turns text into a vector in which texts with similar meaning land close together. Three things matter: choose a multilingual model that handles your languages well (Turkish, in our case), use the same model for indexing and querying, and re-embed the whole index when you change models. That is why every chunk should also store the embedding model's name and version.

Dimension drives storage directly. An example calculation: 1 million chunks × 1,024 dimensions × 4 bytes (float32) ≈ 4 GB of vector data alone; half-precision storage cuts that in half.

## Step 4: Vector database options

| Option                             | Strength                                                   | Watch out for                                     |
| ---------------------------------- | ---------------------------------------------------------- | ------------------------------------------------- |
| PostgreSQL + pgvector              | Existing database, permission filters in SQL, transactions | Needs tuning and partitioning at very large scale |
| Dedicated vector databases         | Focused on vector search, horizontal scaling               | Backup, permissions and monitoring built again    |
| Search engines with vector support | Full-text and vector search in one place                   | Operational load and licensing model              |
| Managed cloud search services      | No setup, quick start                                      | Data location, cost and lock-in                   |

Most companies already run PostgreSQL, so [pgvector](https://github.com/pgvector/pgvector) is a sensible starting point for a first version. It supports HNSW and IVFFlat indexes and cosine (`<=>`), L2 (`<->`) and inner product (`<#>`) distances; an indexed `vector` column can have up to 2,000 dimensions, a `halfvec` column up to 4,000. PostgreSQL also ships a `turkish` text search configuration for full-text search on Turkish documents:

```sql
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
    id             bigserial PRIMARY KEY,
    doc_id         text         NOT NULL,
    doc_version    text         NOT NULL,
    page           int,
    content        text         NOT NULL,
    allowed_groups text[]       NOT NULL,   -- access groups synced from the source system
    embedding      vector(1024) NOT NULL,   -- dimension depends on the chosen model
    tsv            tsvector GENERATED ALWAYS AS (to_tsvector('turkish', content)) STORED
);

CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON doc_chunks USING gin (tsv);
CREATE INDEX ON doc_chunks USING gin (allowed_groups);

-- With approximate indexes the filter runs after the scan; iterative scans (pgvector 0.8.0+) compensate.
SET hnsw.iterative_scan = relaxed_order;

SELECT id, doc_id, page, content
FROM doc_chunks
WHERE allowed_groups && $2::text[]      -- the user's groups, from the identity provider
ORDER BY embedding <=> $1::vector       -- query vector
LIMIT 20;
```

The critical detail is in the comment: with approximate indexes, the filter is applied after the index is scanned. If the user is allowed to see only a few chunks, the query returns fewer results than expected. Iterative scans mitigate this; in multi-tenant setups, a partial index or partition per tenant is another option.

## Step 5: Hybrid search and re-ranking

Vector search captures meaning but can miss procedure codes like "PRS-042", clause numbers and abbreviations; keyword search (BM25 or PostgreSQL full-text search) is the opposite. Running both and merging the results with Reciprocal Rank Fusion (RRF) is the most robust approach in practice. RRF gives each document a score of 1/(k + rank) for its rank in each list and sums the scores; the [2009 study by Cormack and colleagues](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf) uses k = 60. Its advantage is that you never have to normalise the two score scales.

Re-ranking adds one more step: the top few dozen candidates from the merged list are re-scored by a cross-encoder that reads the question and the passage together, and only the best few passages go to the model. It adds latency and cost, so do not switch it on without measuring its effect on the evaluation set.

## A working Python example

The code below runs on the standard library alone (Python 3.10+) and shows the logic of the pipeline: paragraph-aware chunking, a permission filter, vector and BM25 search, fusion with RRF and a prompt with numbered sources. `Embedder` is an interface; the `HashingEmbedder` in the example is only for testing and is replaced with your chosen embedding model in a real project. You can send the generated prompt to any language model; the code is not tied to a provider.

```python
"""A small provider-neutral RAG pipeline with permission filtering (Python 3.10+, standard library only)."""
from __future__ import annotations

import hashlib
import math
import re
from collections import Counter
from dataclasses import dataclass
from typing import Protocol

class Embedder(Protocol):
    def embed(self, texts: list[str]) -> list[list[float]]: ...

def tr_lower(text: str) -> str:
    # str.lower() mishandles Turkish I/İ; map those two by hand first.
    return text.replace("I", "ı").replace("İ", "i").lower()

def tokenize(text: str) -> list[str]:
    return re.findall(r"\w+", tr_lower(text))

class HashingEmbedder:
    """Placeholder for testing. Replace with a multilingual embedding model in production."""

    def __init__(self, dim: int = 512) -> None:
        self.dim = dim

    def embed(self, texts: list[str]) -> list[list[float]]:
        vectors = []
        for text in texts:
            vec = [0.0] * self.dim
            for tok in tokenize(text):
                for gram in {tok[i:i + 4] for i in range(max(1, len(tok) - 3))}:
                    h = int.from_bytes(hashlib.blake2b(gram.encode(), digest_size=8).digest(), "big")
                    vec[h % self.dim] += 1.0 if (h >> 63) & 1 else -1.0
            norm = math.sqrt(sum(v * v for v in vec)) or 1.0
            vectors.append([v / norm for v in vec])
        return vectors

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_title: str
    page: int
    text: str
    allowed_groups: frozenset[str]

def split_text(text: str, max_chars: int = 1200, overlap: int = 200) -> list[str]:
    """Simple chunker that respects paragraph boundaries and overlaps chunks."""
    paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks: list[str] = []
    current = ""
    for para in paragraphs:
        if current and len(current) + len(para) + 2 > max_chars:
            chunks.append(current)
            current = current[-overlap:]  # carry the tail over so context is not cut
        current = f"{current}\n\n{para}".strip()
    if current:
        chunks.append(current)
    return chunks

class HybridIndex:
    def __init__(self, embedder: Embedder, rrf_k: int = 60) -> None:
        self.embedder = embedder
        self.rrf_k = rrf_k
        self.chunks: list[Chunk] = []
        self.vectors: list[list[float]] = []
        self.term_freqs: list[Counter[str]] = []
        self.doc_freq: Counter[str] = Counter()

    def add(self, chunks: list[Chunk]) -> None:
        self.vectors += self.embedder.embed([c.text for c in chunks])
        for c in chunks:
            tf = Counter(tokenize(c.text))
            self.term_freqs.append(tf)
            self.doc_freq.update(tf.keys())
            self.chunks.append(c)

    def _bm25(self, query: str, ids: list[int], k1: float = 1.5, b: float = 0.75) -> dict[int, float]:
        n = len(self.chunks)
        avg_len = sum(sum(tf.values()) for tf in self.term_freqs) / max(n, 1)
        scores: dict[int, float] = {}
        for i in ids:
            tf, length, score = self.term_freqs[i], sum(self.term_freqs[i].values()), 0.0
            for term in set(tokenize(query)):
                if term in tf:
                    idf = math.log(1 + (n - self.doc_freq[term] + 0.5) / (self.doc_freq[term] + 0.5))
                    score += idf * tf[term] * (k1 + 1) / (tf[term] + k1 * (1 - b + b * length / avg_len))
            if score > 0:
                scores[i] = score
        return scores

    def search(self, query: str, user_groups: set[str], top_k: int = 4, pool: int = 20) -> list[Chunk]:
        # 1) The permission filter runs BEFORE ranking: chunks the user cannot see never become candidates.
        ids = [i for i, c in enumerate(self.chunks) if c.allowed_groups & user_groups]
        if not ids:
            return []
        # 2) Semantic search: cosine similarity (vectors are normalised).
        qv = self.embedder.embed([query])[0]
        dense = sorted(ids, key=lambda i: -sum(a * b for a, b in zip(qv, self.vectors[i])))[:pool]
        # 3) Keyword search: BM25.
        bm25 = self._bm25(query, ids)
        sparse = sorted(bm25, key=lambda i: -bm25[i])[:pool]
        # 4) Reciprocal Rank Fusion: merges both rankings without comparing score scales.
        fused: Counter[int] = Counter()
        for ranking in (dense, sparse):
            for rank, i in enumerate(ranking, start=1):
                fused[i] += 1.0 / (self.rrf_k + rank)
        return [self.chunks[i] for i, _ in fused.most_common(top_k)]

def build_prompt(question: str, hits: list[Chunk]) -> str:
    sources = "\n\n".join(
        f"[{n}] {h.doc_title}, p. {h.page}\n{h.text}" for n, h in enumerate(hits, start=1)
    )
    return (
        "Answer only from the sources below. End every sentence with its source as [n]. "
        "If the sources do not contain the answer, say 'This is not in the documents.'\n\n"
        f"SOURCES:\n{sources}\n\nQUESTION: {question}"
    )

if __name__ == "__main__":
    index = HybridIndex(HashingEmbedder())
    docs = [
        ("Travel Policy", 3, "For domestic trips, the accommodation cap depends on the job grade."
         "\n\nReceipts are uploaded to the expense system within 10 working days after the trip.",
         {"all-staff"}),
        ("Payroll Procedure", 7, "Payroll corrections are made only by the HR payroll team.",
         {"hr-payroll"}),
        ("IT Security Policy", 2, "Multi-factor authentication is mandatory for VPN access.",
         {"all-staff"}),
    ]
    for n, (title, page, text, groups) in enumerate(docs):
        index.add([
            Chunk(f"{n}-{k}", title, page, part, frozenset(groups))
            for k, part in enumerate(split_text(text))
        ])

    question = "Within how many days do I have to upload expense receipts?"
    hits = index.search(question, user_groups={"all-staff"})
    for h in hits:
        print(h.chunk_id, h.doc_title, "p.", h.page)
    print(build_prompt(question, hits))  # Send this text to the language model of your choice.
```

Two details deserve attention. Python's `str.lower()` turns "I" into "i", but in Turkish the correct lowercase is "ı"; that small error breaks matching between "IĞDIR" and "ığdır", which is why `tr_lower` exists for Turkish corpora. Second, the permission filter runs before the search: the payroll document never enters the candidate list for a user outside the payroll group. The sample documents are placeholders, not a real company's policies.

## Citations and source attribution

In a corporate assistant, an answer without a source does not build trust even when it is correct. Three rules I recommend:

1. The prompt asks the model to end every sentence with a numbered source and to say explicitly when the sources do not contain the answer.
2. The interface shows each number with the document name, version, page and link; the user opens the original with one click.
3. After generation, code checks that the numbers the model cites are actually among the retrieved chunks. An answer citing a non-existent source is not shown.

"This is not in the documents" is not a bug but a feature; an assistant that says so instead of inventing an answer earns trust faster.

## Access control: document-level ACLs

The most critical design decision in enterprise RAG: permissions are enforced in the retrieval layer, not by the model. An instruction like "do not reveal this if the user is not authorised" is not a security control; any text that enters the model's context can appear in the answer.

- Access control lists (ACLs) are taken from the source system during indexing and re-synced at regular intervals.
- The user's groups come from the identity provider on every query; a field sent by the client is not trusted.
- When a document is deleted or its permissions change at the source, all of its chunks in the index are updated.
- If you cache answers, the cache key includes the user's permission set; otherwise one user's answer can be shown to another.

The same principle applies to agents that call tools; details are in the [AI agents and MCP article](https://www.abdulazizakyol.com/en/blog/ai-agents-and-mcp-architecture-and-security-for-companies/).

## Evaluation: how do you measure accuracy and faithfulness?

Every tuning decision made without measurement is a guess. Build a "golden set" of real user questions with correct answers and correct sources; write the questions together with document owners. Then measure the layers separately:

| Layer      | Metric           | Question it asks                                                |
| ---------- | ---------------- | --------------------------------------------------------------- |
| Retrieval  | Recall\@k, MRR   | Is the right chunk in the top k results, and at what rank?      |
| Generation | Faithfulness     | Is every claim in the answer supported by the retrieved chunks? |
| Generation | Answer relevance | Does the answer actually address the question?                  |
| End to end | Correctness      | Does the answer match the correct answer in the golden set?     |

To measure faithfulness and answer relevance automatically, there are methods that use a language model as the judge, such as the approach in the [Ragas paper](https://arxiv.org/abs/2309.15217); sample the judge's decisions against human review at regular intervals. Re-run the set whenever chunk size, embedding model or prompt changes.

## Data protection and KVKK

Company documents contain personal data: HR files, customer correspondence, contracts. For readers outside Turkey: KVKK is Turkey's Personal Data Protection Law (Law No. 6698), broadly modelled on EU data protection law. Before you start:

- **Narrow the scope:** Decide explicitly which folders are indexed; leave sensitive areas such as HR and legal out of the first version.
- **Decide where data is processed:** If the embedding or language model runs abroad, that is a cross-border transfer under Article 9 of Law No. 6698. The amendment made by Law No. 7499, in force since 1 June 2024, put adequacy decisions and appropriate safeguards such as standard contracts at the centre; a standard contract must be notified to the authority within five working days ([KVKK announcement](https://www.kvkk.gov.tr/Icerik/7834/6698-Sayili-Kisisel-Verilerin-Korunmasi-Kanununda-Yapilan-Degisiklikler-Hakkinda-Kamuoyu-Duyurusu), in Turkish).
- **Protect vectors too:** Vectors are derived from the text; the cautious approach is to protect them at the same level as the source.
- **Design erasure end to end:** On an erasure request, chunks, vectors, caches and logs are deleted together.
- **Manage question logs:** User questions can contain personal data too; set a retention period.

The Turkish data protection authority's [guide on generative AI and personal data](https://www.kvkk.gov.tr/Icerik/8547/uretken-yapay-zeka-ve-kisisel-verilerin-korunmasi-rehberi-15-soruda), published on 24 November 2025, is a good starting point for this assessment. If the system will also serve users in the EU, see the [EU AI Act guide](https://www.abdulazizakyol.com/en/blog/eu-ai-act-compliance-guide-for-turkish-companies/) as well.

## Checklist

- Are the questions factual and is the knowledge changing? Then RAG.
- Only current versions are indexed; every document family has an owner.
- Chunks are stored with their heading path; source, page, version and access groups are in metadata.
- Hybrid search (BM25 + vectors, RRF) is on; re-ranking was added based on measurement.
- The permission filter runs before search; access lists are synced regularly.
- Every answer carries source numbers; citations are verified in code.
- A golden set exists; retrieval and generation metrics are re-measured on every change.
- Data location, cross-border transfer and erasure have been assessed under data protection law.

For the scope and 90-day plan of a first production project, see the [AI roadmap for CIOs](https://www.abdulazizakyol.com/en/blog/ai-roadmap-for-cios-first-production-project-in-90-days/).

## Frequently asked questions

### What is RAG?

RAG (retrieval-augmented generation) is an architecture in which a language model searches a document collection before answering and uses the retrieved passages as context. This lets the model answer with company knowledge that was not in its training data, and cite its sources.

### RAG or fine-tuning?

Use RAG when knowledge changes and answers need citations; use fine-tuning when you want the model to answer in a specific format, tone or domain jargon. For document Q&A the first project is almost always RAG, and document-level permissions and deletion requests are much easier to enforce with RAG.

### Which vector database should I choose for enterprise RAG?

If the company already runs PostgreSQL, the pgvector extension is a good start: it keeps vector search, full-text search, permission filtering and transactions in one database. At very large scale or with heavy concurrent load, dedicated vector databases and search engines are worth evaluating.

### How do you measure the accuracy of RAG answers?

Build a golden set of real user questions with correct answers and correct sources. Measure retrieval with recall@k and MRR, and generation with faithfulness (whether every claim in the answer is supported by the retrieved passages) and answer relevance.

### What should you watch for under data protection law when using RAG?

Narrow the folders you index, determine where the embedding and language models run (if abroad, it is a cross-border transfer under Turkey's KVKK Article 9, and possibly under the GDPR), protect vectors like the source text, delete chunks, vectors, caches and logs together on erasure requests, and set a retention period for question logs.

## Sources

1. [Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)
2. [pgvector: Open-source vector similarity search for Postgres (GitHub)](https://github.com/pgvector/pgvector)
3. [Cormack, Clarke, Büttcher (2009), Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)
4. [Es et al. (2023), Ragas: Automated Evaluation of Retrieval Augmented Generation](https://arxiv.org/abs/2309.15217)
5. [KVKK (Turkish DPA), Guide on Generative AI and Personal Data Protection (in Turkish)](https://www.kvkk.gov.tr/Icerik/8547/uretken-yapay-zeka-ve-kisisel-verilerin-korunmasi-rehberi-15-soruda)
6. [KVKK (Turkish DPA), Announcement on the Amendments to Law No. 6698 (in Turkish)](https://www.kvkk.gov.tr/Icerik/7834/6698-Sayili-Kisisel-Verilerin-Korunmasi-Kanununda-Yapilan-Degisiklikler-Hakkinda-Kamuoyu-Duyurusu)

---

Abdulaziz Akyol is the founder of CX Teknoloji (AI, computer vision and IoT). Canonical version: https://www.abdulazizakyol.com/en/blog/enterprise-rag-an-ai-assistant-that-talks-to-company-documents/