RAG development: AI answers grounded in your own content

RAG suits assistants that must answer from a body of content that is too large or changes too often to put in every prompt:

• Your content is larger than a model accepts in one request, such as a full help center, a product catalog or a set of codes and policies.
• The content changes and answers must follow it. You update the index, not the model.
• Users need to see where an answer came from, so each answer can cite its passages.
• Different users may see different content. Filters at query time can limit what each user retrieves.

A retrieval pipeline adds moving parts. Consider simpler options first:

• Your content is small and fixed. If it fits in the prompt, put it there and skip the index.
• The answers live in structured data, such as orders, stock or account balances. A database query gives exact results, and a model can call it as a tool.
• You want a consistent tone, format or behavior. That is a job for prompting or fine-tuning. Microsoft's guidance notes that fine-tuning does not replace retrieval for current information.
• No one can own the content. A RAG system is only as good as the pages it indexes, so someone has to keep them correct.

Microsoft describes RAG as grounding a model's responses in your own content. Content is split into chunks, because models have input limits and a long page is poorly represented by one vector. Each chunk is turned into an embedding and stored with its text and metadata. At question time the system retrieves the most relevant chunks and gives only those to the model.

• Chunking: Azure AI Search's guidance suggests fixed-size chunks with some overlap as a starting point, or chunks split on headings and sentences. Adding the document title to chunks from the middle of a document helps keep context.
• Hybrid search: Azure AI Search runs keyword and vector queries in one request and merges them with Reciprocal Rank Fusion. Microsoft notes that product codes, jargon, dates and names often match better by keyword. Qdrant combines dense vectors, sparse vectors and BM25 with prefetch and fusion. pgvector's README suggests pairing it with Postgres full-text search.
• Filtering: Qdrant filters on payload fields, and its docs advise creating payload indexes for those fields before loading data. In pgvector, an approximate index applies the WHERE clause after the index scan, so a filtered query can return fewer rows than asked. Its README lists fixes such as iterative scans and partial indexes.
• Reranking: a second stage re-scores the first results. Azure AI Search's semantic ranker reranks the top 50 results. Pinecone offers integrated and standalone reranking with hosted models.
• Separation: Pinecone namespaces keep each tenant's data apart within one index. Azure AI Search supports document-level security trimming and filter-based security at query time.

• A fresh index: content changes, so ingestion runs on a schedule. Azure AI Search supports incremental updates with mergeOrUpload and delete actions, and schema changes such as a changed field definition need a rebuild. A side-by-side index behind an alias avoids downtime.
• A guard on ingestion: a crawl can fail or pull the wrong pages. We use host allow-lists, clean the content, mask fields such as prices that must come from the live system, and abort the load if a run looks wrong, so a bad crawl never replaces a good index.
• Answers only from sources: the model is told to answer only from the retrieved passages and to say when they do not cover the question. An output check runs before the answer is shown. Approved answers to common questions can be served first.
• Citations: each chunk keeps its source URL or section, so the answer can link to it and a reviewer can check it.
• Evaluation: we keep a set of real questions with the passages that should answer them, and run it whenever chunking, the embedding model or the prompt changes. When we rebuilt one knowledge base with better chunking, the test set is what showed whether it helped.

1. Look at the content and the questions first: what the sources are, how often they change, who may see what, and which questions users really ask
2. Build ingestion and retrieval before the chat: check that the right passages come back for real questions before any answer is generated
3. Add answers, citations and guards: answers only from retrieved passages, a source link with each answer, and a check on the output
4. Hand over properly: the ingestion schedule, the index design and the evaluation set documented, with notes on reading a bad answer back to its source