RAG in practice: what ENSET AI taught me, and where retrieval is going

Students don't want answers from a model's general memory. They want answers from their course: this lecture, this lab sheet, this page. And they want to check where the answer came from.

That's what ENSET AI does. It's a self-hosted platform for my school where students and staff upload PDFs, chat with them, and get streamed answers with the source pages attached. It uses RAG, retrieval-augmented generation: instead of hoping the model knows the answer, you find the relevant passages first and give them to the model along with the question.

This article walks through how the pipeline works, what I'd improve, and where I think RAG is going next. The code is public on GitHub.

#The pipeline

RAG has two halves: ingestion, when a document arrives, and retrieval, when a question does.

#Ingestion: from PDF to searchable chunks

1. Check the file first. Before any parser touches an upload, scan_pdf_security verifies the %PDF- signature and looks for active content like embedded JavaScript or auto-run actions. Anything suspicious is blocked. It's not an antivirus, but a RAG system reads every file it's given, so it shouldn't blindly parse whatever it's handed.
2. Split into chunks. The text is cut into pieces of about 1,000 characters with a 200-character overlap, using LangChain's RecursiveCharacterTextSplitter. The overlap stops a sentence that sits across a boundary from being lost.
3. Keep the page number. Every chunk carries its document name and page number. That's what makes citations possible later.
4. Embed and store. Each chunk becomes a vector with the all-MiniLM-L6-v2 model and is stored in its own per-document collection in Qdrant (or Chroma in local development).

One collection per document was a deliberate choice. Deleting a document means dropping one collection, and a user's search can only ever touch the collections they are allowed to see.

#Retrieval: from question to cited answer

1. Check access. Flask verifies that the user can read every selected document before searching. Privacy is enforced on the way in, not filtered on the way out. There's a dedicated privacy test suite for this.

2. Rewrite the question, or skip search. A follow-up like "and what about the second one?" is useless as a search query. A small LLM call rewrites it into a standalone query using the last few messages. The same call can answer __NO_SEARCH__ for greetings or general questions, which saves a pointless search.

3. Search in parallel. The query runs against each selected document's collection at the same time, and the best passages are kept (top 5 by default).

4. Build a tagged context. Passages are wrapped in tags the model can refer to:

   <document id="1" source="network-security.pdf" page="12">
   ...passage text...
   </document>
   

5. Stream the answer. The response streams to the browser over Server-Sent Events, with the source pages attached so the student can open them.

#The architecture decisions that mattered

• One source of truth. PDFs and metadata (in MinIO and PostgreSQL) are the real data. Vectors are derived: a reindex script rebuilds them. So switching embedding models, or losing the vector store, is an inconvenience, not a data loss.
• The public assistant is walled off. The landing page has an optional public chatbot. Its endpoints only ever see context an admin approved as public, never session or document IDs, and it logs only metadata with a hashed IP.
• Several LLM providers. A factory lets the admin switch models, with an automatic fallback order when a provider hits its rate limit.

#What I'd improve

Writing this article made me reread my own pipeline with fresh eyes. Here's what I'd change.

#"If it's not relevant, use your own knowledge"

The prompt tells the model: use the search results if they are relevant, otherwise ignore them and answer from its own knowledge. That's friendly, but it means an answer can look sourced while actually coming from the model's memory. A citation only proves which passages were retrieved, not that the answer came from them.

For a study tool, I now think the better default is stricter: answer from the documents, and say so clearly when they don't contain the answer.

#No threshold, and only one kind of search

• There's no minimum relevance score. The top 5 passages are sent even when all 5 are poor matches.
• Search is vectors only. Embeddings are great at meaning and bad at exact strings. A course code like INF-4302, a function name or an acronym can be missed by semantic search when a plain keyword search would find it instantly.
• Chunks lose their context. A passage saying "this protocol fails when the key is reused" doesn't say which protocol. Out of context, it's hard to retrieve.

#Retrieved text is untrusted input

A PDF is written by someone. If a document contains "ignore previous instructions and…", that text lands in the model's context. It's the same prompt-injection problem I wrote about in securing Agora. ENSET AI's saving grace is that the model can only write an answer. It has no tools to call. The moment a RAG system can act, retrieved text needs to be treated as hostile.

#Where RAG is going

Basic RAG ("embed, retrieve the top-k, stuff the prompt") is where everyone starts. Here's what comes next, and what I'd adopt.

#1. Hybrid search and reranking

Combine vector search with a keyword method like BM25, merge the two result lists, then use a reranker, a model that reads the question and each passage together and scores them far more precisely than embeddings can. Search broadly, then rank carefully. This fixes the course-code problem directly, and it's the first thing I'd add.

#2. Contextual retrieval

Proposed by Anthropic in September 2024: before embedding a chunk, have a model write a sentence or two situating it in its document ("This passage is from the TLS chapter of the network security course, discussing key reuse"). That fixes the "which protocol?" problem.

Their measurements: contextual embeddings cut failed retrievals by 35%, adding contextual BM25 by 49%, and adding reranking by 67%. The gains stack, which is why 1 and 2 belong together.

#3. Adaptive and self-correcting retrieval

Not every question needs a search, and not every search result deserves trust.

• Self-RAG trains the model to decide when to retrieve and to critique whether each passage actually supports its answer.
• Corrective RAG (CRAG) adds an evaluator that grades the retrieved documents. If they're poor, it rewrites the query, searches elsewhere, or discards them instead of answering from bad context.

ENSET AI's __NO_SEARCH__ step is a very small version of this idea. The natural next step is to also grade what comes back, and to say "the documents don't cover this" rather than forcing an answer.

#4. RAFT: training the model to read like a student

RAFT (Retrieval-Augmented Fine-Tuning, UC Berkeley, 2024) attacks the problem from the model's side. It fine-tunes the model on questions paired with a mix of the right document and distractor documents that look related but don't help. The model learns to quote the relevant passage verbatim, reason step by step, and ignore the distractors.

The paper compares it to an open-book exam: RAG gives the model the book, and RAFT teaches it how to use the book, including which pages to skip. For a school that keeps the same courses year after year, fine-tuning a small model on its own material with RAFT is a realistic way to get better answers at lower cost than a large general model.

#5. GraphRAG: questions about the whole corpus

Top-k retrieval is good at "what does page 12 say about X?" and bad at "what are the main themes across this whole course?", because no single passage contains the answer. GraphRAG (Microsoft) extracts entities and relationships into a knowledge graph, groups them into communities, and summarizes those. Broad questions can then be answered from the summaries instead of from a handful of random chunks.

#6. Agentic RAG, and why evaluation comes first

The direction everything is moving in is agentic RAG: a model that plans several searches, follows leads, and combines sources, instead of doing a single lookup. It's powerful, and it brings back every security question from Agora, because now retrieved text can influence actions.

But the most important piece isn't a technique, it's evaluation. Before changing anything, you need a fixed set of real questions with known answers, and metrics like retrieval recall (did the right passage come back?) and faithfulness (is the answer supported by what was retrieved?). Without that, every "improvement" is a guess, and a change that quietly makes answers worse can go unnoticed for months.

#My roadmap for ENSET AI

In order:

1. Build a small evaluation set from real course questions: retrieval recall and faithfulness, run in CI.
2. Add hybrid search and a reranker, and measure the difference.
3. Contextual chunk descriptions at ingestion.
4. A relevance threshold and a stricter prompt: answer from the documents or say they don't cover it.
5. Later: grade retrieved passages CRAG-style, and experiment with RAFT on a small open model.

#What I took from it

Getting a RAG demo working takes an afternoon. Making it trustworthy is the real work: access checks before search, citations a student can verify, a clear boundary between private and public data, and honesty about what a citation does and doesn't prove.

And the biggest gains are rarely where you'd expect. Not a bigger model, but better retrieval, stricter grounding and a way to measure both. Measure before you optimize.