Ibraheem Wagih · Fraud & Financial Crime → Analytics
Projects / Applied GenAI & Evaluation

Market Research RAG Assistant

Individual Machine Learning for Business (MSIN0231) UCL MSc Business Analytics

A retrieval-augmented assistant that takes an industry name and produces a structured ~500-word research brief sourced from Wikipedia. The interesting part is not the generation, it is the evaluation: the app judges its own outputs before anyone reads them, and refuses to invent figures when the data is not there.

2 models
separate generator and LLM-as-judge
2-tier
evaluation: manual adversarial + automated
0
temperature on the judge and input validator
3 fixes
shipped as a direct result of evaluation findings

Problem

Generative research tools fail in a specific, dangerous way: confident, well-formatted briefs containing fabricated numbers. For business use that is worse than no answer. The brief was to build a working research assistant; the real assignment was proving it could be trusted.

Pipeline

Generation and judgement are deliberately separated, with a zero-temperature validator refusing bad inputs before any retrieval spend, and a zero-temperature judge checking every output for hallucination and the word-count constraint before it reaches the user.

User inputindustry or market name
→
Semantic validatorzero-temp LLM rejects non-industry inputs ("red", gibberish) before retrieval
→
RetrievalLangChain WikipediaRetriever, top 5 results
→
GeneratorGemini 2.5 Flash Lite writes the ~500-word brief with "state when data is unavailable" guardrails
→
LLM-as-judgeGemini 2.5 Flash, temp 0, checks hallucination + constraints before display
In-pipeline separation of duties: the model that writes is never the model that approves.

Evaluation Before Claims

A two-tier evaluation strategy came first: manual adversarial inputs to probe failure modes, plus the automated judge scoring every output. The findings drove three shipped fixes: the semantic input-validation layer, a tuned retriever configuration, and a rewritten system prompt that stops the model fabricating figures. The written reflection covered the production path: vector-database RAG over proprietary data, locally hosted models for privacy, and caching for repeat queries.