Agentic RAG

Agentic RAG is a robust Retrieval-Augmented Generation system designed to go beyond simple vector lookups. By leveraging LangGraph, it introduces agentic capabilities such as self-correction, reflection, and external tool use to provide accurate and grounded answers.

Architecture & Tech Stack

• Backend Framework: FastAPI powers the asynchronous API and manages the orchestration of the AI pipeline.
• Agentic Orchestration: Built with LangGraph, the system models the reasoning process as a state graph. This allows the agent to reflect on its own answers, check for hallucinations, and retry if the initial response isn't grounded in the context.
• Advanced Retrieval:
  • Uses Supabase (PostgreSQL + pgvector) for storing document chunks and performing vector similarity searches.
  • Implements Hybrid Search (combining pgvector and Full-Text Search) to improve retrieval accuracy.
  • Utilizes SentenceTransformers for embedding generation and cross-encoders for highly accurate re-ranking of the retrieved chunks.
• Frontend & Real-time Streaming: A React frontend that handles real-time response generation using Server-Sent Events (SSE) for smooth token streaming.
• Web Fallback: Features an integrated DuckDuckGo web search tool. When local document retrieval returns empty, the agent dynamically routes the query to the web to find up-to-date answers.

Overcoming Key Challenges

Building an Agentic RAG system from scratch surfaced several fascinating engineering and security challenges:

1. Handling Async Bottlenecks: Embedding models and re-rankers are notoriously CPU-bound. Ensuring that synchronous models (like SentenceTransformer.encode()) didn't block FastAPI's async event loop required careful thread-pool offloading.
2. Chunking Strategies: Tuning the document chunking service to maintain proper token overlap ensures that context isn't lost at chunk boundaries—a common silent failure mode in standard RAG pipelines.
3. Security & IDOR Protection: The application enforces strict row-level security and local authorization checks to prevent Insecure Direct Object Reference (IDOR) vulnerabilities, ensuring users can only query and stream sessions they own.

Future Enhancements

While the core agentic loop is functional, future iterations will focus on scaling the embedding service (potentially offloading it to dedicated GPU workers) and optimizing the streaming context window for unbounded document uploads.