Enhancing AI Agent Accuracy with Re-rankers in N8N

Central Theme

The video explains how to significantly improve the accuracy of Retrieval-Augmented Generation (RAG) AI agents by implementing re-rankers. It specifically focuses on how this advanced technique works and how to use the new Cohere re-ranker feature within N8N, including its current limitations and a more advanced workaround.

Key Arguments & Findings

• The Problem with Vector Search: Standard vector search compresses the meaning of text into numerical vectors, which can cause “information loss.” As a result, the top results from a vector store are not always the most relevant. Simply increasing the number of results returned (“context stuffing”) can overwhelm the LLM and degrade performance due to the “lost in the middle” problem, where the model struggles to recall information from the middle of a large context.
• The Solution – Re-ranking: Re-ranking introduces a two-stage retrieval process to enhance accuracy:
  1. Stage 1 (Recall): A fast vector search retrieves a large set of potentially relevant documents from the database (e.g., top 25).
  2. Stage 2 (Precision): A more sophisticated and accurate re-ranker model (a cross-encoder) analyzes this smaller set, re-orders them based on true relevance to the query, and returns only the top few (e.g., top 3).

  This maximizes both retrieval recall (getting all potentially useful info) and LLM recall (giving the LLM only the most relevant info).

• N8N Implementation (v1.98+): N8N has a new “re-rank results” toggle on its vector store node, which integrates with Cohere’s re-ranker. However, it has a significant limitation: it is hardcoded to return the top 3 re-ranked results, offering no user control to adjust this number.
• Advanced Workaround: To overcome the limitation and gain more control, you can create a custom workflow. This involves using an HTTP request node to call the Cohere API directly. This method allows you to specify how many initial documents to retrieve and how many top documents to get back from the re-ranker, enabling you to combine it with other techniques like hybrid search and metadata filtering.

Conclusion & Takeaways

Re-ranking is a powerful technique for building more accurate and reliable RAG agents by filtering out irrelevant noise before sending context to an LLM. While N8N’s new built-in feature provides an easy entry point, serious development requires a more custom approach (using HTTP requests) to fine-tune the process and achieve the best possible results.

Mentoring Question

Considering your own projects, where could you apply a two-stage retrieval process (a broad, fast search followed by a precise re-ranking) to improve the quality and relevance of the information your system provides to the user or a downstream process?

Source: https://youtube.com/watch?v=wU847FkGe5A&si=lWvH5ASwokfyjb5D