Coder Legion - Recent posts tagged vector

Beyond Elasticsearch: The Rise of Unified Vector and Keyword Search

Wed, 22 Jan 2025 11:21:23 +0000

This post was written by James Luan, VP of Engineering at Zilliz and key maintainer of the Milvus open source project.

The Evolution of Modern Search Architecture

You're debugging a critical production issue, and you know the solution exists in your documentation. Your vector search returns semantically relevant results about error handling and system recovery but somehow misses that one crucial Stack Overflow answer you saved last month with the exact error code. Sound familiar?

This common frustration highlights a fundamental challenge in modern search technology: the need to balance semantic understanding with precise term matching. The rise of AI and large language models has brought powerful semantic search capabilities through vector embeddings while revealing the continued importance of traditional keyword matching. When supporting customers, for instance, vector search brilliantly handles natural language queries like "my payment isn't going through" by understanding user intent. Still, keyword search remains essential for matching specific product codes and error messages.

Recent experiences with RAG implementations across the industry have revealed interesting insights about search technology. While dense vector embeddings excel at capturing semantic relationships and contextual understanding, traditional BM25 keyword matching still proves superior in certain scenarios, particularly with domain-specific queries and rare terms. This complementary strength of different search approaches has pushed the community toward exploring unified solutions.

However, implementing hybrid search at scale presents its own set of challenges. While frameworks like LangChain or LlamaIndex have made proof-of-concept implementations straightforward, scaling these solutions to production reveals significant architectural complexities. The conventional approach requires maintaining separate vector databases and search engines, resulting in the following:

Data redundancy across multiple storage systems
Complex consistency management between separate indices
Fragmented security controls and access patterns
Increased latency due to cross-system synchronization
Higher operational complexity in high-throughput environments

An Example of Complex Architectures of Hybrid Search

Elasticsearch's Technical Limitations

As the long-standing leader in enterprise search, Elasticsearch has powered everything from e-commerce platforms to logging systems for over a decade. Their dominance in keyword-based search and their extensive enterprise feature set, made them an obvious candidate for bridging the gap between traditional and AI-powered search. Recognizing this opportunity, Elasticsearch made a significant move by introducing vector search capabilities in version 8.0, aiming to unify traditional keyword search with modern vector operations under a single platform.

However, despite its market leadership and ambitious steps toward vector search capabilities, Elasticsearch faces several fundamental constraints that impact its effectiveness in modern search architectures:

Performance and Resource Management

The tight coupling between write operations, index construction, and query processing creates substantial overhead during data ingestion and indexing. This architectural decision leads to CPU and I/O contention, particularly during bulk updates. The "near real-time" search design introduces latency that can impair AI applications requiring rapid interactions or dynamic decision-making.

Architectural Constraints

The JVM-based architecture introduces additional complexity through memory management overhead and garbage collection pauses. Organizations often find themselves over-provisioning resources to maintain stable performance as data volumes grow. The non-cloud-native design, with tightly coupled storage and compute, requires simultaneous scaling of both resources, reducing deployment flexibility.

Scaling Limitations

The shard management system presents a fundamental dilemma: too many shards in small datasets lead to performance degradation. However, too few shards in large datasets limit scalability and cause data distribution imbalances. In multi-replica scenarios, the requirement for independent index construction per shard multiplies compute costs and reduces resource efficiency.

Vector Search Performance

The vector search implementation, built on Lucene's core, shows notable performance gaps compared to specialized engines, particularly in high-dimensional spaces. Internal benchmarks indicate performance degradation becomes acute beyond 128 dimensions, with query latency increasing exponentially in high-concurrency scenarios.

Technical Innovation: A New Approach to Unified Search

Recent advances in search technology have emerged from extensive experience with real-world search challenges across thousands of production deployments. A consistent pattern has emerged through deep engagement with the open-source community. While vector search excels at semantic matching, organizations often need to maintain separate systems for traditional keyword search, leading to architectural complexities and operational overhead.

This insight led to a fundamental reimagining of how unified search capabilities could work in the open source vector database Milvus. The result is Sparse-BM25, introduced by Milvus 2.5, an enhanced approach that leverages vector operations while preserving the precision of traditional keyword matching. This technical innovation builds on Milvus's expertise in vector operations while addressing the broader community's need for unified search capabilities.

Core Technical Architecture

The implementation introduces several key technical innovations:

Advanced text processing pipeline powered by Tantivy, delivering
sophisticated stemming, lemmatization, and stop word filtering
capabilities with support for over 30 languages
Distributed term frequency management system that efficiently handles
corpus-wide statistics at scale, with automatic rebalancing and
optimization
Novel sparse vector generation utilizing Corpus TF for documents and
Query TF with global IDF for queries, optimized for high-cardinality
vocabularies
Custom BM25 distance function optimized for high-dimensional sparse
vector spaces, with configurable saturation parameters

The system implements a sophisticated indexing hierarchy that adapts to query patterns. The WAND-based inverted index provides efficient retrieval for standard queries, while the Block-Max WAND algorithm development promises further optimization for long-tail queries. Through careful pruning of long-tail terms and vector quantization, the system achieves more than 5x performance improvement while keeping recall degradation under 1% and reducing memory usage by over 50%.

The graph index support extends beyond basic HNSW implementation, incorporating dynamic graph pruning and rebuilding based on query patterns. This approach has shown particular effectiveness in handling high-dimensional sparse vectors, where traditional inverted indices struggle with the curse of dimensionality. The system automatically maintains optimal graph connectivity while minimizing index maintenance overhead.

Technical Advances in Modern Search Systems

Algorithm Flexibility

Modern search systems transform similarity calculations into vector distance computations, enabling more complex queries and corpus distance analysis. Recent research like "End-to-End Query Term Weighting" has introduced innovations such as the Term Weighting BERT (TW-BERT) algorithm, which uses BERT to infer n-gram term weights from queries and construct query representations. When combined with BM25 for calculating document relevance, these approaches demonstrate significant improvements over traditional token-based methods in both in-domain and out-of-domain scenarios.

Efficiency Considerations

Advanced search architectures implement lexical search through sparse vectors, combining traditional inverted index compression techniques with lossy compression for dense embeddings. Through careful pruning of long-tail terms and vector quantization, these systems can achieve substantial performance improvements while maintaining high recall rates and reducing resource usage. Ongoing research continues to optimize data compression to further reduce storage costs and query I/O overhead.

Handling Complex Queries

While traditional search engines commonly use WAND (Weak AND) technology to optimize inverted index queries, its effectiveness can diminish with long queries due to excessive inverted list intersections and reduced pruning efficiency. Modern approaches address this by combining sparse embeddings with graph indices like HNSW, showing particular promise for sparse vectors with high dimensionality.

These technical advances in unified vector and keyword search are particularly significant for RAG applications, where queries often combine natural language questions with specific technical terms or metadata filters. Efficiently processing these mixed queries while maintaining high performance is crucial for production deployments.

Looking Forward: The Future of Search

Remember that frustrated developer from our opening scenario? Today's reality is that teams shouldn't have to choose between semantic understanding and precise matching, performance and flexibility, or simplicity and scale. As our industry continues to push the boundaries of AI and search technology, we're moving toward a future where search infrastructure becomes truly intelligent - understanding not just what users are searching for but why they're searching for it.

The next frontier isn't just about combining vector and keyword search - it's about creating systems that can dynamically adapt to different types of queries, learn from user interactions, and seamlessly integrate with the growing ecosystem of AI tools and frameworks. As we continue to build and experiment with these technologies in Milvus, we're not just solving technical challenges - we're reshaping how humans interact with and make sense of the vast amounts of information at our disposal.

This is an incredibly exciting time for the developers and engineers working on these challenges every day. The tools and capabilities we're building today will form the foundation of how future generations will access and understand information. And that's worth getting right.

Learn how to write GenAI applications with Java using the Spring AI framework and utilize RAG for improving answers.

Sun, 22 Sep 2024 09:21:49 +0000

Generative AI (GenAI) is currently a hot topic in the tech world. Retrieval-Augmented Generation (RAG) is a technique that enhances the accuracy and reliability of generative AI models by grounding them in external knowledge sources.

In this blog post, we'll look at how to write GenAI applications with Java using the Spring AI framework and utilize RAG for improving answers.

What is Spring AI?

Spring AI is a framework for building generative AI applications in Java. It provides a set of tools and utilities for working with generative AI models and architectures, such as large language models (LLMs) and retrieval augmented generation (RAG). Spring AI is built on top of the Spring Framework, which is a popular Java framework for building enterprise applications, allowing those already familiar with or involved in the Spring ecosystem the ability to incorporate GenAI strategies to their already existing applications and workflow.

There are also other options for GenAI in Java, such as Langchain4j, but we'll focus on Spring AI for this post.

Creating a project

To get started with Spring AI, you'll need to either create a new project or add the appropriate dependencies to an existing project. You can create a new project using the Spring Initializr at https://start.spring.io/, which is a web-based tool for generating Spring Boot projects.

When creating a new project, you'll need to add the following dependencies:

Spring Web
OpenAI (or other LLM model, such as Mistral, Ollama, etc.)
Neo4j Vector Database (other vector database options also available)
Spring Data Neo4j

If you're adding these dependencies manually to an existing project, you can see the dependency details in today's related Github repository.

The Spring Web dependency allows us to create a REST API for our GenAI application. We need the OpenAI dependency to access the OpenAI model, which is a popular LLM. The Neo4j Vector Database dependency allows us to store and query vectors, which are used for similarity searches. Finally, adding the Spring Data Neo4j dependency provides support for working with Neo4j databases in Spring applications, allowing us to run Cypher queries in Neo4j and map entities to Java objects.

Go ahead and generate the project, and then open it in your favorite IDE. Looking at the pom.xml file, you should see that the milestone repository is included. Since Spring AI is not a general-availability release yet, we need to include the milestone repository to get the pre-release version of the dependencies.

A bit of boilerplate

First thing that we need is a Neo4j database. I like to use the Neo4j Aura free tier because the instance is managed for me, but there are also Docker images and other methods.

Depending on the LLM model you chose, you will also need an API key. For OpenAI, you can get one by signing up at OpenAI.

Once you have a Neo4j database and an API key, you can set up the config in the application.properties file. Here's an example of what that might look like:

spring.ai.openai.api-key=<YOUR API KEY HERE>
spring.neo4j.uri=<NEO4J URI HERE>
spring.neo4j.authentication.username=<NEO4J USERNAME HERE>
spring.neo4j.authentication.password=<NEO4J PASSWORD HERE>
spring.data.neo4j.database=<NEO4J DATABASE NAME HERE>

Note: It's a good idea to keep sensitive information like API keys and passwords in environment variables or other location external to the application. To create environment variables, you can use the export command in the terminal or set them in your IDE.

We can set up Spring Beans for the OpenAI client and the Neo4j vector store that will allow us to access necessary components wherever we need them in our application.

We can put these in our SpringAiApplication class by adding the related code to the class.

The EmbeddingClient bean creates a client for the OpenAI API and passes in our API key environment variable. Lastly, the Neo4jVectorStore bean configures Neo4j as the store for embeddings (vectors). We customize the configuration by specifying the label for the nodes that will store the embeddings, as Spring's default looks for Document entities. We also specify our index name for the embeddings (default is spring-ai-document-index).

Data set

For this example, we'll use a dataset of books and reviews from Goodreads. You can pull a curated version of the dataset from here. The dataset contains information about books, as well as related reviews.

I have already generated embeddings using OpenAI's API, so if you want to generate your own, you will need to comment out the final Cypher statement in the script and instead run the generate-embeddings.py script (or your custom version) to generate and load the review embeddings to Neo4j.

Application model

Next, we need to create a domain model in our application to map to our database model. In this example, we'll create a Book entity that represents a book node. We'll also create a Review entity that represents a review of a book. The Review entity will have an embedding (vector) associated with it, which we'll use for similarity searches.

These entities are standard Spring Data Neo4j code, so I won't show the code here. However, full code for each class is available in the Github repository - Book class, Review class.

We also need a repository interface defined so that we can interact with the database. While we will need to define a custom query, we'll come back and add that in a bit later.

public interface BookRepository extends Neo4jRepository<Book, String> {
}

Next, the core of this application where all the magic happens is the controller class. This class will contain the logic for taking a search phrase provided by the user and calling the Neo4jVectorStore to calculate and return the most similar ones. We can then pass those similar reviews into a Neo4j query to retrieve connected entities, providing additional context in the prompt for the LLM. It will use all the information provided to respond with some similar book recommendations for the original searched phrase.

Controller

Our controller class contains a couple of common annotations, to start. We'll also inject the Neo4jVectorStore and BookRepository beans that we defined earlier, as well as the OpenAiChatClient for our embedding client.

The next thing is to define a string for our prompt. This is the text that we will pass to the LLM to generate the response. We'll use the search phrase provided by the user and the similar reviews we find in the database to populate our prompt parameters in a few minutes. Next, we define the constructor for the controller class, which will inject the necessary beans.

@RestController
@RequestMapping("/")
public class BookController {
    private final ChatClient client;
    private final Neo4jVectorStore vectorStore;
    private final BookRepository repo;

    String prompt = """
            You are a book expert providing recommendations from high-quality book information in the CONTEXT section.
            Please summarize the books provided in the context section.
                        
            CONTEXT:
            {context}
                        
            PHRASE:
            {searchPhrase}
            """;

    public BookController(ChatClient.Builder builder, Neo4jVectorStore vectorStore, BookRepository repo) {
        this.client = builder.build();
        this.vectorStore = vectorStore;
        this.repo = repo;
    }

    //Retrieval Augmented Generation with Neo4j - vector search + retrieval query for related context
    @GetMapping("/rag")
    public String generateResponseWithContext(@RequestParam String searchPhrase) {
        List<Document> results = vectorStore.similaritySearch(SearchRequest.query(searchPhrase).withTopK(10));

        //more code shortly!
    }
}

Finally, we define a method that will be called when a user makes a GET request to the /rag endpoint. This method will first take a search phrase as a query parameter and pass that to the vector store's similaritySearch() method to find similar reviews. I have also added a couple of customization filters to the query.

Then, we map the similar Review nodes back to Document entities because Spring AI expects a general document type. The Neo4jVectorStore class contains methods to convert Document to a custom record, as well as the reverse for record to Document conversion.

Back in our controller method for book recommendations, we now have similar reviews for the user's searched phrase. But reviews (and their accompanying text) aren't really helpful in giving us book recommendations. So now we need to run a query in Neo4j to retrieve the related books for those reviews. This is the retrieval augmented generation (RAG) piece of the application.

Let's write the query in the BookRepository interface to find the books associated with those reviews.

public interface BookRepository extends Neo4jRepository<Book, String> {
    @Query("MATCH (b:Book)<-[rel:WRITTEN_FOR]-(r:Review) " +
            "WHERE r.id IN $reviewIds " +
            "AND r.text <> 'RTC' " +
            "RETURN b, collect(rel), collect(r);")
    List<Book> findBooks(List<String> reviewIds);
}

In the query, we pass in the ids of the reviews from the similarity search ($reviewIds) and pull the Review -> Book pattern for those reviews. We also filter out any reviews that have the text 'RTC' (which is a placeholder for reviews that don't have text). We then return the Book nodes, the relationships, and the Review nodes.

Now we need to call that method in our controller and pass the results to a prompt template. We will pass that to the LLM to generate a response with a book recommendation list based on the user's search phrase (we hope!). :)

//Retrieval Augmented Generation with Neo4j - vector search + retrieval query for related context
@GetMapping("/rag")
public String generateResponseWithContext(@RequestParam String searchPhrase) {
    List<Document> results = vectorStore.similaritySearch(SearchRequest.query(searchPhrase).withTopK(10));

    List<Book> bookList = repo.findBooks(results.stream().map(Document::getId).collect(Collectors.toList()));
    System.out.println("--- Book list ---");
    System.out.println(bookList);

    var template = new PromptTemplate(prompt, Map.of("context", bookList.stream().map(b -> b.toString()).collect(Collectors.joining("\n")), "searchPhrase", searchPhrase));
    System.out.println("----- PROMPT -----");
    System.out.println(template.render());

    return client.prompt(template.create()).call().content();

}

Starting right after the similarity search, we call our new findBooks() method and pass in the list of review ids from the similarity search. The retrieval query returns to a list of books called bookList. Next, we create a prompt template with the prompt string, the context data from the graph, and the user's search phrase, mapping the context and searchPhrase prompt parameters to the graph data (list with each item on new line) and the user's search phrase, respectively. I have also added a System.out.println() to print the prompt to the console so that we can see what is getting passed to the LLM.

Finally, we call the template's create() method to generate the response from the LLM. The returning JSON object has a contents key that contains the response string with the list of book recommendations based on the user's search phrase.

Let's test it out!

Running the application

To run our Goodreads AI application, you can use the ./mvnw spring-boot:run command in the terminal. Once the application is running, you can make a GET request to the /rag endpoint with a search phrase as a query parameter. Some examples are included next.

http ":8080/rag?searchPhrase=happy%20ending"
http ":8080/rag?searchPhrase=encouragement"

Sample call and output + full prompt

Call and returned book recommendations:

jenniferreif@elf-lord springai-goodreads % http ":8080/rag?searchPhrase=encouragement"
    
The books recommended in the context section are all inspirational and encouraging reads. They include "An Autobiography," "The Greatest Gift," "The Cross and the Switchblade," "David and Goliath," "The Art of Recklessness," "90 Minutes in Heaven," "Just Kids," and "The Five People You Meet in Heaven." These books offer great insight, hope, and encouragement to readers looking for uplifting stories and messages.

Application log output:

----- PROMPT -----
You are a book expert providing recommendations from high-quality book information in the CONTEXT section.
Please summarize the books provided in the context section.

CONTEXT:
Book[book_id=125441, title=An Autobiography, isbn=0717806677, isbn13=9780717806676, reviewList=[Review[id=d664f3888ecdfd78ec50606fe5f9b58c, text=Inspiring, rating=5]]]
Book[book_id=1488663, title=The Greatest Gift: The Original Story That Inspired the Christmas Classic It's a Wonderful Life, isbn=0670862045, isbn13=9780670862047, reviewList=[Review[id=b74851666f2ec1841ca5876d977da872, text=Inspiring, rating=4]]]
Book[book_id=772852, title=The Cross and the Switchblade, isbn=0515090255, isbn13=9780515090253, reviewList=[Review[id=f70c68721a0654462bcc6cd68e3259bd, text=encouraging, rating=4]]]
Book[book_id=15751404, title=David and Goliath: Underdogs, Misfits, and the Art of Battling Giants, isbn=0316204366, isbn13=9780316204361, reviewList=[Review[id=8c8bf791101b405c2cd134da193d2701, text=Inspiring, rating=4]]]
Book[book_id=7517330, title=The Art of Recklessness: Poetry as Assertive Force and Contradiction, isbn=1555975623, isbn13=9781555975623, reviewList=[Review[id=2df3600d488e182a3ef06bff7fc82eb8, text=Great insight, great encouragement, and great company., rating=4]]]
Book[book_id=89375, title=90 Minutes in Heaven: A True Story of Death and Life, isbn=0800759494, isbn13=9780800759490, reviewList=[Review[id=85ef80e09c64ebd013aeebdb7292eda9, text=inspiring & hope filled, rating=5]]]
Book[book_id=341879, title=Just Kids, isbn=006621131X, isbn13=9780066211312, reviewList=[Review[id=3eab2a67ac0bde179dd017014cb212bf, text=Inspiring, rating=5], Review[id=d731490fcfefe6547e459bdff204df4a, text=inspiring., rating=5]]]
Book[book_id=3431, title=The Five People You Meet in Heaven, isbn=1401308589, isbn13=9781401308582, reviewList=[Review[id=3ad8018734f177fd1fe47e6ada1c5df8, text=inspirational, rating=4], Review[id=2b94facb74fe09afd1fd54f0db2251c8, text=Inspiring, rating=5]]]

PHRASE:
encouragement

We can see that the LLM generated a response with a list of book recommendations based on the books found in the database (CONTEXT section of prompt). The results of the similarity search + graph retrieval query for the user's search phrase are in the prompt, and the LLM's answer uses that data for a response.

Wrapping Up!

I hope this post helps to get you started with Spring AI and beyond. Happy coding!

Resources

Code (Github repository): Spring AI Goodreads
Webpage: Spring AI project
API: Spring AI - Neo4jVectorStore