Develop a Cloud-Hosted RAG App With an Open Source LLM (2024)

Retrieval-augmented generation (RAG) is often used to develop customized AI applications, including chatbots, recommendation systems and other personalized tools. This system uses the strength of vector databases and large language models (LLMs) to provide high-quality results.

Selecting the right LLM for any RAG model is very important and requires considering factors like cost, privacy concerns and scalability. Commercial LLMs like OpenAI’s GPT-4 and Google’s Gemini are effective but can be expensive and raise data privacy concerns. Some users prefer open source LLMs for their flexibility and cost savings, but they require substantial resources for fine-tuning and deployment, including GPUs and specialized infrastructure. Additionally, managing model updates and scalability can be challenging with local setups.

A better solution is to select an open source LLM and deploy it on the cloud. This approach provides the necessary computational power and scalability without the high costs and complexities of local hosting. It not only saves on initial infrastructural costs but also minimizes maintenance concerns.

Let’s explore a similar approach to develop an application using cloud-hosted open source LLMs and a scalable vector database.

Tools and Technologies

Several tools are required to develop this RAG-based AI application. These include:

• BentoML: BentoML is an open source platform that simplifies deployment of machine learning models into production-ready APIs, ensuring scalability and ease of management.
• LangChain: LangChain is a framework for building applications using LLMs. It offers modular components for easy integration and customization.
• MyScaleDB: MyScaleDB is a high-performance, scalable database optimized for efficient data retrieval and storage, supporting advanced querying capabilities.

In this tutorial, we will extract data from Wikipedia using LangChain’s WikipediaLoader module and build an LLM on that data.

Preparation

Set Up the Environment

Load the Data

Begin by importing the WikipediaLoader from the langchain_community.document_loaders. wikipedia module. You’ll use this loader to fetch documents related to “Albert Einstein” from Wikipedia.

This uses the load method to retrieve the “Albert Einstein” documents, and the print method to print the contents of the first document to verify the loaded data.

Split the Text Into Chunks

Deploy the Models on BentoML

Your data is ready, and the next step is to deploy the models on BentoML and use them in your RAG application. Deploy the LLM first. You’ll need a free BentoML account, and you can sign up for one on BentoCloud if needed. Next, navigate to the Deployments section and click on the Create Deployment button in the top-right corner. A new page will open that looks like this:

Select the bentoml/bentovllm-llama3-8b-instruct-service model from the drop-down menu and click “Submit” in the bottom-right corner. This should start deploying the model. A new page like this will open:

The deployment can take some time. Once it is deployed, copy the endpoint.

Note: BentoML’s free tier only allows the deployment of a single model. If you have a paid plan and can deploy more than one model, follow the steps below. If not, don’t worry — we will use an open source model locally for embeddings.

Deploying the embedding model is very similar to the steps you took to deploy the LLM:

1. Go to the Deployments page.
2. Click the Create Deploymentbutton.
3. Select the sentence-transformers model from the list and click Submit.
4. Once the deployment is complete, copy the endpoint.

Next, go to the API Tokens page and generate a new API key. Now you are ready to use the deployed models in your RAG application.

Define the Embeddings Method

You will define a function called get_embeddings to generate embeddings for the provided text. This function takes three arguments. If the BentoML endpoint and API token are provided, the function uses BentoML’s embedding service; otherwise, it uses the local transformers and torch libraries to load the sentence-transformers/all-MiniLM-L6-v2 model and generate embeddings.

This setup allows flexibility for free-tier BentoML users, who can deploy only one model at a time. If you have a paid version of BentoML and can deploy two models, you can pass the BentoML endpoint and Bento API token to use the deployed embedding model.

Get the Embeddings

Iterate over the text chunks (splits) in batches of 25 to generate embeddings using the get_embeddings function defined above.

This prevents overloading the embedding model with too much data at once, which can be particularly useful for managing memory and computational resources.

Create a DataFrame

Connect to MyScaleDB

Create a Table and Insert Data

Create a Vector Index

The next step is to add a vector index to the embeddings column in the RAG table. The vector index allows for efficient similarity searches, which are essential for retrieval-augmented generation tasks.

Retrieve Relevant Vectors

Note: The distance method takes an embedding column and the embedding vector of the user query to find similar documents by applying cosine similarity.

Connect to BentoML LLM

Perform RAG

Define a function to perform RAG. The function takes a user question and the retrieved context as input. It constructs a prompt for the LLM, instructing it to answer the question based on the provided context. The response from the LLM is then returned as the answer.

Make a Query

If you ask the RAG model about Albert Einstein’s death, the response should look like this:

Conclusion

BentoML stands out as an excellent platform for deploying machine learning models, including LLMs, without the hassle of managing resources. With BentoML, you can quickly deploy and scale your AI applications on the cloud, ensuring they are production-ready and highly accessible. Its simplicity and flexibility make it an ideal choice for developers, enabling them to focus more on innovation and less on deployment complexities.

On the other hand, MyScaleDB is explicitly developed for RAG applications, offering a high-performance SQL vector database. Its familiar SQL syntax makes it easy for developers to integrate and use MyScaleDB in their applications, as the learning curve is minimal. MyScaleDB’s Multi-Scale Tree Graph (MSTG) algorithm significantly outperforms other vector databases in terms of speed and accuracy. Additionally, MyScaleDB offers each new user free storage for up to 5 million vectors, making it a desirable option for developers looking to implement efficient and scalable AI solutions.

What do you think about this project? Share your thoughts on Twitter and Discord.