{"id":16838,"date":"2025-02-05T10:53:31","date_gmt":"2025-02-05T18:53:31","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=16838"},"modified":"2025-06-13T16:36:20","modified_gmt":"2025-06-13T23:36:20","slug":"groq-llm-integration-couchbase-vector-search","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/groq-llm-integration-couchbase-vector-search\/","title":{"rendered":"Integrate Groq’s Fast LLM Inferencing With Couchbase Vector Search"},"content":{"rendered":"

With so many LLMs coming out, a lot of companies are focusing on enhancing the inference speeds for large language models with specialized hardware and optimizations to be able to scale the inference capabilities of these models. One such company making huge strides in this space is Groq<\/a>.<\/span><\/p>\n

In this blog post we will explore Groq and how you integrate Groq’s fast LLM inferencing capabilities with Couchbase Vector Search to create fast and efficient RAG applications. We will also compare the performance of different LLM solutions like OpenAI, Gemini and how they compare with Groq’s inference speeds.<\/span><\/p>\n

What is Groq?<\/span><\/h2>\n

Groq, Inc. is an American technology company specializing in artificial intelligence particularly known for its development of the Language Processing Unit (LPU), an application-specific integrated circuit (ASIC) designed to accelerate AI inference tasks. It is specifically designed to enhance Large Language Models (LLMs)<\/a> with ultra-low latency inference capabilities. Groq Cloud APIs enable developers to integrate state-of-the-art LLMs like Llama3 and Mixtral 8x7B into their applications.<\/span><\/p>\n

What does this mean for developers? It means that Groq APIs can be seamlessly integrated into applications that demand real-time AI processing with quick inference needs.<\/span><\/p>\n

How to Get Started with Groq APIs<\/span><\/h2>\n

To tap into the power of Groq APIs, the first step is to generate an API key. This is a straightforward process that begins with signing up on the Groq Cloud console.<\/span><\/p>\n

\"\"<\/a>
Once you’re signed up, navigate to the API Keys<\/strong>\u00a0section. Here, you’ll have the option to create a new API key<\/a>.<\/span><\/p>\n

\"\"<\/a><\/p>\n

The API Key will allow you to integrate state-of-the-art large language models like Llama3<\/a> and Mixtral<\/a> into your applications. Next, we will be integrating the Groq chat model with LangChain<\/a> in our application.<\/span><\/p>\n

Using Groq as the LLM<\/span><\/h2>\n

You can leverage the Groq API as one of the LLM providers in LangChain:<\/span><\/p>\n

from langchain_groq import ChatGroq\r\n\r\nllm = ChatGroq(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0.3,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model_name=\"mixtral-8x7b-32768\",\r\n)\r\n<\/pre>\n
When you instantiate the ChatGroq<\/em> object you can pass the temperature and the model name. You can take a look at the currently supported models in Groq<\/a>.<\/span><\/p>\n
Building RAG application with Couchbase and Groq<\/span><\/h2>\n
The goal is to create a chat application that allows users to upload PDFs and chat with them. We’ll be using the Couchbase Python SDK and Streamlit to facilitate PDF uploads into Couchbase VectorStore. Additionally, we’ll explore how to use RAG for context-based question-answering from PDFs, all powered by Groq.<\/span><\/p>\n
You can follow the steps mentioned in this tutorial<\/a> on how to set up a Streamlit RAG application powered by Couchbase Vector Search. In this tutorial we leverage Gemini as the LLM. We will replace the implementation for Gemini with Groq.<\/span><\/p>\n
Comparing Groq\u2019s performance<\/span><\/h2>\n
In this blog we also compare the performance of different LLM providers. For this we have built a drop down for the user to be able to select what LLM provider they wish to use for the RAG application. In this example we are using Gemini, OpenAI, Ollama and Groq as the different LLM providers. There is a large list of LLM providers supported by LangChain<\/a>.\u00a0<\/span><\/p>\n
 \u00a0\u00a0\u00a0st.sidebar.subheader(\"Select LLM\")\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm_choice = st.sidebar.selectbox(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"Select LLM\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0[\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"OpenAI\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"Groq\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"Gemini\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"Ollama\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0]\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\r\n\u00a0\u00a0\u00a0\u00a0if llm_choice == \"Gemini\":\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0check_environment_variable(\"GOOGLE_API_KEY\")\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm = GoogleGenerativeAI(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0.3,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model=\"models\/gemini-1.5-pro\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm_without_rag = GoogleGenerativeAI(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model=\"models\/gemini-1.5-pro\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0elif llm_choice == \"Groq\":\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0check_environment_variable(\"GROQ_API_KEY\")\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm = ChatGroq(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0.3,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model_name=\"mixtral-8x7b-32768\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm_without_rag = ChatGroq(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model_name=\"mixtral-8x7b-32768\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0elif llm_choice == \"OpenAI\":\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0check_environment_variable(\"OPENAI_API_KEY\")\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm = ChatOpenAI(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0.3,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model=\"gpt-3.5-turbo\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm_without_rag = ChatOpenAI(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model=\"gpt-3.5-turbo\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0elif llm_choice == \"Ollama\":\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm = Ollama(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0.3,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model = ollama_model,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0base_url = ollama_url\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0llm_without_rag = Ollama(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0temperature=0,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0model = ollama_model,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0base_url = ollama_url\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)<\/pre>\n
In order to highlight Groq\u2019s quick inference speed, we built a way to calculate the inference time for the LLM Response. This measures and records the time taken for each response generation. The results are displayed in a sidebar table, showing the model used and the time taken for each query comparing different LLM providers such as OpenAI, Ollama, Gemini and Groq; through these comparisons, it was found that Groq’s LLM consistently delivered the quickest inference times. This performance benchmark allows users to see the efficiency of various models in real-time.<\/span><\/p>\n

\"\"<\/a><\/p>\n
 if question := st.chat_input(\"Ask a question based on the PDF\"):\r\n\u00a0\u00a0\u00a0\u00a0# Start timing\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0start_time = time.time()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Display user message in chat message container\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.chat_message(\"user\").markdown(question)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Add user message to chat history\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.session_state.messages.append(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\"role\": \"user\", \"content\": question, \"avatar\": \"\ud83d\udc64\"}\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Add placeholder for streaming the response\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0with st.chat_message(\"assistant\", avatar=couchbase_logo):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder = st.empty()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# stream the response from the RAG\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0rag_response = \"\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for chunk in chain.stream(question):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0rag_response += chunk\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder.markdown(rag_response + \"\u258c\")\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder.markdown(rag_response)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.session_state.messages.append(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"role\": \"assistant\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"content\": rag_response,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"avatar\": couchbase_logo,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# stream the response from the pure LLM\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0with st.chat_message(\"ai\", avatar=\"\ud83e\udd16\"):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder_pure_llm = st.empty()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0pure_llm_response = \"\"\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0for chunk in chain_without_rag.stream(question):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0pure_llm_response += chunk\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder_pure_llm.markdown(pure_llm_response + \"\u258c\")\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0message_placeholder_pure_llm.markdown(pure_llm_response)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.session_state.messages.append(\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0{\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"role\": \"assistant\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"content\": pure_llm_response,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"avatar\": \"\ud83e\udd16\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# End timing and calculate duration\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0end_time = time.time()\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0duration = end_time - start_time\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# Display the time taken\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.write(f\"Time taken to generate complete output: {duration:.2f} seconds\")\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.session_state.query_results.append({\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"model\": llm_choice,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"time\": duration\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0})\r\n\r\n\u00a0\u00a0\u00a0 st.sidebar.subheader(\"Query Results\")\r\n\r\n\u00a0\u00a0\u00a0\u00a0table_header = \"| Model | Time (s) |\\n| --- | --- |\\n\"\r\n\r\n\u00a0\u00a0\u00a0\u00a0# create table rows\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0table_rows = \"\"\r\n\r\n\u00a0\u00a0\u00a0\u00a0for idx, result in enumerate(st.session_state.query_results, 1):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0table_rows += f\"| {result['model']} | {result['time']:.2f} |\\n\"\r\n\r\n\u00a0\u00a0\u00a0\u00a0table = table_header + table_rows\u00a0\r\n\r\n\u00a0\u00a0\u00a0\u00a0st.sidebar.markdown(table, unsafe_allow_html=True)\r\n\r\n\u00a0\u00a0\u00a0\u00a0if st.sidebar.button(\"Clear Results\"):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.session_state.query_results = []\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0st.experimental_rerun()<\/pre>\n
As you can see from the results, Groq inference speed is the quickest in comparison with the other LLM providers.<\/span><\/p>\n
Conclusion<\/span><\/h2>\n
LangChain is a great open source framework which provides you a lot of possible options for vector stores, LLM of your choice to build AI powered applications. Groq is at the forefront of being one of the quickest LLM inference engine and it pairs well with AI powered applications that need quick and real time inference. Thus with the power of quick inference of Groq and Couchbase Vector Search you can build production ready and scalable RAG applications.<\/span><\/p>\n