{"id":16210,"date":"2024-08-26T18:40:37","date_gmt":"2024-08-27T01:40:37","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=16210"},"modified":"2024-09-03T11:08:00","modified_gmt":"2024-09-03T18:08:00","slug":"guide-to-data-prep-for-rag","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/guide-to-data-prep-for-rag\/","title":{"rendered":"A Step-by-Step Guide to Preparing Data for Retrieval-Augmented Generation (RAG)"},"content":{"rendered":"

In today’s data-driven world, the ability to efficiently gather and prepare data is crucial for the success of any application. Whether you’re developing a chatbot, a recommendation system, or any AI-driven solution, the quality and structure of your data can make or break your project. In this article, we’ll take you on a journey to explore the process of information gathering and smart chunking, focusing on how to prepare data for Retrieval-Augmented Generation (RAG)<\/a> in any application with your database of choice.<\/span><\/p>\n

\"\"<\/a>

High level overview of converting docs for RAG<\/p><\/div>\n

 <\/p>\n

Data Collection: The Foundation of RAG<\/h2>\n

The Magic of Scrapy<\/h3>\n

Imagine a spider, not the creepy kind, but a diligent librarian spider in the massive library of the internet. This spider, embodied by Scrapy’s Spider<\/em> class, starts at the entrance (the starting URL) and methodically visits every room (webpage), collecting precious books (HTML pages). Whenever it finds a door to another room (a hyperlink), it opens it and continues its exploration, ensuring no room is left unchecked. This is how Scrapy works\u2014systematically and meticulously gathering every piece of information.<\/span><\/p>\n

Leveraging Scrapy for Data Collection<\/h3>\n

Scrapy<\/span><\/a> is a Python-based framework designed to extract data from websites. It’s like giving our librarian spider superpowers. With Scrapy, we can build web spiders that navigate through web pages and extract desired information with precision. In our case, we deploy Scrapy to crawl the Couchbase documentation website and download HTML pages for further processing and analysis.<\/span><\/p>\n

Setting Up Your Scrapy Project<\/h3>\n

Before our spider can start its journey, we need to set up a Scrapy project. Here\u2019s how you do it:<\/span><\/p>\n

    \n
  1. Install Scrapy<\/b>: If you haven’t already installed Scrapy, you can do so using pip:<\/span><\/span><\/span>\n
    pip install scrapy\r\n<\/pre>\n
    \u00a0<\/span><\/span><\/li>\n
  2. Create a New Scrapy Project<\/b>: Set up your new Scrapy project with the following command:<\/span><\/span><\/span>\n
    scrapy startproject couchbase_docs\r\n<\/pre>\n<\/li>\n<\/ol>\n
    Crafting the Spider<\/h3>\n
    With the Scrapy project set up, we now create the spider that will crawl through the Couchbase documentation website and download HTML pages. Here\u2019s how it looks:<\/span><\/p>\n
    from pathlib import Path\r\nimport scrapy\r\n\r\nclass CouchbaseSpider(scrapy.Spider):\r\n\u00a0 \u00a0 name = \"couchbase\"\r\n\u00a0 \u00a0 start_urls = [\"https:\/\/docs.couchbase.com\/home\/index.html\",]\r\n\r\n\u00a0 \u00a0 def parse(self, response):\r\n\u00a0 \u00a0 \u00a0 \u00a0 # Download HTML content of the current page\r\n\u00a0 \u00a0 \u00a0 \u00a0 page = response.url.split(\"\/\")[-1]\r\n\u00a0 \u00a0 \u00a0 \u00a0 filename = f\"{page}.html\"\r\n\u00a0 \u00a0 \u00a0 \u00a0 Path(filename).write_bytes(response.body)\r\n\u00a0 \u00a0 \u00a0 \u00a0 self.log(f\"Saved file {filename}\")\r\n\r\n\u00a0 \u00a0 \u00a0 \u00a0 # Extract links and follow them\r\n\u00a0 \u00a0 \u00a0 \u00a0 for href in response.css(\"ul a::attr(href)\").getall():\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 if href.endswith(\".html\") or \"docs.couchbase.com\" in href:\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 yield response.follow(href, self.parse)<\/pre>\n
    Running the Spider<\/h3>\n
    To run the spider and initiate the data collection process, execute the following command within the Scrapy project directory:<\/span><\/p>\n
    scrapy crawl couchbase\r\n<\/pre>\n
    This command will start the spider, which will begin crawling the specified URLs and saving the HTML content. The spider extracts links from each page and follows them recursively, ensuring comprehensive data collection.<\/span><\/p>\n
    By automating data collection with Scrapy, we ensure all relevant HTML content from the Couchbase documentation website is retrieved efficiently and systematically, laying a solid foundation for further processing and analysis.<\/span><\/p>\n
    Extracting Text Content: Transforming Raw Data<\/h2>\n
    After collecting HTML pages from the Couchbase documentation website, the next crucial step is to extract the text content. This transforms raw data into a usable format for analysis and further processing. Additionally, we may have PDF files containing valuable data, which we’ll also extract. Here, we’ll discuss how to use Python scripts to parse HTML files and PDFs, extract text data, and store it for further processing.<\/span><\/p>\n
    Extracting Text from HTML Pages<\/h3>\n
    To extract text content from HTML pages, we’ll use a Python script that parses the HTML files and retrieves text data enclosed within\u00a0<p>\u00a0tags. This approach captures the main body of text from each page, excluding any HTML markup or structural elements.<\/span><\/p>\n
    Python Function for Text Extraction<\/h4>\n
    Below is a Python function that demonstrates how to extract text content from HTML pages and store it in text files:<\/span><\/p>\n
    from bs4 import BeautifulSoup\r\n\r\ndef get_data(html_content):\r\n\u00a0 \u00a0 soup = BeautifulSoup(html_content, \"html.parser\")\r\n\r\n\u00a0 \u00a0 title = str(soup).split('<title>')[1].split('<\/title>')[0]\r\n\u00a0 \u00a0 if \" | Couchbase Docs\" in title:\r\n\u00a0 \u00a0 \u00a0 \u00a0 title = title[:(title.index(\" | Couchbase Docs\"))].replace(\" \", \"_\")\r\n\u00a0 \u00a0 else:\r\n\u00a0 \u00a0 \u00a0 \u00a0 title = title.replace(\" \", \"_\")\r\n\u00a0 \u00a0 data = \"\"\r\n\u00a0 \u00a0 lines = soup.find_all('p')\r\n\u00a0 \u00a0 for line in lines:\r\n\u00a0 \u00a0 \u00a0 \u00a0 data += \" \" + line.text\r\n\u00a0 \u00a0 return title, data<\/pre>\n
    How to Use?<\/h3>\n
    To use the\u00a0get_data()\u00a0<\/em>function, incorporate it into your Python script or application and provide the HTML content as a parameter. The function will return the extracted text content.<\/span><\/p>\n
    html_content = '<html><head><title>Sample Page<\/title><\/head><body><p>This is a sample paragraph.<\/p><\/body><\/html>'\r\ntitle, text = get_data(html_content)\r\nprint(title)\u00a0 # Output: Sample_Page\r\nprint(text) \u00a0 # Output: This is a sample paragraph.<\/pre>\n
    Extracting Text Content from PDFs<\/h3>\n
    For extracting text content from PDFs, we’ll use a Python script that reads a PDF file and retrieves its data. This process ensures that all relevant textual information is captured for analysis.<\/span><\/p>\n
    Python Function for Text Extraction<\/h4>\n
    Below is a Python function that demonstrates how to extract text content from PDFs:<\/span><\/p>\n
    from PyPDF2 import PdfReader\r\n\r\ndef extract_text_from_pdf(pdf_file):\r\n\u00a0 \u00a0 reader = PdfReader(pdf_file)\r\n\u00a0 \u00a0 text = ''\r\n\u00a0 \u00a0 for page in reader.pages:\r\n\u00a0 \u00a0 \u00a0 \u00a0 text += page.extract_text()\r\n\u00a0 \u00a0 return text<\/pre>\n
    How to Use?<\/h4>\n
    To use the\u00a0extract_text_from_pdf()\u00a0function, incorporate it into your Python script or application and provide the PDF file path as a parameter. The function will return the extracted text content.<\/span><\/p>\n
    pdf_path = 'sample.pdf'\r\ntext = extract_text_from_pdf(pdf_path)<\/pre>\n
    With the text content extracted and saved, we’ve completed the process of getting data from the Couchbase documentation.<\/span><\/p>\n
    Chunking: Making Data Manageable<\/h2>\n
    Imagine you have a lengthy novel and you want to create a summary. Instead of reading the entire book at once, you break it down into chapters, paragraphs, and sentences. This way, you can easily understand and process each part, making the task more manageable. Similarly, chunking in text processing helps in dividing large texts into smaller, meaningful units. By organizing text into manageable chunks, we can facilitate easier processing, retrieval, and analysis of information.<\/span><\/p>\n
    Semantic and Content Chunking for RAG<\/h3>\n
    For Retrieval-Augmented Generation (RAG), chunking is particularly important. We implemented both semantic and content chunking methods to optimize the data for the RAG process, which involves retrieving relevant information and generating responses based on that information.<\/span><\/p>\n
    Recursive Character Text Splitter
    \n<\/b><\/h4>\n
    \"\"<\/a><\/p>\n
    \"\"<\/a><\/p>\n
    Recursive character text splitter<\/span><\/a> chunking by Langchain involves breaking down a piece of text into smaller chunks using recursive patterns within the characters of the text. This technique utilizes separators such as\u00a0\\n\\n<\/code>\u00a0(double newline),\u00a0\\n<\/code> (newline),\u00a0(space), and\u00a0\"\"<\/code>\u00a0(empty string).<\/span><\/p>\n
    Semantic Chunking<\/h4>\n
    \"\"<\/a><\/p>\n
    \"\"<\/a><\/p>\n
    Semantic chunking is a text processing technique that focuses on grouping words or phrases based on their semantic meaning or context. This approach enhances understanding by creating meaningful chunks that capture the underlying relationships in the text. It is particularly useful for tasks that require detailed analysis of text structure and content organization.<\/span><\/p>\n
    Implementation of Semantic and Content Chunking<\/h3>\n
    For our project, we implemented both semantic and content chunking methods. Semantic chunking preserves the hierarchical structure of the text, ensuring that each chunk maintains its contextual integrity. Content chunking was applied to remove redundant chunks and optimize storage and processing efficiency.<\/span><\/p>\n
    Python Implementation<\/h4>\n
    Here\u2019s a Python implementation of semantic and content chunking:<\/span><\/p>\n
    import hashlib\r\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\r\n\r\n# Global set to store unique chunk hash values across all files\r\nglobal_unique_hashes = set()\r\n\r\ndef hash_text(text):\r\n\u00a0 \u00a0 # Generate a hash value for the text using SHA-256\r\n\u00a0 \u00a0 hash_object = hashlib.sha256(text.encode())\r\n\u00a0 \u00a0 return hash_object.hexdigest()\r\n\r\ndef chunk_text(text, title, Chunk_size=2000, Overlap=50, Length_function=len, debug_mode=0):\r\n\u00a0 \u00a0 global global_unique_hashes\r\n\r\n\u00a0 \u00a0 chunks = RecursiveCharacterTextSplitter(\r\n\u00a0 \u00a0 \u00a0 \u00a0 chunk_size=Chunk_size,\r\n\u00a0 \u00a0 \u00a0 \u00a0 chunk_overlap=Overlap,\r\n\u00a0 \u00a0 \u00a0 \u00a0 length_function=Length_function\r\n\u00a0 \u00a0 ).create_documents([text])\r\n\r\n\u00a0 \u00a0 if debug_mode:\r\n\u00a0 \u00a0 \u00a0 \u00a0 for idx, chunk in enumerate(chunks):\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 print(f\"Chunk {idx+1}: {chunk}\\n\")\r\n\u00a0 \u00a0 \u00a0 \u00a0 print('\\n')\r\n\r\n\u00a0 \u00a0 # Deduplication mechanism\r\n\u00a0 \u00a0 unique_chunks = []\r\n\u00a0 \u00a0 for chunk in chunks:\r\n\u00a0 \u00a0 \u00a0 \u00a0 chunk_hash = hash_text(chunk.page_content)\r\n\u00a0 \u00a0 \u00a0 \u00a0 if chunk_hash not in global_unique_hashes:\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 unique_chunks.append(chunk)\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 global_unique_hashes.add(chunk_hash)\r\n\r\n\u00a0 \u00a0 for sentence in unique_chunks:\r\n\u00a0 \u00a0 \u00a0 \u00a0 sentence.page_content = title + \" \" + sentence.page_content\r\n\r\n\u00a0 \u00a0 return unique_chunks<\/pre>\n
    These optimized chunks are then embedded and stored in the Couchbase cluster for efficient retrieval, ensuring seamless integration with the RAG process.<\/span><\/p>\n
    By employing both semantic and content chunking techniques, we effectively structured and optimized textual data for the RAG process and storage in the Couchbase cluster. The next step is to embed the chunks we just generated.<\/span><\/p>\n
    Embedding Chunks: Mapping the Galaxy of Data<\/h2>\n
    Imagine each chunk of text as a star in a vast galaxy. By embedding these chunks, we assign each star a precise location in this galaxy, based on its characteristics and relationships with other stars. This spatial mapping allows us to navigate the galaxy more effectively, finding connections and understanding the broader universe of information.<\/span><\/p>\n
    Embedding Text Chunks for RAG<\/h3>\n
    Embedding text chunks is a crucial step in the Retrieval-Augmented Generation (RAG) process. It involves transforming text into numerical vectors that capture the semantic meaning and context of each chunk, making it easier for machine learning models to analyze and generate responses.<\/span><\/p>\n
    Utilizing the BAAI Model BGE-M3<\/h3>\n
    To embed the chunks, we use the <\/span>BAAI model BGE-M3<\/span><\/a>. This model is capable of embedding text into a high-dimensional vector space, capturing the semantic meaning and context of each chunk.<\/span><\/p>\n
    Embedding Function<\/h3>\n
    The embedding function takes the chunks generated from the previous step and embeds each chunk into a 1024-dimensional vector space using the BAAI model BGE-M3. This process enhances the representation of each chunk, facilitating more accurate and contextually rich analysis.<\/span><\/p>\n
    Python Script for Embedding<\/h3>\n
    Here’s a Python script that demonstrates how to embed text chunks using the BAAI model BGE-M3:<\/span><\/p>\n
    import json\r\nimport numpy as np\r\nfrom json import JSONEncoder\r\nfrom baai_model import BGEM3FlagModel\u00a0 \r\n\r\nembed_model = BGEM3FlagModel('BAAI\/bge-m3', use_fp16=True)\r\n\r\nclass NumpyEncoder(JSONEncoder):\r\n\u00a0 \u00a0 def default(self, obj):\r\n\u00a0 \u00a0 \u00a0 \u00a0 if isinstance(obj, np.ndarray):\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 return obj.tolist()\r\n\u00a0 \u00a0 \u00a0 \u00a0 return JSONEncoder.default(self, obj)\r\n\r\ndef embed(chunks):\r\n\u00a0 \u00a0 embedded_chunks = []\r\n\u00a0 \u00a0 for sentence in chunks:\r\n\u00a0 \u00a0 \u00a0 \u00a0 emb = embed_model.encode(str(sentence.page_content), batch_size=12, max_length=600)['dense_vecs']\r\n\u00a0 \u00a0 \u00a0 \u00a0 embedding = np.array(emb)\r\n\u00a0 \u00a0 \u00a0 \u00a0 np.set_printoptions(suppress=True)\r\n\r\n\u00a0 \u00a0 \u00a0 \u00a0 json_dump = json.dumps(embedding, cls=NumpyEncoder)\r\n\u00a0 \u00a0 \u00a0 \u00a0 embedded_chunk = {\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \"data\": str(sentence.page_content),\r\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \"vector_data\": json.loads(json_dump)\r\n\u00a0 \u00a0 \u00a0 \u00a0 }\r\n\u00a0 \u00a0 \u00a0 \u00a0 embedded_chunks.append(embedded_chunk)\r\n\u00a0 \u00a0 return embedded_chunks<\/pre>\n
    How to Use?<\/h3>\n
    To use the\u00a0embed()\u00a0function, incorporate it into your Python script or application and provide the chunks generated from the previous steps as input. The function will return a list of embedded chunks.<\/span><\/p>\n
    chunks = [\r\n\u00a0 \u00a0 # Assume chunks is a list of text chunks generated previously\r\n\u00a0 \u00a0 {\"page_content\": \"This is the first chunk of text.\"},\r\n\u00a0 \u00a0 {\"page_content\": \"This is the second chunk of text.\"}\r\n]\r\n\r\nembedded_chunks = embed(chunks)<\/pre>\n
    These optimized chunks, now embedded in a high-dimensional vector space, are ready for storage and retrieval, ensuring efficient utilization of resources and seamless integration with the RAG process. By embedding the text chunks, we transform raw text into a format that machine learning models can efficiently process and analyze, enabling more accurate and contextually aware responses in the RAG system.<\/span><\/p>\n
    Storing Embedded Chunks: Ensuring Efficient Retrieval<\/h2>\n
    Once the text chunks have been embedded, the next step is to store these vectors in a database. These embedded chunks can be pushed into vector databases or traditional databases with vector search support, such as Couchbase, Elasticsearch, or Pinecone, to facilitate efficient retrieval for Retrieval-Augmented Generation (RAG) applications.<\/span><\/p>\n
    Vector Databases<\/h3>\n
    Vector databases are designed specifically to handle and search through high-dimensional vectors efficiently. By storing embedded chunks in a vector database, we can leverage advanced search capabilities to quickly retrieve the most relevant information based on the context and semantic meaning of queries.<\/span><\/p>\n
    Integrating with RAG Applications<\/h3>\n
    With the data prepared and stored, it is now ready to be used in RAG applications. The embedded vectors enable these applications to retrieve contextually relevant information and generate more accurate and meaningful responses, enhancing the overall user experience.<\/span><\/p>\n
    Conclusion<\/h2>\n
    By following this guide, we have successfully prepared data for Retrieval-Augmented Generation. We covered data collection using Scrapy, text content extraction from HTML and PDFs, chunking techniques, and embedding text chunks using the BAAI model BGE-M3. These steps ensure that the data is organized, optimized, and ready for use in RAG applications.<\/span><\/p>\n
    For more such technical and engaging content, check out other vector search-related blogs<\/a> on our website and stay tuned for the next part in this series.<\/span><\/p>\n
    References<\/h3>\n