{"id":12617,"date":"2021-12-21T08:00:27","date_gmt":"2021-12-21T16:00:27","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=12617"},"modified":"2025-06-13T17:20:50","modified_gmt":"2025-06-14T00:20:50","slug":"how-couchbase-simplifies-data-science-part-1","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/how-couchbase-simplifies-data-science-part-1\/","title":{"rendered":"How Couchbase Simplifies Data Science (Part 1)"},"content":{"rendered":"

This post was co-authored by Karen Yua<\/span>n<\/span>, a High School Intern.\u00a0<\/span><\/em><\/p>\n

Data science extracts knowledge from data and applies that knowledge to solve problems. In the next two posts, we will learn how the Couchbase Data Platform can meet various data science needs and simplify and reduce the number of tools needed during the process.<\/p>\n

Overview<\/h4>\n

Data science workflows involve several steps, as shown in Figure 1. Data scientists are forced to use different tools for different steps, complicating the process and making it less efficient.<\/p>\n

For example, data scientists perform exploratory data analysis to determine which attributes in the training data are important for their use case. To do this, data scientists usually load training data from a database into a different tool, e.g., a Jupyter notebook. But training datasets are huge and consume a lot of memory. Transferring large datasets also consumes network bandwidth and slows the process. Clearly, the best place to analyze data is the database where the data is stored.<\/strong><\/p>\n

The data scientist needs to read only the essential attributes from the database. This simplifies data analysis, reduces training session memory usages and limits the amount of network data transfer. For example, consider training data with millions of JSON documents; each has ten fields. If only 8 of these fields are needed for training, the data scientist can save ~20% of memory by ignoring the rest (assuming all fields are of the same size).<\/p>\n

In this and the following article, we will learn:<\/p>\n

    \n
  1. How to do exploratory data analysis (EDA) and visualize data science results using the Couchbase Query service. The Query and Analytics services run on the same Couchbase cluster as the training data and predictions. Using these services to analyze the training data and the predictions makes the data science process easy and performant.<\/li>\n
  2. How to efficiently read training data using the Query and Analytics APIs in the Couchbase Python SDK and seamlessly save it to a data structure suitable for machine learning (ML), e.g., a pandas dataframe.<\/li>\n
  3. How Couchbase can meet all data science process storage needs by storing not just the training data and predictions but also ML models (up to 20MB in size).<\/li>\n<\/ol>\n

    \"\"<\/p>\n

    Frame the problem<\/h4>\n

    The data science process (Figure 1) usually starts with a problem definition. In this article, we will use customer churn prediction as an example.<\/p>\n

    The sales team at an online streaming service company wants to know whether increasing the monthly subscription cost can lead to higher revenues. However, increasing the monthly cost too high could lead to higher customer churn.<\/p>\n

    What monthly cost maximizes the revenue while keeping the customer churn in check? To answer this, the data scientist needs to predict the churn scores if the monthly costs are increased by specific amounts, e.g., $1, $2, etc.<\/p>\n

    Collect and prepare relevant data<\/h4>\n

    After defining the problem, the data scientist will collect the right data needed to solve the problem. This raw data may need to be cleaned and pre-processed, e.g., to handle missing values.<\/p>\n

    For this article, we will use the online_streaming.csv<\/em> dataset available here<\/a>. This is a synthetic dataset created by Couchbase. It simulates 500,000 customer records of a hypothetical online streaming service company.<\/p>\n

    Load this dataset into your Couchbase cluster using the steps mentioned in the README here<\/a>.<\/p>\n

    Explore the data<\/h4>\n

    After collecting and preparing the needed data, the data scientist explores it to gain insight into the data. Exploratory data analysis (EDA) is an approach that often employs visualization techniques to uncover the structure of data and extract important variables.<\/p>\n

    As mentioned earlier, the best place to do this analysis is in the database where the data is stored.<\/p>\n

    Data scientists can use Couchbase Query and Analytics services for EDA. These services run on the same Couchbase cluster as the one where the training data is stored. Training data need not be moved elsewhere for analysis. This makes the process simple and efficient.<\/p>\n

    Couchbase Multi-Dimensional Scaling (MDS), described here<\/a>, allows these services to be scaled independently. This results in faster queries. The parallel data management capabilities of the Couchbase Analytics service make data analysis even more efficient.<\/p>\n

    To use the Query features described here, first create a primary index on the online_streaming bucket by running the following command in the Query UI. Select query context<\/em> from the dropdown menu and enter: CREATE PRIMARY INDEX ON<\/em> online_streaming<\/em><\/p>\n

    Instead of creating a primary index, you could also use the Index Advisor feature (ADVISE<\/a> statement in the UI or command line cbq<\/a> tool) in the Couchbase Query service to find appropriate indexes to create, per query, based on the where<\/em> clause.<\/p>\n

    One of the key steps in exploratory data analysis is to understand the attributes in the dataset.
    \nThe INFER<\/em> statement for SQL++ queries can help. It allows users to infer the structure of documents, data types of various attributes, etc. Note that INFER<\/em> is statistical in nature rather than deterministic, as explained     here<\/a>.<\/p>\n

    As seen in the query below, executing the INFER<\/em> statement on the online_streaming bucket shows that there are 500,000 documents in this bucket and the documents contain attributes such as Plan and CustomerID. Expanding on individual attributes gives additional details. E.g., MonthlyCost has three possible values – $5.99, $11.99 and $19.99.<\/p>\n

    \"\"
    \nWe will use logistic regression, a supervised learning algorithm, to train the churn predictor.
    \nThe Churn<\/em> attribute in the dataset will be used as the label (output; target) for the supervised learning. Churn<\/em> is set to either 0 or 1; 1 indicates the customer has churned. The training process tries to learn the relationship between the label (churn) and the inputs (other attributes in the dataset).<\/p>\n

    Query and Analytics Charts (in Couchbase Server 7.0.2 and later) can be used to visualize patterns or correlations between the attributes. This can help decide which attributes to include in the training process. The chart below shows that customers with higher monthly costs are more likely to churn. Clearly, monthly cost is an important attribute to include while training the churn predictor.<\/p>\n

    \"\"<\/p>\n

    Couchbase Query and Analytics services also provide many built-in functions such as mean and standard deviation that can be used for statistical analysis. Other Query functions such as date and string can help with data preprocessing.<\/p>\n

    Train the model<\/h4>\n

    After exploring the data, the data scientist proceeds to train the model. We will train the customer churn predictor using the steps shown in Figure 2. If needed, the features can also be stored in a Couchbase bucket.<\/p>\n

    \"\"<\/p>\n

    Step 1: Efficiently Read Training Data from Couchbase
    \nWe have already identified the important attributes during EDA using the Query service. We will fetch only those attributes while reading the training data. This reduces the size of the data read over the network and the amount of memory needed to store it in the training session.<\/p>\n

    We will use the Query API in Couchbase Python SDK (version 3) to efficiently<\/em> read the entire training dataset by selecting<\/em> only the attributes that are relevant to train our churn predictor.<\/p>\n

    Data read using the query API can be easily stored in a pandas dataframe, as seen in this code.<\/p>\n

    import numpy as np\r\nimport pandas as pd\r\n# Connect to Couchbase cluster using the Python SDK\r\nfrom couchbase.cluster import Cluster, ClusterOptions\r\nfrom couchbase.auth import PasswordAuthenticator\r\n# Fill-in the hostname or IP address, user_name and password for your cluster. \r\n# E.g. Cluster.connect(\"couchbase:\/\/localhost\", ClusterOptions(PasswordAuthenticator(\"Administrator\", # \"password\")))\r\ncluster = Cluster.connect(<host>,\r\nClusterOptions(PasswordAuthenticator(<user_name>, <password>)))\r\n# Connect to online_streaming bucket\r\ncb = cluster.bucket('online_streaming')\r\n# Use the Query API to get all documents from the bucket\r\n# Specify only the needed attributes in the SELECT clause \r\nquery_result = cb.query(\"SELECT c.AverageAge, c.AvgHoursWatchedPerWeek, c.Churn, c.MonthlyCost, c.NumViewersInHousehold, c.Plan FROM `online_streaming` as c\")\r\n# Easily store the read data to a Pandas data frame\r\ndata = pd.DataFrame(list(query_result))\r\ndata.head()<\/pre>\n
    \"\"
    \nRefer to     Couchbase Python SDK documentation<\/a> for information on using the SDK.<\/p>\n
    The Analytics API in the Couchbase Python SDK can also read the training data as described in the next article.<\/p>\n
    Step 2: Extract Features<\/h4>\n
    The data scientist will convert the raw data into suitable features to be passed to the training function. The type of feature engineering techniques depends on the type of data.<\/p>\n
    One of the most common feature engineering steps is to convert categorical data to numerical values using one-hot encoding. We will use this Python code to encode categorical data and create the input (X) and label (Y) dataframes.<\/p>\n
    # Get one-hot encoding for categorical features\r\ncategoricals = data.select_dtypes(include = object).columns\r\ndata = pd.get_dummies(data, columns=categoricals)\r\n# Drop the 'Churn' column since it is a label and not a feature\r\nfeature_names = list(set(list(data.columns)) - set(['Churn']))\r\nX = data[feature_names]\r\nY = data['Churn']<\/pre>\n
    As seen in the X.head()<\/em> output below, one-hot encoding has replaced the Plan column, which was set earlier to one of Basic, Standard or Premium<\/em>, with three numeric columns Plan_Standard, Plan_Basic <\/em>and Plan_Premium.<\/em><\/p>\n
    \"\"<\/p>\n
    Step 3: Train ML Model<\/h4>\n
    Next, the data scientist will proceed with training the ML model relevant to their use case.
    \nWe will use the code below to train a churn predictor using the features created in step two.<\/p>\n
    from sklearn.model_selection import train_test_split\r\n# train_test_split function splits the data into two subsets for training and testing.\r\n# The test_size parameter below ensures that the test subset is 20% of the \r\n# training data.\r\nX_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=101, shuffle=False)\r\n\r\nfrom sklearn.linear_model import LogisticRegression\r\n# We will use Logistic regression to train the churn predictor\r\nlm = LogisticRegression(max_iter=200)\r\nlm.fit(X_train, Y_train)\r\n\r\nlm.score(X_train, Y_train)\r\nlm.score(X_test, Y_test)<\/pre>\n
    Step 4: Store ML Model and its Metadata<\/h4>\n
    The data scientist needs to save the trained model to generate predictions later. This step is also important for reproducing the research. Data science is an iterative process and there could be multiple versions of the model.<\/p>\n
    We will use the code below to store our churn prediction model (version 1) and its metadata in a model_repository<\/em> bucket on Couchbase. Create this bucket on your Couchbase cluster before proceeding.<\/p>\n
    The size of our churn predictor model is less than 1KB. Models up to 20MB in size can be stored in Couchbase as JSON or binary formats as described in the article here<\/a>. The name of features expected by the trained model are stored in the model metadata.<\/p>\n
    import pickle\r\nfrom datetime import datetime\r\n\r\ndef store_model_on_couchbase(model, feature_names, model_id):\r\n# Store model in Binary format\r\nfrom couchbase_core._libcouchbase import FMT_BYTES\r\nbucket = cluster.bucket('model_repository')\r\nmodel_bytes = pickle.dumps(model)\r\nbucket.upsert(model_id, model_bytes, format=FMT_BYTES)\r\nnow = datetime.now()\r\nmodel_metadata = {'model_id': model_id,\r\n'feature_names': list(feature_names), \r\n\"creation time\": now.strftime(\"%d\/%m\/%Y %H:%M:%S\")}\r\n# Store model metadata under a separate key\r\nkey = model_id + \"_metadata\"\r\nbucket.upsert(key, model_metadata)<\/pre>\n
    store_model_on_couchbase(lm, feature_names, 'churn_predictor_model_v1')<\/pre>\n
    Verify that the model and its metadata were successfully stored on Couchbase.<\/p>\n
    \"\"
    \nConclusion<\/h4>\n
    In this article, we learned how Couchbase makes data science easy. Using customer churn prediction as an example, we saw how to perform exploratory analysis using the Query service, how to efficiently read big training datasets using the Python SDK and easily store it in a data structure suitable for ML e.g. pandas data frame. We also saw how to store ML models (up to 20MB in size) and its metadata on Couchbase.<\/p>\n
    In the next article, we will learn how to make predictions, store them on Couchbase and how to use the Query charts to analyze them.<\/p>\n
    Next Steps<\/h4>\n
    If you’re interested in learning more about machine learning and Couchbase, here are some great next steps and resources to get you started:<\/p>\n