There are a plenty of use cases were Linear Regression (or other Regression types) can be used, but let’s focus on the first one related to house prices.<\/p>\n\n\n\n
Imagine we are running a real estate company in a particular region of the country, as we are an old company, there is some data record of which were the houses were sold in the past and for how much.<\/p>\n\n\n\n
In this case, each row in our historical data will look like this:<\/p>\n\n\n
[crayon lang=”js” decode=”true”]{
\n“id”: 7129300520,
\n“date”: “20141013T000000”,
\n“price”: 221900,
\n“bedrooms”: 3,
\n“bathrooms”: 1,
\n“sqft_living”: 1180,
\n“sqft_lot”: 5650,
\n“floors”: 1,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 1180,
\n“sqft_basement”: 0,
\n“yr_built”: 1955,
\n“yr_renovated”: 0,
\n“zipcode”: 98178,
\n“lat”: 47.5112,
\n“long”: -122.257,
\n“sqft_living15”: 1340,
\n“sqft_lot15”: 5650
\n}[\/crayon]<\/p>\n\n\n\n
\u00a0<\/p>\n\n\n\n
The problem – How to price a house<\/h3>\n\n\n\n
Now, imagine you just joined the company and you have to sell the following house:<\/p>\n\n\n
[crayon lang=”js” decode=”true”]{
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n\n\n\n
For how much would you sell it?<\/strong><\/p>\n\n\n\n The question above would be very challenging if you never sold a similar house in the past. Luckily, you have the right tool for the job: A Linear Regression.<\/p>\n\n\n\n Before you go further, you will need to install the following items:<\/p>\n\n\n\n With your Couchbase Server running, go to the administrative portal ( usually at https:\/\/127.0.0.1:8091) and create a new bucket called houses_prices<\/strong><\/p>\n\n\n\n Now, let’s clone our tutorial code:<\/p>\n\n\n [crayon lang=”default” decode=”true”]git clone https:\/\/github.com\/couchbaselabs\/couchbase-spark-mllib-sample.git[\/crayon]<\/p>\n\n\n\n In the root folder there is a file called house_prices_train_data.zip<\/strong>, it is our dataset which I borrowed from an old machine learning course on Coursera<\/a>. Please unzip it and then run the following command:<\/p>\n\n\n [crayon lang=”default” decode=”true”].\/cbimport json -c couchbase:\/\/127.0.0.1 -u YOUR_USER -p YOUR_PASSWORD -b houses_prices -d <PATH_TO_UNZIPED_FILE>\/house_prices_train_data -f list -g key::%id% -t 4[\/crayon]<\/p>\n\n\n\n TIP<\/strong>: If you are not familiar with cbimport\u00a0<\/strong>please check this tutorial<\/a><\/p>\n\n\n\n If your command ran successfully, you should notice that your houses_prices<\/strong> bucket has been populated:<\/p>\n\n\n\n Let’s also quickly add a primary index for it:<\/p>\n\n\n [crayon lang=”default” decode=”true”]CREATE PRIMARY INDEX ON `houses_prices`[\/crayon]<\/p>\n\n\n\n Our environment is ready, it is time to code!<\/p>\n\n\n\n In the LinearRegressionExample<\/a> class we start by creating the Spark context with our bucket credentials:<\/p>\n\n\n [crayon lang=”scala” decode=”true”]val spark = SparkSession and then we load all the data from the database:<\/p>\n\n\n [crayon lang=”scala” decode=”true”]val houses = spark.read.couchbase()[\/crayon]<\/p>\n\n\n\n As Spark uses a lazy approach, the data is not loaded until it is really needed. You can clearly see the beauty of the Couchbase Connector<\/strong> above, we just converted a JSON Document into a Spark Dataframe with zero effort.<\/p>\n\n\n\n In other databases for example, you would be required to export the data to a CSV file with some specific formats, copy it to your machine, load and do some extra procedures to convert it to a dataframe (not to mention the cases where the file generated is too big).<\/p>\n\n\n\n In a real world you would need to do some filtering instead of just grabbing all data, again our connector is there for you, as you can even run some N1QL queries with it:<\/p>\n\n\n [crayon lang=”scala” decode=”true”]\/\/loading documents by its type \/\/loading data using N1QL val schema = StructType( val rdd = spark.sparkContext.couchbaseQuery(query).map( TIP<\/strong>: There are a lot of examples on how to use Couchbase connector here<\/a>.<\/p>\n\n\n\n Our dataframe still looks exactly like what we had in our database:<\/p>\n\n\n [crayon lang=”scala” decode=”true”]houses.show(10)[\/crayon]<\/p>\n\n\n\n There are two different types of data here, “scalar numbers<\/em>” such as bathrooms<\/strong>\u00a0and sqft_living<\/strong>\u00a0and “categorical variables<\/em>” such as zipcode<\/strong>\u00a0and yr_renovated<\/strong>. Those categorical variables are not just simple numbers, they have a much deeper meaning as they describe a property, in the zipcode case, for example, it represents the location of the house.<\/p>\n\n\n\n Linear Regression does not like that kind of categorical variables, so if we really want to use zipcode in our Linear Regression, as it seems to be a relevant field to predict the price of a house, we have to convert it to a\u00a0dummy variable<\/strong>, which is fairly simple process:<\/p>\n\n\n\n Ex:<\/strong><\/p>\n\n\n\n The table above will be transformed to:<\/p>\n\n\n\n That is what we are doing on the line below:<\/p>\n\n\n [crayon lang=”scala” decode=”true”]val df = transformCategoricalFeatures(houses)[\/crayon]<\/p>\n\n\n\n Converting categorical variables is a very standard procedure and Spark already has some utilities to do this work for you:<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ NOTE:<\/strong> The final dataframe will not look exactly like the example shown above as it is already optimized to avoid\u00a0 The Sparse Matrix<\/a> problem.<\/p>\n\n\n\n Now, we can select the fields we would like to use and group them in a vector called features<\/strong>, as this linear regression implementation expects a field called\u00a0label<\/strong>, we also have to rename the\u00a0price<\/strong> column :<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ You can play around with those features removing\/adding them as you wish, later you can try for example remove the “sqft_living<\/em>” feature to see how the algorithm has a much worse performance.<\/p>\n\n\n\n Finally, we will only use houses in which the price is not null to train our machine learning algorithm, as our whole goal is to make our Linear Regression “learn” how to predict the price by a giving set of features.<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ Here is where the magic happens, first we split our data into training (80%<\/em>) and test (20%<\/em>), but for the purpose of this article let’s ignore the test data, then we create our LinearRegression instance and fit<\/strong> our data into it.<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ The lrModel<\/strong>\u00a0variable is already a trained model capable of predicting house prices!<\/em><\/p>\n\n\n\n Before we start predicting things, let’s just check some metrics of our trained model:<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ The one you should care here is called RMSE – Root Mean Squared Error<\/a> which roughly is the\u00a0average deviation of what our model predicts X the actual price sold<\/strong>.<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ On average we miss the actual price by $147556.0841305963<\/em>, which is not bad at all considering we barely did any feature engineering<\/a> or removed any outliers (some houses might have inexplicable high or low prices and it might mess up with your Linear Regression)<\/p>\n\n\n\n There is only one house with a missing price in this dataset, exactly the one that we pointed in the beginning:<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ And now we can finally predict the expected house price:<\/p>\n\n\n\n [crayon lang=”js” decode=”true”]{ \u00a0<\/p>\n\n\n\n Awesome, isn’t it?<\/p>\n\n\n\n For production purpose, you would still need to do a model selection<\/a> first, check other metrics of your regression and save the model instead of training it on the fly, but it’s amazing how much can be done with less than 100 lines of code!<\/p>\n\n\n\n If you have any questions, feel free to ask me on twitter at @deniswsrosa<\/a>\u00a0 or on our forums<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" The past few years we noticed how machine learning had been proven to be a technology in which companies should invest massively, you can easily find dozens of papers talking about how company X saved tons of money by adding some level of AI into their process. Surprisingly I still notice many industries being skeptical […]<\/p>\n","protected":false},"author":8754,"featured_media":18,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[287],"class_list":["post-1191","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"yoast_head":"\n<\/h3>\n\n\n\n
The Answer – Predicting house prices with Linear Regression<\/h3>\n\n\n\n
\n
\u00a0Loading the Dataset<\/h4>\n\n\n\n
![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/bucket_creation-300x258-1.png\")
<\/p>\n\n\n\n![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/filled_bucket-300x135-1.png\")
<\/p>\n\n\n\n![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/index_creation-300x172-1.png\")
<\/p>\n\n\n\n<\/h4>\n\n\n\n
Time to Code!<\/h4>\n\n\n\n
\n .builder()
\n .appName(“SparkSQLExample”)
\n .master(“local[*]”) \/\/ use the JVM as the master, great for testing
\n .config(“spark.couchbase.nodes”, “127.0.0.1”) \/\/ connect to couchbase on localhost
\n .config(“spark.couchbase.bucket.houses_prices”, “”) \/\/ open the houses_prices bucket with empty password
\n .config(“com.couchbase.username”, “YOUR_USER”)
\n .config(“com.couchbase.password”, “YOUR_PASSWORD”)
\n .getOrCreate()[\/crayon]<\/p>\n\n\n\n
\nval airlines = spark.read.couchbase(EqualTo(“type”, “airline”))<\/p>\n
\n\/\/ This query groups airports by country and counts them.
\nval query = N1qlQuery.simple(“” +
\n “select country, count(*) as count ” +
\n “from `travel-sample` ” +
\n “where type = ‘airport’ ” +
\n “group by country ” +
\n “order by count desc”)<\/p>\n
\n StructField(“count”, IntegerType) ::
\n StructField(“country”, StringType) :: Nil
\n)<\/p>\n
\n r => Row(r.value.getInt(“count”), r.value.getString(“country”)))
\nspark.createDataFrame(rdd, schema).show()[\/crayon]<\/p>\n\n\n\n![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/dataframe_data-300x59-1.png\")
<\/p>\n\n\n\n\n
![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/data_before_transformation-300x179-1.png\"){kind=spacer}
<\/p>\n\n\n\n![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/data_after_transformation-300x74-1.png\"){kind=spacer}
<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n0<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n1<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n2<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n3<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n4<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n5<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n6<\/p>\n\n\n\n
\n“id”: 1000001,
\n“date”: “20150422T000000”,
\n“bedrooms”: 6,
\n“bathrooms”: 3,
\n“price”: null,
\n“sqft_living”: 2400,
\n“sqft_lot”: 9373,
\n“floors”: 2,
\n“waterfront”: 0,
\n“view”: 0,
\n“condition”: 3,
\n“grade”: 7,
\n“sqft_above”: 2400,
\n“sqft_basement”: 0,
\n“yr_built”: 1991,
\n“yr_renovated”: 0,
\n“zipcode”: 98002,
\n“lat”: 47.3262,
\n“long”: -122.214,
\n“sqft_living15”: 2060,
\n“sqft_lot15”: 7316
\n}
\n[\/crayon]<\/p>\n7<\/p>\n\n\n\n![\"\"](\"https:\/\/www.couchbase.com\/wp-content\/uploads\/sites\/5\/2026\/05\/predicted_price.png\")
<\/p>\n\n\n\n