couch base server 5.1 and spark connector 2.2.0 is there any way to insert bulk using spark couch base connector ?

Yep this is as expected. All documents in Couchbase need a document ID, unique to that bucket. And when you’re inserting Spark DataFrames directly into Couchbase, as here, you’ll need to make sure that each row in the DataFrame has this unique document id. By default the Spark connector will look…

I tried the mentioned method but it not work, I have RDD[String] then I converted it into data frame as below val dfSchema = Seq(“data”) val a = data.toDF(dfSchema: _*) a.write.mode(saveMode).couchbase(“temp”) // temp is bucket name but it show error “can not resolved symbol saveMode” I used s…

saveMode is a variable referencing a Spark SaveMode, e.g. import org.apache.spark.sql.SaveMode val saveMode = SaveMode.Overwrite

The saveToCouchbase method is documented here https://developer.couchbase.com/documentation/server/current/connectors/spark-2.2/spark-sql.html . Hope this helps :slight_smile:

Thanks for reply, I’m pretty close to upload in bulk but I having some issues I tried following things val saveMode = SaveMode.Overwrite dataframe.write.mode(saveMode).couchbase(Map(“bucket” → “temp”)) but it show following error com.couchbase.client.core.CouchbaseException: Could not find ID…

Couchbase spark bulk insert

Spark Connector

graham.pople April 20, 2018, 2:54pm 2

Hi there. Indeed there is, please see Persisting spark dataframe to couchbase for the recommended way to insert a large Spark DataFrame into Couchbase.

Topic		Replies	Views
Write a Spark DataFrame to Couchbase Spark Connector spark , connections , java	1	2459	October 19, 2017
How to insert big json file into couchbasedb using spark connector Spark Connector spark	2	1235	April 5, 2022
Persisting spark dataframe to couchbase Spark Connector	1	2440	April 12, 2018
Saving a DataFrame to Couchbase Spark Connector	1	3267	May 31, 2017
Leveraging pyspark to write to couchbase Spark Connector	3	1134	April 21, 2022

Couchbase spark bulk insert

Related topics