network details : I am running couchbase server and java program in same machine so there will not be too much latency issues.
amount of RAM on the server : 10 GB
Currently it is taking more than 1 minute to insert 1 million records with 1 node.
I added one more node and did rebalancing, However this slowed up the process further.
Can i improve the performance if i add one more node?.
Any further performance tuning suggestions available?.
You say “Size of record : one million” but “Size of one million record : 1.4 GB”. Those two numbers don’t match. If a record is one million bytes, then one million records would be 1TB.
I am reading from an excel file containing 1 million records. Coverting this data to JSON. After inserting one million records the size of bucket is 1.4 GB.
How much memory is allocated to the bucket?
Does the computer in question have spinning disks or SSDs?
Are you reading the documents from disk before writing them to Couchbase? It’s possible that the server’s I/O system is being swamped by the Java program reading and Couchbase writing.
Total time taken by java program : 71 seconds
Total time taken by java program without couchbase lines : 5 seconds.
(Have commented out couchbase lines and then run the java code).
Total insertion time : 71 - 5 = 66 seconds.
Did you try running the Java program on a separate machine from the Couchbase server?
Can you show the portion of the Java code that writes the records?
Total RAM 10 GB; 9.76 GB allocated for bucket - that’s like 2.4% of total RAM left for everything else including OS and applications. Depending on what else running at the same time, that is potentially problematic, performance-wise - yes? Best practice, we should leave like at least 20% of RAM for OS, I think.