I know you are using an older version of Couchbase, 5.1, but perhaps it is time to consider an upgrade. As the Eventing Service can truly solve your issue at hand. Below I make a test document 40 levels deep (via perl script) and give a simple Eventing function to update an email address at level 40 (a deep nesting).
Running my Eventing function (with 12 workers) on a single 12 phys core box 2,4GHz max boost (non MDS all services are on this box) I process
I get 29,985 items per second where I change all 1M items in a my test set I just created. Obviously a real cluster should perform better. Please compare this to your prior non-Eventing 10K tests in Java (above) which crawl along at under 100 items per second.
You can run this Eventing function stand alone in version 6.5+ -or- if you upgrade you could do what @graham.pople suggests then just use Eventing as a “point tool” to do the final cleanup to avoid any manual work.
** Perl to make a single test document**
#!/usr/bin/perl
# make a deep 40 level document and put in one EMAIL pattern and one other item.
my $max=40;
sub indent {
$cnt = $_[0];
for (my $i=1; $i<=$cnt; $i++) {
printf STDOUT " ";
}
}
print STDOUT "{\n";
for ($i=1; $i<=$max; $i++) {
&indent($i);
printf STDOUT "\"a" . $i . "\": {\n";
if ($i == $max) {
&indent($i+1);
printf STDOUT "\"emailId\": \"ismail.iqbal.ap\@xxmail.com\",\n";
&indent($i+1);
printf STDOUT "\"locale\": \"en_US\"\n";
}
}
for ($i=$max; $i>=1; $i--) {
&indent($i);
printf STDOUT "}\n";
}
printf STDOUT "}\n";
Test Document
{
"a1": {
"a2": {
"a3": {
"a4": {
"a5": {
"a6": {
"a7": {
"a8": {
"a9": {
"a10": {
"a11": {
"a12": {
"a13": {
"a14": {
"a15": {
"a16": {
"a17": {
"a18": {
"a19": {
"a20": {
"a21": {
"a22": {
"a23": {
"a24": {
"a25": {
"a26": {
"a27": {
"a28": {
"a29": {
"a30": {
"a31": {
"a32": {
"a33": {
"a34": {
"a35": {
"a36": {
"a37": {
"a38": {
"a39": {
"a40": {
"emailId": "ismail.iqbal.ap@xxmail.com",
"locale": "en_US"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
}
Eventing Code
function OnUpdate(doc, meta) {
var debug = true;
if (debug) log('1 level 39',doc
.a1.a2.a3.a4.a5.a6.a7.a8.a9.a10
.a11.a12.a13.a14.a15.a16.a17.a18.a19.a20
.a21.a22.a23.a24.a25.a26.a27.a28.a29.a30
.a31.a32.a33.a34.a35.a36.a37.a38.a39);
if (debug) log('2 ALTER emailId');
// derefrence to make easier to work with
var lvl40_hdl = doc
.a1.a2.a3.a4.a5.a6.a7.a8.a9.a10
.a11.a12.a13.a14.a15.a16.a17.a18.a19.a20
.a21.a22.a23.a24.a25.a26.a27.a28.a29.a30
.a31.a32.a33.a34.a35.a36.a37.a38.a39.a40;
// read the property
var oldVal = lvl40_hdl.emailId;
// change @xxx.com to @yyy.com
var newVal = oldVal.replace("@xxmail.com", "@yymail.com");
if (oldVal === newVal) {
// nothing to doc
if (debug) log("meta.id",meta.id,"emailId unchanged",oldVal);
return;
}
// update the local doc
lvl40_hdl.emailId = newVal;
if (debug) log('3 level 39',doc
.a1.a2.a3.a4.a5.a6.a7.a8.a9.a10
.a11.a12.a13.a14.a15.a16.a17.a18.a19.a20
.a21.a22.a23.a24.a25.a26.a27.a28.a29.a30
.a31.a32.a33.a34.a35.a36.a37.a38.a39);
// write back to the source bucket 6.5+ via a binging to
// the alias src_bkt this will update the document.
src_bkt[meta.id] = doc;
}