I’m using CouchBase NoSQL DB but I guess this can happen with any NoSQL DB. Here’s what happens:
I’m checking if a specific key exists and I’m catching the keyNotFound error to now ADD this key into the database. See the code:
// retrieve the document for this connection_id
db.get(connection_id, function(err, result) {
if (err && err.code === 13) {
// Catched a keyNotFound -> define a new document for the voice connection
var voice_c = {
voice_count: '1',
voice_duration: call_duration,
last_contact: call_start
};
// Add this new Voice_c document for this connection_id to DB
db.add(connection_id, voice_c, function(err, result) {
if (err)
throw err; // whilst adding a new voice connection
});
When I get to the db.add step I get an error “Key exists (with a different CAS value)” even though I just checked fragments of a millisecond before if the same key exist (and it didn’t exist).
I couldn’t replicate the error at the same place in my data feed but the second time it happened even earlier, indicating it’s a random event. I’m puzzled as to how this can happen unless there’s a big bug in my code which I just don’t see.
There is no other code running and altering the documents in the NoSQL DB, it all runs locally on my MBP. The DB was flushed and was empty before I started to run my script.
I’ve checked manually in the data feed and when it happened the first time there was indeed the same connection_id about 50 records earlier. But in the second instance the error was thrown when the connection_id was showing up the first time (though it comes up some 19000 records later). Very strange, hope someone can help me how I can avoid getting this error.
Hi, not sure what you mean by executable script that your asking for. Please do note that the script works for 99.9% of the cases, I’m reading 44000 records in and it just gives this strange error on one of them (which is enough to get a runtime error and the script stops).
All the other GET deliver correct data and the script does exactly like intended (and shown in your Ruby script).
So I assume it has something to do with getting some big load onto the CouchBase server that eventually it causes this hiccup. Not easy to replicate though as it happens randomly, when I just run it now again it happened on record 26137, all the previous ones were ok (that makes run correct in 99.996174% of the cases). But unfortunately I can’t either afford to have my script stop nor miss any record.
Okay, I see. Could you also say what is connection_id here? Is this unique key? From the initial description it seems like the script is reusing some ids.
first of all thanks for trying to help me out, missed that in my last comment.
You are correct, connection_id is the unique key that is being used. It actually consists of two mobile numbers with a “-” in between them. Like “01234-56789”
The purpose of this part of the script is to save every occurring A-Party and B-Party mobile number combination as well as there number of calls and total number of seconds they spoke to each other.
Disclaimer: I’m not working for the NSA or any other Government, this is a prototype for a Telco application
Interesting if it is possible to modify script so that it will trace the key origin. Like in error branch for ADD you might add additional GET to see the contents of the document. With logging overall sequence of the keys it let us determine who actually managed to store that key (at least what the database thinks happened with that key)
I’ve changed the script and added an additional GET in the error 12 branch of the code, see below:
// Add this new Voice_c document for this connection_id to DB
db.add(connection_id, voice_c, function(err, result) {
if (err && err.code === 12) {
client.write('3' + ',' + connection_id + '@'); // @@@@@@@@@@@@@@@@@@@@@
// Catched keyExists -> unknown error causing it, just read document again and update record it
db.get(connection_id, function(err, result) {
if (err)
throw err; // errors of 2nd try of db.get for connection_id
}) // end of db.get voice_c
update_voice_c(connection_id, result.value, call_duration, call_start);
} else if (err)
throw err; // whilst adding a new voice connection
}); // end of db.add voice_c
Here is the Update voice_c function I build:
function update_voice_c(connection_id, voice_c, call_duration, call_start) {
// Document with the connection_id is found, now increase voice_count, add call_duration to voice_duration and update the last_contact
voice_c.voice_count++;
voice_c.voice_duration += call_duration;
voice_c.last_contact = call_start;
db.set(connection_id, voice_c, function(err, result) {
if (err)
throw err; // in update_voice_c function
});
// return voice_c;
}
Unfortunately that script crashes at this step:
voice_c.voice_count++;
in the update_voice_c function with this error message:
voice_c.voice_count++;
^
TypeError: Cannot read property ‘voice_count’ of undefined
That seems to indicate to me that the GET actually didn’t retrieve anything at all, despite getting an error 12 in the step before which seems to be wrong (key exists with different CAS value).
Could you inspect RAW value of the broken key after the failure?
For example it can be done with cbc tool from libcouchbase2-bin package.
$ cbc cat foobar
"foobar" Size:82 Flags:0 CAS:ec3926a879e00000
{“voice_count”:“1”,“voice_duration”:42,“last_contact”:“2014-03-26 10:19:40 +0300”}
Again, appreciate your commitment to help Sergej but I don’t know how to access that value before or after the crash. I wish someone officially would comment on this as well.
Am I posting this in the wrong area or are there no official CouchBase support in here?
It’s kind of sad, I’m trying to built a prototype and I’m not even scratching the surface of what the final application would have to handle (records in the excess of more than 1 billion per day easily on a mid range Telco) and I’m already running into what seems to me like either a bug or some serious timing issue of the database whilst reading some mere 500 records per second in (that would be 43 million per day).
Would be great if CouchBase would invest more resources into monitoring their forum and less into sending me a monthly reminder email how my project moves along and if I want to buy the Enterprise version. Guess what, in order to do that I first need a client, for a client I need a working MVP and right now I’m stuck at an easy entry point.
Hey Andreas,
Can you log the value result after your add operation, and additionally against the subsequent get operation? Additionally, it may be useful if you could provide a small example app that exhibits this issue to allow us to narrow down potential issues, whether its in your application, the Node.js driver or even Couchbase Server.
Cheers, Brett
I think I know where the problem is coming from. When I log the err directly as the next line of code after the GET then it’s being printed as “undefined”. So it seems that it’s a timing issue, as the I/O process of a previous ADD (with the same key) isn’t finished when it goes to the next line of the function to execute, it will in some cases where high I/O load is happening (I do see up to 1000 transactions per second in the max) still work on getting the key back when I do check with GET.
Maybe I illustrate this for better understanding:
ADD for key 123 is still in the process (from an earlier record)
GET for key 123 is coming back with keyNotFound
Create the initial object/document in the meantime
ADD for key 123 is finished (the one from the earlier record)
ADD for key 123 with the initial (new) object/document only to find out that
Error for the ADD as keyExists (with another CAS)
Does this make sense? When I put console.log there it’s slowing code execution down (so the I/O process has time to finish), so in those cases I couldn’t repeat the errors anymore.
Would it make sense (and actually make the code execution faster) to write the steps after the GET call (the ones where I check for error codes) as a callback of GET? How would my piece of code look like then (I have to admit that I haven’t inherited the principle of nested callbacks yet)?
Or am I talking rubbish here? Will node.js always continue with the next line of my code ONLY when it has the err and result of the GET?
Hey a4xrbj1,
Your best bet would be to create a recursive function that performs the logic you want so that in the case of encountering an keyAlreadyExists ADD after the GET, you can restart the process (and hopefully complete successfully). In the case of a highly contended object you may end up trying a couple times before success, but this logic will allow your application to handle concurrent writes to the same object successfully.
Cheers, Brett
