http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7161 Couchbase Server We have just recently found that under load, doing a series of writes, waiting until they&#39;re durable, and then querying a view with stale=false gives incorrect results. I should note that this is definitely tied to throughput load. Under minimal load, it works fine. Under a lot of load, we see the incorrect results. This could be a client issue, but I admit that it sounds more like a server side issue at the moment. We&#39;re going to try to reproduce it with another client before raising a server issue. It&#39;s <a href="http://www.couchbase.com/issues/browse/JCBC-142" title="Observe Tests show that something is wrong in the observe impl"><strike>JCBC-142</strike></a> at the moment. MB-7161

under higher sets per second clients dont get all view results after observing for all mutations

Bug Critical Resolved Incomplete Public Mark Nunberg Farshid Ghods Sun, 11 Nov 2012 13:22:21 -0600 Mon, 19 Nov 2012 23:25:08 -0600 Mon, 19 Nov 2012 22:09:43 -0600 2.0-beta-2 2.0 view-engine 0 6 if possible please run the stale=false query with debug=true <br/> <br/> more information about running queries with such option <br/> <br/> <a href="http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them">http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them</a> Please attach your PHP tests where we repro&#39;d this after the JCBC issue, and assign back to Farshid. I&#39;ve reproduced this issue with both php and java, so definitely a server issue. <br/> <br/> In a nutshell: <br/> <br/> PHP &#39;test&#39; which fails (really an ad-hoc script to reproduce this on something other than the Java SDK): <br/> <br/> <a href="http://www.couchbase.com/issues/secure/attachment/15765/observe-test.php">http://www.couchbase.com/issues/secure/attachment/15765/observe-test.php</a> <br/> <br/> Modified Java tests - functions a lot like the php test.. <br/> <br/> <a href="https://github.com/mnunberg/couchbase-java-client/commit/3d788ab9d3a88c1dc20717c4dd110e3a8bb5f5bc">https://github.com/mnunberg/couchbase-java-client/commit/3d788ab9d3a88c1dc20717c4dd110e3a8bb5f5bc</a> <br/> <br/> <br/> I should note something interesting here: <br/> <br/> 1) - I could not reproduce when running under a single node cluster (sometimes the keys failed to endure within the constraints, but the ones which did were always returned by the view <br/> <br/> 2) - Under a two node cluster, this happens a lot more. The most interesting note (And what gives it away as a server issue) is this: <br/> <br/> About half of the time that the test failed, it failed by the view returning *Exactly* half the amount of expected rows; <br/> <br/> For example; if I would set 500 keys (i.e. ensure that they persisted) and then request the view using stale=false, I would often get back *exactly* 250. <br/> <br/> I&#39;ve reproduced this behavior under both Java and PHP Farshid, this is a blocker. has there been any update on this correctness issue? Deep, <br/> <br/> can you please reproduce this issue using the steps given by Mark. <br/> Is someone sure/has evidence this is a view-engine issue? Like checking if things were really persisted to disk before the query? <br/> Can bet a lunch the problem is not there. <br/> Looking at the code, I think we are not doing &quot;full_set&quot; queries here. <br/> <br/> Mark, can you please try the same queries with full_set=true option. <br/> <br/> Also, can you give me an idea of how many number of items you are loading/mutating. I have tried with 100K items. Adding server logs here would be helpful. <br/> <br/> Mark, please run collect_info on your server node e.g. <br/> <br/> cd /opt/couchbase/bin/ <br/> ./cbcollect_info out_file.zip <br/> <br/> and attach the out_file.zip here. @deep: Have you run this?? the view is not a development view, thus full_set is not needed. Even if it were, the results should be deterministic. I would think the first step would be for you to try running it yourself. Is there something you&#39;re missing to be able to run this and gather whatever info you need? <br/> <br/> @filipe: I won&#39;t take your bet. :) I don&#39;t know that it is a view-engine issue. In fact, there are some clues that it may not be. Here&#39;s what we know thus far: <br/> * The issue is only manifested when there is a large number of items <br/> * The issue only comes up if there is more than one server in the cluster (evidence that it could be in merging) <br/> * The issue has been reproduced by two separate client libraries, verified by SDK QE (Mark) before escalating it <br/> <br/> From what I know of our architecture, it could certainly be in ep-engine or couchstore. The client library is getting back (via the observe command) that data has been persisted, but perhaps it hasn&#39;t really been persisted? The other thing that could be happening is something at the view merging layer, since it only manifests itself when we have two or more nodes? to give comments/instructions to Deep, etc. on next info to get. So, to nail it down, there&#39;s several things that need to be done: <br/> <br/> 1) Check that&#39;s there&#39;s no indexing errors due to runtime JavaScript errors (check mapreduce_errors.1 log file on each node). This is unlikely to be the issues apparently; <br/> <br/> 2) To know if stuff is not persisted to disk after observe unblocks the client, we need to know the ID of all keys, and know in which vbuckets (per node) keys will end up (running the hash function, or just analysis the db vbucket files after a few minutes). Adding a new key (document) to vbucket database file increases its seq number. So at query time, if we pass ?debug=true we&#39;ll get information about the seq numbers of all vbucket database files at the time the query arrived - this will allows us to see if these sequences are lower than what we expected. <br/> <br/> Deep, do you think you can help on this? (Someone from SDKs can also due it, appreciated) <br/> <br/> As soon as I have a list of all keys (and the result of the hash function to know in which vbucket they end up on each node) and the output of a query with ?debug=true missing some rows, I&#39;ll mention here, and add an entry to the to the wiki [1], how to analysis this issue based on all this information, so that in the future others can do this autonomously. <br/> <br/> [1] <a href="http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them">http://hub.internal.couchbase.com/confluence/display/QA/Debugging+view+engine+issues+and+reporting+them</a> <br/> See comment above Deep. Thanks. Matt, <br/> <br/> I tested yesterday with a similar reproducer I wrote in python but couldn&#39;t reproduce except for dev views(without full_set). <br/> <br/> I have taken up the java reproducer by Mark and set it up (with my limited knowledge of java) as below: <br/> <br/> &gt; git clone <a href="https://github.com/mnunberg/couchbase-java-client.git">https://github.com/mnunberg/couchbase-java-client.git</a> <br/> &gt; cd couchbase-java-client <br/> (Modified the SERVER_URI to my server in ViewTest.java) <br/> &gt; ant compile jar <br/> and Run the test <br/> &gt; java -cp .:build/ivy/lib/couchbase-client/common/*:build/jars/* org.junit.runner.JUnitCore com.couchbase.client.ViewTest <br/> <br/> The test testObserveWithStaleFalse is always passing on a 2-node(and 4-node) setup (1024 vbuckets, 1 replica). I am using build#1949. <br/> I tried with docAmount as 500, 2500 and running the same test multiple times. <br/> <br/> Please let me know if I am doing anything different/incorrect. <br/> <br/> <br/> <br/> <br/> <br/> Deep: can you share your environment with mark?<br/> <br/> Mark: can you try to repro there? Mark, <br/> <br/> as part of your testing can you also grab the query results by appending ?debug=true <br/> &gt;&gt;comments from filipe&lt;&lt;&lt; <br/> As soon as I have a list of all keys (and the result of the hash function to know in which vbucket they end up on each node) and the output of a query with ?debug=true missing some rows, <br/> <br/> also when you say you ran a process to consume lots of cpu is this on the same node as the one couchbase server is running ? Mark may need logins to the servers. Can you help there Farshid? The script has been modified to dump the entire view output (with debug=true) into a file called &#39;view_results.out&#39; in the same directory. <br/> <br/> Yes the cpuburn is running on one of the cluster nodes (which also happens to be the one running the client) <br/> <br/> I don&#39;t need SSH access to those other machines. I&#39;ve worked around this by adding the current machine (10.3.2.100) to the cluster; making it a three node cluster. <br/> <br/> The server version for all those nodes is the same (1945 or such). Thanks Mark. if you were able to replicate this issue did you happen to catprue the view_results.out ? <br/> <br/> <br/> also starting another cpu intensive process or any other process that eats up i/o on couchbase cluster which is doing indexing or other things is not recommended. in this small scale however if enough cpu is left for the server to perform the indexing should happen right after persistence. <br/> <br/> upload view results here : <a href="http://s3.amazonaws.com/bugdb/jira/MB-7161/view-results.out">http://s3.amazonaws.com/bugdb/jira/MB-7161/view-results.out</a> Sorry, but can you get the raw JSON? <br/> <br/> It&#39;s a lot easier like this, as it makes it easier to paste long arrays into a javascript interpreter or python or even erlang. <br/> <br/> From what I see, the view engine didn&#39;t miss anything, that is, it indexed all stuff persisted in vbucket database files. <br/> <br/> Here&#39;s how I see it: <br/> <br/> In debug_info object, you have an entry for each node. That entry is an object. One of its keys is &quot;wanted_seqs&quot;. This corresponds to the seq number of each vbucket database file (active vbuckets) at the time the query request was received. <br/> Then another field (again, per node), is named &quot;indexable_seqs&quot;. This field tells us the seq number of each vbucket database file (active vbuckets) after the updater ran (for stale=false queries). <br/> <br/> As we can see (by eye), for each node, indexable_seqs == wanted_seqs. This means the view engine didn&#39;t miss any persisted changes to the database file. <br/> <br/> If the output were in JSON, we could just paste these hash objects into a javascript console (or erlang), and no need to do the eyeballing, just compared directly the seq numbers via == or &gt;=. <br/> <br/> What I suggest, after talking with Farshid, is right after the observe command returns, to send the ep-engine &quot;stop/pause persistence&quot; command to all nodes. And then use couch_dbdump against all database files to find out / confirm what documents are missing. I was able to reproduce this with the php reproducer with 5 instances of burnP6 running on 1 node(making erlang and memcached starve). <br/> We have some rows missing in view output. Now I picked up one of the rows.. and checked that its indeed there in the couchdb file<br/> <br/> e.g one of the missing key observeseq_1 exists in 145.couch.1<br/> <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:/opt/couchbase/var/lib/couchbase/data/default# ../../../../../bin/couch_dbdump 145.couch.1 <br/> Doc seq: 27<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;id: observeseq_1<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rev: 27<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;content_meta: 128<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cas: 5032553164630947, expiry: 0, flags: 0<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data: (snappy) {&quot;php_value&quot;:&quot;observeseq_1&quot;}<br/> <br/> Total docs: 1<br/> <br/> For 145.couch.1 the Doc seq is 27, but in the view query debug output for vbucket 145, we have:<br/> indexable_seqs - 26<br/> wanted_seqs - 26<br/> <br/> so that means when the view query request was made Doc Seq was 26 and it later got updated to 27.<br/> <br/> If I do a stale=false query after this, I get back the missing rows. i.e. in this case now doc seq 27 is picked up from vbucket 145.<br/> <br/> So somewhere due to slowness of the system things go out of sync..observe says the key has been persisted and client code issues a view query. but view engine doesn&#39;t yet see the updated DocSeq.<br/> <br/> I modified the php reproducer to output JSON for query debug information. The modified code and output in json is attached.<br/> Thanks Deep, excellent analysis. <br/> <br/> So my gut feeling is that observe replies to the client after stuff is written to database file, but *before* ep-engine/couchkvstore notifies the erlang side that the vbucket database has a new header/snapshot. <br/> <br/> Will confirm with Chiyoung/Mike if this is the case. We do not respond until after the notification takes place so I don&#39;t think that would be the issue. We can discuss more offline. See comment/question above Chiyoung/Mike. bug-scrub mtg: <br/> <br/> - feels like a race with mccouch notification / processing. The design, though, is synchronous (?). <br/> <br/> - this scenario also artificially saturates server CPU (&quot;cpuburn&quot; ?). <br/> <br/> - need to understand what the cost / risk of a fix. <br/> Deep, this might have been a race/timing issue. <br/> <br/> You did the couchdb_dump run right after observe returned and before the query? (My guess is not, unless you have Flash powers). <br/> <br/> Discussing with Chiyoung, without code changes to debug this issue, we can&#39;t reliably find out the source of the problem. My idea is to make a change (not to commit/submit to gerrit) that disables persistence (on all nodes) right *before observe returns and after writing the stuff to disk and synchronously notifying mccouch*. Filipe, couchdb_dump was done after the stale=false view query. So that&#39;s in steady state, long after the test is finished. Deep, <br/> <br/> I ran &quot;observe-test.php&quot; with 5 processes of burnP6 on a single node many times, and got the following output: <br/> <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ php observe-test.php <br/> Have ID observe <br/> Ooops.. key observeseq_0 didn&#39;t persist <br/> Ooops.. key observeseq_2 didn&#39;t persist <br/> Ooops.. key observeseq_3 didn&#39;t persist <br/> ... <br/> Ooops.. key observeseq_297 didn&#39;t persist <br/> Ooops.. key observeseq_298 didn&#39;t persist <br/> Ooops.. key observeseq_299 didn&#39;t persist <br/> Found key observeseq_48, but we didn&#39;t manage to persist it <br/> Found key observeseq_82, but we didn&#39;t manage to persist it <br/> Found key observeseq_87, but we didn&#39;t manage to persist it <br/> Found key observeseq_94, but we didn&#39;t manage to persist it <br/> Found key observeseq_100, but we didn&#39;t manage to persist it <br/> Found key observeseq_105, but we didn&#39;t manage to persist it <br/> Found key observeseq_113, but we didn&#39;t manage to persist it <br/> Found key observeseq_140, but we didn&#39;t manage to persist it <br/> Found key observeseq_145, but we didn&#39;t manage to persist it <br/> Found key observeseq_156, but we didn&#39;t manage to persist it <br/> Found key observeseq_199, but we didn&#39;t manage to persist it <br/> Found key observeseq_225, but we didn&#39;t manage to persist it <br/> Found key observeseq_233, but we didn&#39;t manage to persist it <br/> Found key observeseq_236, but we didn&#39;t manage to persist it <br/> Found key observeseq_260, but we didn&#39;t manage to persist it <br/> Found key observeseq_273, but we didn&#39;t manage to persist it <br/> <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> Total rows: 284 <br/> <br/> <br/> I looked at the php test code and saw the following code snippet: <br/> <br/> # Ensure that they&#39;ve all persisted (sometimes not all of them will). <br/> $status = $cb-&gt;keyDurabilityMulti($casvals, array(&quot;persist_to&quot; =&gt; 2, &quot;replicate_to&quot; =&gt; 1)); <br/> <br/> I guess that &quot;keyDurabilityMulti()&quot; doesn&#39;t wait until all the keys are persisted. If this is true, I don&#39;t see anything wrong from my tests. <br/> <br/> You are right Chiyoung, keyDurabilityMulti doesn&#39;t wait until all the keys are persisted. But it returns a status array using which the test figures out which keys were persisted and which weren&#39;t. So in the view query output, the test expects only those keys which were persisted. <br/> <br/> $didnt_persist = array(); <br/> foreach ($status as $k =&gt; $v) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;# For each of those keys which did not yet persist, store them in <br/> &nbsp;&nbsp;&nbsp;&nbsp;# a lookup. These are the keys we don&#39;t expect to show up in our view <br/> &nbsp;&nbsp;&nbsp;&nbsp;if (!$v) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;printf(&quot;Ooops.. key %s didn&#39;t persist\n&quot;, $k); <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$didnt_persist[$k] = true; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> } <br/> <br/> The problematic case is the output like below: <br/> <br/> <br/> Have ID observe <br/> Hrrm.. value observeseq_1 still has key {&quot;php_value&quot;:&quot;observeseq_1&quot;} <br/> Hrrm.. value observeseq_4 still has key {&quot;php_value&quot;:&quot;observeseq_4&quot;} <br/> Hrrm.. value observeseq_8 still has key {&quot;php_value&quot;:&quot;observeseq_8&quot;} <br/> Hrrm.. value observeseq_11 still has key {&quot;php_value&quot;:&quot;observeseq_11&quot;} <br/> Hrrm.. value observeseq_14 still has key {&quot;php_value&quot;:&quot;observeseq_14&quot;} <br/> ...... <br/> Hrrm.. value observeseq_18 still has key {&quot;php_value&quot;:&quot;observeseq_18&quot;} <br/> Hrrm.. value observeseq_20 still has key {&quot;php_value&quot;:&quot;observeseq_20&quot;} <br/> Hrrm.. value observeseq_25 still has key {&quot;php_value&quot;:&quot;observeseq_25&quot;} <br/> Hrrm.. value observeseq_29 still has key {&quot;php_value&quot;:&quot;observeseq_29&quot;} <br/> Hrrm.. value observeseq_33 still has key {&quot;php_value&quot;:&quot;observeseq_33&quot;} <br/> <br/> Total keys that should have been deleted, but weren&#39;t: 94 <br/> Total rows: 206 <br/> <br/> This output is generated when the test detects a key which was reported as persisted but didn&#39;t show up in view query results(as all the keys returned by view query are deleted). <br/> <br/> With a single node, I wasn&#39;t able to repro this error. I have a 3 node cluster and on one of the nodes I am running 5 burnP6 and on the same node I run the php code. Thanks for your quick response. <br/> <br/> I set up three node cluster (10.5.2.35, 10.5.2.36, 10.5.2.37) and was running 5 burnP6 instances and the php client on 10.5.2.37. However, I didn&#39;t see any issues in 100 runs yet: <br/> <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ ps aux | grep burn <br/> root 21640 73.0 0.0 100 16 pts/0 R 23:16 9:25 burnP6 <br/> root 21641 75.5 0.0 100 16 pts/0 R 23:16 9:44 burnP6 <br/> root 21642 81.2 0.0 100 12 pts/0 R 23:16 10:29 burnP6 <br/> root 21643 74.9 0.0 100 12 pts/0 R 23:16 9:40 burnP6 <br/> root 21644 80.1 0.0 100 16 pts/0 R 23:16 10:20 burnP6 <br/> <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ ./observe-test.sh &gt; result.txt <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ <br/> <a href='mailto:chiyoung@ubu-1706'>chiyoung@ubu-1706</a>:~$ cat result.txt | grep &quot;Total keys that should have been deleted&quot; <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> ... <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> Total keys that should have been deleted, but weren&#39;t: 0 <br/> <br/> <br/> Can you give me an access to your cluster? I can take a look more tomorrow. <br/> <br/> Okay, I saw the same issue after iterating the test in 1000 times: <br/> <br/> Hrrm.. value observeseq_0 still has key {&quot;php_value&quot;:&quot;observeseq_0&quot;} <br/> Hrrm.. value observeseq_10 still has key {&quot;php_value&quot;:&quot;observeseq_10&quot;} <br/> Total keys that should have been deleted, but weren&#39;t: 2 <br/> Total rows: 292 <br/> Unfortunately, adding a delay is not a workaround for the lack of correctness. How long do we delay? We&#39;ll never know. One can&#39;t tell by the results in a real deployment since there&#39;d be other actors in the system. Deep has been working on reproducing this under higher mutations deterministically and he is going to update the ticket shortly This can also be reproduced by running the php test when data load is in progress(20k items, 500 sets/sec) on 1 node and without any CPU burning. <br/> And is much more consistently reproducible this way. <br/> e.g. <br/> On a 3 node cluster, start load on a node using mcsoda(no cpu burning): <br/> ./lib/perf_engines/mcsoda.py 10.3.121.100 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 ratio-misses=0 min-value-size=256,512 max-items=20000 exit-after-creates=1 prefix=aone <br/> <br/> Run the test while load is in progress and the same problem happens: <br/> <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:~/php-tests# ./runany.pl misc/observe-test.php <br/> php misc/observe-test.php <br/> Have ID observe <br/> Hrrm.. value observeseq_8 still has key {&quot;php_value&quot;:&quot;observeseq_8&quot;} <br/> Hrrm.. value observeseq_18 still has key {&quot;php_value&quot;:&quot;observeseq_18&quot;} <br/> Hrrm.. value observeseq_39 still has key {&quot;php_value&quot;:&quot;observeseq_39&quot;} <br/> Hrrm.. value observeseq_58 still has key {&quot;php_value&quot;:&quot;observeseq_58&quot;} <br/> Hrrm.. value observeseq_81 still has key {&quot;php_value&quot;:&quot;observeseq_81&quot;} <br/> Hrrm.. value observeseq_82 still has key {&quot;php_value&quot;:&quot;observeseq_82&quot;} <br/> Hrrm.. value observeseq_91 still has key {&quot;php_value&quot;:&quot;observeseq_91&quot;} <br/> Hrrm.. value observeseq_92 still has key {&quot;php_value&quot;:&quot;observeseq_92&quot;} <br/> Hrrm.. value observeseq_94 still has key {&quot;php_value&quot;:&quot;observeseq_94&quot;} <br/> Hrrm.. value observeseq_97 still has key {&quot;php_value&quot;:&quot;observeseq_97&quot;} <br/> .... <br/> Hrrm.. value observeseq_265 still has key {&quot;php_value&quot;:&quot;observeseq_265&quot;} <br/> Hrrm.. value observeseq_266 still has key {&quot;php_value&quot;:&quot;observeseq_266&quot;} <br/> Hrrm.. value observeseq_270 still has key {&quot;php_value&quot;:&quot;observeseq_270&quot;} <br/> Hrrm.. value observeseq_275 still has key {&quot;php_value&quot;:&quot;observeseq_275&quot;} <br/> Hrrm.. value observeseq_276 still has key {&quot;php_value&quot;:&quot;observeseq_276&quot;} <br/> Hrrm.. value observeseq_288 still has key {&quot;php_value&quot;:&quot;observeseq_288&quot;} <br/> Total keys that should have been deleted, but weren&#39;t: 46 <br/> Total rows: 1662 <br/> Deep, <br/> <br/> do you have a python test which can be run with cluster_run configuration to reproduce this issue ? looks like no intraserver-component race.<br/> <br/> Chiyoung provided explanation of test-case with observe with 2-persist / 1-replica and how client mistakenly thinks active server has persisted. bug-scrub - down to critical, as it&#39;s no longer a server internal issue. Set up the three node cluster (10.5.2.35, 10.5.2.36, 10.5.2.37) and ran 5 instances of burnP6 on 10.5.2.37 <br/> <br/> I ran &quot;observe-test.php&quot; and got the following ep-engine logs when the test was failed: <br/> <br/> 1) From 10.5.2.37 where active vbucket 764 is placed: <br/> <br/> Mon Nov 19 10:54:15.275393 PST 3: PersistenceCallback: Set the seq no 345 for key &#39;observeseq_228&#39; and vbucket 764: seq no 345 from DB <br/> <br/> 2) From 10.5.2.35 where replica vbucket 764 is placed: <br/> <br/> Mon Nov 19 10:54:14.694152 PST 3: PersistenceCallback: Set the seq no 198 for key &#39;observeseq_228&#39; and vbucket 764 <br/> Mon Nov 19 10:54:14.722671 PST 3: OBSERVE: key &#39;observeseq_228&#39; and vbucket 764 was persisted <br/> <br/> 3) From the view debug output: <br/> <br/> &quot;wanted_seqs&quot;: <br/> &quot;0764&quot; : 344 <br/> <br/> From the test, I confirmed that the php client gets the response indicating that &quot;observeseq_228&quot; was persisted in vbucket 764, but it was actually persisted in replica vbucket only on 10.5.2.35. However, the view indexer will look at the active vbucket 764 on 10.5.2.37 where the item wasn&#39;t persisted yet. <br/> <br/> It seems to me that this is a client library issue (especially libcouchbase). <br/> <br/> <br/> &nbsp; I haven&#39;t been able to determine any specific client bug as far as libcouchbase is concerned with respect to observe. <br/> <br/> There are some things that should be noted, though. <br/> <br/> The observe as implemented in libcouchbase is a simple low-level observe; that is, libcouchbase will effectively &quot;broadcast&quot; requests to the relevant nodes, and will invoke a php-ext level callback for each observe response. <br/> <br/> Then the php-ext will go ahead and count the number of times those callbacks were received with a successful response (i.e. it was persisted and exists with the user-specified CAS). <br/> <br/> This logic is implemented here: <br/> <a href="https://github.com/couchbase/php-ext-couchbase/blob/ac35571a79c87281ea8fe3f03f89b7f8316ae71c/observe.c#L489">https://github.com/couchbase/php-ext-couchbase/blob/ac35571a79c87281ea8fe3f03f89b7f8316ae71c/observe.c#L489</a> <br/> <br/> Basically sitting and waiting until durability requirements have been satisfied. <br/> <br/> Despite the PHP client lacking any kind of &#39;master-first&#39; logic, my initial tests were implemented with a bucket employing a single replica. The test script requests persistence to two nodes; considering the fact that there are only two nodes for any given vbucket, a persistence requirement of two effectively includes waiting for the item to be persisted to the master. <br/> <br/> The script may of course be modified to persist to an arbitrary number of nodes (if the count is too many, the php client will warn, and cap the number to the maximum available (persist_to &lt;= (num_replicas + 1) &lt;= 4); <br/> <br/> I&#39;ve modified observe.c in the php ext (on 10.3.121.100) to print out keys as they receive positive responses from the master, and indeed I can reproduce this issue when such responses are received. <br/> <br/> I&#39;ll actually go ahead now and dump the packets from libcouchbase to the screen as well (i.e. hex dump). Hopefully this should remove confusion about this.. <br/> <br/> Now as far as the keys not being replicated to the master go; the php ext only knows what libcouchbase tells it, and libcouchbase only knows what it receives on the memcached connection. <br/> <br/> Unless libcouchbase is doing something seriously wrong with the protocol handling, I&#39;m guessing the problem isn&#39;t there. From the logs: <br/> <br/> Ooops.. key observeseq_599 didn&#39;t persist <br/> <br/> [libcouchbase] observe_response_handler:459 Response from 10.3.3.95 <br/> MAGIC=RES OP=92 STATUS=SUCCESS KLEN=0 EXTLEN=0 NBODY=27 OPAQUE=4a9 CAS=4020000 <br/> Body: <br/> [0000] 03 40 00 0E 6F 62 73 65 72 76 65 73 65 71 5F 35 ....obse rveseq.5 <br/> [0010] 39 39 00 00 1B 84 08 DD EC B6 E7 99...... ... <br/> Key observeseq_599 persisted to master.. <br/> <br/> <br/> The key status in this message is 00 which corresponds to &quot;Found, Not Persisted&quot;, but the log message reports that the item was persisted to master. Can you look at the libcouchbase implementation to see if this is the possible error? Please see my comment. You are correct; the logging statement was placed at the wrong location (after *any* success status, rather than specifically a persisted one), and the persistence check in the php extension code checked against the wrong criteria (typo). <br/> <br/> I&#39;m not sure how the Java client failed, but I can&#39;t reproduce it with PHP anymore in light of those items which I&#39;ve fixed up. <br/> <br/> I&#39;ve filed a bug in PHP for this (<a href="http://www.couchbase.com/issues/browse/PCBC-148" title="Key persistence checks wrong criterion"><strike>PCBC-148</strike></a>, <a href="http://www.couchbase.com/issues/browse/PCBC-149" title="persist_to should imply master when greater than zero"><strike>PCBC-149</strike></a>), and there&#39;s already <a href="http://www.couchbase.com/issues/browse/JCBC-142" title="Observe Tests show that something is wrong in the observe impl"><strike>JCBC-142</strike></a> (from whence the bug originated). <br/> <br/> Additionally, means to test this (be it via the mock, or via the cluster) should be available. In reality there&#39;s usually a sub-millisecond time difference between having a key persisted and having it replicated.. <br/> <br/> So it seems that thanks to our extra debugging, it&#39;s no longer a server issue Thanks for following up on this and I&#39;m glad we have a fix. The Java client is a separate issue that I will take care it tomorrow. Please close this issue as invalid if there is no server fix needed. If you have any other questions let me know. I think we have enough evidence indicating that this issue was NOT caused by the server engine side. The separate bugs were created for client sides. Great to know we can put this issue to rest. I&#39;m glad we resolved where it was. <br/> <br/> Mike: thanks for stepping through it and reviewing everything Mark submitted. It surely helped to get everyone looking at the same thing. <br/> <br/> Mark: thanks for repro&#39;ing the Java issue and then continuing to iterate providing enough info so the issue could be definitely identified. Also, great followup by filing the related issues. Date of First Response Sun, 11 Nov 2012 16:15:10 -0600 Rank 3555 Time In Status