[MB-10946] last_sent_seqno > end_seqno can be sent when items are deduped Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UPR
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Tommie McAfee Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: upr
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Created 3 items with same key (diff values) and let them persist to disk.
Then requested the first 2 mutations. However, I only get the 3rd mutation, as seen in upr stats:

 eq_uprq:mystream:stream_0_end_seqno: 2
 eq_uprq:mystream:stream_0_flags: 0
 eq_uprq:mystream:stream_0_high_seqno: 0
 eq_uprq:mystream:stream_0_items_ready: false
 eq_uprq:mystream:stream_0_last_sent_seqno: 3
 eq_uprq:mystream:stream_0_memory: 1
 eq_uprq:mystream:stream_0_opaque: 469762303
 eq_uprq:mystream:stream_0_start_seqno: 1


Not sure what to expect here. The protocol spec notes that we send up to end_seqno and maybe STREAM_END could be sent.

I'm also wondering if there're any implications to not being able roll back to some older mutation.


 Comments   
Comment by Tommie McAfee [ 23/Apr/14 ]
pyupr test: http://review.couchbase.org/#/c/36239/1




[MB-10945] Make low priority the default setting for bucket. Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Pavel Blagodov
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
Make low priority the default setting for bucket. Please refer to MB-10938 for background discussion.




[MB-10944] Support of stale=false queries Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: query-preview
Affects Version/s: cbq-DP3, cbq-DP4
Fix Version/s: None
Security Level: Public

Type: Story Priority: Critical
Reporter: Pavel Paulau Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
stale=false queries in view engine are not really consistent but critical for competitive benchmarking.




[MB-10943] Source node auto failed over during initial data load with XDCR Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Pavel Paulau Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-601

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2680 v2
Memory = 256 GB
Disk = RAID 10

Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.17.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.18.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.19.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.20.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.21.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.22.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.23.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.24.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.25.zip
https://s3.amazonaws.com/bugdb/jira/MB-10943/172.23.100.26.zip
Is this a Regression?: Yes

 Description   
Steps:

1. Two clusters, each has 5 nodes.
2. 2 buckets
3. Enable XDCR replication
4. Start loading data, moderate rate about 50K inserts/sec

172.23.100.17 has failed.




[MB-10941] VIew queries return errors on " key with invalid state " Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Ketaki Gangal Assignee: Sarath Lakshman
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-594-rel

Triage: Untriaged
Is this a Regression?: Unknown

 Description   

Output from

======================================================================
FAIL: test_employee_dataset_startkey_endkey_queries_rebalance_in (view.viewquerytests.ViewQueryTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pytests/view/viewquerytests.py", line 855, in test_employee_dataset_startkey_endkey_queries_rebalance_in
    self._query_all_views(data_set.views, gen_load)
  File "pytests/view/viewquerytests.py", line 1979, in _query_all_views
    self._check_view_intergrity(views)
  File "pytests/view/viewquerytests.py", line 1997, in _check_view_intergrity

Failing test : ./testrunner -i /tmp/centos64.ini max-dupe-result-count=10,get-cbcollect-info=True,num-tries=60,attempt-num=60,get-delays=True,GROUP=P0 -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,GROUP=P0

Test information : http://qa.sc.couchbase.com/job/centos_x64-02-01-view-query-2-P0/lastCompletedBuild/testReport/


Uploading logs.
    self.fail(self._form_report_failure(failures, views))
AssertionError:
****************** Error report *********************
Failure message is: Exception: DEBUG INFO: [{'msg': 'missing ids from memcached', 'details': ["Error expected in results for key with invalid state {'key_vb_state': 'active', 'key_cas': '1556457554741465', 'key_exptime': '0', 'key_is_dirty': '1', 'key_flags': '0', 'key_valid': 'dirty'}"]}]

Test case info:

        Test uses employee data set:
            -documents are structured as {"name": name<string>,
                                       "join_yr" : year<int>,
                                       "join_mo" : month<int>,
                                       "join_day" : day<int>,
                                       "email": email<string>,
                                       "job_title" : title<string>,
                                       "type" : type<string>,
                                       "desc" : desc<tring>}
        Steps to repro:
            1. Start load data
            3. start rebalance in
            4. Start querying
        
Views structure are:
Views : ['test_view-4c1ca79 : map_fn=function (doc) { if(doc.job_title !== undefined) emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] ); }, reduce_fn=None', 'test_view-2f2ed85 : map_fn=function (doc) { if(doc.job_title !== undefined) { var myregexp = new RegExp("^UI "); if(doc.job_title.match(myregexp)){ emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}, reduce_fn=None', 'test_view-b513a78 : map_fn=function (doc) { if(doc.job_title !== undefined) { var myregexp = new RegExp("^System "); if(doc.job_title.match(myregexp)){ emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}, reduce_fn=None', 'test_view-99552ed : map_fn=function (doc) { if(doc.job_title !== undefined) { var myregexp = new RegExp("^Senior "); if(doc.job_title.match(myregexp)){ emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}, reduce_fn=None', 'test_view-36d10e0 : map_fn=function (doc) { if(doc.job_title !== undefined) emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] ); }, reduce_fn=_count', 'test_view-1b106a9 : map_fn=function (doc, meta) { if(doc.job_title !== undefined) { var myregexp = new RegExp("^System "); if(doc.job_title.match(myregexp)) { emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}, reduce_fn=None']





[MB-10940] Rebalance with views fails with "function_clause, [{couch_set_view_group,handle_info" on qe-sanity-p0 view tests Created: 23/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Ketaki Gangal Assignee: Mike Wiederhold
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-601-rel

Triage: Untriaged
Is this a Regression?: Yes

 Description   
Failed test : ./testrunner -i /tmp/4-centos64-view-simple-3.0.ini -t view.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,num_nodes_to_add=1,skip_rebalance=true,docs-per-day=1


Test information : http://qa.sc.couchbase.com/job/centos_x64--00_01--qe-sanity-P0/267/consoleFull

Reports : http://qa.sc.couchbase.com/job/centos_x64--00_01--qe-sanity-P0/lastSuccessfulBuild/artifact/logs/testrunner-14-Apr-22_22-23-41/

Opening a new bug inplace of re-opening older bugs as advised by dev.

 Comments   
Comment by Ketaki Gangal [ 23/Apr/14 ]
Above runs with tap, seen likewise failures on the UPR tests as well.

http://qa.sc.couchbase.com/job/centOS_x64-upr-make-simple/5/consoleFull




[MB-10939] Bucket goes in Pending state after Rebalance-in operation Created: 23/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket, ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Meenakshi Goel Assignee: Aliaksey Artamonau
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-603-rel

Attachments: Text File log.txt    
Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Link:
UPR - http://qa.sc.couchbase.com/job/ubuntu_x64--65_02--view_query_extended-P1/74/consoleFull
TAP - http://qa.sc.couchbase.com/job/centos_x64--29_01--new_view_all-P1/46/consoleFull

*Observed with both TAP and UPR Replication

Steps to Reproduce:
1. Add Node to cluster
2. Add 1 default bucket
3. Delete Bucket
4. Remove the node added from cluster
5. Add Node to the cluster again
6. Try to create bucket

Expected Behaviour : Bucket should get created successfully
Observed Behaviour: Bucket and nodes goes in Pending state

Logs:
2014-04-23 03:01:49,996] - [rest_client:109] INFO - vbucket map is not ready for bucket default after waiting 60 seconds
[2014-04-23 03:02:50,196] - [rest_client:109] INFO - vbucket map is not ready for bucket default after waiting 60 seconds
[2014-04-23 03:02:50,196] - [task:199] ERROR - Unexpected error: vbucket map is not ready for bucket default

[ns_server:debug,2014-04-23T4:13:25.484,ns_1@172.23.105.230:<0.21112.0>:janitor_agent:query_vbucket_states_loop_next_step:107]Waiting for "default" on 'ns_1@172.23.105.231'
[ns_server:debug,2014-04-23T4:13:26.487,ns_1@172.23.105.230:<0.21112.0>:janitor_agent:query_vbucket_states_loop:102]Exception from query_vbucket_states of "default":'ns_1@172.23.105.231'
warming_up


[ns_server:warn,2014-04-23T3:48:16.834,ns_1@172.23.105.231:<0.10492.0>:ns_memcached:connect:1271]Unable to connect: {error,
                       {badmatch,
                           {memcached_error,auth_error,<<"Auth failure">>}}}, retrying.
[ns_server:info,2014-04-23T3:48:17.836,ns_1@172.23.105.231:ns_memcached-default<0.10491.0>:ns_memcached:handle_cast:697]Failed to establish ns_memcached connection: {error,
                                              couldnt_connect_to_memcached}
[stats:error,2014-04-23T3:48:17.837,ns_1@172.23.105.231:<0.10499.0>:stats_collector:handle_info:119]Exception in stats collector: {exit,
                               {{bad_return_value,
                                 {stop,{error,couldnt_connect_to_memcached}}},
                                {gen_server,call,
                                 ['ns_memcached-default',
                                  {stats,<<>>},
                                  180000]}},
                               [{gen_server,call,3,
                                 [{file,"gen_server.erl"},{line,188}]},
                                {ns_memcached,do_call,3,
                                 [{file,"src/ns_memcached.erl"},{line,1412}]},
                                {stats_collector,grab_all_stats,1,
                                 [{file,"src/stats_collector.erl"},{line,83}]},
                                {stats_collector,handle_info,2,
                                 [{file,"src/stats_collector.erl"},
                                  {line,111}]},
                                {gen_server,handle_msg,5,
                                 [{file,"gen_server.erl"},{line,604}]},
                                {proc_lib,init_p_do_apply,3,
                                 [{file,"proc_lib.erl"},{line,239}]}]}

[ns_server:warn,2014-04-23T3:48:17.837,ns_1@172.23.105.231:<0.10509.0>:compaction_daemon:do_chain_compactors:725]Compactor for database `default` (pid [{type,database},
                                       {important,true},
                                       {name,<<"default">>},
                                       {fa,
                                        {#Fun<compaction_daemon.19.117966711>,
 [<<"default">>,
                                          {config,
                                           {30,18446744073709551616},
                                           {30,18446744073709551616},
                                           undefined,false,false,
                                           {daemon_config,30,131072}},
                                          false,
                                          {[{type,bucket}]}]}}]) terminated unexpectedly: {{bad_return_value,
                                                                                            {stop,
                                                                                             {error,
                                                                                              couldnt_connect_to_memcached}}},
                                                                                           {gen_server,
                                                                                            call,
                                                                                            [{'ns_memcached-default',
                                                                                              'ns_1@172.23.105.231'},
                                                                                             {raw_stats,
                                                                                              <<"diskinfo">>,
                                                                                              #Fun<compaction_daemon.11.117966711>,
                                                                                              {<<"0">>,
                                                                                               <<"0">>}},
                                                                                             180000]}}
[ns_server:error,2014-04-23T3:48:17.838,ns_1@172.23.105.231:compaction_daemon<0.10340.0>:compaction_daemon:handle_info:471]Compactor <0.10508.0> exited unexpectedly: {{bad_return_value,
                                             {stop,
                                              {error,
                                               couldnt_connect_to_memcached}}},
                                            {gen_server,call,
                                             [{'ns_memcached-default',
                                               'ns_1@172.23.105.231'},
                                              {raw_stats,<<"diskinfo">>,
                                               #Fun<compaction_daemon.11.117966711>,
                                               {<<"0">>,<<"0">>}},
                                              180000]}}. Moving to the next bucket.

Uploading Logs.

 Comments   
Comment by Meenakshi Goel [ 23/Apr/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-10939/7f6c45fe/172.23.105.230-010.zip
https://s3.amazonaws.com/bugdb/jira/MB-10939/0d3b37e0/172.23.105.231-011.zip
https://s3.amazonaws.com/bugdb/jira/MB-10939/e0c9457f/172.23.105.232-012.zip
https://s3.amazonaws.com/bugdb/jira/MB-10939/208fd98a/172.23.105.245-014.zip
Comment by Mike Wiederhold [ 23/Apr/14 ]
I noticed this last night and it appears to be an ns_server regression.
Comment by Mike Wiederhold [ 23/Apr/14 ]
Promoting to test blocker since this causes make simple-test failures. It appears to be caused by one of these changes.

http://factory.couchbase.com/job/make-simple-github/46/changes#detail3
Comment by Aliaksey Artamonau [ 23/Apr/14 ]
http://review.couchbase.org/36237




[MB-10938] Provide performance guidelines in documentation on setting the bucket priority Created: 23/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
The UI already presents High priority as the default choice but this message should be reinforced in documentation as well that low priority setting should not be used as default choice.

Using low priority as the default choice for buckets adversely impacts bucket performance. For example if there are 8 low priority buckets and 1 high priority bucket, then the 8 low priority buckets have a throughput of 30-40% of the high priority bucket.

This also means that there should not be a case where the customer has only Low priority buckets since all buckets suffer from decreased performance in this case.

 Comments   
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
I think low should be the default. the case you mentioned is expected high pri buckets is supposed to be able to take more resources compared to low priority buckets. however having every bucket in low pri vs having every bucket in high priority should not have any difference. we should verify that.
Comment by Venu Uppalapati [ 23/Apr/14 ]
throughput between all low priority and all high priority buckets matches. However, UI mentions High priority bucket as the default, will create another issue to fix this. We should provide some guidelines to the end user in our documentation so that users do not choose either Low or High priority buckets with the wrong expectation. will update the title of this issue.
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
thanks agreed. we should make low the default. that was the previous default (3) so lets keep that.




[MB-10937] Commit failures after changing the symlink of data directory(earlier mounted on SSD) to new location(mounted on EBS) Created: 22/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Task Priority: Critical
Reporter: Abhishek Singh Assignee: Abhishek Singh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: [info] OS Name : Linux 3.2.0-58-virtual
[info] OS Version : Ubuntu 12.04.4 LTS

Attachments: PNG File Screen Shot 2014-04-23 at 9.33.36 am.png    

 Description   
Background:

One of the customer was facing disk space issue during rebalance because AWS instances don't have enough space on SSD drives. Request was to see if we could:

* Mount a new EBS volume on AWS inatance
* Copy over the bucket data from SSD mount to EBS volume.
* Cleanup SSD data dir location and symlink it to data dir now residing on EBS volume.

Ran a sample test on AWS, cluster info:

* 3 node cluster, 7.5G RAM, 28GB SSD, ~100GB EBS, 2 cores(Instance type m3.large)

Observation:

Getting commit failures after doing the symlink operation and disk write queue isn't draining anymore.

Steps to reproduce:

* Set up a 3 node cluster.
* After installing Couchbase run `/opt/couchbase/bin/couchbase-cli node-init --node-init-data-path=/mnt/data --node-init-index-path=/mnt/data -c 127.0.0.1 -u ops -p password` to change the data directory to SSD mount i.e. /mnt/
* Run cbworkloadgen like `/opt/couchbase/bin/cbworkloadgen --prefix=junk1 -j -u ops -p password -t 8 -v -s 100 -r .99 -i 10000000`.
* After approx 4M items were written in 'default' bucket, I stopped cbworkloadgen and moved the data from `/mnt/data` to `/opt/couchbase/var/lib/couchbase/data` & then deleted /mnt/data. (Note: During this period there was no operation this cluster to make sure all open file-handlers are getting closed and data loss is minimal).
* Create symlink `ln -s /opt/couchbase/var/lib/couchbase/data/ /mnt/data` and re-ran the cbworkloadgen job.

cbworkloadgen operation were failing now with pop on Admin UI saying "Write Commit Failure. Disk write failed for item in Bucket "default" on node 10.61.156.183."

 Comments   
Comment by Perry Krug [ 22/Apr/14 ]
What happens if you restart the memcached process?
Comment by Abhishek Singh [ 23/Apr/14 ]
After Couchbase restart, the Admin UI doesn't load
Comment by Perry Krug [ 23/Apr/14 ]
Which directory was originally directed at the SSD and which one did you symlink?
Comment by Abhishek Singh [ 23/Apr/14 ]
I did mention those in ticket details.

At first data dir was /mnt/data (running on 28G SSD)
Then I copied the data from /mnt/data over to /opt/couchbase/var/lib/couchbase/data/ (/ mounted on EBS 100G)
Deleted /mnt/data to create symlink

symlink done using `ln -s /opt/couchbase/var/lib/couchbase/data/ /mnt/data`
Comment by Perry Krug [ 23/Apr/14 ]
Sorry that I missed those steps. The output from the collect_info exiting is not directly related, but may point to something...did you run it as root?
Comment by Abhishek Singh [ 23/Apr/14 ]
Ahh you may be right about the cbcollect_info issue, I didn't run it as root. Ran it as 'ubuntu' user and 'ubuntu' user didn't have read perms to `/opt/couchbase/var/lib/couchbase/logs`. I would grab the logs once again
Comment by Chiyoung Seo [ 23/Apr/14 ]
Abhishek,

Please close this ticket if you don't see commit failures anymore.




[MB-10936] UPR Issue:: Rebalance fails for delta recovery of 2 nodes in a 6 node cluster after 1 graceful+ 1 hard failover Created: 22/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Abhinav Dangeti
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: versions 3.0.0-597, centos-64

Attachments: GZip Archive rebalance_fail_after_delta_recovery.tar.gz    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
UPR, vbuckets=64, versions 3.0.0-597, centos-64

Cluster Nodes: 10.6.2.144, 10.6.2.145, 10.6.2.147, 10.6.2.148, 10.6.2.149, 10.6.2.150

1. 6 Nodes added to cluster
2. Add 1 default bucket (replica=1)
3. Add 10-20 K items and mutate
4. Using UI graceful failover 1 node 10.6.2.149
5. Using UI failover another node (which says hard) 10.6.2.150
6. Add-back node and make recovery type as delta
7. rebalance the nodes

Expectation: Rebalance will succeed

Observation: Rebalance fails

Error

Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.20300.0>,
{upr_wait_for_data_move_failed,"default",43,
'ns_1@10.6.2.145',
['ns_1@10.6.2.149'],
{error,no_stats_for_this_vbucket}}}}

 Comments   
Comment by Anil Kumar [ 23/Apr/14 ]
Parag - make sure to add fixversion.
Comment by Aliaksey Artamonau [ 23/Apr/14 ]
At first glance looks similar to MB-10928.
Comment by Artem Stemkovski [ 23/Apr/14 ]
I'm seeing this in the logs:

[ns_server:debug,2014-04-22T20:15:02.413,ns_1@10.6.2.149:upr_consumer_conn-default-ns_1@10.6.2.148<0.16475.0>:upr_proxy:handle_packet:118]Proxy packet: RESPONSE: 0x51 (upr_add_stream) vbucket = 0 opaque = 0x2D status = 0x23 (rollback)
81 51 00 00
04 00 00 23
00 00 00 04
00 00 00 2D
00 00 00 00
00 00 00 00
00 00 00 DF

[ns_server:debug,2014-04-22T20:15:02.413,ns_1@10.6.2.149:upr_consumer_conn-default-ns_1@10.6.2.148<0.16475.0>:upr_proxy:handle_packet:118]Proxy packet: RESPONSE: 0x51 (upr_add_stream) vbucket = 0 opaque = 0x2E status = 0x23 (rollback)
81 51 00 00
04 00 00 23
00 00 00 04
00 00 00 2E
00 00 00 00
00 00 00 00
00 00 00 E0

so this is just another way to repro MB-10928
Comment by Mike Wiederhold [ 23/Apr/14 ]
Abhinav,

Since you did the rollback implementation can you take a look at this issue? Let me know if you have any questions about what is going on.
Comment by Mike Wiederhold [ 23/Apr/14 ]
Looks like a duplicate of MB-10928 since both bugs show an issue with the add stream command returning a rollback error.




[MB-10935] memcached gets out of memory Created: 22/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Artem Stemkovski Assignee: Mike Wiederhold
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
2 nodes, 1 rebalanced bucket
RAM Quota = 1000
load data with the following command:

cbc-pillowfight --host 127.0.0.1:9000 -b default --num-threads 4 --min-size 1024 --max-size 1024 --ratio 50 -Q 4 -I 10000 --loop

memory for both nodes grow constantly
on one of the nodes the memory gets up to quota and stabilizes there
for the other one it grows bigger than quota allows and the persistence and UPR replication gets broken

[ns_server:debug,2014-04-22T18:50:35.559,n_1@127.0.0.1:upr_consumer_conn-default-n_0@10.17.32.209<0.13193.0>:upr_proxy:handle_packet:118]Proxy packet: RESPONSE: 0x57 (upr_mutation) vbucket = 0 opaque = 0xEF010000 status = 0x82 (enomem)
81 57 00 00
00 00 00 82
00 00 00 0D
EF 01 00 00
00 00 00 00
00 00 00 00
4F 75 74 20
6F 66 20 6D
65 6D 6F 72
79


 Comments   
Comment by Pavel Paulau [ 22/Apr/14 ]
Dup MB-10930?
Comment by Chiyoung Seo [ 23/Apr/14 ]
Yes, duplicate of MB-10930




[MB-10934] UPR Issue:: Rebalance Fails after Failover of a node blocked due to firewall restrictions Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Artem Stemkovski
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GZip Archive failover_firewall_failure_to_rebalance.tar.gz    
Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Yes

 Description   
UPR, 3.0.0-597, 64 vbuckets, 7 node cluster (10.6.2.144, 10.6.2.145, 10.6.2.146, 10.6.2.147, 10.6.2.148, 10.6.2.149, 10.6.2.150)

Cluster nodes::
1. Add 7 nodes to cluster
2. Create default bucket
3. Add 1000 items
4. Make a node unreable due to firewall restriction
5. Failover the node
6. Rebalance the node

Expectation:: The failover succeeds with rebalance

Observation: Failover happens but rebalance fails

Rebalance exited with reason {{{{{child_interrupted,
{'EXIT',<13287.1469.0>,normal}},
[{upr_replicator,spawn_and_wait,1},
{upr_replicator,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
['upr_replicator-default-ns_1@10.6.2.146',
{setup_replication,",-"},
infinity]}},
{gen_server,call,
['replication_manager-default',
{change_vbucket_replication,46,undefined},
infinity]}},
{gen_server,call,
[{'janitor_agent-default','ns_1@10.6.2.147'},
{get_tap_docs_estimate_many_taps,45,[<<>>]},
infinity]}}


 Comments   
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
sending to Aliaksey in Alk's absence.




[MB-10933] "Create dev. view" dialog doesn't disappear when pressing "esc" after "Creating more than 10 views" warning Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Pavel Paulau Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: JPEG File 10views.jpeg    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Steps:
1. Create 10 views.
2. Add 1 more view.
3. Observe "Creating more than 10 views per design document may decrease performance. Please, confirm." dialog.
4. Instead of clicking "Cancel" press "esc" button.





[MB-10932] Disable force-respawn of memcached upon certain configuration changes Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Improvement Priority: Major
Reporter: Perry Krug Assignee: Cihan Biyikoglu
Resolution: Unresolved Votes: 0
Labels: supportability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Given the addition of the GIO and bucket priority in 3.0, I would like to ask that we disable ns_server's forced respawn of the memcached process when the following items are changed (and therefore allow them to be changed if they cannot currently):
-GIO thread count
-Bucket IO priority
-memcached connections
-memcached worker threads

These are very frequently tuned items by the field, support and customers and it would be extremely helpful to allow the new values to be "set" but not yet applied until either a memcached restart occurs or a new node is balanced in




[MB-10931] Lack of NRU info for read-heavy items. Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 2.5.1
Fix Version/s: techdebt-backlog
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Patrick Varley Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: customer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
In DGM with a high read rate and replicas the resident ratios between active vbuckets and replica can get way out of balance which can cause a problem when there is a failover.

To solve this we should update the LRU for replica when the document is read.

Adding on to this our LRU should take into consideration how many replicas there are and evict the 2nd and 3rd replicas over the 1st.




[MB-10930] Possible memory leak in 3.0 (same workload, regression from 2.5.1) Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Perry Krug Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Yes

 Description   
Steps to reproduce:
-Create 4-node cluster (3.0-597)
-Load beer-sample
-Run this load-generator from 2 separate clients (same command):
cbc pillowfight --host <IP> -C HTTP -b beer-sample --num-threads 4 --min-size 1024 --max-size 1024 --ratio 50 -Q 4 -I 10000 --loop
-Observe memory usage constantly increasing. On 2.5.1, it stays flat at ~30mb per node. On 3.0, it is already up to >700mb per node after an hour

Let me know which stats are most useful to grab

 Comments   
Comment by Pavel Paulau [ 22/Apr/14 ]
/opt/couchbase/bin/cbstats -b beer-sample 127.0.0.1:11210 allocator
/opt/couchbase/bin/cbstats -b beer-sample 127.0.0.1:11210 raw memory

What OS are you using?
Comment by Perry Krug [ 22/Apr/14 ]
The stats are below.

I'm on CentOS 5.8. I see mem_used as growing very high, i.e. not just fragmentation of the allocator.

[root@ip-10-198-37-111 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b beer-sample allocator
------------------------------------------------
MALLOC: 190669456 ( 181.8 MiB) Bytes in use by application
MALLOC: + 16826368 ( 16.0 MiB) Bytes in page heap freelist
MALLOC: + 13831048 ( 13.2 MiB) Bytes in central cache freelist
MALLOC: + 8872240 ( 8.5 MiB) Bytes in transfer cache freelist
MALLOC: + 13070520 ( 12.5 MiB) Bytes in thread cache freelists
MALLOC: + 1765528 ( 1.7 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 245035160 ( 233.7 MiB) Actual memory used (physical + swap)
MALLOC: + 0 ( 0.0 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 245035160 ( 233.7 MiB) Virtual address space used
MALLOC:
MALLOC: 10568 Spans in use
MALLOC: 10 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Total size of freelists for per-thread caches,
transfer cache, and central cache, by size class
------------------------------------------------
class 1 [ 8 bytes ] : 1928 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 851 objs; 0.0 MiB; 0.0 cum MiB
class 3 [ 32 bytes ] : 47433 objs; 1.4 MiB; 1.5 cum MiB
class 4 [ 48 bytes ] : 45061 objs; 2.1 MiB; 3.5 cum MiB
class 5 [ 64 bytes ] : 23587 objs; 1.4 MiB; 5.0 cum MiB
class 6 [ 80 bytes ] : 25006 objs; 1.9 MiB; 6.9 cum MiB
class 7 [ 96 bytes ] : 16257 objs; 1.5 MiB; 8.4 cum MiB
class 8 [ 112 bytes ] : 3751 objs; 0.4 MiB; 8.8 cum MiB
class 9 [ 128 bytes ] : 6362 objs; 0.8 MiB; 9.6 cum MiB
class 10 [ 144 bytes ] : 2331 objs; 0.3 MiB; 9.9 cum MiB
class 11 [ 160 bytes ] : 8818 objs; 1.3 MiB; 11.2 cum MiB
class 12 [ 176 bytes ] : 7526 objs; 1.3 MiB; 12.5 cum MiB
class 13 [ 192 bytes ] : 7392 objs; 1.4 MiB; 13.8 cum MiB
class 14 [ 208 bytes ] : 7561 objs; 1.5 MiB; 15.3 cum MiB
class 15 [ 224 bytes ] : 73 objs; 0.0 MiB; 15.3 cum MiB
class 16 [ 240 bytes ] : 266 objs; 0.1 MiB; 15.4 cum MiB
class 17 [ 256 bytes ] : 188 objs; 0.0 MiB; 15.5 cum MiB
class 18 [ 288 bytes ] : 219 objs; 0.1 MiB; 15.5 cum MiB
class 19 [ 320 bytes ] : 135 objs; 0.0 MiB; 15.6 cum MiB
class 20 [ 352 bytes ] : 223 objs; 0.1 MiB; 15.6 cum MiB
class 21 [ 384 bytes ] : 158 objs; 0.1 MiB; 15.7 cum MiB
class 22 [ 416 bytes ] : 77 objs; 0.0 MiB; 15.7 cum MiB
class 23 [ 448 bytes ] : 100 objs; 0.0 MiB; 15.8 cum MiB
class 24 [ 480 bytes ] : 72 objs; 0.0 MiB; 15.8 cum MiB
class 25 [ 512 bytes ] : 2928 objs; 1.4 MiB; 17.2 cum MiB
class 26 [ 576 bytes ] : 166 objs; 0.1 MiB; 17.3 cum MiB
class 27 [ 640 bytes ] : 94 objs; 0.1 MiB; 17.4 cum MiB
class 28 [ 704 bytes ] : 91 objs; 0.1 MiB; 17.4 cum MiB
class 29 [ 768 bytes ] : 63 objs; 0.0 MiB; 17.5 cum MiB
class 30 [ 832 bytes ] : 39 objs; 0.0 MiB; 17.5 cum MiB
class 31 [ 896 bytes ] : 36 objs; 0.0 MiB; 17.5 cum MiB
class 32 [ 960 bytes ] : 51 objs; 0.0 MiB; 17.6 cum MiB
class 33 [ 1024 bytes ] : 39 objs; 0.0 MiB; 17.6 cum MiB
class 34 [ 1152 bytes ] : 44 objs; 0.0 MiB; 17.7 cum MiB
class 35 [ 1280 bytes ] : 49 objs; 0.1 MiB; 17.7 cum MiB
class 36 [ 1408 bytes ] : 44 objs; 0.1 MiB; 17.8 cum MiB
class 37 [ 1536 bytes ] : 10 objs; 0.0 MiB; 17.8 cum MiB
class 38 [ 1792 bytes ] : 16 objs; 0.0 MiB; 17.8 cum MiB
class 39 [ 2048 bytes ] : 37 objs; 0.1 MiB; 17.9 cum MiB
class 40 [ 2304 bytes ] : 2449 objs; 5.4 MiB; 23.3 cum MiB
class 41 [ 2560 bytes ] : 17 objs; 0.0 MiB; 23.3 cum MiB
class 42 [ 2816 bytes ] : 8 objs; 0.0 MiB; 23.4 cum MiB
class 43 [ 3072 bytes ] : 5 objs; 0.0 MiB; 23.4 cum MiB
class 45 [ 4096 bytes ] : 23 objs; 0.1 MiB; 23.5 cum MiB
class 46 [ 4608 bytes ] : 18 objs; 0.1 MiB; 23.5 cum MiB
class 50 [ 8192 bytes ] : 15 objs; 0.1 MiB; 23.7 cum MiB
class 51 [ 9216 bytes ] : 47 objs; 0.4 MiB; 24.1 cum MiB
class 55 [ 16384 bytes ] : 9 objs; 0.1 MiB; 24.2 cum MiB
class 58 [ 26624 bytes ] : 46 objs; 1.2 MiB; 25.4 cum MiB
class 59 [ 32768 bytes ] : 12 objs; 0.4 MiB; 25.8 cum MiB
class 60 [ 40960 bytes ] : 1 objs; 0.0 MiB; 25.8 cum MiB
class 63 [ 65536 bytes ] : 11 objs; 0.7 MiB; 26.5 cum MiB
class 71 [ 131072 bytes ] : 9 objs; 1.1 MiB; 27.6 cum MiB
class 72 [ 139264 bytes ] : 9 objs; 1.2 MiB; 28.8 cum MiB
class 87 [ 262144 bytes ] : 6 objs; 1.5 MiB; 30.3 cum MiB
------------------------------------------------
PageHeap: 12 sizes; 16.0 MiB free; 0.0 MiB unmapped
------------------------------------------------
     1 pages * 22 spans ~ 0.2 MiB; 0.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     2 pages * 456 spans ~ 7.1 MiB; 7.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     3 pages * 10 spans ~ 0.2 MiB; 7.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     4 pages * 105 spans ~ 3.3 MiB; 10.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     5 pages * 1 spans ~ 0.0 MiB; 10.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     6 pages * 34 spans ~ 1.6 MiB; 12.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     7 pages * 1 spans ~ 0.1 MiB; 12.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     8 pages * 11 spans ~ 0.7 MiB; 13.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
     9 pages * 1 spans ~ 0.1 MiB; 13.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
    10 pages * 6 spans ~ 0.5 MiB; 13.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
    12 pages * 5 spans ~ 0.5 MiB; 14.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
    14 pages * 3 spans ~ 0.3 MiB; 14.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
>255 large * 1 spans ~ 1.5 MiB; 16.0 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum

[root@ip-10-198-37-111 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b beer-sample raw memory
 bytes: 996201320
 ep_kv_size: 17277858
 ep_max_size: 1048576000
 ep_mem_high_wat: 891289600
 ep_mem_low_wat: 786432000
 ep_mem_tracker_enabled: true
 ep_oom_errors: 0
 ep_overhead: 14844868
 ep_tmp_oom_errors: 1332869
 ep_value_size: 16572294
 mem_used: 996201320
 tcmalloc_current_thread_cache_bytes: 13101984
 tcmalloc_max_thread_cache_bytes: 33554432
 tcmalloc_unmapped_bytes: 0
 total_allocated_bytes: 190657512
 total_fragmentation_bytes: 35785752
 total_free_bytes: 16826368
 total_heap_bytes: 243269632
[root@ip-10-198-37-111 ~]# /opt/couchbase/bin/cbstats localhost:11210 -b beer-sample all
 accepting_conns: 1
 auth_cmds: 20
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 25
 bytes: 996199224
 bytes_read: 416992920448
 bytes_written: 398956774349
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 114532214
 cmd_set: 114538973
 conn_yields: 8134691
 connection_structures: 10500
 curr_connections: 24
 curr_conns_on_port_11207: 1
 curr_conns_on_port_11209: 19
 curr_conns_on_port_11210: 4
 curr_items: 4282
 curr_items_tot: 8561
 curr_temp_items: 0
 daemon_connections: 3
 decr_hits: 0
 decr_misses: 0
 delete_hits: 0
 delete_misses: 0
 ep_access_scanner_last_runtime: 0
 ep_access_scanner_num_items: 0
 ep_access_scanner_task_time: 2014-04-23 21:31:01
 ep_allow_data_loss_during_shutdown: 1
 ep_alog_block_size: 4096
 ep_alog_path: /opt/couchbase/var/lib/couchbase/data/beer-sample/access.log
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10
 ep_backend: couchdb
 ep_bg_fetch_delay: 0
 ep_bg_fetched: 0
 ep_bg_meta_fetched: 0
 ep_bg_remaining_jobs: 0
 ep_bucket_priority: HIGH
 ep_chk_max_items: 5000
 ep_chk_period: 1800
 ep_chk_persistence_remains: 0
 ep_chk_persistence_timeout: 10
 ep_chk_remover_stime: 5
 ep_commit_num: 704216
 ep_commit_time: 43
 ep_commit_time_total: 18446758699936
 ep_config_file:
 ep_conflict_resolution_type: seqno
 ep_couch_bucket: beer-sample
 ep_couch_host: 127.0.0.1
 ep_couch_port: 11213
 ep_couch_reconnect_sleeptime: 250
 ep_couch_response_timeout: 180000
 ep_data_traffic_enabled: 0
 ep_db_data_size: 2728318
 ep_db_file_size: 14829056
 ep_dbname: /opt/couchbase/var/lib/couchbase/data/beer-sample
 ep_degraded_mode: 0
 ep_diskqueue_drain: 14503042
 ep_diskqueue_fill: 15096110
 ep_diskqueue_items: 7433
 ep_diskqueue_memory: 42700896
 ep_diskqueue_pending: 1277468472
 ep_exp_pager_stime: 3600
 ep_expired_access: 0
 ep_expired_pager: 0
 ep_failpartialwarmup: 0
 ep_flush_all: false
 ep_flush_duration_total: 20962
 ep_flushall_enabled: 0
 ep_flusher_state: running
 ep_flusher_todo: 0
 ep_getl_default_timeout: 15
 ep_getl_max_timeout: 30
 ep_ht_locks: 5
 ep_ht_size: 3079
 ep_initfile:
 ep_io_num_read: 0
 ep_io_num_write: 6781076
 ep_io_read_bytes: 0
 ep_io_write_bytes: 14030785916
 ep_item_begin_failed: 0
 ep_item_commit_failed: 0
 ep_item_eviction_policy: value_only
 ep_item_flush_expired: 0
 ep_item_flush_failed: 0
 ep_item_num_based_new_chk: 1
 ep_items_rm_from_checkpoints: 9224575
 ep_keep_closed_chks: 0
 ep_kv_size: 17275796
 ep_max_bg_remaining_jobs: 0
 ep_max_checkpoints: 2
 ep_max_failover_entries: 25
 ep_max_item_size: 20971520
 ep_max_num_shards: 4
 ep_max_num_workers: 8
 ep_max_size: 1048576000
 ep_max_threads: 0
 ep_max_vbuckets: 1024
 ep_mem_high_wat: 891289600
 ep_mem_low_wat: 786432000
 ep_mem_tracker_enabled: true
 ep_meta_data_memory: 705564
 ep_mlog_compactor_runs: 0
 ep_mutation_mem_threshold: 95
 ep_num_access_scanner_runs: 0
 ep_num_eject_failures: 911792
 ep_num_expiry_pager_runs: 2
 ep_num_non_resident: 6061
 ep_num_not_my_vbuckets: 0
 ep_num_ops_del_meta: 0
 ep_num_ops_del_meta_res_fail: 0
 ep_num_ops_del_ret_meta: 0
 ep_num_ops_get_meta: 0
 ep_num_ops_get_meta_on_set_meta: 0
 ep_num_ops_set_meta: 0
 ep_num_ops_set_meta_res_fail: 0
 ep_num_ops_set_ret_meta: 0
 ep_num_pager_runs: 316
 ep_num_value_ejects: 11965
 ep_num_workers: 4
 ep_oom_errors: 0
 ep_overhead: 14844776
 ep_pager_active_vb_pcnt: 40
 ep_pending_compactions: 0
 ep_pending_ops: 0
 ep_pending_ops_max: 0
 ep_pending_ops_max_duration: 0
 ep_pending_ops_total: 0
 ep_postInitfile:
 ep_queue_size: 7433
 ep_rollback_count: 0
 ep_startup_time: 1398202260
 ep_storage_age: 1
 ep_storage_age_highwat: 10
 ep_tap_ack_grace_period: 300
 ep_tap_ack_initial_sequence_number: 1
 ep_tap_ack_interval: 1000
 ep_tap_ack_window_size: 10
 ep_tap_backfill_resident: 0.9
 ep_tap_backlog_limit: 5000
 ep_tap_backoff_period: 5
 ep_tap_bg_fetch_requeued: 0
 ep_tap_bg_fetched: 0
 ep_tap_bg_max_pending: 500
 ep_tap_keepalive: 300
 ep_tap_noop_interval: 20
 ep_tap_requeue_sleep_time: 0.1
 ep_tap_throttle_cap_pcnt: 10
 ep_tap_throttle_queue_cap: 1000000
 ep_tap_throttle_threshold: 90
 ep_tmp_oom_errors: 2017238
 ep_total_cache_size: 5830564
 ep_total_del_items: 0
 ep_total_enqueued: 15096110
 ep_total_new_items: 8561
 ep_total_persisted: 6781076
 ep_uncommitted_items: 0
 ep_uuid: be040fbae2d968d1f68fccf1c71afecf
 ep_value_size: 16570232
 ep_vb0: 0
 ep_vb_snapshot_total: 666892
 ep_vb_total: 512
 ep_vbucket_del: 256
 ep_vbucket_del_avg_walltime: 43625
 ep_vbucket_del_fail: 0
 ep_vbucket_del_max_walltime: 187768
 ep_version: 2.1.1r-603-ge81d80b
 ep_waitforwarmup: 0
 ep_warmup: 1
 ep_warmup_batch_size: 1000
 ep_warmup_dups: 0
 ep_warmup_min_items_threshold: 100
 ep_warmup_min_memory_threshold: 100
 ep_warmup_oom: 0
 ep_warmup_thread: complete
 ep_warmup_time: 377606
 get_hits: 114532214
 get_misses: 0
 incr_hits: 0
 incr_misses: 0
 libevent: 2.0.11-stable
 listen_disabled_num: 0
 max_conns_on_port_11207: 10000
 max_conns_on_port_11209: 1000
 max_conns_on_port_11210: 10000
 mem_used: 996199224
 memcached_version: 2.0.1-macosx-175-g6cf6630
 pid: 3964
 pointer_size: 64
 rejected_conns: 0
 rusage_system: 4784.595018
 rusage_user: 5175.215430
 tap_checkpoint_end_received: 443414
 tap_checkpoint_end_sent: 413590
 tap_checkpoint_start_received: 465368
 tap_checkpoint_start_sent: 463014
 tap_connect_received: 3
 tap_mutation_received: 80478530
 tap_mutation_sent: 69079050
 tap_opaque_received: 518
 tap_opaque_sent: 518
 threads: 4
 time: 1398211964
 total_connections: 2101
 uptime: 9831
 vb_active_curr_items: 4282
 vb_active_eject: 1782
 vb_active_expired: 0
 vb_active_ht_memory: 6424576
 vb_active_itm_memory: 5477495
 vb_active_meta_data_memory: 352495
 vb_active_num: 256
 vb_active_num_non_resident: 1782
 vb_active_ops_create: 4282
 vb_active_ops_delete: 0
 vb_active_ops_reject: 0
 vb_active_ops_update: 3726889
 vb_active_perc_mem_resident: 58
 vb_active_queue_age: 0
 vb_active_queue_drain: 7740006
 vb_active_queue_fill: 8012762
 vb_active_queue_memory: 19638432
 vb_active_queue_pending: 587516424
 vb_active_queue_size: 4239
 vb_dead_num: 0
 vb_pending_curr_items: 0
 vb_pending_eject: 0
 vb_pending_expired: 0
 vb_pending_ht_memory: 0
 vb_pending_itm_memory: 0
 vb_pending_meta_data_memory: 0
 vb_pending_num: 0
 vb_pending_num_non_resident: 0
 vb_pending_ops_create: 0
 vb_pending_ops_delete: 0
 vb_pending_ops_reject: 0
 vb_pending_ops_update: 0
 vb_pending_perc_mem_resident: 0
 vb_pending_queue_age: 0
 vb_pending_queue_drain: 0
 vb_pending_queue_fill: 0
 vb_pending_queue_memory: 0
 vb_pending_queue_pending: 0
 vb_pending_queue_size: 0
 vb_replica_curr_items: 4279
 vb_replica_eject: 10183
 vb_replica_expired: 0
 vb_replica_ht_memory: 6424576
 vb_replica_itm_memory: 353069
 vb_replica_meta_data_memory: 353069
 vb_replica_num: 256
 vb_replica_num_non_resident: 4279
 vb_replica_ops_create: 4279
 vb_replica_ops_delete: 0
 vb_replica_ops_reject: 0
 vb_replica_ops_update: 3045626
 vb_replica_perc_mem_resident: 0
 vb_replica_queue_age: 0
 vb_replica_queue_drain: 6763036
 vb_replica_queue_fill: 7083348
 vb_replica_queue_memory: 23062464
 vb_replica_queue_pending: 689952048
 vb_replica_queue_size: 3194
 version: 3.0.0-597-rel




[MB-10929] Print logs after running ep-engine make test when the test fails Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Mike Wiederhold Assignee: Phil Labee
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
We need to print out the logs when make test fails otherwise we won't know why there is a failure. This can be done by doing the following after running make test.

cat build/ep-engine/Testing/Temporary/LastTest.log




[MB-10928] Upgrade to UPR fails because add_stream command returns rollback response Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Aliaksey Artamonau Assignee: Abhinav Dangeti
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File debug.1    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
[ns_server:debug,2014-04-22T13:53:41.410,n_2@127.0.0.1:upr_consumer_conn-default-n_3@127.0.0.1<0.12530.0>:upr_proxy:handle_packet:118]Proxy packet: RESPONSE: 0x51 (upr_add_stream) vbucket = 0 opaque = 0x20 status = 0x23 (rollback)
81 51 00 00
04 00 00 23
00 00 00 04
00 00 00 20
00 00 00 00
00 00 00 00
00 00 00 01


 Comments   
Comment by Artem Stemkovski [ 23/Apr/14 ]
looks like the same issue as MB-10936




[MB-10927] couchbase-cli throws out incorrect output when actually command executed successfully in --cluster-init Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: test-execution
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Thuan Nguyen
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: ubuntu 12.04 64-bit

Triage: Triaged
Is this a Regression?: Unknown

 Description   
Install couchbase server 3.0.0-594 on ubuntu 12.04 64-bit
In CLI test, the option --cluster-init test failed due to incorrect output printout.
The command executed successfully but output saying "ERROR: command: cluster-init: 192.168.171.151:8091, [Errno 111] Connection refused"
I said the command executed ok and change the configuration of couchbase server due to the next command with new credentials and port was also executed successfully
as in following run


root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
ERROR: command: cluster-init: 192.168.171.151:8091, [Errno 111] Connection refused
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
SUCCESS: init localhost
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init --cluster=localhost:8099 -u Administrator1 -p password1 --cluster-username=Administrator --cluster-password=password --cluster-port=8091 --cluster-ramsize=300
ERROR: command: cluster-init: localhost:8099, [Errno 111] Connection refused
root@ubuntu:~# /opt/couchbase/bin/couchbase-cli cluster-init -c 192.168.171.151:8091 --cluster-username=Administrator1 --cluster-password=password1 --cluster-port=8099 -u Administrator -p password --cluster-ramsize=300
SUCCESS: init 192.168.171.151


 Comments   
Comment by Bin Cui [ 23/Apr/14 ]
It is up to ns_server to return connection refused error.




[MB-10926] RPM package warns twice about "Transparent hugepages may be used." Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: installer
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Minor
Reporter: Pavel Paulau Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0-594

Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Yes

 Description   
# rpm -i /tmp/couchbase-server-enterprise_centos6_x86_64_3.0.0-594-rel.rpm
Warning: Transparent hugepages may be used. To disable the usage
of transparent hugepages, set the kernel settings at runtime with
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Warning: Transparent hugepages may be used. To disable the usage
of transparent hugepages, set the kernel settings at runtime with
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Minimum RAM required : 4 GB
System RAM configured : 252.23 GB

Minimum number of processors required : 4 cores
Number of processors on the system : 40 cores

Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://atlas-s315:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.


 Comments   
Comment by Pavel Paulau [ 22/Apr/14 ]
There are two instructions but nobody needs two warnings.

Especially considering the fact that THP are disabled:

# cat /sys/kernel/mm/transparent_hugepage/enabled
always [never]
# cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always [never]
Comment by Bin Cui [ 22/Apr/14 ]
http://review.couchbase.org/#/c/36158/

We do need to check if the value is always or never. But for Centos, there are possible two locations to check for. Thats why we have these false warning twice.




[MB-10925] UI View Access for Read-Only User Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UI, view-engine
Affects Version/s: 2.5.1
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Jeff Dillon Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: devX
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
View (data) access is currently disabled for the read-only user. The ask here is to allow view access as a read-only user. Customers can run view queries through an SDK client application, but the ask is for a light weight web interface to quickly get data from views.

 Comments   
Comment by Perry Krug [ 22/Apr/14 ]
We had a very specific discussion around this and decided that "read-only" also meant that any data stored in Couchbase should not be accessible. IMO that still stands. If and when we get a more robust user-management system it might be appropriate to allow different configurations around this, but at this time I wouldn't think it's appropriate to change.
Comment by Aleksey Kondratenko [ 22/Apr/14 ]
I'm pretty sure it's duplicated somewhere
Comment by Matt Ingenthron [ 22/Apr/14 ]
I'd not gotten an opportunity to have input in that earlier discussion, but I would argue that one of the primary use cases for this is for developers to access views to experiment. It's not a huge deal if we don't have it (they could use a SDK, after all), but it's an annoyance to not have this UI feature. I'd support a way to define such a user, if we can support it.
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
if we are talking about the read only user created in the UI - that is a read only user that has no rights to access data. they can look at views but I believe that is where we stop. giving this user data access would change its role.
I think we need a bucket level read only user for what you are asking to do. isn't that right?
thanks
Comment by Matt Ingenthron [ 23/Apr/14 ]
This was opened by Jeff Dillon, so I'm not sure if you're asking him or me. From my perspective, yes a way to associate a user with read only permissions for both statistics and data (including views) on a bucket would be good.

That said, I think there's something higher level where we need to rethink authentication/authorization rather than try to handle these one off. Probably better plan the future and figure out what the milestones in between are.
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
sounds good. Anil owns that.




[MB-10924] unable to run latest builds on mac(3.0.0-597) Created: 22/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: installer, ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Andrei Baranouski Assignee: Wayne Siu
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File erl_crash.dump    
Issue Links:
Relates to
relates to MB-10808 Server not starting up on Mac OS and ... Open
Triage: Untriaged
Is this a Regression?: Unknown

 Description   

 
./MacOS/Couchbase\ Server
2014-04-22 18:51:56.804 Couchbase Server[4815:507] Launched server task -- pid = 4816
sh: -c: line 0: syntax error near unexpected token `1'
sh: -c: line 0: `/Users/andrei/Downloads/couchbase-server-enterprise_x86_64_3.0.0-597-rel (1)/Couchbase Server.app/Contents/Resources/couchbase-core/lib/erlang/erts-5.8.5/bin/epmd -daemon'

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{shutdown,{kernel,start,[normal,[]]}}})
2014-04-22 18:51:57.698 Couchbase Server[4815:507] Task terminated with status 1


 Comments   
Comment by Aleksey Kondratenko [ 22/Apr/14 ]
Pretty sure it's duplicated somewhere. But certainly not ns-server bug




[MB-10923] Stream request with on-disk items doesn't work properly Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket, UPR
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Volker Mische Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File no-flag.log     Text File on-disk-only-flag.log    
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
When doing a stream request with the on-disk items only flag (0x02) set, ep-engine does something strange that leads to a timeout in the UPR client.

Please note that I know little about ep-engine, but here's what I found out. I've attached to log files. The no-flag.log is without the flag set, the on-disk-only-flag.log with the flag set.

You can see in the on-disk-only-flag.log that a stream for vBuclet 0 is created twice:

memcached<0.86.0>: Tue Apr 22 15:50:29.918776 CEST 3: (beer-sample) UPR (Producer) eq_uprq:Indexer mapreduce_view: beer-sample _design/beer (prod/main) - Stream created for vbucket 0
memcached<0.86.0>: Tue Apr 22 15:50:38.292332 CEST 3: (beer-sample) UPR (Producer) eq_uprq:Indexer mapreduce_view: beer-sample _design/beer (prod/main) - Stream created for vbucket 0

It seems that the UPR client receives the snapshot information already with the first stream request. It then waits for more data. The current timeout is set to 5 seconds, which is not enough, as the second stream, that seems to contain the mutations starts 8 seconds later.




[MB-10922] Linux binaries for generate_cert and vbmap is installed on SmarOS Created: 22/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Trond Norbye Assignee: Trond Norbye
Resolution: Fixed Votes: 0
Labels: #SmartOS
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: SmartOS

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
In ns_server/deps the CMakeLists.txt lists assumes that all UNIX target's is linux. You need a:

ELSEIF (CMAKE_SYSTEM_NAME MATCHES "SunOS")
  ADD_SUBDIRECTORY (generate_cert)
  ADD_SUBDIRECTORY (vbmap)

before the UNIX match (ideally we should just check if you have go or gccgo and if you have just rebuild it and then fallback to the prebuilt things...)

The go build still fails with:

CMake Error at ns_server/deps/generate_cert/CMakeLists.txt:3 (GoBuild):
  Unknown CMake command "GoBuild".




 Comments   
Comment by Chris Hillery [ 22/Apr/14 ]
Sorry, but as smartos is not a supported platform (nor one I can test on), this is low priority.
Comment by Trond Norbye [ 22/Apr/14 ]
There is ongoing pressure from marketing to make it a supported platform... All we need is the "good old fallback" to actually build it if its not there, so you can test it by ignoring the plain install rules for the precompiled bits..
Comment by Chris Hillery [ 22/Apr/14 ]
FYI, the reason I left that as a fallback is that one of those binaries crashed when compiled with GCC go on Linux.
Comment by Trond Norbye [ 22/Apr/14 ]
Ok.. But the linux binaries currently installed does not run at all on non-linux platforms. I've verified that the gccgo binaries works just fine on smartos (I was able to start our server on smaros when I built the binaries by hand and installed them... So at least for non-linux targets we shouldn't install linux binaries...
Comment by Trond Norbye [ 23/Apr/14 ]
http://review.couchbase.org/#/c/36203/
http://review.couchbase.org/#/c/36202/




[MB-10921] Possibly file descriptor leak? Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: view-engine
Affects Version/s: 2.2.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Trond Norbye Assignee: Filipe Manana
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
I ran df and du on that server and noticed similar figures (full console log at the end of this email). du reports 68GB on /var/opt/couchbase, whereas df reports ~140GB of disk usage. The lsof command shows that there are several files which have been deleted but are still opened by beam.smp. Those files are in /var/opt/couchbase/.delete/, and their total size amounts to the “Other Data” (roughly 70GB).
 
I’ve never noticed that before, yet recently we started playing with CB views. I wonder if that can be related. Also note that at the time I did the investigation, there had been no activity on the cluster for several hours: no get/set on the buckets, and no compaction or indexing was ongoing.
Are you aware of this problem? What can we do about it?

beam.smp 55872 couchbase 19u REG 8,17 16136013537 4849671 /var/opt/couchbase/.delete/babe701b000ce862e58ca2edd1b8098b (deleted)
beam.smp 55872 couchbase 34u REG 8,17 1029765807 4849668 /var/opt/couchbase/.delete/5c6df85a423263523471f6e20d82ce07 (deleted)
beam.smp 55872 couchbase 51u REG 8,17 1063802330 4849728 /var/opt/couchbase/.delete/c2a11ea6f3e70f8d222ceae9ed482b13 (deleted)
beam.smp 55872 couchbase 55u REG 8,17 403075242 4849667 /var/opt/couchbase/.delete/6af0b53325bf4f2cd1df34b476ee4bb6 (deleted)
beam.smp 55872 couchbase 56r REG 8,17 403075242 4849667 /var/opt/couchbase/.delete/6af0b53325bf4f2cd1df34b476ee4bb6 (deleted)
beam.smp 55872 couchbase 57u REG 8,17 861075170 4849666 /var/opt/couchbase/.delete/72a08b8a613198cd3a340ae15690b7f1 (deleted)
beam.smp 55872 couchbase 58r REG 8,17 861075170 4849666 /var/opt/couchbase/.delete/72a08b8a613198cd3a340ae15690b7f1 (deleted)
beam.smp 55872 couchbase 59r REG 8,17 1029765807 4849668 /var/opt/couchbase/.delete/5c6df85a423263523471f6e20d82ce07 (deleted)
beam.smp 55872 couchbase 60r REG 8,17 896931996 4849672 /var/opt/couchbase/.delete/3b1b7aae4af60e9e720ad0f0d3c0182c (deleted)
beam.smp 55872 couchbase 63r REG 8,17 976476432 4849766 /var/opt/couchbase/.delete/6f5736b1ed9ba232084ee7f0aa5bd011 (deleted)
beam.smp 55872 couchbase 66u REG 8,17 18656904860 4849675 /var/opt/couchbase/.delete/fcaf4193727374b471c990a017a20800 (deleted)
beam.smp 55872 couchbase 67u REG 8,17 662227221 4849726 /var/opt/couchbase/.delete/4e7bbc192f20def5d99447b431591076 (deleted)
beam.smp 55872 couchbase 70u REG 8,17 896931996 4849672 /var/opt/couchbase/.delete/3b1b7aae4af60e9e720ad0f0d3c0182c (deleted)
beam.smp 55872 couchbase 74r REG 8,17 662227221 4849726 /var/opt/couchbase/.delete/4e7bbc192f20def5d99447b431591076 (deleted)
beam.smp 55872 couchbase 75u REG 8,17 1896522981 4849670 /var/opt/couchbase/.delete/3ce0c5999854691fe8e3dacc39fa20dd (deleted)
beam.smp 55872 couchbase 81u REG 8,17 976476432 4849766 /var/opt/couchbase/.delete/6f5736b1ed9ba232084ee7f0aa5bd011 (deleted)
beam.smp 55872 couchbase 82r REG 8,17 1063802330 4849728 /var/opt/couchbase/.delete/c2a11ea6f3e70f8d222ceae9ed482b13 (deleted)
beam.smp 55872 couchbase 83u REG 8,17 1263063280 4849673 /var/opt/couchbase/.delete/e06facd62f73b20505d2fdeab5f66faa (deleted)
beam.smp 55872 couchbase 85u REG 8,17 1000218613 4849767 /var/opt/couchbase/.delete/0c4fb6d5cd7d65a4bae915a4626ccc2b (deleted)
beam.smp 55872 couchbase 87r REG 8,17 1000218613 4849767 /var/opt/couchbase/.delete/0c4fb6d5cd7d65a4bae915a4626ccc2b (deleted)
beam.smp 55872 couchbase 90u REG 8,17 830450260 4849841 /var/opt/couchbase/.delete/7ac46b314e4e30f81cdf0cd664bb174a (deleted)
beam.smp 55872 couchbase 95r REG 8,17 1263063280 4849673 /var/opt/couchbase/.delete/e06facd62f73b20505d2fdeab5f66faa (deleted)
beam.smp 55872 couchbase 96r REG 8,17 1896522981 4849670 /var/opt/couchbase/.delete/3ce0c5999854691fe8e3dacc39fa20dd (deleted)
beam.smp 55872 couchbase 97u REG 8,17 1400132620 4849719 /var/opt/couchbase/.delete/e8eaade7b2ee5ba7a3115f712eba623e (deleted)
beam.smp 55872 couchbase 103r REG 8,17 16136013537 4849671 /var/opt/couchbase/.delete/babe701b000ce862e58ca2edd1b8098b (deleted)
beam.smp 55872 couchbase 104u REG 8,17 1254021993 4849695 /var/opt/couchbase/.delete/f77992cdae28194411b825fa52c560cd (deleted)
beam.smp 55872 couchbase 105r REG 8,17 1254021993 4849695 /var/opt/couchbase/.delete/f77992cdae28194411b825fa52c560cd (deleted)
beam.smp 55872 couchbase 106r REG 8,17 1400132620 4849719 /var/opt/couchbase/.delete/e8eaade7b2ee5ba7a3115f712eba623e (deleted)
beam.smp 55872 couchbase 108u REG 8,17 1371453421 4849793 /var/opt/couchbase/.delete/9b8b199920075102e52742c49233c57c (deleted)
beam.smp 55872 couchbase 109r REG 8,17 1371453421 4849793 /var/opt/couchbase/.delete/9b8b199920075102e52742c49233c57c (deleted)
beam.smp 55872 couchbase 111r REG 8,17 18656904860 4849675 /var/opt/couchbase/.delete/fcaf4193727374b471c990a017a20800 (deleted)
beam.smp 55872 couchbase 115u REG 8,17 16442158432 4849708 /var/opt/couchbase/.delete/2b70b084bd9d0a1790de9b3ee6c78f69 (deleted)
beam.smp 55872 couchbase 116r REG 8,17 16442158432 4849708 /var/opt/couchbase/.delete/2b70b084bd9d0a1790de9b3ee6c78f69 (deleted)
beam.smp 55872 couchbase 151r REG 8,17 830450260 4849841 /var/opt/couchbase/.delete/7ac46b314e4e30f81cdf0cd664bb174a (deleted)
beam.smp 55872 couchbase 181u REG 8,17 770014022 4849751 /var/opt/couchbase/.delete/d35ac74521ae4c1d455c60240e1c41e1 (deleted)
beam.smp 55872 couchbase 182r REG 8,17 770014022 4849751 /var/opt/couchbase/.delete/d35ac74521ae4c1d455c60240e1c41e1 (deleted)
beam.smp 55872 couchbase 184u REG 8,17 775017865 4849786 /var/opt/couchbase/.delete/2a85b841a373ee149290b0ec906aae55 (deleted)
beam.smp 55872 couchbase 185r REG 8,17 775017865 4849786 /var/opt/couchbase/.delete/2a85b841a373ee149290b0ec906aae55 (deleted)

 Comments   
Comment by Volker Mische [ 22/Apr/14 ]
Filipe, could you have a look at this?

I also found in the bug tracker an issue that we needed to patch Erlang because of file descriptor leaks (CBD-753 [1]). Could it be related?

[1]: http://www.couchbase.com/issues/browse/CBD-753
Comment by Trond Norbye [ 22/Apr/14 ]
From the comments in that bug that seems to be blocker for 2.1 testing and this is 2.2...
Comment by Volker Mische [ 22/Apr/14 ]
Trond, though Erlang was patched, so it's independent of the Couchbase version, but depends on Erlang. Though I guess you use Erlang >= R16 anyway (which should have that patch).

Could you also try it with 2.5? Perhaps it has been fixed already.
Comment by Filipe Manana [ 22/Apr/14 ]
There's no useful information here to work on or conclude anything.

First of all, it may be database files. Both database and view files are renamed to uuids and moved to .delete directory. And before 3.0, database compaction is orchestrated in erlang land (rename + delete).

Second, we had in the past such leaks, one caused by Erlang itself (whence a patched R14/R15 is needed, or R16 unpatched) and others caused by CouchDB upstream code, which got fixed before 2.0 (and in Apache CouchDB 1.2) - geocouch is based on a copy of CouchDB's view engine that is much older than Apache CouchDB 1.x whence suffers the same leaks issue (not closing files after compactions amongst other cases).

Given there's no concrete steps to reproduce this, nor has anyone observed this recently, I can't exclude the possibility of him using the geo/spatial views or running an unpatched Erlang.
Comment by Trond Norbye [ 22/Apr/14 ]
Volker: We ship a bundled erlang in our releases don't we?

Filipe: I forwarded the email to you, Volker and Alk April the 8th with all the information I had. We can ask back for more information if that helps us pinpoint where it is..
Comment by Volker Mische [ 22/Apr/14 ]
Trond: I wasn't aware that the "I" isn't you :)

I would ask to try it again on 2.5.1 and if it's still there the steps to reproduce it.
Comment by Trond Norbye [ 22/Apr/14 ]
The customer have the system running. Are we sure there is no commands to run on the erlang thing to gather more information on the current state?




[MB-10920] unable to start tuq if there are no buckets Created: 22/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: query-preview
Affects Version/s: 2.5.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Iryna Mironava Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Operating System: Centos 64-bit
Is this a Regression?: Unknown

 Description   
node is initialized but has no buckets
[root@kiwi-r116 tuqtng]# ./tuqtng -couchbase http://localhost:8091
10:26:56.520415 Info line disabled false
10:26:56.522641 FATAL: Unable to run server, err: Unable to access site http://localhost:8091, err: HTTP error 401 Unauthorized getting "http://localhost:8091/pools": -- main.main() at main.go:76




[MB-10919] UPR Issue :: Rebalance-in/out leads to out of sync failover logs between active vs replica Created: 21/Apr/14  Updated: 23/Apr/14  Resolved: 22/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Mike Wiederhold
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 10.6.2.144,10.6.2.145, 10.6.2.146, 10.6.2.147

 3.0.0 enterprise edition (build-593)

Attachments: GZip Archive rebalance_in_failover_logs.tar.gz     GZip Archive rebalance_out_failover_log.tar.gz    
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Attached
Is this a Regression?: Unknown

 Description   
Centos-6, 64 v-buckets, UPR enabled, 3.0.0 enterprise edition (build-593)

Rebalance-in

1. 3 Nodes initially
2. Add 1 default
3. Add 1000 items
4. Rebalance-in 1 node
5. Wait for replication to be over
6. Wait for disk queues to be empty

Compare Active Vs Replica failovers UUID and seqno

Expectation: The UUID and seq no’s are similar

Nodes that Rebalanced-in :: 10.6.2.147

Actual: Difference in UUID and seqno are found different
 bucket default, vbucket vb_58 :: Original node 10.6.2.146 replica :: UUID 130190895682164, Change node 10.6.2.147 active UUID 175368009930834

 bucket default, vbucket vb_60 :: Original node 10.6.2.146 replica :: UUID 196726257035839, Change node 10.6.2.147 active UUID 214601589581880

 bucket default, vbucket vb_61 :: Original node 10.6.2.146 replica :: UUID 28459121955904, Change node 10.6.2.147 active UUID 136370844239760

 bucket default, vbucket vb_61 :: Original node 10.6.2.146 replica :: seq 0, Change node 10.6.2.147 active :: seq 16

 bucket default, vbucket vb_62 :: Original node 10.6.2.146 replica :: seq 0, Change node 10.6.2.147 active :: seq 15

 bucket default, vbucket vb_63 :: Original node 10.6.2.146 replica :: UUID 270597828829037, Change node 10.6.2.147 active UUID 108239843453069

 bucket default, vbucket vb_54 :: Original node 10.6.2.147 active :: seq 0, Change node 10.6.2.145 replica :: seq 16

 bucket default, vbucket vb_21 :: Original node 10.6.2.144 replica :: UUID 86383886656703, Change node 10.6.2.145 active UUID 193425227034066

 bucket default, vbucket vb_39 :: Original node 10.6.2.146 active :: UUID 163037669257214, Change node 10.6.2.145 replica UUID 220414430905962

 bucket default, vbucket vb_39 :: Original node 10.6.2.146 active :: seq 0, Change node 10.6.2.145 replica :: seq 14


Rebalance-out

1. 4 Nodes initially
2. Add 1 default
3. Add 1000 items
4. Rebalance-out 1 node
5. Wait for replication to be over
6. Wait for disk queues to be empty

Compare Active Vs Replica failovers UUID and seqno

Expectation: The UUID and seq no’s are similar

Nodes that Rebalanced-out :: 10.6.2.147

Actual: Difference in UUID and seqno are found different
 bucket default, vbucket vb_21 :: Original node 10.6.2.146 replica :: UUID 165824422930582, Change node 10.6.2.144 active UUID 254327835483436
 bucket default, vbucket vb_16 :: Original node 10.6.2.146 replica :: UUID 51478899239274, Change node 10.6.2.144 active UUID 266150421735029
 bucket default, vbucket vb_59 :: Original node 10.6.2.146 active :: UUID 140676620793932, Change node 10.6.2.145 replica UUID 74025060289246
 bucket default, vbucket vb_54 :: Original node 10.6.2.146 active :: seq 0, Change node 10.6.2.145 replica :: seq 16
 bucket default, vbucket vb_60 :: Original node 10.6.2.146 active :: seq 0, Change node 10.6.2.145 replica :: seq 15
 bucket default, vbucket vb_61 :: Original node 10.6.2.146 active :: seq 0, Change node 10.6.2.145 replica :: seq 16
 bucket default, vbucket vb_63 :: Original node 10.6.2.146 active :: UUID 109662588567642, Change node 10.6.2.145 replica UUID 43438854051435
 bucket default, vbucket vb_39 :: Original node 10.6.2.146 replica :: UUID 21671881275397, Change node 10.6.2.145 active UUID 1841774905165



 Comments   
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
Mike found some issue on ep-engine side. So assigned this ticket to him.
Comment by Mike Wiederhold [ 22/Apr/14 ]
http://review.couchbase.org/36157
Comment by Parag Agarwal [ 23/Apr/14 ]
Mike

The issue seems fixed in latest build 603, but I see it happening after normal failover. I have logs for it. Should we reopen this bug or close this chapter with another bug?
Comment by Mike Wiederhold [ 23/Apr/14 ]
Open another and describe exactly why you think it is incorrect during a failover case. The reason I say to describe why you think it's wrong is because the behavior your seeing may actually be the correct behavior, but it's hard for me to tell from your short description above.




[MB-10918] upr-notifier stream hangs when start_seq >0 and vb_uuid is missing Created: 21/Apr/14  Updated: 21/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UPR
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Tommie McAfee Assignee: Mike Wiederhold
Resolution: Unresolved Votes: 0
Labels: upr
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
NotifierStreams are protected from rollback by setting high_seqno to start_seqno.
This causes needRollback in in failover-table.cc to always return false.

Problem is this will skip case where vb_uuid doesn't not exist in failovertable, because under normal cases producer stream requests returns rollback when client tries to stream mutations past seqno 0 when vbuuid does not exist.

Result is that notifier stream is created, but is never notified of stream_end because it is not subscribed to any vbucket.



 Comments   
Comment by Tommie McAfee [ 21/Apr/14 ]
test added in pyupr to repro: http://review.couchbase.org/#/c/36118/2
Comment by Tommie McAfee [ 21/Apr/14 ]
I propose a fix here:
http://review.couchbase.org/#/c/36124/

only verified with pyupr




[MB-10917] CBBackup/CBRestore needs to support compression Created: 21/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: tools
Affects Version/s: 3.0, 2.5.1
Fix Version/s: feature-backlog
Security Level: Public

Type: Improvement Priority: Major
Reporter: Don Pinto Assignee: Anil Kumar
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate

 Description   
Customer X wants to backup terabytes of data. Wants our backup tool to support compression.

Today's options are :
1. Use 3rd party backup tools on the archive
2. Use storage level compression tools






[MB-10915] Impossible to store a key with an empty value Created: 21/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Jens Alfke Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
It's impossible to store a key/value pair with an empty (zero-length) value -- it always deletes the key. This has been awkward for me so far in a few places. (For example, when using a db as a map/reduce index, the map function will commonly emit a null value, but the key still needs to be stored. I've had to work around this by changing the value to a single 0 byte when storing, and then detecting that value when reading the doc and changing it back to a NULL pointer.)

The doc-comment for fdb_set says "Setting bodylen to 0 and body to NULL is equivalent to calling fdb_del", which to me implies that if bodylen is 0 but body is _not_ NULL, it should store a zero-length value. But this doesn't work in practice. The implementation tests for bodylen==0 and goes to the delete code-path.




[MB-10914] {UPR} ::Control connection to memcached on 'ns_1@IP' disconnected with some other crashes before upr_replicator:init/1, upr_proxy:init/1, replication_manager:init/1 Created: 21/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Meenakshi Goel Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-594-rel

Attachments: Text File log.txt    
Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Jenkins Link:
http://qa.sc.couchbase.com/job/ubuntu_x64--65_01--view_query_negative-P1/17/console

Notes:
Failed Test will not deterministically reproduce the error.
No Core dumps are observed on the machines.
Please refer log file log.txt attached.

Logs:
[user:info,2014-04-21T1:42:50.417,ns_1@172.23.106.196:ns_memcached-default<0.22998.7>:ns_memcached:terminate:821]Control connection to memcached on 'ns_1@172.23.106.196' disconnected: {badmatch,
                                                                        {error,
                                                                         couldnt_connect_to_memcached}}
[error_logger:error,2014-04-21T1:42:50.419,ns_1@172.23.106.196:error_logger<0.6.0>:ale_error_logger_handler:log_msg:119]** Generic server <0.22998.7> terminating
** Last message in was {'EXIT',<0.23030.7>,
                           {badmatch,{error,couldnt_connect_to_memcached}}}
** When Server state == {state,1,0,0,
                               {[],[]},
                               {[],[]},
                               {[],[]},
                               connected,
                               {1398,69765,399403},
                               "default",#Port<0.424036>,
                               {interval,#Ref<0.0.28.134215>},
                               [{<0.23031.7>,#Ref<0.0.28.136461>},
                                {<0.23029.7>,#Ref<0.0.28.134508>},
                                {<0.23032.7>,#Ref<0.0.28.134245>}],
                               []}
** Reason for termination ==
** {badmatch,{error,couldnt_connect_to_memcached}}

[error_logger:error,2014-04-21T1:42:50.420,ns_1@172.23.106.196:error_logger<0.6.0>:ale_error_logger_handler:log_report:115]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_memcached:init/1
    pid: <0.22998.7>
    registered_name: []
    exception exit: {badmatch,{error,couldnt_connect_to_memcached}}
      in function gen_server:init_it/6
    ancestors: ['single_bucket_sup-default',<0.22992.7>]
    messages: []
    links: [<0.23029.7>,<0.23031.7>,<0.23032.7>,<0.307.0>,<0.22993.7>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 196418
    stack_size: 24
    reductions: 22902
  neighbours:
    neighbour: [{pid,<0.23032.7>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.22992.7>]},
                  {messages,[]},
                  {links,[<0.22998.7>,#Port<0.424043>]},
                  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,46368},
                  {stack_size,24},
                  {reductions,4663}]
    neighbour: [{pid,<0.23031.7>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.22992.7>]},
                  {messages,[]},
                  {links,[<0.22998.7>,#Port<0.424044>]},
                  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,10946},
                  {stack_size,24},
                  {reductions,44938}]
    neighbour: [{pid,<0.23029.7>},
                  {registered_name,[]},
                  {initial_call,{erlang,apply,['Argument__1','Argument__2']}},
                  {current_function,{gen,do_call,4}},
                  {ancestors,['ns_memcached-default',
                              'single_bucket_sup-default',<0.22992.7>]},
                  {messages,[]},
                  {links,[<0.22998.7>,#Port<0.424045>]},
  {dictionary,[]},
                  {trap_exit,false},
                  {status,waiting},
                  {heap_size,10946},
                  {stack_size,24},
                  {reductions,11546}]


Uploading logs.

 Comments   
Comment by Meenakshi Goel [ 21/Apr/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-10914/fd10746b/172.23.106.196-4212014-20-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10914/e390d261/172.23.106.197-4212014-22-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10914/29d793bb/172.23.106.198-4212014-24-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10914/d5568c08/172.23.106.199-4212014-26-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10914/22245367/172.23.106.200-4212014-29-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10914/5fabfeeb/172.23.106.201-4212014-211-diag.zip




[MB-10913] Mutations (1 deleted items) not replicated, caused items mismatch on destination cluster Created: 21/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Sangharsh Agarwal Assignee: Aliaksey Artamonau
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0 531, Centos 32 bit

Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.121.65 : https://s3.amazonaws.com/bugdb/jira/MB-10913/6715c078/10.3.121.65-4182014-422-diag.zip
10.3.121.65 : https://s3.amazonaws.com/bugdb/jira/MB-10913/b39f3625/10.3.121.65-4182014-420-couch.tar.gz
10.3.3.207 : https://s3.amazonaws.com/bugdb/jira/MB-10913/4eb29318/10.3.3.207-4182014-420-couch.tar.gz
10.3.3.207 : https://s3.amazonaws.com/bugdb/jira/MB-10913/f81636e3/10.3.3.207-4182014-424-diag.zip
10.3.3.209 : https://s3.amazonaws.com/bugdb/jira/MB-10913/106b5aed/10.3.3.209-4182014-420-couch.tar.gz
10.3.3.209 : https://s3.amazonaws.com/bugdb/jira/MB-10913/d8cb5a74/10.3.3.209-4182014-425-diag.zip
10.3.3.210 : https://s3.amazonaws.com/bugdb/jira/MB-10913/7ffea465/10.3.3.210-4182014-420-couch.tar.gz
10.3.3.210 : https://s3.amazonaws.com/bugdb/jira/MB-10913/b8104406/10.3.3.210-4182014-426-diag.zip


[Destination]
10.3.4.177 : https://s3.amazonaws.com/bugdb/jira/MB-10913/88b8d050/10.3.4.177-4182014-427-diag.zip
10.3.4.177 : https://s3.amazonaws.com/bugdb/jira/MB-10913/e6388a6b/10.3.4.177-4182014-420-couch.tar.gz
10.3.121.62 : https://s3.amazonaws.com/bugdb/jira/MB-10913/289ad162/10.3.121.62-4182014-429-diag.zip
10.3.121.62 : https://s3.amazonaws.com/bugdb/jira/MB-10913/aec8a352/10.3.121.62-4182014-420-couch.tar.gz
10.3.2.204 : https://s3.amazonaws.com/bugdb/jira/MB-10913/316af914/10.3.2.204-4182014-429-diag.zip
10.3.2.204 : https://s3.amazonaws.com/bugdb/jira/MB-10913/c6294b49/10.3.2.204-4182014-421-couch.tar.gz
10.3.3.208 : https://s3.amazonaws.com/bugdb/jira/MB-10913/619c133a/10.3.3.208-4182014-420-couch.tar.gz
10.3.3.208 : https://s3.amazonaws.com/bugdb/jira/MB-10913/807ad34c/10.3.3.208-4182014-428-diag.zip
Is this a Regression?: Unknown

 Description   
[Jenkins]
http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/29/consoleFull

[Test]
./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,expires=60,standard_buckets=1,sasl_buckets=2,default_bucket=False,doc-ops=delete,GROUP=CHAIN;P1


[Test Logs]
[2014-04-18 03:54:11,221] - [rest_client:790] INFO - adding remote cluster hostname:10.3.4.177:8091 with username:password Administrator:password name:cluster1
[2014-04-18 03:54:11,295] - [rest_client:836] INFO - starting replication type:continuous from sasl_bucket_1 to sasl_bucket_1 in the remote cluster cluster1
[2014-04-18 03:54:11,506] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
[2014-04-18 03:54:16,513] - [rest_client:836] INFO - starting replication type:continuous from sasl_bucket_2 to sasl_bucket_2 in the remote cluster cluster1
[2014-04-18 03:54:16,653] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
[2014-04-18 03:54:21,659] - [rest_client:836] INFO - starting replication type:continuous from standard_bucket_1 to standard_bucket_1 in the remote cluster cluster1
[2014-04-18 03:54:21,824] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...
..
..
..
2014-04-18 04:11:16,490] - [task:420] WARNING - Not Ready: curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
[2014-04-18 04:11:16,542] - [task:420] WARNING - Not Ready: vb_active_curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
[2014-04-18 04:11:21,585] - [task:420] WARNING - Not Ready: curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket
[2014-04-18 04:11:21,622] - [task:420] WARNING - Not Ready: vb_active_curr_items 70001 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', sasl_bucket_1 bucket


[Test Steps] - No UPR.
1. Create 4-4 Nodes SRC and DEST.
2. Create 1 sasl, 2 standard buckets.
3. Configure XMEM Uni XDCR for both buckets.
4. Load 1M Items each bucket.
5. Delete 30% items on each bucket.
6. expected items on Destination = 70000 while 70001 items on destination cluster (for sasl_bucket_1). Following key loadOne97310 has mismatch. Keys shows deleted from Source but not deleted from Destination, also it shows higher sequence number of Source, showing mutations were not replicated properly.

2014-04-18 04:18:16,419] - [task:1169] ERROR - ===== Verifying rev_ids failed for key: loadOne97310 =====
[2014-04-18 04:18:16,419] - [task:1170] ERROR - deleted mismatch: Source deleted:1, Destination deleted:0, Error Count:1
[2014-04-18 04:18:16,420] - [task:1170] ERROR - seqno mismatch: Source seqno:2, Destination seqno:1, Error Count:2
[2014-04-18 04:18:16,421] - [task:1170] ERROR - cas mismatch: Source cas:17748657181694094, Destination cas:17748657181694093, Error Count:3
[2014-04-18 04:18:16,423] - [task:1171] ERROR - Source meta data: {'deleted': 1, 'seqno': 2, 'cas': 17748657181694094, 'flags': 0, 'expiration': 1397818797}
[2014-04-18 04:18:16,424] - [task:1172] ERROR - Dest meta data: {'deleted': 0, 'seqno': 1, 'cas': 17748657181694093, 'flags': 0, 'expiration': 0}

Items were properly replicated on other 2 buckets i.e. standard_bucekt_1, sasl_bucket_2. But not sasl_bucket_1.

 Comments   
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
Bucket data-files, cb collect logs, and keys is mentioned.
Comment by Aruna Piravi [ 21/Apr/14 ]
Could possibly be a duplicate of MB-10856.
Comment by Cihan Biyikoglu [ 23/Apr/14 ]
sending over to Ailaksey,
Comment by Aruna Piravi [ 23/Apr/14 ]
Thanks Cihan.
Comment by Aliaksey Artamonau [ 23/Apr/14 ]
I'd wait until MB-10856 is resolved.




[MB-10912] curr_items are not expected number of items on destination cluster Created: 21/Apr/14  Updated: 21/Apr/14

Status: Open
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Aleksey Kondratenko
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0 591.

Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.121.65 : https://s3.amazonaws.com/bugdb/jira/MB-10912/3d973e33/10.3.121.65-4182014-327-diag.zip
10.3.121.65 : https://s3.amazonaws.com/bugdb/jira/MB-10912/e8c46be1/10.3.121.65-4182014-325-couch.tar.gz
10.3.3.207 : https://s3.amazonaws.com/bugdb/jira/MB-10912/1dc784cb/10.3.3.207-4182014-325-couch.tar.gz
10.3.3.207 : https://s3.amazonaws.com/bugdb/jira/MB-10912/d6067609/10.3.3.207-4182014-329-diag.zip
10.3.3.209 : https://s3.amazonaws.com/bugdb/jira/MB-10912/89d0a503/10.3.3.209-4182014-330-diag.zip
10.3.3.209 : https://s3.amazonaws.com/bugdb/jira/MB-10912/eb4804a3/10.3.3.209-4182014-325-couch.tar.gz
10.3.3.210 : https://s3.amazonaws.com/bugdb/jira/MB-10912/db73c085/10.3.3.210-4182014-325-couch.tar.gz
10.3.3.210 : https://s3.amazonaws.com/bugdb/jira/MB-10912/e225c957/10.3.3.210-4182014-331-diag.zip


[Dest]
10.3.4.177 : https://s3.amazonaws.com/bugdb/jira/MB-10912/81aac8cf/10.3.4.177-4182014-325-couch.tar.gz
10.3.4.177 : https://s3.amazonaws.com/bugdb/jira/MB-10912/b376051a/10.3.4.177-4182014-332-diag.zip
10.3.121.62 : https://s3.amazonaws.com/bugdb/jira/MB-10912/bdd6f5e9/10.3.121.62-4182014-325-couch.tar.gz
10.3.121.62 : https://s3.amazonaws.com/bugdb/jira/MB-10912/d4e5d541/10.3.121.62-4182014-334-diag.zip
10.3.2.204 : https://s3.amazonaws.com/bugdb/jira/MB-10912/2e8d9c32/10.3.2.204-4182014-334-diag.zip
10.3.2.204 : https://s3.amazonaws.com/bugdb/jira/MB-10912/3b707290/10.3.2.204-4182014-325-couch.tar.gz
10.3.3.208 : https://s3.amazonaws.com/bugdb/jira/MB-10912/412475b1/10.3.3.208-4182014-325-couch.tar.gz
10.3.3.208 : https://s3.amazonaws.com/bugdb/jira/MB-10912/5a455ac3/10.3.3.208-4182014-333-diag.zip
Is this a Regression?: Unknown

 Description   
[Jenkins, test #2]
http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/29/consoleFull

[Test]
./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,doc-ops=delete-delete,GROUP=CHAIN;P1

[Test Logs Duration]
[2014-04-18 03:07:05,150] - [rest_client:790] INFO - adding remote cluster hostname:10.3.4.177:8091 with username:password Administrator:password name:cluster1
[2014-04-18 03:07:05,204] - [rest_client:836] INFO - starting replication type:continuous from default to default in the remote cluster cluster1
[2014-04-18 03:07:05,272] - [xdcrbasetests:355] INFO - sleep for 5 secs. ...

..
..
..
[2014-04-18 03:25:14,590] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:25:19,637] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket


[Test Steps] - Non UPR
1. Create 4-4 Nodes SRC and DEST.
2. 1 default bucket.
3. Configure CAPI Uni XDCR for both buckets.
4. Load 1M Items each bucket.
5. delete 30% items on each bucket of Source
6. Wait for stats curr_items to be expected items i.e. 70000 -> Failed, curr_items stat shows 69999 items. 1 items missing.

[2014-04-18 03:21:16,826] - [task:1179] ERROR - Key:loadOne336 Memcached error #1 'Not found': for vbucket :315 to mc 10.3.2.204:11210, Error Count:1

vbucket 315 exists on Source node i.e. 10.3.3.207, from the logs it shows that items was put in the outgoing batch:

[xdcr_trace:debug,2014-04-18T3:07:48.976,ns_1@10.3.3.207:<0.9736.20>:xdc_vbucket_rep_worker:local_process_batch:110]added mutation loadOne36636@31 (rev = 1-..) to outgoing batch
[xdcr_trace:deb



 Comments   
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
One items is not replicated, there is Memcached error for one key i.e. loadOne336 on 10.3.2.204:11210

[2014-04-18 03:21:01,776] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:21:06,860] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:21:11,902] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket


[2014-04-18 03:21:16,826] - [task:1179] ERROR - Key:loadOne336 Memcached error #1 'Not found': for vbucket :315 to mc 10.3.2.204:11210, Error Count:1


[2014-04-18 03:21:16,955] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:21:21,996] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:21:27,044] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', default bucket
[2014-04-18 03:21:32,115] - [task:420] WARNING - Not Ready: vb_active_curr_items 69999 == 70000 expected on '10.3.4.177:8091''10.3.3.208:8091''10.3.121.62:8091''10.3.2.204:8091', defau

It seems that key -> loadOne336 is not replicated.
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
Data files are also uploaded along with log files.
Comment by Pavel Paulau [ 21/Apr/14 ]
MB-10856




[MB-10911] TAP UniXDCR (xmem mode), time taken by ep_queue_size to 0 is significant Created: 21/Apr/14  Updated: 21/Apr/14  Resolved: 21/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: cross-datacenter-replication, storage-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Sangharsh Agarwal Assignee: Sangharsh Agarwal
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 3.0.0 - 591

Issue Links:
Duplicate
duplicates MB-10875 Flusher queue doesn't get flushed Reopened
Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: [Source]
10.3.121.65 : https://s3.amazonaws.com/bugdb/jira/MB-10911/5ba07ffa/10.3.121.65-4182014-258-diag.zip
10.3.3.210 : https://s3.amazonaws.com/bugdb/jira/MB-10911/c70b7506/10.3.3.210-4182014-31-diag.zip
10.3.3.207 : https://s3.amazonaws.com/bugdb/jira/MB-10911/a83a7060/10.3.3.207-4182014-259-diag.zip
10.3.3.209 : https://s3.amazonaws.com/bugdb/jira/MB-10911/5a07c1b9/10.3.3.209-4182014-30-diag.zip

[Destination]
10.3.4.177 : https://s3.amazonaws.com/bugdb/jira/MB-10911/96a92d36/10.3.4.177-4182014-32-diag.zip
10.3.3.208 : https://s3.amazonaws.com/bugdb/jira/MB-10911/e4c76759/10.3.3.208-4182014-33-diag.zip
10.3.121.62 : https://s3.amazonaws.com/bugdb/jira/MB-10911/69625ebd/10.3.121.62-4182014-33-diag.zip
10.3.2.204 : https://s3.amazonaws.com/bugdb/jira/MB-10911/595f521b/10.3.2.204-4182014-34-diag.zip
Is this a Regression?: Unknown

 Description   
[Jenkins]
http://qa.hq.northscale.net/job/centos_x64--31_01--uniXDCR-P1/29/consoleFull

[Test]
./testrunner -i /tmp/ubuntu-64-2.0-uniXDCR.ini GROUP=CHAIN,num_items=50000,get-cbcollect-info=True -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,doc-ops=update-delete,sasl_buckets=1,replication_type=xmem,GROUP=CHAIN;P0;xmem


[Test Logs]
2014-04-18 02:55:39 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 1748 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:55:41 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 1907 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:55:44 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 1478 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:55:46 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 1569 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:55:49 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 975 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:55:51 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 898 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:55:54 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 202 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:55:56 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 91 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:55:59 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:01 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:04 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:06 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:09 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:11 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:14 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:16 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:19 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:24 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:26 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:29 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:32 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:35 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:37 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:40 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:45 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:50 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:56:55 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
2014-04-18 02:56:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 43 == 0 expected on '10.3.121.65:8091', default bucket
2014-04-18 02:57:00 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4 == 0 expected on '10.3.121.65:8091', sasl_bucket_1 bucket
ERROR
[('testrunner', 331, '<module>', 'result = unittest.TextTestRunner(verbosity=2).run(suite)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/suite.py', 70, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/suite.py', 108, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/xdcr/uniXDCR.py', 42, 'load_with_async_ops', 'self._wait_for_stats_all_buckets(self.src_nodes)'), ('pytests/xdcr/xdcrbasetests.py', 1241, '_wait_for_stats_all_buckets', 'is_verified = self._poll_for_condition(verify)'), ('pytests/xdcr/xdcrbasetests.py', 901, '_poll_for_condition', 'return self._poll_for_condition_rec(condition, interval, num_itr)'), ('pytests/xdcr/xdcrbasetests.py', 907, '_poll_for_condition_rec', 'if condition():'), ('pytests/xdcr/xdcrbasetests.py', 1234, 'verify', 'task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Fri Apr 18 02:57:01 2014

[Test Steps]
1. Create 4-4 Nodes SRC and DEST.
2. Create 1 sasl, 1 default bucket.
3. Configure XMEM Uni XDCR for both buckets.
4. Load 1M Items each bucket.
5. Update 30%, delete 30% items on each bucket.
6. Wait for stats ep_queue_size to 0 -> Timeout here.

 Comments   
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
Installed using , xdcr_upr=False.

python scripts/install.py -i /tmp/ubuntu-64-2.0-uniXDCR.ini -p version=3.0.0-591-rel,product=cb,parallel=True,get-logs=True,xdcr_upr=False
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
Alk, let me know if other artifiacts needed here. Checked the ns_server.debug.log, at node "10.3.121.65:8091", but couldn't found any error on logs.

Seems persistence layer is slow to drain items queue to disk.
Comment by Sangharsh Agarwal [ 21/Apr/14 ]
Venu, Can you please check from ep_engine point of view.
Comment by Pavel Paulau [ 21/Apr/14 ]
ep-engine stats are broken.
See MB-10875.
Comment by Aruna Piravi [ 21/Apr/14 ]
Duplicate of MB-10875 which is a test blocker.




[MB-10910] Rebalance with views after failover fails due to "wait_checkpoint_persisted_failed" Created: 20/Apr/14  Updated: 20/Apr/14  Resolved: 20/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Pavel Paulau Assignee: Pavel Paulau
Resolution: Duplicate Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-594

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630
Memory = 64 GB
Disk = 2 x SSD

Issue Links:
Duplicate
duplicates MB-10514 During rebalance, UPR stream gets stu... Reopened
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: http://ci.sc.couchbase.com/job/apollo-64/1247/artifact/
Is this a Regression?: Yes

 Description   
Rebalance after failover, 3 -> 4 nodes, 1 bucket x 100M x 2KB, DGM, 1 x 1 views, 10K ops/sec, 400 qps

Rebalance exited with reason {unexpected_exit,
                              {'EXIT',<0.16719.261>,
                               {wait_checkpoint_persisted_failed,"bucket-1",
                                892,2204,
                                [{'ns_1@172.23.96.18',
                                  {'EXIT',
                                   {{{{badmatch,
                                       {error,
                                        {{function_clause,
                                          [{couch_set_view_group,handle_info,
                                            [{'DOWN',#Ref<16800.0.690.88151>,
                                              process,<16800.29051.86>,normal},
                                             {state,
                                              {"/ssd2",<<"bucket-1">>,
                                               {set_view_group,
                                                <<185,152,194,156,196,175,136,
                                                  156,249,192,91,24,8,146,109,
                                                  97>>,
                                                nil,<<"bucket-1">>,
                                                <<"_design/A">>,[],
                                                [{set_view,0,
                                                  <<"\n function(doc, meta) {\n emit(doc.city, null);\n }\n ">>,
                                                  undefined,
                                                  {mapreduce_view,
                                                   [<<"id_by_city">>],
                                                   nil,[],[]}}],
                                                nil,nil,
                                                {set_view_index_header,2,0,0,
                                                 0,0,[],nil,[],false,[],nil,
                                                 [],[]},
                                                main,nil,nil,nil,[],
                                                mapreduce_view,".view",prod,
                                                couch_set_view_stats_prod,0,
                                                nil}},
                                              <16800.28934.86>,
                                              {set_view_group,
                                               <<185,152,194,156,196,175,136,
                                                 156,249,192,91,24,8,146,109,97>>,
                                               <16800.28930.86>,
                                               <<"bucket-1">>,<<"_design/A">>,
                                               [],
                                               [{set_view,0,
                                                 <<"\n function(doc, meta) {\n emit(doc.city, null);\n }\n ">>,
                                                 #Ref<16800.0.690.85787>,
                                                 {mapreduce_view,
                                                  [<<"id_by_city">>],
                                                  {btree,<16800.28930.86>,
                                                   {872023493,
                                                    <<0,0,94,204,32,255,254,
                                                      254,191,207,240,120,0,0,
                                                      0,3,255,128,0,0,0,0,0,0,
                                                      0,0,3,199,65,254,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0,0,0,0,0,0,0,0,0,0,
                                                      0,0,0>>,
                                                    115226299},
                                                   identity,identity,
                                                   #Fun<mapreduce_view.14.59760005>,
                                                   #Fun<mapreduce_view.13.116710578>,
                                                   7168,6144,true},
                                                  [],[]}}],
                                               {btree,<16800.28930.86>,
                                                {757316813,
                                                 <<0,0,94,204,32,255,254,254,
                                                   191,207,240,120,0,0,0,3,255,
                                                   128,0,0,0,0,0,0,0,0,3,199,
                                                   65,254,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0>>,
                                                 98005067},
                                                identity,identity,
                                                #Fun<couch_btree.1.34546481>,
                                                #Fun<couch_set_view_group.15.54553809>,
                                                7168,6144,true},
                                               <16800.28933.86>,
                                               {set_view_index_header,2,1024,
                                                179769313486231580793728973728767334576404131515618582199733964182346700693311000189374238233220773621413668943491054022143327228131710028156464112654519069829469374388748562738720335524163543296024799521113246188546084449778835326689833933543997895438175469077364855610760285917179803342821150185391109701632,
                                                7484401160755199293711447201500438591369787779463561409980691526969024494823315590706649929017925717627609195968322657071845869263978468466858275373827242994228699112960690437335703692189245416448148245614649742243301249193480738942045004516930273328675008179153418008263268875886110850416640,
                                                0,
                                                [{820,0},
                                                 {821,0},
                                                 {822,0},
                                                 {823,0},
                                                 {824,0},
                                                 {825,98576},
                                                 {826,98613},
                                                 {827,98636},
                                                 {828,98649},
                                                 {829,98744},
                                                 {830,98776},
                                                 {831,98779},
                                                 {832,98834},
                                                 {833,0},
                                                 {834,0},
                                                 {835,0},
                                                 {836,0},
                                                 {837,0},
                                                 {838,98538},
                                                 {839,0},
                                                 {840,98596},
                                                 {841,98622},
                                                 {842,98599},
                                                 {843,0},
                                                 {844,0},
                                                 {845,0},
                                                 {846,98663},
                                                 {847,98725},
                                                 {848,98727},
                                                 {849,98793},
                                                 {850,0},
                                                 {851,0},
                                                 {852,0},
                                                 {891,0},
                                                 {892,0},
                                                 {893,0},
                                                 {894,0},
                                                 {895,0},
                                                 {896,0},
                                                 {897,0},
                                                 {898,0},
                                                 {899,0},
                                                 {900,0},
                                                 {901,0},
                                                 {902,0},
                                                 {903,0},
                                                 {904,0},
                                                 {905,0},
                                                 {906,0},
                                                 {907,0},
                                                 {908,0},
                                                 {909,0},
                                                 {910,0},
                                                 {911,0},
                                                 {912,0},
                                                 {913,0},
                                                 {914,0},
                                                 {915,0},
                                                 {916,0},
                                                 {917,0},
                                                 {918,0},
                                                 {919,0},
                                                 {920,0},
                                                 {921,0},
                                                 {922,0},
                                                 {923,0},
                                                 {924,0},
                                                 {925,0},
                                                 {926,0},
                                                 {927,98576},
                                                 {928,98605},
                                                 {929,98654},
                                                 {930,98651},
                                                 {931,98704},
                                                 {932,98721},
                                                 {933,98738},
                                                 {934,98713},
                                                 {935,98782},
                                                 {936,98779},
                                                 {937,98760},
                                                 {968,0},
                                                 {969,0},
                                                 {970,0},
                                                 {971,98877},
                                                 {972,98914},
                                                 {973,98961},
                                                 {974,99028},
                                                 {975,0},
                                                 {976,0},
                                                 {977,0},
                                                 {978,0},
                                                 {979,0},
                                                 {980,98539},
                                                 {981,98574},
                                                 {982,98586},
                                                 {983,98664},
                                                 {984,98658},
                                                 {985,98722},
                                                 {986,98704},
                                                 {987,98676},
                                                 {988,0},
                                                 {989,0},
                                                 {990,98535},
                                                 {991,98594},
                                                 {992,98591},
                                                 {993,98649},
                                                 {994,98653},
                                                 {995,98711},
                                                 {996,98717},
                                                 {997,98774},
                                                 {998,0},
                                                 {999,98396},
                                                 {1000,0},
                                                 {1001,98395},
                                                 {1002,98430},
                                                 {1003,98456},
                                                 {1004,98502},
                                                 {1005,98528},
                                                 {1006,98570},
                                                 {1007,98602},
                                                 {1008,0},
                                                 {1009,98412},
                                                 {1010,98354},
                                                 {1011,98346},
                                                 {1012,98366},
                                                 {1013,98360},
                                                 {1014,98358},
                                                 {1015,98378},
                                                 {1016,98345},
                                                 {1017,98405},
                                                 {1018,98375},
                                                 {1019,98428},
                                                 {1020,98454},
                                                 {1021,98460},
                                                 {1022,98494},
                                                 {1023,98521}],
                                                <<0,0,45,35,188,205,0,0,5,215,
                                                  112,75,0,0,94,204,32,255,254,
                                                  254,191,207,240,120,0,0,0,3,
                                                  255,128,0,0,0,0,0,0,0,0,3,
                                                  199,65,254,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                  0,0,0,0,0,0,0,0,0,0>>,
                                                [<<0,0,51,250,5,197,0,0,6,222,
                                                   54,187,0,0,94,204,32,255,
                                                   254,254,191,207,240,120,0,
                                                   0,0,3,255,128,0,0,0,0,0,0,
                                                   0,0,3,199,65,254,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0,0,0,0,0,0,0,
                                                   0,0,0,0,0,0,0>>],
                                                true,[],nil,[],
                                                [{820,[{0,0}]},
                                                 {821,[{0,0}]},
                                                 {822,[{0,0}]},
                                                 {823,[{0,0}]},
                                                 {824,[{0,0}]},
                                                 {825,[{0,0}]},
                                                 {826,[{0,0}]},
                                                 {827,[{0,0}]},
                                                 {828,[{0,0}]},
                                                 {829,[{0,0}]},
                                                 {830,[{0,0}]},
                                                 {831,[{0,0}]},
                                                 {832,[{0,0}]},
                                                 {833,[{0,0}]},
                                                 {834,[{0,0}]},
                                                 {835,[{0,0}]},
                                                 {836,[{0,0}]},
                                                 {837,[{0,0}]},
                                                 {838,[{0,0}]},
                                                 {839,[{0,0}]},
                                                 {840,[{0,0}]},
                                                 {841,[{0,0}]},
                                                 {842,[{0,0}]},
                                                 {843,[{0,0}]},
                                                 {844,[{0,0}]},
                                                 {845,[{0,0}]},
                                                 {846,[{0,0}]},
                                                 {847,[{0,0}]},
                                                 {848,[{0,0}]},
                                                 {849,[{0,0}]},
                                                 {850,[{0,0}]},
                                                 {851,[{0,0}]},
                                                 {852,[{0,0}]},
                                                 {891,[{0,0}]},
                                                 {892,[{0,0}]},
                                                 {893,[{0,0}]},
                                                 {894,[{0,0}]},
                                                 {895,[{0,0}]},
                                                 {896,[{0,0}]},
                                                 {897,[{0,0}]},
                                                 {898,[{0,0}]},
                                                 {899,[{0,0}]},
                                                 {900,[{0,0}]},
                                                 {901,[{0,0}]},
                                                 {902,[{0,0}]},
                                                 {903,[{0,0}]},
                                                 {904,[{0,0}]},
                                                 {905,[{0,0}]},
                                                 {906,[{0,0}]},
                                                 {907,[{0,0}]},
                                                 {908,[{0,0}]},
                                                 {909,[{0,0}]},
                                                 {910,[{0,0}]},
                                                 {911,[{0,0}]},
                                                 {912,[{0,0}]},
                                                 {913,[{0,0}]},
                                                 {914,[{0,0}]},
                                                 {915,[{0,0}]},
                                                 {916,[{0,0}]},
                                                 {917,[{0,0}]},
                                                 {918,[{0,0}]},
                                                 {919,[{0,0}]},
                                                 {920,[{0,0}]},
                                                 {921,[{0,0}]},
                                                 {922,[{0,0}]},
                                                 {923,[{0,0}]},
                                                 {924,[{0,0}]},
                                                 {925,[{0,0}]},
                                                 {926,[{0,0}]},
                                                 {927,[{0,0}]},
                                                 {928,[{0,0}]},
                                                 {929,[{0,0}]},
                                                 {930,[{0,0}]},
                                                 {931,[{0,0}]},
                                                 {932,[{0,0}]},
                                                 {933,[{0,0}]},
                                                 {934,[{0,0}]},
                                                 {935,[{0,0}]},
                                                 {936,[{0,0}]},
                                                 {937,[{0,0}]},
                                                 {968,[{0,0}]},
                                                 {969,[{0,0}]},
                                                 {970,[{0,0}]},
                                                 {971,[{0,0}]},
                                                 {972,[{0,0}]},
                                                 {973,[{0,0}]},
                                                 {974,[{0,0}]},
                                                 {975,[{0,0}]},
                                                 {976,[{0,0}]},
                                                 {977,[{0,0}]},
                                                 {978,[{0,0}]},
                                                 {979,[{0,0}]},
                                                 {980,[{0,0}]},
                                                 {981,[{0,0}]},
                                                 {982,[{0,0}]},
                                                 {983,[{0,0}]},
                                                 {984,[{0,0}]},
                                                 {985,[{0,0}]},
                                                 {986,[{0,0}]},
                                                 {987,[{0,0}]},
                                                 {988,[{0,0}]},
                                                 {989,[{0,0}]},
                                                 {990,[{0,0}]},
                                                 {991,[{0,0}]},
                                                 {992,[{0,0}]},
                                                 {993,[{0,0}]},
                                                 {994,[{0,0}]},
                                                 {995,[{0,0}]},
                                                 {996,[{0,0}]},
                                                 {997,[{0,0}]},
                                                 {998,[{0,0}]},
                                                 {999,[{0,0}]},
                                                 {1000,[{0,0}]},
                                                 {1001,[{0,0}]},
                                                 {1002,[{0,0}]},
                                                 {1003,[{0,0}]},
                                                 {1004,[{0,0}]},
                                                 {1005,[{0,0}]},
                                                 {1006,[{0,0}]},
                                                 {1007,[{0,0}]},
                                                 {1008,[{0,0}]},
                                                 {1014,[{0,0}]},
                                                 {1015,[{0,0}]},
                                                 {1016,[{0,0}]},
                                                 {1017,[{206630118054190,0}]},
                                                 {1018,[{58851646531965,0}]},
                                                 {1019,
                                                  [{118920551874785,98345},
                                                   {103801475651071,0}]},
                                                 {1020,[{0,0}]},
                                                 {1021,[{14726742042937,0}]},
                                                 {1022,
                                                  [{210477493706492,98367},
                                                   {85820590538866,0}]},
                                                 {1023,
                                                  [{92085762404913,98351},
                                                   {277016319418037,0}]}]},
                                               main,nil,<16800.28934.86>,nil,
                                               "/ssd2/@indexes/bucket-1/main_b998c29cc4af889cf9c05b1808926d61.view.1",
                                               mapreduce_view,".view",prod,
                                               couch_set_view_stats_prod,
                                               872062976,<16800.28949.86>},
                                              nil,false,not_running,nil,nil,
                                              nil,0,[],nil,false,undefined,
                                              true,true,
                                              [171,172,173,174,175,176,177,
                                               178,179,180,181,182,183,184,
                                               185,186,187,188,189,190,191,
                                               192,193,194,195,196,197,198,
                                               199,200,201,202,203,204,205,
                                               206,207,208,209,210,211,212,
                                               213,214,215,216,217,218,219,
                                               220,221,222,223,224,225,226,
                                               227,228,229,230,231,232,233,
                                               234,235,236,237,238,239,240,
                                               241,242,243,244,245,246,247,
                                               248,249,250,251,252,253,254,
                                               255,427,428,429,430,431,432,
                                               433,434,435,436,437,438,439,
                                               440,441,442,443,444,445,446,
                                               447,448,449,450,451,452,453,
                                               454,455,456,457,458,459,460,
                                               461,462,463,464,465,466,467,
                                               468,469,470,471,472,473,474,
                                               475,476,477,478,479,480,481,
                                               482,483,484,485,486,487,488,
                                               489,490,491,492,493,494,495,
                                               496,497,498,499,500,501,502,
                                               503,504,505,506,507,508,509,
                                               510,511,682,683,684,685,686,
                                               687,688,689,690,691,692,693,
                                               694,695,696,697,698,699,700,
                                               701,702,703,704,705,706,707,
                                               708,709,710,711,712,713,714,
                                               715,716,717,718,719,720,721,
                                               722,723,724,725,726,727,728,
                                               729,730,731,732,733,734,735,
                                               736,737,738,739,740,741,742,
                                               743,744,745,746,747,748,749,
                                               750,751,752,753,754,755,756,
                                               757,758,759,760,761,762,763,
                                               764,765,766,767],
                                              [],
                                              {dict,0,16,16,8,80,48,
                                               {[],[],[],[],[],[],[],[],[],[],
                                                [],[],[],[],[],[]},
                                               {{[],[],[],[],[],[],[],[],[],
                                                 [],[],[],[],[],[],[]}}},
                                              nil,3000}]},
                                           {gen_server,handle_msg,5},
                                           {proc_lib,init_p_do_apply,3}]},
                                         {gen_server,call,
                                          [<16800.28929.86>,
                                           {monitor_partition_update,891,
                                            #Ref<16800.0.695.30261>,
                                            <16800.10511.87>},
                                           infinity]}}}},
                                      [{capi_set_view_manager,handle_call,3},
                                       {gen_server,handle_msg,5},
                                       {gen_server,init_it,6},
                                       {proc_lib,init_p_do_apply,3}]},
                                     {gen_server,call,
                                      ['capi_set_view_manager-bucket-1',
                                       {wait_index_updated,891},
                                       infinity]}},
                                    {gen_server,call,
                                     [{'janitor_agent-bucket-1',
                                       'ns_1@172.23.96.18'},
                                      {if_rebalance,<0.15317.163>,
                                       {wait_checkpoint_persisted,892,2204}},
                                      infinity]}}}}]}}}


 Comments   
Comment by Pavel Paulau [ 20/Apr/14 ]
MB-10514




[MB-10909] Crash with unaligned memory access on ARM Created: 19/Apr/14  Updated: 20/Apr/14  Resolved: 20/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: forestdb
Affects Version/s: 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Jens Alfke Assignee: Chiyoung Seo
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: iPhone 5, iOS 7.1

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
When I run an optimized build of Couchbase Lite on an iPhone, it's crashing while opening a database:

* thread #1: tid = 0x12b82, 0x000f2f62 Worker Bee`crc32_8 + 42, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_ARM_DA_ALIGN, address=0x18aaa085)
    frame #0: 0x000f2f62 Worker Bee`crc32_8 + 42
    frame #1: 0x000efb2c Worker Bee`_file_hash(hash*, hash_elem*) + 32
    frame #2: 0x000fa4c0 Worker Bee`hash_find + 12
    frame #3: 0x000efde8 Worker Bee`filemgr_open + 84
    frame #4: 0x000f45c2 Worker Bee`___lldb_unnamed_function2572$$Worker Bee + 114
    frame #5: 0x000f452c Worker Bee`fdb_open + 52

ARM requires non-byte memory accesses to be aligned, and the crash is due to a 32-bit load from an odd address.

The problem seems to be in _file_hash. It looks like it hashes only the last 8 bytes of the file path, but by counting back 8 bytes from the end of the string it's likely to start at an odd address. I'll try modifying this to align the address. (BTW, it looks like this was done for speed, but CRC32 is a fairly slow hash function. Something like murmurhash (https://code.google.com/p/smhasher/) would be a lot faster for in-memory use. CRC32 is still appropriate for use on disk.)

I have no idea why I only ran into this when using an optimized build; I've been running a debug build earlier today in the same app on the same device with no problems.

 Comments   
Comment by Jens Alfke [ 19/Apr/14 ]
http://review.couchbase.org/#/c/36040/
Comment by Chiyoung Seo [ 20/Apr/14 ]
Thanks a lot for catching a bug and good fix. That's quite interesting to know that ARM requires that constraint.

Yeah, as you mentioned, murmurhash is known to be a fast hash solution, and is also widely used in implementing a bloom filter.

The fix was merged.




[MB-10908] beam.smp RSS grows to 50GB during delta recovery causing OOM killer invocation and rebalance failure Created: 19/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: ns_server, view-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Pavel Paulau Assignee: Sarath Lakshman
Resolution: Unresolved Votes: 0
Labels: performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Builds 3.0.0-585+

Platform = Physical
OS = CentOS 6.5
CPU = Intel Xeon E5-2630
Memory = 64 GB
Disk = 2 x SSD

Attachments: PNG File beam.smp_rss_594.png     PNG File beam.smp_rss.png    
Issue Links:
Dependency
depends on MB-10472 Handle multiple snapshots properly in... Open
Gantt: start-finish
is triggered by MB-10514 During rebalance, UPR stream gets stu... Reopened
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Build 3.0.0-385
http://ci.sc.couchbase.com/job/apollo-64/1238/artifact/

Build 3.0.0-394
http://ci.sc.couchbase.com/job/apollo-64/1248/artifact/
Is this a Regression?: Yes

 Description   
Delta rebalance after failover, 3 -> 4 nodes, 1 bucket x 100M x 2KB, DGM, 1 ddoc with 1 view, 10K mixed ops/sec, 400 qps

Steps:
1. "Failover" one node.
2. Add it back.
3. Enable delta recovery mode.
4. Wait predefined time (20 minutes).
5. Trigger cluster rebalance, wait for rebalance to finish.

 Comments   
Comment by Aliaksey Artamonau [ 21/Apr/14 ]
High memory usage is due to error_logger that has to format huge messages. Initially this is happens because capi_set_view_manager receives unexpected messages while waiting for index to be updated:

[ns_server:debug,2014-04-20T20:37:18.832,ns_1@172.23.96.18:<0.17410.54>:capi_set_view_manager:do_wait_index_updated:550]Got unexpected message from ddoc monitoring. Assuming that's shutdown: {updater_error,
                                                                        {error,
                                                                         <<"New seq smaller or equal than old seq.">>,
                                                                         823,
                                                                         100427,
                                                                         100424}}

[ns_server:debug,2014-04-20T20:18:48.539,ns_1@172.23.96.18:<0.26961.49>:capi_set_view_manager:do_wait_index_updated:550]Got unexpected message from ddoc monitoring. Assuming that's shutdown: {updater_error,
                                                                        {timeout,
                                                                         {gen_server,
                                                                          call,
                                                                          [<0.18598.49>,
                                                                           {get_stream_event,
                                                                            8274}]}}}

[ns_server:debug,2014-04-20T20:21:34.273,ns_1@172.23.96.18:<0.26151.50>:capi_set_view_manager:do_wait_index_updated:550]Got unexpected message from ddoc monitoring. Assuming that's shutdown: {updater_error,
                                                                        {error,
                                                                         vbucket_stream_already_exists}}

[ns_server:debug,2014-04-20T20:22:14.930,ns_1@172.23.96.18:<0.3348.51>:capi_set_view_manager:do_wait_index_updated:550]Got unexpected message from ddoc monitoring. Assuming that's shutdown: {updater_error,
                                                                        function_clause}

This causes capi_set_view_manager to terminate. This also leads to abnormal termination of different couch_set_view* processes. Those guys tend to have huge states. And pretty printing huge terms is known memory hog in erlang. BTW, these states contain user data. So view engine guys might want to define format_status/2 function in their modules to prevent user data from being logged.

Then it gets even worse when memcached gets killed by OOM killer. As a consequence ebucketmigrator_srv's start to terminate. So their last messages get logged. And these messages are quite likely to be huge tcp packets which overwhelms the logger even more.

So I'll try to mitigate it somehow on the logger side, but I'd like view engine guys to look into what caused in the first place.
Comment by Sarath Lakshman [ 21/Apr/14 ]
Above updater_errors are related to the issue, MB-10514. It seems to be partially resolved by now (Rebalance with TAP replication is broken. Otherwise with UPR replication, it looks fine). Once MB-10514 is fixed, we can expect all memory hogging related to updater error logging to go away. As of now with UPR replication, this shouldn't be a problem. We are logging partition versions for 1024 frequently and which floods log files. We may need to reconsider partition version logging format.
Comment by Sriram Melkote [ 21/Apr/14 ]
Hi Sarath, let's fix this irrespective of MB-10514, perhaps by adding format_status as Aliaksey suggests, which can be the resolution to this issue.
Comment by Aliaksey Artamonau [ 21/Apr/14 ]
Let me be clear. In this particular case it's not about what you log deliberately. It's about what gen_server logs when the process crashes. I found that in one case logging couch_set_view_group's state resulted in almost 1 gigabyte of log output because of all the documents in the state
Comment by Aliaksey Artamonau [ 21/Apr/14 ]
Merged few fixes on the logger side:

http://review.couchbase.org/#/c/36115/
http://review.couchbase.org/#/c/36116/
Comment by Sarath Lakshman [ 22/Apr/14 ]
Currently we have some documents buffering code that is very inefficient. The document loader for index updater buffers entire set of documents for an UPR snapshot in a list buffer before processing. I am hoping that might be the major process state that has size > 1Gig. Volker is working on fixing this problem (MB-10472) and avoiding buffering of documents. No other processes seem to have large in-memory state. I will take a detailed look if any other modules have large states and whether logging can be restricted by format_status().
Comment by Sarath Lakshman [ 22/Apr/14 ]
Thanks Aliaksey. The fix for avoiding logging larger objects will help us for now.
I have a bit of confusion about you mentioned above that couch_set_view_group's state was significantly larger and could see all documents in it. I doubt if it is couch_set_view_group's state. We do not keep any list of documents in couch_set_view_group's state. We have a non-genserver process which keeps a large buffer of documents. Probably it is that process. Is there a way to trim data structures held by a non genserver process being logged ?
Comment by Sarath Lakshman [ 22/Apr/14 ]
Pavel, I think performance testing without MB-10472 doesn't make sense. Also, if you want to run tests, note that you should turn on UPR for replication. Above updater errors and crash logging is due to TAP replication related problem.
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
I double checked and the process I talked about is indeed not a couch_set_view_group. It seems to be a couch_upr_client:

[error_logger:error,2014-04-20T20:43:17.014,ns_1@172.23.96.18:error_logger<0.6.0>:ale_error_logger_handler:log_msg:119]** Generic server <0.9849.55> terminating
** Last message in was {'EXIT',<0.9807.55>,
                        {function_clause,
                         [{couch_set_view_group,handle_info,
                           [{'DOWN',#Ref<0.0.585.159426>,process,<0.4726.64>,
                             normal},

    exception exit: {updater_died,
                        {updater_error,
                            {timeout,
                                {gen_server,call,
                                    [<0.9849.55>,{get_stream_event,3124}]}}}}

And it is a gen_server.
Comment by Sarath Lakshman [ 22/Apr/14 ]
Thanks Aliaksey.
Ideally couch_upr_client should throttle at 10MB. I will have a detailed look.
Comment by Sarath Lakshman [ 23/Apr/14 ]
The couch_upr_client gen_server never crashed. Call to couch_upr_client timed out. Even if the couch_upr_client crashes, its state should not have more than 10MB worth of documents. So I think it is something else. Aliaksey, It would be great if you could upload the logs that has the user documents logged.
Comment by Sriram Melkote [ 23/Apr/14 ]
Reducing to critical, as with the current limit on logger side, this should be not so bad - will keep this open however, so we fix the root cause
Comment by Aliaksey Artamonau [ 23/Apr/14 ]
The excerpt I posted above was from Pavel's logs.
Comment by Aliaksey Artamonau [ 23/Apr/14 ]
And that timeout on get_stream_event cass was unrelated to crash. I just added it to show that crashed gen_server was indeed couch_upr_client.




[MB-10907] Missing UPR config :: UUID difference observed in (active vs replica) vbuckets after online upgrade 2.5.1 ==> 3.0.0-593 Created: 19/Apr/14  Updated: 22/Apr/14

Status: Reopened
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Aliaksey Artamonau
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GZip Archive online_upgrade_logs_2.5.1_3.0.0.tar.gz    
Triage: Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump: 1. Install 2.5.1 on 10.6.2.144, 10.6.2.145
2. Add 10.6.2.145 to 10.6.2.144 and rebalance
3. Add default bucket with 1024 vbuckets to cluster
4. Add ~ 1000 items to the buckets
5. Online upgrade cluster to 3.0.0-593 with 10.6.2.146 as our extra node
6. Finally cluster has node 10.6.2.145, 10.6.2.146

Check V-bucket id for active and replica v-bucket, after all replication is complete, plus disk queue drained.

Expectation: Should be same according to UPR

Actual Result: Different as per observation. Without an upgrade they are same.

Note that: In the build we tested UPR was not turned due to missing COUCHBASE_REPL_TYPE = upr and we operating with TAP. This case will occur during upgrade and we have to fix the config before upgrade.

Example of difference in UUID

On 10.6.2.145 where vb_9 is active
 vb_9:high_seqno: 14

 vb_9:purge_seqno: 0

 vb_9:uuid: 18881518640852

On 10.6.2.146 where vb_9 is replica
 vb_9:high_seqno: 14

 vb_9:purge_seqno: 0

 vb_9:uuid: 120602843033209





Is this a Regression?: No

 Comments   
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
I can't find any evidence that there was even an attempt to upgrade replications to upr after rebalance. I assume that you forgot to set COUCHBASE_REPL_TYPE environment variable accordingly.
Comment by Parag Agarwal [ 22/Apr/14 ]
isn't UPR switched on by default? for version like 3.0.0-593 or do we need to set explicitly?
Comment by Parag Agarwal [ 22/Apr/14 ]
I will re-run the scenario by adding it in the config file and let you know the results. But I think if this is not on by default, we should have it on to avoid this scenario at least.
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
You need to set it explicitly.
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
Please also note that currently it's known that upgrade to UPR is broken: MB-10928.
Comment by Parag Agarwal [ 22/Apr/14 ]
Thanks for the update, I am going to change the scope of this bug due to the issue observed was missing COUCHBASE_REPL_TYPE = upr
Comment by Parag Agarwal [ 22/Apr/14 ]
Changing scope of the bug as per comments from the dev. We need to fix the config of our install package to switch on UPR by default.




[MB-10906] CBTransfer in CSV mode for backup data acts like STDOUT Created: 18/Apr/14  Updated: 21/Apr/14  Resolved: 21/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: tools
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Minor
Reporter: Parag Agarwal Assignee: Bin Cui
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Scenario

I was trying to take csv format output for backup file, it was trying to give stdout and not save in a file. In my case the file /tmp/data.txt did not get created. The only work around was to redirect the output.

[root@palm-10307 bin]# ./cbtransfer /tmp/bucket/ csv:/tmp/data.txt
id,flags,expiration,cas,value,rev,vbid
ddd,0,0,3358190305579503,"{""click"":""to edit"",""new in 2.0"":""there are no reserved field names""}",49,17
  [####################] 100.0% (1/estimated 1 msgs)
bucket: default, msgs transferred...
       : total | last | per sec
 byte : 68 | 68 | 1028.6



 Comments   
Comment by Bin Cui [ 21/Apr/14 ]
If you specify csv:// with filename suffix with .csv, cbtransfer will generate csv file. If you specify cvs:// without .csv, cbtransfer will generate csv but write to stdout. csv://file.txt falls into this category.
Comment by Parag Agarwal [ 21/Apr/14 ]
Explanation sounds reasonable.




[MB-10905] memcached cannot load libraries on mac when run as install/bin/couchbase-server Created: 18/Apr/14  Updated: 21/Apr/14

Status: Open
Project: Couchbase Server
Component/s: build, storage-engine
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Artem Stemkovski Assignee: Phil Labee
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
though it works without any issues if run as ns_server/cluster_run

=========================PROGRESS REPORT=========================
          supervisor: {local,ns_child_ports_sup}
             started: [{pid,<0.272.0>},
                       {name,
                           {memcached,
                               "/Users/artem/Work/couchbase/install/bin/memcached",
                               ["-C",
                                "/Users/artem/Work/couchbase/install/var/lib/couchbase/config/memcached.json"],
                               [{env,
                                    [{"EVENT_NOSELECT","1"},
                                     {"MEMCACHED_TOP_KEYS","100"},
                                     {"ISASL_PWFILE",
                                      "/Users/artem/Work/couchbase/install/var/lib/couchbase/isasl.pw"}]},
                                use_stdio,stderr_to_stdout,exit_status,
                                port_server_send_eol,stream],
                               [{"/Users/artem/Work/couchbase/install/var/lib/couchbase/config/memcached.json",
                                 <<"{\"interfaces\":[{\"host\":\"*\",\"port\":11210,\"maxconn\":10000},{\"host\":\"*\",\"port\":11209,\"maxconn\":1000},{\"host\":\"*\",\"port\":11207,\"maxconn\":10000,\"ssl\":{\"key\":\"/Users/artem/Work/couchbase/install/var/lib/couchbase/config/memcached-key.pem\",\"cert\":\"/Users/artem/Work/couchbase/install/var/lib/couchbase/config/memcached-cert.pem\"}}],\"extensions\":[{\"module\":\"/Users/artem/Work/couchbase/install/lib/memcached/stdin_term_handler.so\",\"config\":\"\"},{\"module\":\"/Users/artem/Work/couchbase/install/lib/memcached/file_logger.so\",\"config\":\"cyclesize=10485760;sleeptime=19;filename=/Users/artem/Work/couchbase/install/var/lib/couchbase/logs/memcached.log\"}],\"engine\":{\"module\":\"/Users/artem/Work/couchbase/install/lib/memcached/bucket_engine.so\",\"config\":\"admin=_admin;default_bucket_name=default;auto_create=false\"},\"verbosity\":0}">>}]}},
                       {mfargs,
                           {supervisor_cushion,start_link,
                               [memcached,5000,infinity,ns_port_server,
                                start_link,
                                [#Fun<ns_child_ports_sup.4.41629644>]]}},
                       {restart_type,permanent},
                       {shutdown,86400000},
                       {child_type,worker}]

[ns_server:info,2014-04-18T13:26:43.754,babysitter_of_ns_1@127.0.0.1:<0.273.0>:ns_port_server:log:169]memcached<0.273.0>: dyld: Library not loaded: libJSON_checker.dylib
memcached<0.273.0>: Referenced from: /Users/artem/Work/couchbase/install/bin/memcached
memcached<0.273.0>: Reason: image not found





[MB-10904] CBTransfer can read replica from couchstore and http:// Created: 18/Apr/14  Updated: 23/Apr/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Parag Agarwal Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
Currently we can cbtansfer for replica only from sqllite files (i.e. from backup). However, the following command does not work (for couchstore and http://)

Example of couchstore not working for replica

[root@palm-10307 bin]# /opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data csv:/tmp/default.5838520c-c744-11e3-99de-6003089eed5e.csv -b default -u Administrator -p password --source-vbucket-state=replica --destination-vbucket-state=replica -i 59
error: only --source-vbucket-state=active is supported by this source: couchstore-files:///opt/couchbase/var/lib/couchbase/data


This request is to improve the cbtransfer to support http:// and couchstore files as well. Would be great to have it for data comparison analysis for our testing.


 Comments   
Comment by Bin Cui [ 21/Apr/14 ]
http://review.couchbase.org/#/c/36123/

We will allow to read nonactive data out of couchstore files, like we do for sqlite files. However, we won't be able tro read replica from http:// or from cluster directly because ep-engine will block such requests.




[MB-10903] [Doc] Global I/O Manager Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Global I/O Manager

http://www.couchbase.com/issues/browse/MB-9036






[MB-10902] [Doc] Progress indicator for Warm-up Operation Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Progress indicator for Warm-up Operation -

http://www.couchbase.com/issues/browse/MB-8989




[MB-10901] [Doc] Tunable Memory - Optional removal of keys & metadata from memory Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Microsoft Word TunableMemory_TestPlan_1.1.docx     Microsoft Word TunableMemory_TestPlan.docx    
Flagged:
Release Note

 Description   
Tunable Memory - Optional removal of keys & metadata from memory

http://www.couchbase.com/issues/browse/CBD-1034

MB-10151




[MB-10900] [Doc] Bucket Priority for Disk I/O optimization Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Bucket Priority for Disk I/O optimization

http://www.couchbase.com/issues/browse/MB-10369
http://www.couchbase.com/issues/browse/MB-10307
http://www.couchbase.com/issues/browse/MB-10849




[MB-10899] [Doc] Support immediate and eventual consistency level for indexes (stale=false) Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Support immediate and eventual consistency level for indexes (stale=false)






[MB-10898] [Doc] Password encryption between Client and Server for Admin ports credentials Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Password encryption between Client and Server for Admin ports credentials

http://www.couchbase.com/issues/browse/MB-10088
http://www.couchbase.com/issues/browse/MB-9198




[MB-10897] [Doc] Logging/serviceability improvements Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Logging/serviceability improvements

http://www.couchbase.com/issues/browse/MB-10088
http://www.couchbase.com/issues/browse/MB-9198
http://www.couchbase.com/issues/browse/MB-10085




[MB-10896] [Doc] Unified Protocol for Replication (UPR) Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Unified Protocol for Replication (UPR)


 Comments   
Comment by Matt Ingenthron [ 18/Apr/14 ]
The description and the summary here don't seem to match. Is this about documenting UPR, or incremental backup?

Note that UPR is, to my knowledge, a private interface in 3.0. This means it should be documented still, but perhaps in a different way.
Comment by Anil Kumar [ 18/Apr/14 ]
This is for UPR. You're right it is a private interface and we'll have concepts, stats etc documented. If you have any suggestion please let us know.




[MB-10895] [Doc] Incremental backup and restore (Differential, Cumulative) Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Incremental backup and restore (Differential, Cumulative).

http://www.couchbase.com/issues/browse/MB-10176





[MB-10894] [Doc] Data encryption between client and server for bucket data - couchbase buckets Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Data encryption between client and server for bucket data - couchbase buckets.

https://www.couchbase.com/issues/browse/MB-10082
https://www.couchbase.com/issues/browse/MB-10083
https://www.couchbase.com/issues/browse/MB-10084




[MB-10893] [Doc] XDCR - pause and resume Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
XDCR - pause and resume

https://www.couchbase.com/issues/browse/MB-5487




[MB-10891] [Doc] Gracefully failover - Node is in Repair mode Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Gracefully failover - Node is in Repair mode.

ns_server - https://www.couchbase.com/issues/browse/MB-9980

tool - https://www.couchbase.com/issues/browse/MB-10892





[MB-10890] [Doc] Delta node recovery for failed over node Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Delta node recovery for failed over node.

ns_server - https://www.couchbase.com/issues/browse/MB-9979

ui - https://www.couchbase.com/issues/browse/MB-10150

tools - https://www.couchbase.com/issues/browse/MB-10456




[MB-10889] [Doc] Index building without persisting to disk on source [using UPR] Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Index building without persisting to disk on source [using UPR] .

http://www.couchbase.com/issues/browse/MB-8903.




[MB-10888] [Doc] XDCR replication without persisting to disk on source Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Ruth Harris
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
XDCR replication without persisting to disk on source.

http://www.couchbase.com/issues/browse/MB-9981
https://www.couchbase.com/issues/browse/MB-10400






[MB-10887] [Doc] Cluster-wide diagnostics gathering - collect_info from UI across cluster Created: 18/Apr/14  Updated: 18/Apr/14  Due: 23/May/14

Status: Open
Project: Couchbase Server
Component/s: documentation
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Anil Kumar Assignee: Amy Kurtzman
Resolution: Unresolved Votes: 0
Labels: 3.0-Beta
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Flagged:
Release Note

 Description   
Cluster-wide diagnostics gathering tool on Admin UI.

http://www.couchbase.com/issues/browse/MB-10086





switch to R16 for 3.0 (was: investigate possibility of switching to R16B03) (MB-9998)

[MB-10886] Windows - Confirm if it's on R16. Created: 17/Apr/14  Updated: 18/Apr/14  Due: 23/Apr/14  Resolved: 17/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Technical task Priority: Critical
Reporter: Wayne Siu Assignee: Chris Hillery
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If not,
a. Install R16 on Windows
b. update the path to pick up R16.

 Comments   
Comment by Chris Hillery [ 17/Apr/14 ]
3.0 build are using R16B03-1.




switch to R16 for 3.0 (was: investigate possibility of switching to R16B03) (MB-9998)

[MB-10885] Linux - Upgrade the manifests to pick up R16 Created: 17/Apr/14  Updated: 21/Apr/14  Due: 23/Apr/14  Resolved: 21/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Technical task Priority: Critical
Reporter: Wayne Siu Assignee: Chris Hillery
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Comments   
Comment by Chris Hillery [ 21/Apr/14 ]
http://review.couchbase.org/36100
http://review.couchbase.org/36106
http://review.couchbase.org/36112




[MB-10884] Erlang Upgrade to R16 (R16B03-1) Created: 17/Apr/14  Updated: 18/Apr/14  Due: 23/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: build
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Wayne Siu Assignee: Chris Hillery
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates MB-9998 switch to R16 for 3.0 (was: investiga... Closed

 Comments   
Comment by Sriram Melkote [ 18/Apr/14 ]
Let's track this on the older bug MB-9998 so there's only one place for R16 upgrade related information




[MB-10883] Difference in Rev id: Disk vs (Disk+Memory) Output for CBTransfer in CSV mode Created: 17/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: tools
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Bin Cui
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 6, version:: 3.0.0-587

Triage: Untriaged
Is this a Regression?: Unknown

 Description   
1. Add a default bucket to a cluster with 1 node
2. Add item to the cluster and wait till the disk queue is 0

Expectation:

The following should dump the same output

From (memory+disk)

/opt/couchbase/bin/cbtransfer http://10.6.2.144:8091 csv:/tmp/default.15f6a721-c688-11e3-8c97-6003089eed5e.csv -b default -u Administrator -p password --single-node

From (disk)

/opt/couchbase/bin/cbtransfer couchstore-files:///opt/couchbase/var/lib/couchbase/data csv:/tmp/default.16d409e6-c688-11e3-94cc-6003089eed5e.csv -b default -u Administrator -p password

See how the rev ids are different

[root@palm-10307 ~]# cat /tmp/default.15f6a721-c688-11e3-8c97-6003089eed5e.csv
id,flags,expiration,cas,value,rev,vbid
dd,0,0,3271998131903809,"{""click"":""to edit"",""new in 2.0"":""there are no reserved field names""}",49,663
[root@palm-10307 ~]# cat /tmp/default.16d409e6-c688-11e3-94cc-6003089eed5e.csv
id,flags,expiration,cas,value,rev,vbid
dd,0,0,3271998131903809,"{""click"":""to edit"",""new in 2.0"":""there are no reserved field names""}",1,663
[root@palm-10307 ~]#

Strangely for any number of items the rev id difference is 49 and 1 only.






 Comments   
Comment by Bin Cui [ 18/Apr/14 ]
http://review.couchbase.org/#/c/36024/




[MB-10882] upr_notifier should not crash on server start and on server shutdown Created: 17/Apr/14  Updated: 18/Apr/14  Resolved: 18/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Task Priority: Critical
Reporter: Artem Stemkovski Assignee: Artem Stemkovski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
cause:

on server start: it tries to create connection before memcached is ready for it

on server shutdown: after memcached closes the socket the upr_notifier exits with normal
then it gets restarted by the sup and crashes with

=========================CRASH REPORT=========================
  crasher:
    initial call: upr_proxy:init/1
    pid: <0.15704.0>
    registered_name: []
    exception exit: {{badmatch,{error,econnrefused}},
                     [{mc_replication,connect,3,
                                      [{file,"src/mc_replication.erl"},
                                       {line,27}]},
                      {upr_proxy,connect,4,
                                 [{file,"src/upr_proxy.erl"},{line,147}]},
                      {upr_proxy,init,1,
                                 [{file,"src/upr_proxy.erl"},{line,46}]},
                      {gen_server,init_it,6,
                                  [{file,"gen_server.erl"},{line,304}]},
                      {proc_lib,init_p_do_apply,3,
                                [{file,"proc_lib.erl"},{line,239}]}]}
      in function gen_server:init_it/6 (gen_server.erl, line 328)
    ancestors: ['single_bucket_sup-gamesim-sample',<0.12788.0>]
    messages: []
    links: [<0.12789.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 75113
    stack_size: 27
    reductions: 6122
  neighbours:

 Comments   
Comment by Artem Stemkovski [ 18/Apr/14 ]
http://review.couchbase.org/36023




[MB-10881] Expose UPR stats in UI Created: 17/Apr/14  Updated: 17/Apr/14

Status: Open
Project: Couchbase Server
Component/s: None
Affects Version/s: 3.0
Fix Version/s: None
Security Level: Public

Type: Task Priority: Critical
Reporter: Artem Stemkovski Assignee: Artem Stemkovski
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
- rebalance details
- graphs




[MB-10880] CLI for N1QL needs multiple lines query support Created: 17/Apr/14  Updated: 17/Apr/14

Status: Open
Project: Couchbase Server
Component/s: query-preview
Affects Version/s: 2.5.0
Fix Version/s: cbq-DP4
Security Level: Public

Type: Improvement Priority: Critical
Reporter: Ilam Siva Assignee: Gerald Sangudi
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 64-bit


 Description   
Customers like to type multi-line queries at the command prompt in CLI. We need to be able to support this basic use-case with our CLI.

SELECT fname
FROM tutorial
WHERE lname = "Smith"

is a simple example of multiple lines at command prompt. Obviously more complex queries can be typed as multi-line as well.




[MB-10879] Rebalance fails sporadically on employee dataset test (make simple-test) Created: 17/Apr/14  Updated: 23/Apr/14  Resolved: 23/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Mike Wiederhold Assignee: Sarath Lakshman
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: GZip Archive logs.tar.gz    
Issue Links:
Relates to
relates to MB-10514 During rebalance, UPR stream gets stu... Reopened
Triage: Untriaged
Is this a Regression?: Unknown

 Description   
Fails on employee dataset test with 64 vbuckets. It doesn't happen frequently. I see it most often however on centos.

 Comments   
Comment by Mike Wiederhold [ 22/Apr/14 ]
This looks view engine related. See the error message below:

[rebalance:info,2014-04-22T17:58:19.648,n_0@10.5.2.33:<0.3650.0>:ebucketmigrator_srv:process_upstream:976]TAP stream is not doing backfill
[rebalance:info,2014-04-22T17:58:19.653,n_0@10.5.2.33:<0.3650.0>:ebucketmigrator_srv:terminate:727]Skipping close ack for successfull takover

[rebalance:info,2014-04-22T17:58:19.663,n_0@10.5.2.33:<0.3654.0>:janitor_agent:set_vbucket_state:387]Doing vbucket 127 state change: {'n_2@127.0.0.1',active,undefined,undefined}
[ns_server:warn,2014-04-22T17:58:19.693,n_0@10.5.2.33:capi_set_view_manager-default<0.852.0>:capi_set_view_manager:handle_info:302]Remote server node {'capi_ddoc_replication_srv-default','n_2@127.0.0.1'} process down: killed
[ns_server:error,2014-04-22T17:58:19.716,n_0@10.5.2.33:<0.2631.0>:ns_single_vbucket_mover:spawn_and_wait:107]Got unexpected exit signal {'EXIT',<0.2692.0>,
                            {{{{badmatch,
                                {error,
                                 {{function_clause,
                                   [{couch_set_view_group,handle_info,
                                     [{'DOWN',#Ref<13706.0.0.20276>,process,
                                       <13706.1550.0>,normal},
                                      {state,
                                       {"/home/jenkins/couchbase/ns_server/data/n_2/data",
                                        <<"default">>,
                                        {set_view_group,
                                         <<223,177,167,252,146,233,92,11,210,
                                           66,6,189,181,169,123,106>>,
                                         nil,<<"default">>,
                                         <<"_design/test_view-65b578a">>,[],
                                         [{set_view,0,
                                           <<"function (doc) { if(doc.job_title !== undefined) { var myregexp = new RegExp(\"^Senior \"); if(doc.job_title.match(myregexp)){ emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}">>,
                                           undefined,
                                           {mapreduce_view,
                                            [<<"test_view-65b578a">>],
                                            nil,[],[]}}],
                                         nil,nil,
                                         {set_view_index_header,2,0,0,0,0,[],
                                          nil,[],false,[],nil,[],[]},
                                         main,nil,nil,nil,[],mapreduce_view,
                                         ".view",prod,
                                         couch_set_view_stats_prod,0,nil}},
                                       <13706.1394.0>,
                                       {set_view_group,
                                        <<223,177,167,252,146,233,92,11,210,66,
                                          6,189,181,169,123,106>>,
                                        <13706.1345.0>,<<"default">>,
                                        <<"_design/test_view-65b578a">>,[],
                                        [{set_view,0,
                                          <<"function (doc) { if(doc.job_title !== undefined) { var myregexp = new RegExp(\"^Senior \"); if(doc.job_title.match(myregexp)){ emit([doc.join_yr, doc.join_mo, doc.join_day], [doc.name, doc.email] );}}}">>,
                                          #Ref<13706.0.0.14417>,
                                          {mapreduce_view,
                                           [<<"test_view-65b578a">>],
                                           {btree,<13706.1345.0>,nil,
                                            identity,identity,
                                            #Fun<mapreduce_view.14.84993316>,
                                            #Fun<mapreduce_view.13.84993316>,
                                            7168,6144,true},
                                           [],[]}}],
                                        {btree,<13706.1345.0>,nil,identity,
                                         identity,
                                         #Fun<couch_btree.1.46073246>,
                                         #Fun<couch_set_view_group.15.12362153>,
                                         7168,6144,true},
                                        <13706.1349.0>,
                                        {set_view_index_header,2,128,
                                         9903520314283042199192993792,
                                         340282366911015600335977731165779918848,
                                         0,
                                         [{84,0},
                                          {85,0},
                                          {86,0},
                                          {87,0},
                                          {88,0},
                                          {89,0},
                                          {90,0},
                                          {91,0},
                                          {92,0},
                                          {93,0},
                                          {94,0},
                                          {95,0},
                                          {96,0},
                                          {97,0},
                                          {98,0},
                                          {99,0},
                                          {100,0},
                                          {101,0},
                                          {102,0},
                                          {103,0},
                                          {104,0},
                                          {105,0},
                                          {106,0},
                                          {107,0},
                                          {108,0},
                                          {109,0},
                                          {110,0},
                                          {111,0},
                                          {112,0},
                                          {113,0},
                                          {114,0},
                                          {115,0},
                                          {116,0},
                                          {117,0},
                                          {118,0},
                                          {119,0},
                                          {120,0},
                                          {121,0},
                                          {122,0},
                                          {123,0},
                                          {124,0},
                                          {125,0},
                                          {126,0},
                                          {127,0}],
                                         nil,
                                         [nil],
                                         true,[],nil,[],
                                         [{84,[{0,0}]},
                                          {85,[{0,0}]},
                                          {86,[{0,0}]},
                                          {87,[{0,0}]},
                                          {88,[{0,0}]},
                                          {89,[{0,0}]},
                                          {90,[{0,0}]},
                                          {91,[{0,0}]},
                                          {92,[{0,0}]},
                                          {93,[{0,0}]},
                                          {94,[{0,0}]},
                                          {95,[{0,0}]},
                                          {96,[{0,0}]},
                                          {97,[{0,0}]},
                                          {98,[{0,0}]},
                                          {99,[{0,0}]},
                                          {100,[{0,0}]},
                                          {101,[{0,0}]},
                                          {102,[{0,0}]},
                                          {103,[{0,0}]},
                                          {104,[{0,0}]},
                                          {105,[{0,0}]},
                                          {106,[{0,0}]},
                                          {107,[{0,0}]},
                                          {108,[{0,0}]},
                                          {109,[{0,0}]},
                                          {110,[{0,0}]},
                                          {111,[{0,0}]},
                                          {112,[{0,0}]},
                                          {113,[{0,0}]},
                                          {114,[{0,0}]},
                                          {115,[{0,0}]},
                                          {116,[{0,0}]},
                                          {117,[{0,0}]},
                                          {118,[{0,0}]},
                                          {119,[{0,0}]},
                                          {120,[{0,0}]},
                                          {121,[{0,0}]},
                                          {122,[{0,0}]},
                                          {123,[{0,0}]},
                                          {124,[{0,0}]},
                                          {125,[{0,0}]},
                                          {126,[{0,0}]},
                                          {127,[{0,0}]}]},
                                        main,nil,<13706.1394.0>,nil,
                                        "/home/jenkins/couchbase/ns_server/data/n_2/data/@indexes/default/main_dfb1a7fc92e95c0bd24206bdb5a97b6a.view.1",
                                        mapreduce_view,".view",prod,
                                        couch_set_view_stats_prod,188416,
                                        <13706.1350.0>},
                                       nil,false,not_running,nil,nil,nil,0,[],
                                       nil,false,undefined,true,true,[],[],
                                       {dict,0,16,16,8,80,48,
                                        {[],[],[],[],[],[],[],[],[],[],[],[],
                                         [],[],[],[]},
                                        {{[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]}}},
                                       nil,3000}],
                                     [{file,
                                       "/home/jenkins/couchbase/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},
                                      {line,990}]},
                                    {gen_server,handle_msg,5,
                                     [{file,"gen_server.erl"},{line,597}]},
                                    {proc_lib,init_p_do_apply,3,
                                     [{file,"proc_lib.erl"},{line,227}]}]},
                                  {gen_server,call,
                                   [<13706.1342.0>,
                                    {monitor_partition_update,85,
                                     #Ref<13706.0.0.53221>,<13706.2594.0>},
                                    infinity]}}}},
                               [{capi_set_view_manager,handle_call,3,
                                 [{file,"src/capi_set_view_manager.erl"},
                                  {line,217}]},
                                {gen_server,handle_msg,5,
                                 [{file,"gen_server.erl"},{line,578}]},
                                {gen_server,init_it,6,
                                 [{file,"gen_server.erl"},{line,297}]},
                                {proc_lib,init_p_do_apply,3,
                                 [{file,"proc_lib.erl"},{line,227}]}]},
                              {gen_server,call,
                               ['capi_set_view_manager-default',
                                {wait_index_updated,103},
                                infinity]}},
                             {gen_server,call,
                              [{'janitor_agent-default','n_2@127.0.0.1'},
                               {if_rebalance,<0.2564.0>,
                                {wait_index_updated,124}},
                               infinity]}}}
Comment by Sarath Lakshman [ 23/Apr/14 ]
Is this test run with TAP replication enabled ?
Comment by Mike Wiederhold [ 23/Apr/14 ]
Yes, this is with tap replication.
Comment by Sarath Lakshman [ 23/Apr/14 ]
UPR stream still gets stuck with TAP replication (MB-10514). This is depended on MB-10514. You can see get_stream_event timeouts in the logs.
Comment by Mike Wiederhold [ 23/Apr/14 ]
Per Sarath's comments I'm closing this as a duplicate of MB-10514.




[MB-10878] Provide message on the UI that bucket will be reloaded if priority is changed at runtime. Created: 17/Apr/14  Updated: 22/Apr/14

Status: Open
Project: Couchbase Server
Component/s: UI
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Venu Uppalapati Assignee: Pavel Blagodov
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Triage: Untriaged
Is this a Regression?: No

 Description   
It is possible to change bucket priority at runtime. Provide message on the UI that bucket will be reloaded if priority is changed at runtime.

 Comments   
Comment by Venu Uppalapati [ 22/Apr/14 ]
In build # 597, I see the message warning about bucket reload, however, it is seen after I make the change in priority and save and again open the bucket properties edit window. The message is then seen every time bucket properties edit window is opened and 'save' is clicked.




[MB-10877] Some cli tools may halt with an error but exit(0) Created: 17/Apr/14  Updated: 17/Apr/14

Status: Open
Project: Couchbase Server
Component/s: tools
Affects Version/s: 2.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1, 2.5.1
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Major
Reporter: Jim Walker Assignee: Bin Cui
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 3h
Time Spent: Not Specified
Original Estimate: 3h

Issue Links:
Duplicate
Triage: Untriaged
Is this a Regression?: No

 Description   
pump_tap.py has a number of error conditions and it was noted in the field that some scripts driving cbbackup were hitting these errors but never flagging failure as we hit exit(0)

The errors affect "provide_batch" and the first check of "if self.tap_done". This check always returns 0 as the return code, however tap_done may be true because of an error.

This effects any tool which uses pump_tap.py provide_batch, certainly seen on cbbackup.

A patch is available which fixes this issue and has been tested. I'm in the process of pushing the patch to review

 Comments   
Comment by Jim Walker [ 17/Apr/14 ]
Code in review

http://review.couchbase.org/#/c/35936/
Comment by Bin Cui [ 17/Apr/14 ]
See CBSE-1090




[MB-10876] Items seems to be not getting purged from ep-engine after expiry Created: 17/Apr/14  Updated: 18/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Meenakshi Goel Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: 3.0.0-587-rel

Attachments: Text File cbstats.txt     Text File Stats.txt     Text File ViewQuery.txt    
Triage: Triaged
Operating System: Ubuntu 64-bit
Is this a Regression?: Yes

 Description   
Seems items not getting purged from ep-engine even after expiry

Jenkins Job Link:
http://qa.hq.northscale.net/job/centos_x64--44_02--replica_read_tests-P0/41/consoleFull

Test to Reproduce:
./testrunner -i /tmp/replica_read.ini get-logs=True,wait_timeout=180,GROUP=P0,get-cbcollect-info=True,get-delays=true -t newmemcapable.GetrTests.getr_test,nodes_init=4,GROUP=P0,expiration=60,wait_expiration=true,error=Not found for vbucket,descr=#simple getr replica_count=1 expiration=60 flags = 0 docs_ops=create cluster ops = None

Logs:
2014-04-16 22:43:27 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 1000 == 0 expected on '172.23.105.230:8091''172.23.105.231:8091''172.23.105.232:8091''172.23.105.245:8091', default bucket
2014-04-16 22:43:27 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 1000 == 0 expected on '172.23.105.230:8091''172.23.105.231:8091''172.23.105.232:8091''172.23.105.245:8091', default bucket
2014-04-16 22:43:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 1000 == 0 expected on '172.23.105.230:8091''172.23.105.231:8091''172.23.105.232:8091''172.23.105.245:8091', default bucket
2014-04-16 22:43:28 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items_tot 2000 == 0 expected on '172.23.105.230:8091''172.23.105.231:8091''172.23.105.232:8091''172.23.105.245:8091', default bucket

/opt/couchbase/bin/couch_dbinfo master.couch.1
DB Info (master.couch.1) - header at 12288
   file format version: 11
   update_seq: 3
   no documents
   B-tree size: 116 bytes
   total disk size: 12.1 kB

cbstats:
 /opt/couchbase/bin/cbstats 172.23.105.230:11210 all | grep items
 curr_items: 249
 curr_items_tot: 510
 curr_temp_items: 0
 ep_access_scanner_num_items: 0
 ep_chk_max_items: 5000
 ep_diskqueue_items: 0
 ep_items_rm_from_checkpoints: 1038
 ep_total_del_items: 0
 ep_total_new_items: 510
 ep_uncommitted_items: 0
 ep_warmup_min_items_threshold: 100
 vb_active_curr_items: 249
 vb_pending_curr_items: 0
 vb_replica_curr_items: 261

Notes:
Please refer attached Stats.txt
Created View and attaching ViewQuery Logs if of any help.
Uploading logs

 Comments   
Comment by Meenakshi Goel [ 17/Apr/14 ]
https://s3.amazonaws.com/bugdb/jira/MB-10876/7f6c45fe/172.23.105.230-011.zip
https://s3.amazonaws.com/bugdb/jira/MB-10876/0d3b37e0/172.23.105.231-012.zip
https://s3.amazonaws.com/bugdb/jira/MB-10876/e0c9457f/172.23.105.232-012.zip
https://s3.amazonaws.com/bugdb/jira/MB-10876/13f68e9c/172.23.105.245-014.zip




[MB-10875] Flusher queue doesn't get flushed Created: 16/Apr/14  Updated: 23/Apr/14

Status: Reopened
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Aruna Piravi Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: ep-engine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centOS, 64 bit , build 585, seen in builds as old as 555.

Attachments: Zip Archive 10.3.4.186-4162014-1742-diag.zip     Zip Archive 10.3.4.187-4162014-1744-diag.zip     Zip Archive 10.3.4.188-4162014-1753-diag.zip     Zip Archive 10.3.4.189-4162014-1755-diag.zip     PNG File Disk_Write.png     PNG File Screen Shot 2014-04-16 at 5.11.17 PM.png    
Issue Links:
Duplicate
is duplicated by MB-10824 1 replica item gets stuck in disk wri... Closed
is duplicated by MB-10911 TAP UniXDCR (xmem mode), time taken b... Closed
Triage: Triaged
Is this a Regression?: Yes

 Description   
Scenario
--------------
- flusher queue always has a few items and never goes to zero even after minutes of waiting
- seen both on cluster_run and servers on vms
- reproducible consistently with
./testrunner -i cluster_run.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=1000,rdirection=unidirection,ctopology=chain,doc-ops=delete-delete
- seen with 1000, 100 items
- not seen in some cases like
./testrunner -i cluster_run.ini -t xdcr.uniXDCR.unidirectional.load_with_ops,replicas=1,items=10000,value_size=128,ctopology=chain,rdirection=unidirection,doc-ops=update-delete
- consistently seen after warmup
- all tests that wait for final verification after drain queue size becomes 0 are timing out.

Setup
---------
Source cluster : 10.3.4.186, 10.3.4.187
Destination : 10.3.4.188, 10.3.4.189

setup live at 10.3.4.187:8091 , default login credentials for SSH and GUI.


GDB info on .187
----------------------------
Thread 13 (Thread 0x7effd79ad700 (LWP 4381)):
#0 0x00007effd904174d in read () from /lib64/libc.so.6
#1 0x00007effd8fd7fe8 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007effd8fd9aee in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007effd8fce1ca in _IO_getline_info_internal () from /lib64/libc.so.6
#4 0x00007effd8fcd029 in fgets () from /lib64/libc.so.6
#5 0x00007effd79ae8b1 in check_stdin_thread (arg=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#6 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4070) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#7 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00007effd904eb6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7effd6d98700 (LWP 4382)):
#0 0x00007effd9eb198e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007effdb0f78eb in cb_cond_timedwait (cond=0x7effd6face60, mutex=0x7effd6face20, ms=<value optimized out>) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007effd6d9c548 in logger_thead_main (arg=0x2a14b00) at /home/buildbot/centos-5-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4080) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007effd904eb6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7effd618a700 (LWP 4383)):
#0 0x00007effd904f163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007effda68e376 in epoll_dispatch (base=0xbd60500, tv=<value optimized out>) at epoll.c:404
#2 0x00007effda679c44 in event_base_loop (base=0xbd60500, flags=<value optimized out>) at event.c:1558
#3 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4190) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007effd904eb6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7effd5789700 (LWP 4384)):
#0 0x00007effd904f163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007effda68e376 in epoll_dispatch (base=0xbd60280, tv=<value optimized out>) at epoll.c:404
#2 0x00007effda679c44 in event_base_loop (base=0xbd60280, flags=<value optimized out>) at event.c:1558
#3 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4180) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007effd904eb6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7effd4d88700 (LWP 4385)):
#0 0x00007effd904f163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007effda68e376 in epoll_dispatch (base=0xbd60c80, tv=<value optimized out>) at epoll.c:404
#2 0x00007effda679c44 in event_base_loop (base=0xbd60c80, flags=<value optimized out>) at event.c:1558
#3 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4170) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007effd904eb6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7effd4387700 (LWP 4386)):
#0 0x00007effd904f163 in epoll_wait () from /lib64/libc.so.6
#1 0x00007effda68e376 in epoll_dispatch (base=0xbd60a00, tv=<value optimized out>) at epoll.c:404
#2 0x00007effda679c44 in event_base_loop (base=0xbd60a00, flags=<value optimized out>) at event.c:1558
#3 0x00007effdb0f7b6f in platform_thread_wrap (arg=0x29d4160) at /home/buildbot/centos-5-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007effd9ead9d1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007effd904eb6d in clone () from /lib64/libc.so.6


[root@centos-64-x64 ~]# /opt/couchbase/bin/cbstats localhost:11210 all
 accepting_conns: 1
 auth_cmds: 6
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 13
 bytes: 35154528
 bytes_read: 1277423
 bytes_written: 300104578
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 0
 cmd_set: 500
 conn_yields: 85
 connection_structures: 10500
 curr_connections: 17
 curr_conns_on_port_11207: 2
 curr_conns_on_port_11209: 12
 curr_conns_on_port_11210: 3
 curr_items: 350
 curr_items_tot: 700
 curr_temp_items: 0
 daemon_connections: 6
 decr_hits: 0
 decr_misses: 0
 delete_hits: 150
 delete_misses: 0
 ep_access_scanner_last_runtime: 0
 ep_access_scanner_num_items: 0
 ep_access_scanner_task_time: 2014-04-17 23:30:57
 ep_allow_data_loss_during_shutdown: 1
 ep_alog_block_size: 4096
 ep_alog_path: /opt/couchbase/var/lib/couchbase/data/default/access.log
 ep_alog_sleep_time: 1440
 ep_alog_task_time: 10
 ep_backend: couchdb
 ep_bg_fetch_delay: 0
 ep_bg_fetched: 0
 ep_bg_meta_fetched: 0
 ep_bg_remaining_jobs: 0
 ep_bucket_priority: LOW
 ep_chk_max_items: 5000
 ep_chk_period: 1800
 ep_chk_persistence_remains: 0
 ep_chk_persistence_timeout: 10
 ep_chk_remover_stime: 5
 ep_commit_num: 2378
 ep_commit_time: 0
 ep_commit_time_total: 5599
 ep_config_file:
 ep_conflict_resolution_type: seqno
 ep_couch_bucket: default
 ep_couch_host: 127.0.0.1
 ep_couch_port: 11213
 ep_couch_reconnect_sleeptime: 250
 ep_couch_response_timeout: 180000
 ep_data_traffic_enabled: 0
 ep_db_data_size: 274320
 ep_db_file_size: 16666010
 ep_dbname: /opt/couchbase/var/lib/couchbase/data/default
 ep_degraded_mode: 0
 ep_diskqueue_drain: 1272
 ep_diskqueue_fill: 1290
 ep_diskqueue_items: 18
 ep_diskqueue_memory: 1296
 ep_diskqueue_pending: 9036
 ep_exp_pager_stime: 3600
 ep_expired_access: 0
 ep_expired_pager: 0
 ep_failpartialwarmup: 0
 ep_flush_all: false
 ep_flush_duration_total: 8
 ep_flushall_enabled: 0
 ep_flusher_state: running
 ep_flusher_todo: 0
 ep_getl_default_timeout: 15
 ep_getl_max_timeout: 30
 ep_ht_locks: 5
 ep_ht_size: 3079
 ep_initfile:
 ep_io_num_read: 0
 ep_io_num_write: 1265
 ep_io_read_bytes: 0
 ep_io_write_bytes: 261510
 ep_item_begin_failed: 0
 ep_item_commit_failed: 0
 ep_item_eviction_policy: value_only
 ep_item_flush_expired: 0
 ep_item_flush_failed: 0
 ep_item_num_based_new_chk: 1
 ep_items_rm_from_checkpoints: 2402
 ep_keep_closed_chks: 0
 ep_kv_size: 235440
 ep_max_bg_remaining_jobs: 0
 ep_max_checkpoints: 2
 ep_max_failover_entries: 25
 ep_max_item_size: 20971520
 ep_max_num_shards: 4
 ep_max_num_workers: 3
 ep_max_size: 2169503744
 ep_max_threads: 0
 ep_max_vbuckets: 1024
 ep_mem_high_wat: 1844078182
 ep_mem_low_wat: 1627127808
 ep_mem_tracker_enabled: true
 ep_meta_data_memory: 46090
 ep_mlog_compactor_runs: 0
 ep_mutation_mem_threshold: 95
 ep_num_access_scanner_runs: 0
 ep_num_eject_failures: 0
 ep_num_expiry_pager_runs: 2
 ep_num_non_resident: 0
 ep_num_not_my_vbuckets: 0
 ep_num_ops_del_meta: 0
 ep_num_ops_del_meta_res_fail: 0
 ep_num_ops_del_ret_meta: 0
 ep_num_ops_get_meta: 0
 ep_num_ops_get_meta_on_set_meta: 0
 ep_num_ops_set_meta: 0
 ep_num_ops_set_meta_res_fail: 0
 ep_num_ops_set_ret_meta: 0
 ep_num_pager_runs: 0
 ep_num_value_ejects: 0
 ep_num_workers: 4
 ep_oom_errors: 0
 ep_overhead: 27466374
 ep_pager_active_vb_pcnt: 40
 ep_pending_compactions: 0
 ep_pending_ops: 0
 ep_pending_ops_max: 0
 ep_pending_ops_max_duration: 0
 ep_pending_ops_total: 0
 ep_postInitfile:
 ep_queue_size: 18
 ep_rollback_count: 0
 ep_startup_time: 1397691056
 ep_storage_age: 0
 ep_storage_age_highwat: 1
 ep_tap_ack_grace_period: 300
 ep_tap_ack_initial_sequence_number: 1
 ep_tap_ack_interval: 1000
 ep_tap_ack_window_size: 10
 ep_tap_backfill_resident: 0.9
 ep_tap_backlog_limit: 5000
 ep_tap_backoff_period: 5
 ep_tap_bg_fetch_requeued: 0
 ep_tap_bg_fetched: 0
 ep_tap_bg_max_pending: 500
 ep_tap_keepalive: 300
 ep_tap_noop_interval: 20
 ep_tap_requeue_sleep_time: 0.1
 ep_tap_throttle_cap_pcnt: 10
 ep_tap_throttle_queue_cap: 1000000
 ep_tap_throttle_threshold: 90
 ep_tmp_oom_errors: 0
 ep_total_cache_size: 226690
 ep_total_del_items: 265
 ep_total_enqueued: 1290
 ep_total_new_items: 965
 ep_total_persisted: 1230
 ep_uncommitted_items: 0
 ep_uuid: 28b4c2f6d709668a060c9c6489f4003d
 ep_value_size: 189350
 ep_vb0: 0
 ep_vb_snapshot_total: 1704
 ep_vb_total: 1024
 ep_vbucket_del: 512
 ep_vbucket_del_avg_walltime: 67098
 ep_vbucket_del_fail: 0
 ep_vbucket_del_max_walltime: 2666206
 ep_version: 2.1.1r-602-g0e4754a
 ep_waitforwarmup: 0
 ep_warmup: 1
 ep_warmup_batch_size: 1000
 ep_warmup_dups: 0
 ep_warmup_min_items_threshold: 100
 ep_warmup_min_memory_threshold: 100
 ep_warmup_oom: 0
 ep_warmup_thread: complete
 ep_warmup_time: 461934
 get_hits: 0
 get_misses: 0
 incr_hits: 0
 incr_misses: 0
 libevent: 2.0.11-stable
 listen_disabled_num: 0
 max_conns_on_port_11207: 10000
 max_conns_on_port_11209: 1000
 max_conns_on_port_11210: 10000
 mem_used: 35154528
 memcached_version: 2.0.1-macosx-171-g493f088
 pid: 4380
 pointer_size: 64
 rejected_conns: 0
 rusage_system: 265.075702
 rusage_user: 860.965113
 tap_checkpoint_end_received: 335
 tap_checkpoint_end_sent: 322
 tap_checkpoint_start_received: 847
 tap_checkpoint_start_sent: 1346
 tap_connect_received: 2
 tap_delete_received: 150
 tap_delete_sent: 150
 tap_mutation_received: 500
 tap_mutation_sent: 500
 tap_opaque_received: 1026
 tap_opaque_sent: 1028
 threads: 4
 time: 1397698341
 total_connections: 99
 uptime: 7316
 vb_active_curr_items: 350
 vb_active_eject: 0
 vb_active_expired: 0
 vb_active_ht_memory: 12849152
 vb_active_itm_memory: 113345
 vb_active_meta_data_memory: 23045
 vb_active_num: 512
 vb_active_num_non_resident: 0
 vb_active_ops_create: 468
 vb_active_ops_delete: 118
 vb_active_ops_reject: 0
 vb_active_ops_update: 0
 vb_active_perc_mem_resident: 100
 vb_active_queue_age: 115921000
 vb_active_queue_drain: 625
 vb_active_queue_fill: 641
 vb_active_queue_memory: 1152
 vb_active_queue_pending: 8062
 vb_active_queue_size: 16
 vb_dead_num: 0
 vb_pending_curr_items: 0
 vb_pending_eject: 0
 vb_pending_expired: 0
 vb_pending_ht_memory: 0
 vb_pending_itm_memory: 0
 vb_pending_meta_data_memory: 0
 vb_pending_num: 0
 vb_pending_num_non_resident: 0
 vb_pending_ops_create: 0
 vb_pending_ops_delete: 0
 vb_pending_ops_reject: 0
 vb_pending_ops_update: 0
 vb_pending_perc_mem_resident: 0
 vb_pending_queue_age: 0
 vb_pending_queue_drain: 0
 vb_pending_queue_fill: 0
 vb_pending_queue_memory: 0
 vb_pending_queue_pending: 0
 vb_pending_queue_size: 0
 vb_replica_curr_items: 350
 vb_replica_eject: 0
 vb_replica_expired: 0
 vb_replica_ht_memory: 12849152
 vb_replica_itm_memory: 113345
 vb_replica_meta_data_memory: 23045
 vb_replica_num: 512
 vb_replica_num_non_resident: 0
 vb_replica_ops_create: 497
 vb_replica_ops_delete: 147
 vb_replica_ops_reject: 0
 vb_replica_ops_update: 0
 vb_replica_perc_mem_resident: 100
 vb_replica_queue_age: 14490000
 vb_replica_queue_drain: 647
 vb_replica_queue_fill: 649
 vb_replica_queue_memory: 144
 vb_replica_queue_pending: 974
 vb_replica_queue_size: 2
 version: 3.0.0-585-rel


Attaching cbcollect info

 Comments   
Comment by Pavel Paulau [ 16/Apr/14 ]
MB-10824?
Comment by Aruna Piravi [ 16/Apr/14 ]
Yeah, looks like it.
Comment by Aruna Piravi [ 17/Apr/14 ]
I don't mind this being closed as duplicate if that's what it looks like to Sundar too. But I have the system running if either Sundar or Abhinav wants to take a look at. Either way pls consider this issue or 10824 to be test blocker since most of our tests are failing consistently.
Comment by Pavel Paulau [ 17/Apr/14 ]
Likewise I have live cluster with this issue.)

Agree with blocker status as well.

So up to Sundar/Abhinav...

Comment by Pavel Paulau [ 17/Apr/14 ]
I closed my ticket as duplicate.
It's too expensive to keep my cluster for debugging.
Comment by Sundar Sridharan [ 17/Apr/14 ]
Looks like this could be a symptom of MB-10539
Can we verify this with a build having fix for above?
thanks
Comment by Pavel Paulau [ 17/Apr/14 ]
It happened to me in build with fixes from MB-10539.
Comment by Aruna Piravi [ 17/Apr/14 ]
This test was on 3.0.0-585 that contains this fix - http://builder.hq.couchbase.com/#/buildinfo/couchbase-server-enterprise_x86_64_3.0.0-585-rel.rpm
Comment by Sundar Sridharan [ 17/Apr/14 ]
We found this to be a stat only issue. It can be seen that even when disk queue does not go back down, the disk updates per second does show movement indicating that items are getting flushed to disk. thanks
Comment by Meenakshi Goel [ 18/Apr/14 ]
Seen likewise failures during CAS value manipulation tests, 1 item getting stuck in disk write queue.
Build Tested - 3.0.0-590-rel, Please refer attached screenshot "Disk_Write.png".
http://qa.hq.northscale.net/job/centos_x64--28_01--cas-P1/17/console
Comment by Abhinav Dangeti [ 22/Apr/14 ]
http://review.couchbase.org/#/c/36154/
Comment by Abhinav Dangeti [ 22/Apr/14 ]
Fix has been merged.
Comment by Sundar Sridharan [ 22/Apr/14 ]
Fix is merged, assigning to Aruna as per Maria's instructions.
Comment by Pavel Paulau [ 23/Apr/14 ]
The problem is still there:

# cat /opt/couchbase/VERSION.txt
3.0.0-601-rel

# grep ep-engine /opt/couchbase/manifest.xml
  <project name="ep-engine" path="ep-engine" revision="d5a9a2620c0c46e400c3e3b797f7e540fd24f731"/>

ep_queue_size: 608

Live cluster: 172.23.96.15 Administrator:password, root:couchbase
Comment by Aruna Piravi [ 23/Apr/14 ]
+1 for reopening. I'm seeing this issue too on 603-rel.

2014-04-23 07:53:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.45:11210 default
2014-04-23 07:53:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 5519 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:53:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.46:11210 default
2014-04-23 07:53:42 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 4885 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:53:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:53:47 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:53:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:53:52 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:53:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:53:57 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:54:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:54:02 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:54:07 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:54:07 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:54:12 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:54:12 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:54:17 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:54:17 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
2014-04-23 07:54:22 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 832 == 0 expected on '172.23.106.45:8091', default bucket
2014-04-23 07:54:22 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: ep_queue_size 605 == 0 expected on '172.23.106.46:8091', default bucket
Comment by Aruna Piravi [ 23/Apr/14 ]
Try:

./testrunner -i /tmp/XDCR-sanity.ini -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=10000,rdirection=unidirection,ctopology=chain,doc-ops=delete-delete,replication_type=xmem




[MB-10874] before we can kill mccouch ep-engine needs to start doing cleanup of stale vbucket revisions Created: 16/Apr/14  Updated: 17/Apr/14

Status: Open
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Task Priority: Critical
Reporter: Aleksey Kondratenko Assignee: Chiyoung Seo
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
If compaction failed in the middle it's possible that some older revisions of vbuckets will be on disk. E.g. 0.couch.1 and 0.couch.2. As well as 0.couch.1.compact.

Couchdb has the code that opens latest revision and deletes unfinished .compact files as well as old vbucket revisions.

Before we will be able to kill mccouch we will need to duplicate this logic somewhere in ep-engine.

Corresponding couchdb code is here: https://github.com/couchbase/couchdb/blob/44734ef03daa158c0ebcac842b9e1f228e943054/src/couchdb/couch_db.erl#L83 and here: https://github.com/couchbase/couchdb/blob/44734ef03daa158c0ebcac842b9e1f228e943054/src/couchdb/couch_db_updater.erl#L45

 Comments   
Comment by Chiyoung Seo [ 17/Apr/14 ]
We will work on this with a high priority soon.




[MB-10873] UPR :: Vbucket UUID changes during Graceful failover with Add-back of Node via (full/delta) recovery Created: 16/Apr/14  Updated: 22/Apr/14  Resolved: 22/Apr/14

Status: Resolved
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Aliaksey Artamonau
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos6, version 3.0.0-584

Attachments: GZip Archive graceful_delta.tar.gz     GZip Archive graceful_full.tar.gz     Text File scenario_information_failover_addback_delta_recovery.txt     Text File scenario_information_failover_addback_full_recovery.txt    
Triage: Untriaged
Operating System: Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Will attached asap
Is this a Regression?: Unknown

 Description   
Scenario running with vbuckets=32 per bucket, replica=1, running UPR

1. Add 5 nodes
2. Add 1 default buckets with replica=1
3. Add 10 items
4. Graceful Failover 1 node
5. Add Back 1 node with delta Recovery = full/delta
Nodes that participated in the cluster

10.6.2.144
10.6.2.147
10.6.2.148
10.6.2.149
10.6.2.150

The node chosen during Failover and recovery is 10.6.2.150

Expected: The UUIDs of vbuckets will not change before failover-add-on-recovery and after ailover-add-on-recovery

Observed: The UUIDs changed for the vbuckets belonging to the node that participated in failover-add-back-recovery. This is true for when recovery type is either full or delta

For delta/full scenarios, atattched .txt file that has missing uuids info and the details of bucket, node, vbucket

 Comments   
Comment by Parag Agarwal [ 16/Apr/14 ]
Logs

Scenario where recoverType=delta

https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_delta_10.6.2.144-4162014-1658-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_delta_10.6.2.147-4162014-1659-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_delta_10.6.2.148-4162014-170-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_delta_10.6.2.149-4162014-171-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_delta_10.6.2.150-4162014-172-diag.zip


Scenario where recoverType=full

https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_full_10.6.2.144-4162014-1718-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_full_10.6.2.147-4162014-1719-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_full_10.6.2.148-4162014-1720-diag.zip
https://s3.amazonaws.com/bugdb/jira/MB-10873/graceful_full_10.6.2.149-4162014-1721-diag.zip
Comment by Andrei Baranouski [ 17/Apr/14 ]

vb_28 : {'uuid': 'Expected 162816295095290 :: Actual 126122070037587'}

 vb_29 : {'uuid': 'Expected 217402359481926 :: Actual 188586946904115'}

 vb_26 : {'uuid': 'Expected 76205354514325 :: Actual 201466466282756'}

 vb_27 : {'uuid': 'Expected 10316033053601 :: Actual 129697191560595'}

 vb_31 : {'uuid': 'Expected 56894599871883 :: Actual 271417789529123'}

 vb_30 : {'uuid': 'Expected 77677303413706 :: Actual 84498997010846'}


these 'Actual' values are duplicated in your stats after failover( the same vbuckets in diff nodes)
Comment by Aliaksey Artamonau [ 22/Apr/14 ]
Likely a duplicate of MB-10919.




[MB-10872] create memcached bucket crashes couchbase server in 3.0.0-584 Created: 16/Apr/14  Updated: 17/Apr/14  Resolved: 17/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Test Blocker
Reporter: Thuan Nguyen Assignee: Thuan Nguyen
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: ubuntu 12.04 64bit

Triage: Triaged
Operating System: Ubuntu 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump: Manifest file from this build http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_3.0.0-584-rel.rpm.manifest.xml
Is this a Regression?: Unknown

 Description   
Install couchbase server 3.0.0-584 in one ubuntu server 12.04 64-bit
Create memcached bucket. Couchbase server crashed and hang

Back trace from memcached

Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/librt.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libstdc++.so.6
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libgcc_s.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/file_logger.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/file_logger.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/default_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/default_engine.so

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff7366f000
0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) t a a bt

Thread 8 (Thread 0x7f3e2aa06700 (LWP 15558)):
#0 0x00007f3e2af06ffd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2ae9aff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f3e2ae9c03e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f3e2ae9018a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f3e2ae8f06b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007f3e2aa07a91 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6 check_stdin_thread (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/daemon/stdin_check.c:38
#7 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6080) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#8 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f3e2a001700 (LWP 15559)):
#0 0x00007f3e2be0a0fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f3e2d08718b in cb_cond_timedwait (cond=0x7f3e2a205240, mutex=0x7f3e2a205200, ms=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:156
#2 0x00007f3e2a0041e8 in logger_thead_main (arg=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/extensions/loggers/file_logger.c:372
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6090) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f3e295f3700 (LWP 15570)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066500, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066500, flags=<optimized out>) at event.c:1558
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce61a0) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f3e28df2700 (LWP 15571)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066280, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066280, flags=<optimized out>) at event.c:1558
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6190) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f3e285f1700 (LWP 15572)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066c80, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066c80, flags=<optimized out>) at event.c:1558
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6180) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f3e27df0700 (LWP 15573)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066a00, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066a00, flags=<optimized out>) at event.c:1558
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6170) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f3e275ef700 (LWP 15574)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066780, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066780, flags=<optimized out>) at event.c:1558
#3 0x00007f3e2d08709f in platform_thread_wrap (arg=0x2ce6160) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/platform/src/cb_pthreads.c:19
#4 0x00007f3e2be05e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007f3e2af143fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f3e2d8a4740 (LWP 15552)):
#0 0x00007f3e2af14a93 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f3e2c61d5a6 in epoll_dispatch (base=0xc066000, tv=<optimized out>) at epoll.c:404
#2 0x00007f3e2c608a04 in event_base_loop (base=0xc066000, flags=<optimized out>) at event.c:1558
#3 0x000000000040fde6 in main (argc=<optimized out>, argv=<optimized out>) at /home/buildbot/ubuntu-1004-x64-300-builder/build/build/memcached/daemon/memcached.c:8567

Tested on build 3.0.0-585, got the same issue



 Comments   
Comment by Aleksey Kondratenko [ 17/Apr/14 ]
There's great chance that it was ns_server bug that I fixed yesterday.

Please note for the future that logs must be attached always even if you have core dump from memcached (which is not clear if you actually have it)
Comment by Thuan Nguyen [ 17/Apr/14 ]
It does not have any coredump. I attached gdb to memcached process ID to generate the back trace
Comment by Aleksey Kondratenko [ 17/Apr/14 ]
I believe that testing with build that has this commit (http://review.couchbase.org/35917) should find everything working fine.
Comment by Chiyoung Seo [ 17/Apr/14 ]
Thuan,

As Alk suggested above, please test it with the build that has the fix.
Comment by Thuan Nguyen [ 17/Apr/14 ]
Tested in build 3.0.0-588, the test was able to create memcached bucket




[MB-10871] UPR:: Rebalance-out/in node results in change in vbucket UUID Created: 16/Apr/14  Updated: 22/Apr/14  Resolved: 22/Apr/14

Status: Closed
Project: Couchbase Server
Component/s: ns_server
Affects Version/s: 3.0
Fix Version/s: 3.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Parag Agarwal Assignee: Aliaksey Artamonau
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos-6

Attachments: File rebalance_in_detail     GZip Archive rebalance_in.tar.gz     GZip Archive rebalance_out.tar.gz    
Is this a Regression?: Yes

 Description   
Using UPR, centos-64 bit, Version:: 3.0.0-584

1. Add 5 nodes to cluster
2. Create default bucket with replica=1
3. Add 50000 items
4. Rebalance out 1 node

Cluster Nodes

10.6.2.144
10.6.2.147
10.6.2.148
10.6.2.149
10.6.2.150

Rebalanced-out Node:: 10.6.2.150

Expected UUIDs of buckets is the same after we rebalance out

Observed Difference in UUIDs which are a part of 10.6.2.150

 vb_28 : {'uuid': 'Expected 170724551552615 :: Actual 10627939076719'}

 vb_29 : {'uuid': 'Expected 45359036031519 :: Actual 59640339796757'}

 vb_26 : {'uuid': 'Expected 133471459893767 :: Actual 114252468159585''}

 vb_27 : {'uuid': 'Expected 104126006114437 :: Actual 86374568011280'}

 vb_6 : {'uuid': 'Expected 175130199501286 :: Actual 23856384718339'}

 vb_13 : {'uuid': 'Expected 84680084219325 :: Actual 233007425883197'}

 vb_31 : {'uuid': 'Expected 170201395300887 :: Actual 174005664911105'}

 vb_30 : {'uuid': 'Expected 83362933025898 :: Actual 229555825112404'}

Dump of Bucket, Node, Vbucket Information (uuid, high_seqno, purge_seqno)

Before we rebalanced-out
______________________

----- Bucket default -----

-------------Node 10.6.2.147------------

   for vbucket vb_9

            :: for key high_seqno = 1558

            :: for key uuid = 224689180220009

            :: for key purge_seqno = 0

   for vbucket vb_8

            :: for key purge_seqno = 0

            :: for key uuid = 24173416366515

            :: for key high_seqno = 1569

   for vbucket vb_1

            :: for key purge_seqno = 0

            :: for key uuid = 3382665152986

            :: for key high_seqno = 1569

   for vbucket vb_0

            :: for key purge_seqno = 0

            :: for key uuid = 252550884809196

            :: for key high_seqno = 1551

   for vbucket vb_7

            :: for key high_seqno = 1551

            :: for key uuid = 160369294390021

            :: for key purge_seqno = 0

   for vbucket vb_11

            :: for key purge_seqno = 0

            :: for key uuid = 91922888028629

            :: for key high_seqno = 1560

   for vbucket vb_10

            :: for key high_seqno = 1568

            :: for key uuid = 98217872738904

            :: for key purge_seqno = 0

   for vbucket vb_13

            :: for key purge_seqno = 0

            :: for key uuid = 84680084219325

            :: for key high_seqno = 1568

   for vbucket vb_12

            :: for key high_seqno = 1560

            :: for key uuid = 75150401842511

            :: for key purge_seqno = 0

   for vbucket vb_15

            :: for key purge_seqno = 0

            :: for key uuid = 21343486345542

            :: for key high_seqno = 1569

   for vbucket vb_16

            :: for key high_seqno = 1560

            :: for key uuid = 113002939605931

            :: for key purge_seqno = 0

   for vbucket vb_21

            :: for key purge_seqno = 0

            :: for key uuid = 131979757081143

            :: for key high_seqno = 1559

   for vbucket vb_28

            :: for key high_seqno = 1553

            :: for key uuid = 170724551552615

            :: for key purge_seqno = 0

-------------Node 10.6.2.144------------

   for vbucket vb_8

            :: for key purge_seqno = 0

            :: for key uuid = 24173416366515

            :: for key high_seqno = 1569

   for vbucket vb_1

            :: for key purge_seqno = 0

            :: for key uuid = 3382665152986

            :: for key high_seqno = 1569

   for vbucket vb_0

            :: for key purge_seqno = 0

            :: for key uuid = 252550884809196

            :: for key high_seqno = 1551

   for vbucket vb_3

            :: for key high_seqno = 1570

            :: for key uuid = 189410128542953

            :: for key purge_seqno = 0

   for vbucket vb_2

            :: for key high_seqno = 1552

            :: for key uuid = 202175152188264

            :: for key purge_seqno = 0

   for vbucket vb_5

            :: for key high_seqno = 1552

            :: for key uuid = 272505847148341

            :: for key purge_seqno = 0

   for vbucket vb_4

            :: for key purge_seqno = 0

            :: for key uuid = 170782289546886

            :: for key high_seqno = 1570

   for vbucket vb_7

            :: for key high_seqno = 1551

            :: for key uuid = 160369294390021

            :: for key purge_seqno = 0

   for vbucket vb_6

            :: for key purge_seqno = 0

            :: for key uuid = 175130199501286

            :: for key high_seqno = 1569

   for vbucket vb_14

            :: for key purge_seqno = 0

            :: for key uuid = 9666779467087

            :: for key high_seqno = 1558

   for vbucket vb_20

            :: for key high_seqno = 1569

            :: for key uuid = 226337594275581

            :: for key purge_seqno = 0

   for vbucket vb_26

            :: for key purge_seqno = 0

            :: for key uuid = 133471459893767

            :: for key high_seqno = 1569

   for vbucket vb_27

            :: for key purge_seqno = 0

            :: for key uuid = 104126006114437

            :: for key high_seqno = 1553

-------------Node 10.6.2.150------------

   for vbucket vb_5

            :: for key high_seqno = 1552

            :: for key uuid = 272505847148341

            :: for key purge_seqno = 0

   for vbucket vb_6

            :: for key purge_seqno = 0

            :: for key uuid = 175130199501286

            :: for key high_seqno = 1569

   for vbucket vb_28

            :: for key high_seqno = 1553

            :: for key uuid = 170724551552615

            :: for key purge_seqno = 0

   for vbucket vb_29

            :: for key purge_seqno = 0

            :: for key uuid = 45359036031519

            :: for key high_seqno = 1569

   for vbucket vb_13

            :: for key purge_seqno = 0

            :: for key uuid = 84680084219325

            :: for key high_seqno = 1568

   for vbucket vb_26

            :: for key purge_seqno = 0

            :: for key uuid = 133471459893767

            :: for key high_seqno = 1569

   for vbucket vb_19

            :: for key purge_seqno = 0

            :: for key uuid = 251498636389290

            :: for key high_seqno = 1569

   for vbucket vb_27

            :: for key purge_seqno = 0

            :: for key uuid = 104126006114437

            :: for key high_seqno = 1553

   for vbucket vb_24

            :: for key high_seqno = 1571

            :: for key uuid = 184565815683126

            :: for key purge_seqno = 0

   for vbucket vb_25

            :: for key high_seqno = 1552

            :: for key uuid = 266765133502323

            :: for key purge_seqno = 0

   for vbucket vb_31

            :: for key purge_seqno = 0

            :: for key uuid = 170201395300887

            :: for key high_seqno = 1571

   for vbucket vb_30

            :: for key high_seqno = 1552

            :: for key uuid = 83362933025898

            :: for key purge_seqno = 0

-------------Node 10.6.2.148------------

   for vbucket vb_9

            :: for key high_seqno = 1558

            :: for key uuid = 224689180220009

            :: for key purge_seqno = 0

   for vbucket vb_10

            :: for key high_seqno = 1568

            :: for key uuid = 98217872738904

            :: for key purge_seqno = 0

   for vbucket vb_3

            :: for key high_seqno = 1570

            :: for key uuid = 189410128542953

            :: for key purge_seqno = 0

   for vbucket vb_2

            :: for key high_seqno = 1552

            :: for key uuid = 202175152188264

            :: for key purge_seqno = 0

   for vbucket vb_29

            :: for key purge_seqno = 0

            :: for key uuid = 45359036031519

            :: for key high_seqno = 1569

   for vbucket vb_15

            :: for key purge_seqno = 0

            :: for key uuid = 21343486345542

            :: for key high_seqno = 1569

   for vbucket vb_14

            :: for key purge_seqno = 0

            :: for key uuid = 9666779467087

            :: for key high_seqno = 1558

   for vbucket vb_17

            :: for key purge_seqno = 0

            :: for key uuid = 50245200692736

            :: for key high_seqno = 1570

   for vbucket vb_16

            :: for key high_seqno = 1560

            :: for key uuid = 113002939605931

            :: for key purge_seqno = 0

   for vbucket vb_19

            :: for key purge_seqno = 0

            :: for key uuid = 251498636389290

            :: for key high_seqno = 1569

   for vbucket vb_18

            :: for key high_seqno = 1559

            :: for key uuid = 5501229932749

            :: for key purge_seqno = 0

   for vbucket vb_22

            :: for key high_seqno = 1570

            :: for key uuid = 247498759341926

            :: for key purge_seqno = 0

   for vbucket vb_23

            :: for key purge_seqno = 0

            :: for key uuid = 74620381341917

            :: for key high_seqno = 1560

-------------Node 10.6.2.149------------

   for vbucket vb_4

            :: for key purge_seqno = 0

            :: for key uuid = 170782289546886

            :: for key high_seqno = 1570

   for vbucket vb_11

            :: for key purge_seqno = 0

            :: for key uuid = 91922888028629

            :: for key high_seqno = 1560

   for vbucket vb_12

            :: for key purge_seqno = 0

            :: for key uuid = 75150401842511

            :: for key high_seqno = 1560

   for vbucket vb_17

            :: for key purge_seqno = 0

            :: for key uuid = 50245200692736

            :: for key high_seqno = 1570

   for vbucket vb_18

            :: for key high_seqno = 1559

            :: for key uuid = 5501229932749

            :: for key purge_seqno = 0

   for vbucket vb_20

            :: for key high_seqno = 1569

            :: for key uuid = 226337594275581

            :: for key purge_seqno = 0

   for vbucket vb_21

            :: for key purge_seqno = 0

            :: for key uuid = 131979757081143

            :: for key high_seqno = 1559

   for vbucket vb_22

            :: for key high_seqno = 1570

            :: for key uuid = 247498759341926

            :: for key purge_seqno = 0

   for vbucket vb_23

            :: for key purge_seqno = 0

            :: for key uuid = 74620381341917

            :: for key high_seqno = 1560

   for vbucket vb_24

            :: for key high_seqno = 1571

            :: for key uuid = 184565815683126

            :: for key purge_seqno = 0

   for vbucket vb_25

            :: for key high_seqno = 1552

            :: for key uuid = 266765133502323

            :: for key purge_seqno = 0

   for vbucket vb_31

            :: for key purge_seqno = 0

            :: for key uuid = 170201395300887

            :: for key high_seqno = 1571

   for vbucket vb_30

            :: for key high_seqno = 1552

            :: for key uuid = 83362933025898

            :: for key purge_seqno = 0

After we rebalanced-out
____________________

----- Bucket default -----

-------------Node 10.6.2.147------------

   for vbucket vb_9

            :: for key high_seqno = 2337

            :: for key uuid = 224689180220009

            :: for key purge_seqno = 0

   for vbucket vb_8

            :: for key purge_seqno = 0

            :: for key uuid = 24173416366515

            :: for key high_seqno = 2354

   for vbucket vb_1

            :: for key purge_seqno = 0

            :: for key uuid = 3382665152986

            :: for key high_seqno = 2355

   for vbucket vb_0

            :: for key purge_seqno = 0

            :: for key uuid = 252550884809196

            :: for key high_seqno = 2326

   for vbucket vb_7

            :: for key high_seqno = 2326

            :: for key uuid = 160369294390021

            :: for key purge_seqno = 0

   for vbucket vb_6

            :: for key purge_seqno = 0

            :: for key uuid = 23856384718339

            :: for key high_seqno = 2355

   for vbucket vb_11

            :: for key purge_seqno = 0

            :: for key uuid = 91922888028629

            :: for key high_seqno = 2339

   for vbucket vb_10

            :: for key high_seqno = 2353

            :: for key uuid = 98217872738904

            :: for key purge_seqno = 0

   for vbucket vb_12

            :: for key high_seqno = 2339

            :: for key uuid = 75150401842511

            :: for key purge_seqno = 0

   for vbucket vb_15

            :: for key purge_seqno = 0

            :: for key uuid = 21343486345542

            :: for key high_seqno = 2354

   for vbucket vb_26

            :: for key purge_seqno = 0

            :: for key uuid = 114252468159585

            :: for key high_seqno = 2355

   for vbucket vb_16

            :: for key high_seqno = 2339

            :: for key uuid = 113002939605931

            :: for key purge_seqno = 0

   for vbucket vb_21

            :: for key purge_seqno = 0

            :: for key uuid = 131979757081143

            :: for key high_seqno = 2338

   for vbucket vb_24

            :: for key high_seqno = 2357

            :: for key uuid = 184565815683126

            :: for key purge_seqno = 0

   for vbucket vb_31

            :: for key purge_seqno = 0

            :: for key uuid = 174005664911105

            :: for key high_seqno = 2357

   for vbucket vb_28

            :: for key high_seqno = 2328

            :: for key uuid = 10627939076719

            :: for key purge_seqno = 0

-------------Node 10.6.2.144------------

   for vbucket vb_8

            :: for key purge_seqno = 0

            :: for key uuid = 24173416366515

            :: for key high_seqno = 2354

   for vbucket vb_1

            :: for key purge_seqno = 0

            :: for key uuid = 3382665152986

            :: for key high_seqno = 2355

   for vbucket vb_0

            :: for key purge_seqno = 0

            :: for key uuid = 252550884809196

            :: for key high_seqno = 2326

   for vbucket vb_3

            :: for key high_seqno = 2356

            :: for key uuid = 189410128542953

            :: for key purge_seqno = 0

   for vbucket vb_2

            :: for key high_seqno = 2327

            :: for key uuid = 202175152188264

            :: for key purge_seqno = 0

   for vbucket vb_5

            :: for key high_seqno = 2327

            :: for key uuid = 272505847148341

            :: for key purge_seqno = 0

   for vbucket vb_4

            :: for key purge_seqno = 0

            :: for key uuid = 170782289546886

            :: for key high_seqno = 2356

   for vbucket vb_7

            :: for key high_seqno = 2326

            :: for key uuid = 160369294390021

            :: for key purge_seqno = 0

   for vbucket vb_28

            :: for key high_seqno = 2328

            :: for key uuid = 10627939076719

            :: for key purge_seqno = 0

   for vbucket vb_29

            :: for key purge_seqno = 0

            :: for key uuid = 59640339796757

            :: for key high_seqno = 2355

   for vbucket vb_13

            :: for key purge_seqno = 0

            :: for key uuid = 233007425883197

            :: for key high_seqno = 2353

   for vbucket vb_14

            :: for key purge_seqno = 0

            :: for key uuid = 9666779467087

            :: for key high_seqno = 2337

   for vbucket vb_20

            :: for key high_seqno = 2354

            :: for key uuid = 226337594275581

            :: for key purge_seqno = 0

   for vbucket vb_30

            :: for key high_seqno = 2327

            :: for key uuid = 229555825112404

            :: for key purge_seqno = 0

   for vbucket vb_26

            :: for key purge_seqno = 0

            :: for key uuid = 114252468159585

            :: for key high_seqno = 2355

   for vbucket vb_27

            :: for key purge_seqno = 0

            :: for key uuid = 86374568011280

            :: for key high_seqno = 2328

-------------Node 10.6.2.148------------

   for vbucket vb_9

            :: for key high_seqno = 2337

            :: for key uuid = 224689180220009

            :: for key purge_seqno = 0

   for vbucket vb_10

            :: for key high_seqno = 2353

            :: for key uuid = 98217872738904

            :: for key purge_seqno = 0

   for vbucket vb_3

            :: for key high_seqno = 2356

            :: for key uuid = 189410128542953

            :: for key purge_seqno = 0

   for vbucket vb_2

            :: for key high_seqno = 2327

            :: for key uuid = 202175152188264

            :: for key purge_seqno = 0

   for vbucket vb_29

            :: for key purge_seqno = 0

            :: for key uuid = 59640339796757

            :: for key high_seqno = 2355

   for vbucket vb_13

            :: for key purge_seqno = 0

            :: for key uuid = 233007425883197

            :: for key high_seqno = 2353

   for vbucket vb_15

            :: for key purge_seqno = 0

            :: for key uuid = 21343486345542

            :: for key high_seqno = 2354

   for vbucket vb_14

            :: for key purge_seqno = 0

            :: for key uuid = 9666779467087

            :: for key high_seqno = 2337

   for vbucket vb_17

            :: for key purge_seqno = 0

            :: for key uuid = 50245200692736

            :: for key high_seqno = 2355

   for vbucket vb_16

            :: for key high_seqno = 2339

            :: for key uuid = 113002939605931

            :: for key purge_seqno = 0

   for vbucket vb_19

            :: for key purge_seqno = 0

            :: for key uuid = 251498636389290

            :: for key high_seqno = 2354

   for vbucket vb_18

            :: for key high_seqno = 2338

            :: for key uuid = 5501229932749

            :: for key purge_seqno = 0

   for vbucket vb_22

            :: for key high_seqno = 2355

            :: for key uuid = 247498759341926

            :: for key purge_seqno = 0

   for vbucket vb_23

            :: for key purge_seqno = 0

            :: for key uuid = 74620381341917

            :: for key high_seqno = 2338

   for vbucket vb_25

            :: for key high_seqno = 2327

            :: for key uuid = 266765133502323

            :: for key purge_seqno = 0

   for vbucket vb_27

            :: for key purge_seqno = 0

            :: for key uuid = 86374568011280

            :: for key high