[MB-7878] cbstats does not work on port 11210 in non-root installation Created: 06/Mar/13  Updated: 31/Oct/13  Resolved: 25/Oct/13

Status: Closed
Project: Couchbase Server
Component/s: installer, tools
Affects Version/s: 2.0.1
Fix Version/s: 2.5.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Thuan Nguyen Assignee: Aruna Piravi
Resolution: Fixed Votes: 0
Labels: non-root
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 5.6 64bit in ec2


 Description   
Download couchbase server 2.0.1-170
Login as non-root user (test)
Follow instruction in https://github.com/couchbase/voltron/blob/2.0.1/README-non-root.md to install couchbase server
Start couchbase server
Open web browser to do initial setup.
Go to opt/couchbase/bin/tools
Run cbstats at port 11210, failed to get info

[test@ip-10-33-3-109 ~]$ ./opt/couchbase/bin/cbstats 10.33.3.109:11210 all
Stats '' are not available from the requested engine.
[test@ip-10-33-3-109 ~]$ ./opt/couchbase/bin/cbstats 10.33.3.109:11210 all -b default
Traceback (most recent call last):
  File "/home/test/opt/couchbase/lib/python/cbstats", line 428, in ?
    main()
  File "/home/test/opt/couchbase/lib/python/cbstats", line 425, in main
    c.execute()
  File "/home/test/opt/couchbase/lib/python/clitool.py", line 58, in execute
    f[0](mc, *args[2:], **opts.__dict__)
  File "/home/test/opt/couchbase/lib/python/cbstats", line 32, in g
    mc.sasl_auth_plain(bucket, password)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 262, in sasl_auth_plain
    return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 258, in sasl_auth_start
    return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 104, in _doCmd
    return self._handleSingleResponse(opaque)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 97, in _handleSingleResponse
    cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 89, in _handleKeyedResponse
    cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg()
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 71, in _recvMsg
    raise exceptions.EOFError("Got empty data (remote died?).")
EOFError: Got empty data (remote died?).
[test@ip-10-33-3-109 ~]$ ./opt/couchbase/bin/cbstats 10.33.3.109:11210 all -b sasl -p password
Traceback (most recent call last):
  File "/home/test/opt/couchbase/lib/python/cbstats", line 428, in ?
    main()
  File "/home/test/opt/couchbase/lib/python/cbstats", line 425, in main
    c.execute()
  File "/home/test/opt/couchbase/lib/python/clitool.py", line 58, in execute
    f[0](mc, *args[2:], **opts.__dict__)
  File "/home/test/opt/couchbase/lib/python/cbstats", line 32, in g
    mc.sasl_auth_plain(bucket, password)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 262, in sasl_auth_plain
    return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password]))
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 258, in sasl_auth_start
    return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 104, in _doCmd
    return self._handleSingleResponse(opaque)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 97, in _handleSingleResponse
    cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque)
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 89, in _handleKeyedResponse
    cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg()
  File "/home/test/opt/couchbase/lib/python/mc_bin_client.py", line 71, in _recvMsg
    raise exceptions.EOFError("Got empty data (remote died?).")
EOFError: Got empty data (remote died?).


Load items to both bucket, items loaed

grep process

[test@ip-10-33-3-109 ~]$ ps aux | grep memcached
test 2047 0.3 6.0 706412 474232 ? Ssl 02:09 3:21 /home/test/opt/couchbase/bin/memcached.bin -X /home/test/opt/couchbase/lib/memcached/stdin_term_handler.so -X /home/test/opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/home/test/opt/couchbase/var/lib/couchbase/logs/memcached.log -l 0.0.0.0:11210,0.0.0.0:11209:1000 -p 11210 -E /home/test/opt/couchbase/lib/memcached/bucket_engine.so -B binary -r -c 10000 -e admin=_admin;default_bucket_name=default;auto_create=false
test 6963 0.0 0.0 61180 812 pts/3 S+ 18:09 0:00 grep memcached
[test@ip-10-33-3-109 ~]$ ps aux | grep couchbase
test 1881 8.1 5.8 850812 456204 pts/1 Sl+ 02:07 78:47 /home/test/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 -- -root /home/test/opt/couchbase/lib/erlang -progname erl -- -home /home/test -- -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -run ns_bootstrap -- -couch_ini /home/test/opt/couchbase/etc/couchdb/default.ini /home/test/opt/couchbase/etc/couchdb/default.d/capi.ini /home/test/opt/couchbase/etc/couchdb/default.d/geocouch.ini /home/test/opt/couchbase/etc/couchdb/local.ini -ns_server config_path "/home/test/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/home/test/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server nodefile "/home/test/opt/couchbase/var/lib/couchbase/couchbase-server.node" -ns_server cookiefile "/home/test/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server enable_mlockall true
test 1892 0.0 0.0 10572 568 ? S 02:07 0:00 /home/test/opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
test 1931 0.0 0.0 3800 548 ? Ss 02:07 0:00 /home/test/opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup
test 1932 0.0 0.0 3796 412 ? Ss 02:07 0:00 /home/test/opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
test 1933 0.0 0.0 38348 2016 ? Ss 02:07 0:00 /home/test/opt/couchbase/lib/erlang/lib/ssl-4.1.6/priv/bin/ssl_esock
test 2046 0.0 0.0 158172 3504 ? Ssl 02:09 0:34 /home/test/opt/couchbase/bin/moxi.bin -Z port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200 -z url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming -p 0 -Y y -O stderr
test 2047 0.3 6.0 706412 474232 ? Ssl 02:09 3:21 /home/test/opt/couchbase/bin/memcached.bin -X /home/test/opt/couchbase/lib/memcached/stdin_term_handler.so -X /home/test/opt/couchbase/lib/memcached/file_logger.so,cyclesize=104857600;sleeptime=19;filename=/home/test/opt/couchbase/var/lib/couchbase/logs/memcached.log -l 0.0.0.0:11210,0.0.0.0:11209:1000 -p 11210 -E /home/test/opt/couchbase/lib/memcached/bucket_engine.so -B binary -r -c 10000 -e admin=_admin;default_bucket_name=default;auto_create=false
test 2073 0.0 0.0 6016 580 ? Ss 02:09 0:07 /home/test/opt/couchbase/bin/sigar_port.bin
test 6032 0.0 0.0 13612 920 pts/0 T 15:03 0:00 /home/test/opt/couchbase/bin/sqlite3.bin
test 6965 0.0 0.0 61180 812 pts/3 S+ 18:09 0:00 grep couchbase
[test@ip-10-

Moxi port works ok

[test@ip-10-33-3-109 ~]$ ./opt/couchbase/bin/cbstats 10.118.163.79:11211 all -b sasl -p password
 accepting_conns: 1
 auth_cmds: 3078
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 12
 bytes: 222778816
 bytes_read: 184970587
 bytes_written: 2074537677
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 1
 cmd_set: 14174
 conn_yields: 14895
 connection_structures: 1006
 curr_connections: 26
 curr_conns_on_port_11209: 19
 curr_conns_on_port_11210: 5
 curr_items: 164042
 curr_items_tot: 314174
 curr_temp_items: 0
 daemon_connections: 4
 decr_hits:


Link to collect info from 2 nodes

https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201303/2nodes_201-170_cbstats_failed_11210_non_root_20130306.tgz

 Comments   
Comment by Farshid Ghods (Inactive) [ 06/Mar/13 ]
most likelty memcached is not listening in 0.0.0.0

Tony,

please post instructions on how anyone can install with non-root and then assign to ep-engine team
Comment by Maria McDuff (Inactive) [ 13/May/13 ]
Bin, can you take a look at this issue?
Comment by kzeller [ 13/May/13 ]
Added to RN 2.0.2 Can update if becomes fix:

<command>cbstats</command> will not work on port 11210 if you installed Couchbase Server without root-level permissions.
Comment by kzeller [ 25/Jun/13 ]
removing doc component as already in doc for 2.1.0

Save as open issue for Eng/QA
Comment by Wayne Siu [ 01/Jul/13 ]
Tony, can you rerun the test using the steps listed below? Thanks.

Installation steps:

- rpm2cpio couchbase-server-enterprise_x86_64_2.1.0-718-rel.rpm | cpio --extract --make-directories --no-absolute-filenames
- cd opt/couchbase/
- copy couchbase_init.d to etc/couchbase_init.d, and remove command used by root.
- copy reloc_wrapper.sh to bin/install/reloc_wrapper.sh, and remove references to pwd.
- ./bin/install/reloc.sh /home/abhinav/opt/couchbase
- rm -f /home/username/etc/init.d/couchbase-server
- ln -s /home/username/opt/couchbase/etc/couchbase_init.d /home/username/etc/init.d/couchbase-server
- ln -s /home/username/couchbase/logs /home/username/opt/couchbase/var/lib/couchbase/logs

Then to start couchbase-server:

- /home/username/etc/init.d/couchbase-server start
Starting couchbase-serverchown: changing ownership of `/home/username/opt/couchbase/var/lib/couchbase/couchbase-server.pid': Operation not permitted
chown: changing ownership of `/home/username/opt/couchbase/var/lib/couchbase/couchbase-server.node': Operation not permitted
chown: changing ownership of `/home/username/opt/couchbase/var/lib/couchbase/couchbase-server.cookie': Operation not permitted
                                                           [ OK ]

Running cbstats:

[username@orange-11602 ~]$ opt/couchbase/bin/cbstats localhost:11210 raw memory
 bytes: 31727960
 ep_kv_size: 0
 ep_max_data_size: 10485760000
 ep_mem_high_wat: 8912896000
 ep_mem_low_wat: 7864320000
 ep_mem_tracker_enabled: true
 ep_oom_errors: 0
 ep_overhead: 27347928
 ep_tmp_oom_errors: 0
 ep_value_size: 0
 mem_used: 31727960
 tcmalloc_current_thread_cache_bytes: 962520
 tcmalloc_max_thread_cache_bytes: 4194304
 tcmalloc_unmapped_bytes: 0
 total_allocated_bytes: 50362584
 total_fragmentation_bytes: 3573544
 total_free_bytes: 589824
 total_heap_bytes: 54525952
Comment by Maria McDuff (Inactive) [ 25/Oct/13 ]
aruna,
pls try in latest 2.5.0 build using the steps above and update this bug.
Comment by Wayne Siu [ 25/Oct/13 ]
Aruna,
The steps listed above were for releases prior to officially supporting the non-root/non-sudo feature. We should now test using the instructions documented in our doc to see if all the tools work (under non-root installation).
Comment by Aruna Piravi [ 25/Oct/13 ]
Wayne and Maria, I figured out the correct documentation for the non sudo, non root installation (there are many versions available on JIRA) but this seems to be latest and is working - http://docs.couchbase.com/couchbase-manual-2.2/#installing-on-centosrhel-as-non-root-non-sudo. I arrived at this through the install script automated by Abhinav.

Results -

Tested on both v2.2.0 (build821) and 2.5.0 (build 839). cbstats works fine for non-root user without sudo privileges.

[nonroot@olive-012 ~]$ opt/couchbase/bin/cbstats 10.3.3.118:11210 raw memory
 bytes: 33036360
 ep_kv_size: 10866064
 ep_max_data_size: 3145728000
 ep_mem_high_wat: 2673868800
 ep_mem_low_wat: 2359296000
 ep_mem_tracker_enabled: true
 ep_oom_errors: 0
 ep_overhead: 18214476
 ep_tmp_oom_errors: 0
 ep_value_size: 5659470
 mem_used: 33036360
 tcmalloc_current_thread_cache_bytes: 3180680
 tcmalloc_max_thread_cache_bytes: 4194304
 tcmalloc_unmapped_bytes: 0
 total_allocated_bytes: 107614656
 total_fragmentation_bytes: 6688320
 total_free_bytes: 2088960
 total_heap_bytes: 116391936

Hence marking this item as resolved. Thanks.
Comment by Aruna Piravi [ 25/Oct/13 ]
Verified on couchbase server v2.5.0 and v2.2.0
Generated at Wed Nov 26 12:30:21 CST 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.