Moxi Pegs Out CPU

David,

It would be be very helpful if you could answer the following:

how big your cluster is?

what is the version of membase that you are running?

if you telnet to the system in question and run the following command and send us the output, I would be able to help you further.

/opt/membase/bin/ep_engine/management/stats host:port all
 

Thanks

Bhawana

 Our cluster consists of two servers running version 1.6.0_5_gd128355.  These are windows machines so no telnet.

dgottlieb,

You may go to each of your nodes and execute the stats command at

c:\cd Program Files\Membase\Server\bin\ep_engine\management

usage: stats.exe 10.x.x.x:11210 all

Thanks

Bhawana

 Server 1:

C:\Program Files\Membase\Server\bin\ep_engine\management>stats.exe 10.x.x.74:11210 all auth_cmds:             1 auth_errors:           0 bucket_conns:          3 bytes:                 0 bytes_read:            11621454 bytes_written:         533084836 cas_badval:            0 cas_hits:              0 cas_misses:            0 cmd_flush:             0 cmd_get:               0 cmd_set:               0 conn_yields:           0 connection_structures: 33 curr_connections:      10 curr_items:            0 daemon_connections:    5 decr_hits:             0 decr_misses:           0 delete_hits:           0 delete_misses:         0 engine_maxbytes:       134217728 evictions:             0 get_hits:              0 get_misses:            0 incr_hits:             0 incr_misses:           0 libevent:              1.4.13-stable limit_maxbytes:        67108864 pid:                   4204 pointer_size:          32 reclaimed:             0 rejected_conns:        0 threads:               4 time:                  1288705200 total_connections:     48 total_items:           0 uptime:                325486 version:               1.4.4_298_g250909b

Server 2:

C:\Program Files\Membase\Server\bin\ep_engine\management>stats eswstestmt02:11210 all auth_cmds:             0 auth_errors:           0 bucket_conns:          2 bytes:                 0 bytes_read:            21004 bytes_written:         937301 cas_badval:            0 cas_hits:              0 cas_misses:            0 cmd_flush:             0 cmd_get:               0 cmd_set:               0 conn_yields:           0 connection_structures: 17 curr_connections:      17 curr_items:            0 daemon_connections:    5 decr_hits:             0 decr_misses:           0 delete_hits:           0 delete_misses:         0 engine_maxbytes:       134217728 evictions:             0 get_hits:              0 get_misses:            0 incr_hits:             0 incr_misses:           0 libevent:              1.4.13-stable limit_maxbytes:        67108864 pid:                   1272 pointer_size:          32 reclaimed:             0 rejected_conns:        0 threads:               4 time:                  1288809999 total_connections:     17 total_items:           0 uptime:                577  

 I haven't heard back with a response to this issue.  Have you been able to research our issue?

Thanks.

David Gottlieb

EBSCO Industries, Inc.

David,

We are looking into it and will get back to you as soon as we have an answer.

Thanks

Bhawana

Hi David,

Another thing to look at (and if you could send to us) -- can you run a "stats proxy" against the problem moxi?  Please see: http://wiki.membase.org/display/membase/Moxi+Statistics for some more information on getting statistics from moxi.

Also, after restarting the moxi process, does the problem resurface quickly afterwards?  On windows, you can use tasklist.exe to find a process ID and taskkill.exe.  The moxi program will be automatically restarted by the membase process watchdog system.

Thanks,

Steve

 Steve,

After looking at the wiki article on Moxi stats I am still unsure as to how to retrieve these.  Keep in mind this is a Windows machine.  Can you give me the series of commands at the command line I need to run?

When the Moxi service spins out of control I end up restarting the MembaseServer Windows service which knocks it back down.  The odd thing is that when this happens, it happens on both servers in the cluster at the same time.

On your Windows server, you can run the following:

telnet 10.X.X.XXX 11211

(In the telnet window write "stats proxy")

OR 

If you have access to a UNIX machine you can run the following command:

echo "stats proxy" | nc 10.X.X.XXX 11211 | sort
 

and you will get the stats for the moxi server running on your windows machine.

Bhawana

STAT 11211:www.summitstands.com_eswstest2:behavior:port 0STAT 11211:www.summitstands.com_eswstest2:behavior:bucketSTAT 11211:www.summitstands.com_eswstest2:behavior:port_listen 11211STAT 11211:www.summitstands.com_eswstest2:behavior:default_bucket_name defaultSTAT 11211:www.summitstands.com_eswstest2:behavior-0:downstream_weight 0STAT 11211:www.summitstands.com_eswstest2:behavior-0:downstream_retry 1STAT 11211:www.summitstands.com_eswstest2:behavior-0:downstream_protocol 8STAT 11211:www.summitstands.com_eswstest2:behavior-0:downstream_timeout 0STAT 11211:www.summitstands.com_eswstest2:behavior-0:usr www.summitstands.com_eswstest2STAT 11211:www.summitstands.com_eswstest2:behavior-0:host 192.73.31.74STAT 11211:www.summitstands.com_eswstest2:behavior-0:port 11210STAT 11211:www.summitstands.com_eswstest2:behavior-0:bucketSTAT 11211:www.summitstands.com_eswstest2:behavior-1:downstream_weight 0STAT 11211:www.summitstands.com_eswstest2:behavior-1:downstream_retry 1STAT 11211:www.summitstands.com_eswstest2:behavior-1:downstream_protocol 8STAT 11211:www.summitstands.com_eswstest2:behavior-1:downstream_timeout 0STAT 11211:www.summitstands.com_eswstest2:behavior-1:usr www.summitstands.com_eswstest2STAT 11211:www.summitstands.com_eswstest2:behavior-1:host 192.73.31.76STAT 11211:www.summitstands.com_eswstest2:behavior-1:port 11210STAT 11211:www.summitstands.com_eswstest2:behavior-1:bucketSTAT 11211:www.summitstands.com_eswstest2:stats:listening 1STAT 11211:www.summitstands.com_eswstest2:stats:listening_failed 0STAT 11211:www.summitstands.com_eswstest2:frontcache:max 0STAT 11211:www.summitstands.com_eswstest2:frontcache:oldest_live 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_get_hits 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_get_expires 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_get_misses 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_get_bytes 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_adds 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_add_skips 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_add_fails 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_add_bytes 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_deletes 0STAT 11211:www.summitstands.com_eswstest2:frontcache:tot_evictions 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:num_upstream 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_upstream 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:num_downstream_conn 10STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_conn 10STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_conn_acquired 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_conn_released 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_released 169STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_reserved 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_reserved_time 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:max_downstream_reserved_time 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_freed 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_quit_server0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_max_reached0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_create_failed 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_connect 10STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_connect_failed 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_connect_timeout 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_connect_interval 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_connect_max_reached 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_waiting_errors 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_auth 10STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_auth_failed0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_bucket 10STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_bucket_failed 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_propagate_failed 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_close_on_upstream_close 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_downstream_timeout 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_wait_queue_timeout 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_assign_downstream 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_assign_upstream 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_assign_recursion 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_reset_upstream_avail 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_multiget_keys 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_multiget_keys_dedupe 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_multiget_bytes_dedupe 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_optimize_sets 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_optimize_self 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_retry 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_retry_time 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:max_retry_time 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_retry_vbucket 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_upstream_paused 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:tot_upstream_unpaused 164STAT 11211:www.summitstands.com_eswstest2:pstd_stats:err_oom 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:err_upstream_write_prep 0STAT 11211:www.summitstands.com_eswstest2:pstd_stats:err_downstream_write_prep 0STAT 11211:default:info:port 11211STAT 11211:default:info:name defaultSTAT 11211:default:info:config 192.73.31.74:11210,192.73.31.76:11210STAT 11211:default:info:config_ver 2STAT 11211:default:info:behaviors_num 2STAT 11211:default:behavior:downstream_max 4STAT 11211:default:behavior:downstream_conn_max 0STAT 11211:default:behavior:downstream_weight 0STAT 11211:default:behavior:downstream_retry 1STAT 11211:default:behavior:downstream_protocol 8STAT 11211:default:behavior:downstream_timeout 0STAT 11211:default:behavior:wait_queue_timeout 0STAT 11211:default:behavior:time_stats 0STAT 11211:default:behavior:connect_max_errors 0STAT 11211:default:behavior:connect_retry_interval 0STAT 11211:default:behavior:front_cache_max 200STAT 11211:default:behavior:front_cache_lifespan 0STAT 11211:default:behavior:front_cache_specSTAT 11211:default:behavior:front_cache_unspecSTAT 11211:default:behavior:key_stats_max 4000STAT 11211:default:behavior:key_stats_lifespan 0STAT 11211:default:behavior:key_stats_specSTAT 11211:default:behavior:key_stats_unspecSTAT 11211:default:behavior:optimize_setSTAT 11211:default:behavior:usr defaultSTAT 11211:default:behavior:hostSTAT 11211:default:behavior:port 0STAT 11211:default:behavior:bucketSTAT 11211:default:behavior:port_listen 11211STAT 11211:default:behavior:default_bucket_name defaultSTAT 11211:default:behavior-0:downstream_weight 0STAT 11211:default:behavior-0:downstream_retry 1STAT 11211:default:behavior-0:downstream_protocol 8STAT 11211:default:behavior-0:downstream_timeout 0STAT 11211:default:behavior-0:usr defaultSTAT 11211:default:behavior-0:host 192.73.31.74STAT 11211:default:behavior-0:port 11210STAT 11211:default:behavior-0:bucketSTAT 11211:default:behavior-1:downstream_weight 0STAT 11211:default:behavior-1:downstream_retry 1STAT 11211:default:behavior-1:downstream_protocol 8STAT 11211:default:behavior-1:downstream_timeout 0STAT 11211:default:behavior-1:usr defaultSTAT 11211:default:behavior-1:host 192.73.31.76STAT 11211:default:behavior-1:port 11210STAT 11211:default:behavior-1:bucketSTAT 11211:default:stats:listening 1STAT 11211:default:stats:listening_failed 0STAT 11211:default:frontcache:max 0STAT 11211:default:frontcache:oldest_live 0STAT 11211:default:frontcache:tot_get_hits 0STAT 11211:default:frontcache:tot_get_expires 0STAT 11211:default:frontcache:tot_get_misses 0STAT 11211:default:frontcache:tot_get_bytes 0STAT 11211:default:frontcache:tot_adds 0STAT 11211:default:frontcache:tot_add_skips 0STAT 11211:default:frontcache:tot_add_fails 0STAT 11211:default:frontcache:tot_add_bytes 0STAT 11211:default:frontcache:tot_deletes 0STAT 11211:default:frontcache:tot_evictions 0STAT 11211:default:pstd_stats:num_upstream 17STAT 11211:default:pstd_stats:tot_upstream 21STAT 11211:default:pstd_stats:num_downstream_conn 0STAT 11211:default:pstd_stats:tot_downstream_conn 0STAT 11211:default:pstd_stats:tot_downstream_conn_acquired 0STAT 11211:default:pstd_stats:tot_downstream_conn_released 0STAT 11211:default:pstd_stats:tot_downstream_released 0STAT 11211:default:pstd_stats:tot_downstream_reserved 0STAT 11211:default:pstd_stats:tot_downstream_reserved_time 0STAT 11211:default:pstd_stats:max_downstream_reserved_time 0STAT 11211:default:pstd_stats:tot_downstream_freed 0STAT 11211:default:pstd_stats:tot_downstream_quit_server 0STAT 11211:default:pstd_stats:tot_downstream_max_reached 0STAT 11211:default:pstd_stats:tot_downstream_create_failed 0STAT 11211:default:pstd_stats:tot_downstream_connect 0STAT 11211:default:pstd_stats:tot_downstream_connect_failed 0STAT 11211:default:pstd_stats:tot_downstream_connect_timeout 0STAT 11211:default:pstd_stats:tot_downstream_connect_interval 0STAT 11211:default:pstd_stats:tot_downstream_connect_max_reached 0STAT 11211:default:pstd_stats:tot_downstream_waiting_errors 0STAT 11211:default:pstd_stats:tot_downstream_auth 0STAT 11211:default:pstd_stats:tot_downstream_auth_failed 0STAT 11211:default:pstd_stats:tot_downstream_bucket 0STAT 11211:default:pstd_stats:tot_downstream_bucket_failed 0STAT 11211:default:pstd_stats:tot_downstream_propagate_failed 0STAT 11211:default:pstd_stats:tot_downstream_close_on_upstream_close 0STAT 11211:default:pstd_stats:tot_downstream_timeout 0STAT 11211:default:pstd_stats:tot_wait_queue_timeout 0STAT 11211:default:pstd_stats:tot_assign_downstream 0STAT 11211:default:pstd_stats:tot_assign_upstream 0STAT 11211:default:pstd_stats:tot_assign_recursion 0STAT 11211:default:pstd_stats:tot_reset_upstream_avail 0STAT 11211:default:pstd_stats:tot_multiget_keys 0STAT 11211:default:pstd_stats:tot_multiget_keys_dedupe 0STAT 11211:default:pstd_stats:tot_multiget_bytes_dedupe 0STAT 11211:default:pstd_stats:tot_optimize_sets 0STAT 11211:default:pstd_stats:tot_optimize_self 0STAT 11211:default:pstd_stats:tot_retry 0STAT 11211:default:pstd_stats:tot_retry_time 0STAT 11211:default:pstd_stats:max_retry_time 0STAT 11211:default:pstd_stats:tot_retry_vbucket 0STAT 11211:default:pstd_stats:tot_upstream_paused 0STAT 11211:default:pstd_stats:tot_upstream_unpaused 0STAT 11211:default:pstd_stats:err_oom 0STAT 11211:default:pstd_stats:err_upstream_write_prep 0STAT 11211:default:pstd_stats:err_downstream_write_prep 0STAT 11211:default:pstd_stats_cmd:regular_ERROR:seen 10STAT 11211:default:pstd_stats_cmd:regular_ERROR:read_bytes 128STAT 11211:[ <NULL_BUCKET> ]:info:port 11211STAT 11211:[ <NULL_BUCKET> ]:info:name [ <NULL_BUCKET> ]STAT 11211:[ <NULL_BUCKET> ]:info:configSTAT 11211:[ <NULL_BUCKET> ]:info:config_ver 0STAT 11211:[ <NULL_BUCKET> ]:info:behaviors_num 0STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_max 4STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_conn_max 0STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_weight 0STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_retry 1STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_protocol 8STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_timeout 0STAT 11211:[ <NULL_BUCKET> ]:behavior:wait_queue_timeout 0STAT 11211:[ <NULL_BUCKET> ]:behavior:time_stats 0STAT 11211:[ <NULL_BUCKET> ]:behavior:connect_max_errors 0STAT 11211:[ <NULL_BUCKET> ]:behavior:connect_retry_interval 0STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_max 200STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_lifespan 0STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_specSTAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_unspecSTAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_max 4000STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_lifespan 0STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_specSTAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_unspecSTAT 11211:[ <NULL_BUCKET> ]:behavior:optimize_setSTAT 11211:[ <NULL_BUCKET> ]:behavior:usr AdministratorSTAT 11211:[ <NULL_BUCKET> ]:behavior:hostSTAT 11211:[ <NULL_BUCKET> ]:behavior:port 0STAT 11211:[ <NULL_BUCKET> ]:behavior:bucketSTAT 11211:[ <NULL_BUCKET> ]:behavior:port_listen 11211STAT 11211:[ <NULL_BUCKET> ]:behavior:default_bucket_name defaultSTAT 11211:[ <NULL_BUCKET> ]:stats:listening 1STAT 11211:[ <NULL_BUCKET> ]:stats:listening_failed 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:max 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:oldest_live 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_hits 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_expires 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_misses 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_bytes 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_adds 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_skips 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_fails 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_bytes 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_deletes 0STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_evictions 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:num_upstream 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:num_downstream_conn 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn_acquired 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn_released 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_released 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_reserved 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_reserved_time 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:max_downstream_reserved_time 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_freed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_quit_server 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_max_reached 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_create_failed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_failed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_timeout 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_interval 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_max_reached 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_waiting_errors 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_auth 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_auth_failed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_bucket 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_bucket_failed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_propagate_failed 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_close_on_upstream_close 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_timeout 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_wait_queue_timeout 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_downstream 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_upstream 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_recursion 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_reset_upstream_avail 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_keys 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_keys_dedupe 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_bytes_dedupe 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_optimize_sets 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_optimize_self 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry_time 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:max_retry_time 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry_vbucket 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_paused 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_unpaused 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_oom 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_upstream_write_prep 0STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_downstream_write_prep 0END

Thanks -- strangely, the first part of the stats looks clipped off.  But, in what's there, there's no smoking gun that output.

Hearing that both your moxi's hit 100% cpu at exactly/near the same time is important, though.  Some more questions...

I see you've got 2 buckets -- "default" (a memcached bucket) and a summit-related bucket.  Is the summit bucket a memcached-type or membase-type bucket (I can't tell from the clipped ouput).  Is your app using both buckets actively or just one (and which one)?

Also, what client were you using?  For example, the enyim .Net client?

Does the 100% cpu happen right away, even right after a restart?

Any clues to help us reproduce this over here are greatly appreciated!

Best,

Steve

 The summit bucket is a Memcached bucket type and the app is using only the summit bucket.  Default is not used.

We are using the .NET Membase 2.10 client.

The 100% does not happen right away.  Here is one thing that we have been able to consistently reproduce.  Output caching.  We wrote a custom output cache provider that uses Membase for the storage instead of local cache.  This is a new feature in .NET 4.0.  We are using Telerik web controls as well which does do some output caching of its own.  I think we may have been able to pin point the fact that this happens only when output caching is turned on.

 Any word on this?  Haven't heard back.

A quick update -- the inhouse efforts to reproduce this haven't yielded much yet.

Do you have any pointers to source (or can you provide some in private channel) on the output-caching custom code?  I'm wondering if it's doing anything obvious that could be visible from source inspection that gives an "a-ha" on why it induces this condition.

 Steve.  Working on getting code.

Here is an error we are receiving that I thought I'd post.  Any thoughts?

2010-11-16 15:34:19,549 [5] ERROR Membase.BucketConfigListener (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Error resolving streaming uri

System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it <replaced ip for security>:8091

   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)

   at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)

   --- End of inner exception stack trace ---

   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)

   at System.Net.WebClient.DownloadString(Uri address)

   at Membase.ConfigHelper.DeserializeUri[T](Uri uri) in d:\EnyimMemcached\Membase\ConfigHelper.cs:line 42

   at Membase.ConfigHelper.GetBucketsRoot(Uri poolUri) in d:\EnyimMemcached\Membase\ConfigHelper.cs:line 50

   at Membase.ConfigHelper.ResolveBucket(Uri poolUri, String name) in d:\EnyimMemcached\Membase\ConfigHelper.cs:line 71

   at Membase.BucketConfigListener.ResolveBucketUri(WebClientWithTimeout client, Uri uri) in d:\EnyimMemcached\Membase\BucketConfigListener.cs:line 141

2010-11-16 15:34:19,636 [5] WARN  Membase.MessageStreamListener (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Could not resolve url http://<replacedipforsecurity:8091/pools/default; trying the next in the list

2010-11-16 15:34:19,731 [5] INFO  Membase.MembasePool (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Received new configuration.

2010-11-16 15:34:19,744 [5] INFO  Membase.MembasePool (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - No vbucket. Server count: 2

2010-11-16 15:34:22,385 [1] ERROR Ebsco.Caching.Membase.MembaseCache (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Membase store operation returned false for key: www.summitstands.com_ApplicationSettings_www.summitstands.com.Default

2010-11-16 15:34:22,386 [1] WARN  Ebsco.Caching.Cache (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Error setting key "ApplicationSettings_www.summitstands.com.Default" for cache provider type "MembaseCache"

2010-11-16 15:34:23,427 [1] ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl (null) SessionID: (null) ShoppingCartID:(null) UserID:(null) - Could not init pool.

System.TimeoutException: Could not connect to <replaced ip for security>:11211

   at Enyim.Caching.Memcached.PooledSocket..ctor(IPEndPoint endpoint, TimeSpan connectionTimeout, TimeSpan receiveTimeout) in d:\EnyimMemcached\Enyim.Caching\Memcached\PooledSocket.cs:line 49

   at Enyim.Caching.Memcached.Protocol.Binary.BinaryNode.CreateSocket() in d:\EnyimMemcached\Enyim.Caching\Memcached\Protocol\Binary\BinaryNode.cs:line 35

   at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket() in d:\EnyimMemcached\Enyim.Caching\Memcached\MemcachedNode.cs:line 242

Dave, the Enyim client shouldn't be using Moxi at all.  For some reason you are connecting to the server on port 11211:

System.TimeoutException: Could not connect to <replaced ip for security>:11211

Have you changed the default port in the client-side configuration?  It "should" be going to 11210.

-Are the client and server installed onthe same or different machines?

-Just to confirm that it is the Moxi process (and not memcached) taking up %100 CPU? 

Perry

 Perry,

If Enyim doesn't use it then why is Moxi installed?  Can we some how remove it?  That's what seems to be causing the problem.  I haven't changed the default port.  As a matter of fact, our client-side configuration doesn't reference that port anywhere:

 <membase>

<!--

Bucket is optional and can be specified in the constructor, so you 

can use the same configuration to connect to different buckets in the cluster.

You must provide userName and password if the server is "secured" (membase is secured by default).

-->

<servers bucket="bucket1" bucketPassword="password" userName="Administrator" password="password">

<!--

Provide at least 2-3 urls from your cluster. 

Usually you only need to change the host, except when connecting to older versions.

Membase uses port 8091, while the NorthScale Memcached Server (1.0.3) uses 8080

-->

<add uri="http://111.222.333.72:8091/pools/default" />

<add uri="http://111.222.333.73:8091/pools/default" />

</servers>

<!-- the client waits deadTmimeout time to reconnect to the pool when all the specified urls are down -->

Are the client and server installed on the same machine?  What client are you speaking of?  The API?  The application we are using to test this lives on a separate machine.  The machines that have Membase on them are our middle tier boxes.  Is there some other "client" that we need to install?  I guess I don't understand that question.

Yes, Moxi is what will peg out the CPU but for some reason, we noticed behavior yesterday that is associated with the error I just previously posted; Membase will just go unresponsive.  If we try to go to the admin console web site, it doesn't respond and our logs fill up with errors not being able to get/set keys.  Once I restart the Membase service, it comes back.

I have sent you two emails from bhawana-at-membase-com. Please check your junk mail folder.

bhawana