Home | Forums | Membase | Membase Server 1.6.x

Huge memory leak

7 replies [Last post]
Sat, 11/06/2010 - 05:00
diego
Offline
Joined: 11/06/2010
Groups: None

Notice the memory usage on the moxi process. Using centos5 64bit and the memory leak issue is present in all 4 of my membase cluster after a period of time. They were running the 1.6.0 release with only a single memcached based buckets of 6GB each for 24GB total.

4759 membase   15   0 8888m 8.5g 1268 S 10.0 36.2 291:14.08 moxi
 4760 membase   15   0 1517m 1.3g  892 S  6.0  5.4  57:41.38 memcached
 4724 membase   25   0  246m  53m 2260 S  2.0  0.2   6:26.97 beam.smp

Top
Sat, 11/06/2010 - 05:02
diego
Offline
Joined: 11/06/2010
Groups: None

More on how the membase cluster is used. The 4 membase servers are setup to be round-robined via my tcp load balancer to the moxi proxy 11211 port.

Top
Sat, 11/06/2010 - 09:07
bhawana@membase
Offline
Joined: 10/29/2010
Groups: None

deigo,

 

Can you please run some stats on your moxy.


Please run the following command and let us know the output:

echo "stats proxy" | nc 10.X.X.XXX 11211 | sort

 

Thanks

Bhawana

 

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!

Top
Sat, 11/06/2010 - 09:07
bhawana@membase
Offline
Joined: 10/29/2010
Groups: None

deigo,

 

Can you please run some stats on your moxy.


Please run the following command and let us know the output:

echo "stats proxy" | nc 10.X.X.XXX 11211 | sort

 

Thanks

Bhawana

 

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!

Top
Sat, 11/06/2010 - 19:44
diego
Offline
Joined: 11/06/2010
Groups: None

Here is the "stats" output

 

stats
STAT delete_misses 0
STAT rejected_conns 0
STAT connection_structures 225
STAT reclaimed 86915498
STAT limit_maxbytes 67108864
STAT decr_hits 0
STAT curr_connections 225
STAT rusage_system 9339.666016
STAT cas_misses 0
STAT bytes 1935705794
STAT get_misses 75373828
STAT bytes_read 146157365475
STAT get_hits 699269234
STAT decr_misses 0
STAT rusage_user 7383.335938
STAT bucket_conns 195
STAT cmd_flush 0
STAT uptime 88189
STAT total_items 150937726
STAT incr_hits 0
STAT time 1289097305
STAT daemon_connections 30
STAT pointer_size 64
STAT version 1.4.4_298_g250909b
STAT engine_maxbytes 18874368000
STAT evictions 0
STAT total_connections 6208
STAT curr_items 30620806
STAT delete_hits 0
STAT cas_hits 0
STAT auth_errors 0
STAT threads 12
STAT pid 18694
STAT auth_cmds 277
STAT cas_badval 0
STAT cmd_set 150937726
STAT cmd_get 774643062
STAT conn_yields 0
STAT bytes_written 204352512367
STAT libevent 1.4.13-stable
STAT incr_misses 0
END

 

 

Here is the "stats proxy" output of a server at 126MB moxi ram usage. Another server which has been running about 12 hours has only 11MB ram usage. I restarted the 8+GB processes so no longer have stats to them.

 

stats proxy
STAT basic:version 1.6.0
STAT basic:nthreads 5
STAT basic:hostname tofu1.fanfiction.net
STAT memcached:settings:maxbytes 67108864
STAT memcached:settings:maxconns 1024
STAT memcached:settings:tcpport 0
STAT memcached:settings:udpport -2
STAT memcached:settings:inter NULL
STAT memcached:settings:verbosity 0
STAT memcached:settings:oldest 0
STAT memcached:settings:evictions on
STAT memcached:settings:domain_socket NULL
STAT memcached:settings:umask 700
STAT memcached:settings:growth_factor 1.25
STAT memcached:settings:chunk_size 48
STAT memcached:settings:num_threads 5
STAT memcached:settings:stat_key_prefix :
STAT memcached:settings:detail_enabled no
STAT memcached:settings:reqs_per_event 20
STAT memcached:settings:cas_enabled yes
STAT memcached:settings:tcp_backlog 1024
STAT memcached:settings:binding_protocol auto-negotiate
STAT memcached:stats:pid 18693
STAT memcached:stats:uptime 88370
STAT memcached:stats:time 1289097486
STAT memcached:stats:version 1.6.0
STAT memcached:stats:pointer_size 64
STAT memcached:stats:rusage_user 3922.389706
STAT memcached:stats:rusage_system 7146.197612
STAT memcached:stats:curr_connections 62
STAT memcached:stats:total_connections 9764818
STAT memcached:stats:connection_structures 10201
STAT memcached:stats:cmd_get 0
STAT memcached:stats:cmd_set 0
STAT memcached:stats:cmd_flush 0
STAT memcached:stats:get_hits 0
STAT memcached:stats:get_misses 0
STAT memcached:stats:delete_misses 0
STAT memcached:stats:delete_hits 0
STAT memcached:stats:incr_misses 0
STAT memcached:stats:incr_hits 0
STAT memcached:stats:decr_misses 0
STAT memcached:stats:decr_hits 0
STAT memcached:stats:cas_misses 0
STAT memcached:stats:cas_hits 0
STAT memcached:stats:cas_badval 0
STAT memcached:stats:bytes_read 170201299044
STAT memcached:stats:bytes_written 93838055589
STAT memcached:stats:limit_maxbytes 67108864
STAT memcached:stats:accepting_conns 1
STAT memcached:stats:listen_disabled_num 2
STAT memcached:stats:threads 5
STAT memcached:stats:conn_yields 6
STAT proxy_main:conf_type dynamic
STAT proxy_main:behavior:cycle 0
STAT proxy_main:behavior:downstream_max 4
STAT proxy_main:behavior:downstream_conn_max 0
STAT proxy_main:behavior:downstream_weight 0
STAT proxy_main:behavior:downstream_retry 1
STAT proxy_main:behavior:downstream_protocol 8
STAT proxy_main:behavior:downstream_timeout 0
STAT proxy_main:behavior:wait_queue_timeout 0
STAT proxy_main:behavior:time_stats 0
STAT proxy_main:behavior:connect_max_errors 0
STAT proxy_main:behavior:connect_retry_interval 0
STAT proxy_main:behavior:front_cache_max 200
STAT proxy_main:behavior:front_cache_lifespan 0
STAT proxy_main:behavior:front_cache_spec
STAT proxy_main:behavior:front_cache_unspec
STAT proxy_main:behavior:key_stats_max 4000
STAT proxy_main:behavior:key_stats_lifespan 0
STAT proxy_main:behavior:key_stats_spec
STAT proxy_main:behavior:key_stats_unspec
STAT proxy_main:behavior:optimize_set
STAT proxy_main:behavior:usr admin
STAT proxy_main:behavior:host
STAT proxy_main:behavior:port 0
STAT proxy_main:behavior:bucket
STAT proxy_main:behavior:port_listen 11211
STAT proxy_main:behavior:default_bucket_name default
STAT proxy_main:stats:stat_configs 8
STAT proxy_main:stats:stat_config_fails 0
STAT proxy_main:stats:stat_proxy_starts 2
STAT proxy_main:stats:stat_proxy_start_fails 0
STAT proxy_main:stats:stat_proxy_existings 7
STAT proxy_main:stats:stat_proxy_shutdowns 0
STAT 11211:default:info:port 11211
STAT 11211:default:info:name default
STAT 11211:default:info:config 208.121.227.111:11210,208.121.227.113:11210,208.121.227.114:11210
STAT 11211:default:info:config_ver 8
STAT 11211:default:info:behaviors_num 3
STAT 11211:default:behavior:downstream_max 4
STAT 11211:default:behavior:downstream_conn_max 0
STAT 11211:default:behavior:downstream_weight 0
STAT 11211:default:behavior:downstream_retry 1
STAT 11211:default:behavior:downstream_protocol 8
STAT 11211:default:behavior:downstream_timeout 0
STAT 11211:default:behavior:wait_queue_timeout 0
STAT 11211:default:behavior:time_stats 0
STAT 11211:default:behavior:connect_max_errors 0
STAT 11211:default:behavior:connect_retry_interval 0
STAT 11211:default:behavior:front_cache_max 200
STAT 11211:default:behavior:front_cache_lifespan 0
STAT 11211:default:behavior:front_cache_spec
STAT 11211:default:behavior:front_cache_unspec
STAT 11211:default:behavior:key_stats_max 4000
STAT 11211:default:behavior:key_stats_lifespan 0
STAT 11211:default:behavior:key_stats_spec
STAT 11211:default:behavior:key_stats_unspec
STAT 11211:default:behavior:optimize_set
STAT 11211:default:behavior:usr default
STAT 11211:default:behavior:host
STAT 11211:default:behavior:port 0
STAT 11211:default:behavior:bucket
STAT 11211:default:behavior:port_listen 11211
STAT 11211:default:behavior:default_bucket_name default
STAT 11211:default:behavior-0:downstream_weight 0
STAT 11211:default:behavior-0:downstream_retry 1
STAT 11211:default:behavior-0:downstream_protocol 8
STAT 11211:default:behavior-0:downstream_timeout 0
STAT 11211:default:behavior-0:usr default
STAT 11211:default:behavior-0:host 208.121.227.111
STAT 11211:default:behavior-0:port 11210
STAT 11211:default:behavior-0:bucket
STAT 11211:default:behavior-1:downstream_weight 0
STAT 11211:default:behavior-1:downstream_retry 1
STAT 11211:default:behavior-1:downstream_protocol 8
STAT 11211:default:behavior-1:downstream_timeout 0
STAT 11211:default:behavior-1:usr default
STAT 11211:default:behavior-1:host 209.151.227.113
STAT 11211:default:behavior-1:port 11210
STAT 11211:default:behavior-1:bucket
STAT 11211:default:behavior-2:downstream_weight 0
STAT 11211:default:behavior-2:downstream_retry 1
STAT 11211:default:behavior-2:downstream_protocol 8
STAT 11211:default:behavior-2:downstream_timeout 0
STAT 11211:default:behavior-2:usr default
STAT 11211:default:behavior-2:host 209.151.227.114
STAT 11211:default:behavior-2:port 11210
STAT 11211:default:behavior-2:bucket
STAT 11211:default:stats:listening 2
STAT 11211:default:stats:listening_failed 0
STAT 11211:default:frontcache:max 0
STAT 11211:default:frontcache:oldest_live 0
STAT 11211:default:frontcache:tot_get_hits 0
STAT 11211:default:frontcache:tot_get_expires 0
STAT 11211:default:frontcache:tot_get_misses 0
STAT 11211:default:frontcache:tot_get_bytes 0
STAT 11211:default:frontcache:tot_adds 0
STAT 11211:default:frontcache:tot_add_skips 0
STAT 11211:default:frontcache:tot_add_fails 0
STAT 11211:default:frontcache:tot_add_bytes 0
STAT 11211:default:frontcache:tot_deletes 0
STAT 11211:default:frontcache:tot_evictions 0
STAT 11211:default:pstd_stats:num_upstream 11
STAT 11211:default:pstd_stats:tot_upstream 9735563
STAT 11211:default:pstd_stats:num_downstream_conn 29201
STAT 11211:default:pstd_stats:tot_downstream_conn 29251
STAT 11211:default:pstd_stats:tot_downstream_conn_acquired 257977304
STAT 11211:default:pstd_stats:tot_downstream_conn_released 257948101
STAT 11211:default:pstd_stats:tot_downstream_released 257977417
STAT 11211:default:pstd_stats:tot_downstream_reserved 257977302
STAT 11211:default:pstd_stats:tot_downstream_reserved_time 0
STAT 11211:default:pstd_stats:max_downstream_reserved_time 0
STAT 11211:default:pstd_stats:tot_downstream_freed 99
STAT 11211:default:pstd_stats:tot_downstream_quit_server 29203
STAT 11211:default:pstd_stats:tot_downstream_max_reached 20621
STAT 11211:default:pstd_stats:tot_downstream_create_failed 0
STAT 11211:default:pstd_stats:tot_downstream_connect 18603
STAT 11211:default:pstd_stats:tot_downstream_connect_failed 29153
STAT 11211:default:pstd_stats:tot_downstream_connect_timeout 0
STAT 11211:default:pstd_stats:tot_downstream_connect_interval 0
STAT 11211:default:pstd_stats:tot_downstream_connect_max_reached 0
STAT 11211:default:pstd_stats:tot_downstream_waiting_errors 0
STAT 11211:default:pstd_stats:tot_downstream_auth 98
STAT 11211:default:pstd_stats:tot_downstream_auth_failed 18505
STAT 11211:default:pstd_stats:tot_downstream_bucket 98
STAT 11211:default:pstd_stats:tot_downstream_bucket_failed 0
STAT 11211:default:pstd_stats:tot_downstream_propagate_failed 29153
STAT 11211:default:pstd_stats:tot_downstream_close_on_upstream_close 0
STAT 11211:default:pstd_stats:tot_downstream_timeout 0
STAT 11211:default:pstd_stats:tot_wait_queue_timeout 0
STAT 11211:default:pstd_stats:tot_assign_downstream 257977302
STAT 11211:default:pstd_stats:tot_assign_upstream 257977302
STAT 11211:default:pstd_stats:tot_assign_recursion 0
STAT 11211:default:pstd_stats:tot_reset_upstream_avail 0
STAT 11211:default:pstd_stats:tot_multiget_keys 0
STAT 11211:default:pstd_stats:tot_multiget_keys_dedupe 0
STAT 11211:default:pstd_stats:tot_multiget_bytes_dedupe 0
STAT 11211:default:pstd_stats:tot_optimize_sets 0
STAT 11211:default:pstd_stats:tot_optimize_self 0
STAT 11211:default:pstd_stats:tot_retry 45
STAT 11211:default:pstd_stats:tot_retry_time 0
STAT 11211:default:pstd_stats:max_retry_time 0
STAT 11211:default:pstd_stats:tot_retry_vbucket 0
STAT 11211:default:pstd_stats:tot_upstream_paused 257977257
STAT 11211:default:pstd_stats:tot_upstream_unpaused 257977257
STAT 11211:default:pstd_stats:err_oom 0
STAT 11211:default:pstd_stats:err_upstream_write_prep 0
STAT 11211:default:pstd_stats:err_downstream_write_prep 0
STAT 11211:default:pstd_stats_cmd:regular_get:seen 215352919
STAT 11211:default:pstd_stats_cmd:regular_get_key:hits 194135189
STAT 11211:default:pstd_stats_cmd:regular_get_key:write_bytes 42754129985
STAT 11211:default:pstd_stats_cmd:regular_set:seen 42601837
STAT 11211:default:pstd_stats_cmd:regular_set:read_bytes 28140678620
STAT 11211:default:pstd_stats_cmd:regular_delete:seen 22500
STAT 11211:default:pstd_stats_cmd:regular_delete:read_bytes 877500
STAT 11211:default:pstd_stats_cmd:regular_stats:seen 7
STAT 11211:default:pstd_stats_cmd:regular_stats:read_bytes 71
STAT 11211:default:pstd_stats_cmd:regular_quit:seen 9466240
STAT 11211:default:pstd_stats_cmd:regular_quit:read_bytes 37864960
STAT 11211:default:pstd_stats_cmd:regular_ERROR:seen 6
STAT 11211:default:pstd_stats_cmd:regular_ERROR:read_bytes 58
STAT 11211:[ <NULL_BUCKET> ]:info:port 11211
STAT 11211:[ <NULL_BUCKET> ]:info:name [ <NULL_BUCKET> ]
STAT 11211:[ <NULL_BUCKET> ]:info:config
STAT 11211:[ <NULL_BUCKET> ]:info:config_ver 0
STAT 11211:[ <NULL_BUCKET> ]:info:behaviors_num 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_max 4
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_conn_max 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_weight 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_retry 1
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_protocol 8
STAT 11211:[ <NULL_BUCKET> ]:behavior:downstream_timeout 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:wait_queue_timeout 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:time_stats 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:connect_max_errors 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:connect_retry_interval 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_max 200
STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_lifespan 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_spec
STAT 11211:[ <NULL_BUCKET> ]:behavior:front_cache_unspec
STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_max 4000
STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_lifespan 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_spec
STAT 11211:[ <NULL_BUCKET> ]:behavior:key_stats_unspec
STAT 11211:[ <NULL_BUCKET> ]:behavior:optimize_set
STAT 11211:[ <NULL_BUCKET> ]:behavior:usr admin
STAT 11211:[ <NULL_BUCKET> ]:behavior:host
STAT 11211:[ <NULL_BUCKET> ]:behavior:port 0
STAT 11211:[ <NULL_BUCKET> ]:behavior:bucket
STAT 11211:[ <NULL_BUCKET> ]:behavior:port_listen 11211
STAT 11211:[ <NULL_BUCKET> ]:behavior:default_bucket_name default
STAT 11211:[ <NULL_BUCKET> ]:stats:listening 2
STAT 11211:[ <NULL_BUCKET> ]:stats:listening_failed 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:max 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:oldest_live 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_hits 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_expires 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_misses 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_get_bytes 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_adds 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_skips 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_fails 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_add_bytes 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_deletes 0
STAT 11211:[ <NULL_BUCKET> ]:frontcache:tot_evictions 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:num_upstream 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:num_downstream_conn 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn_acquired 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_conn_released 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_released 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_reserved 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_reserved_time 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:max_downstream_reserved_time 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_freed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_quit_server 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_max_reached 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_create_failed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_failed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_timeout 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_interval 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_connect_max_reached 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_waiting_errors 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_auth 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_auth_failed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_bucket 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_bucket_failed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_propagate_failed 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_close_on_upstream_close 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_downstream_timeout 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_wait_queue_timeout 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_downstream 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_upstream 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_assign_recursion 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_reset_upstream_avail 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_keys 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_keys_dedupe 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_multiget_bytes_dedupe 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_optimize_sets 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_optimize_self 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry_time 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:max_retry_time 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_retry_vbucket 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_paused 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:tot_upstream_unpaused 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_oom 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_upstream_write_prep 0
STAT 11211:[ <NULL_BUCKET> ]:pstd_stats:err_downstream_write_prep 0
END

 

 

 

 

Top
Mon, 11/08/2010 - 12:23
bhawana@membase
Offline
Joined: 10/29/2010
Groups: None

Please let us know when you see memory leak again and report it with the "stats proxy" .

-Bhawana

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!

Top
Mon, 11/08/2010 - 12:23
bhawana@membase
Offline
Joined: 10/29/2010
Groups: None

Please let us know when you see memory leak again and report it with the "stats proxy" .

-Bhawana

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!

Top
Mon, 11/08/2010 - 14:37
steve
Offline
Joined: 03/15/2010
Groups: None

Hi, in addition, there seems to be some sort of connection leak.

From looking at the "stats proxy" output (thanks for supplying that), here are the suspicious parts...

 STAT memcached:stats:connection_structures 10201

And here...

 STAT 11211:default:pstd_stats:num_downstream_conn 29201
 STAT 11211:default:pstd_stats:tot_downstream_conn 29251

Are you seeing lots of client-side errors (your application sees many "SERVER_ERROR" responses from moxi)?

The "stats proxy" output shows that moxi has had trouble connecting and/or authenticating to the memcached servers (and is apparently not free'ing up resources correctly on an auth failure).  The auth failure, however, should not be happening.

 STAT 11211:default:pstd_stats:tot_downstream_connect_failed 29153
 STAT 11211:default:pstd_stats:tot_downstream_auth_failed 18505
 STAT 11211:default:pstd_stats:tot_downstream_propagate_failed 29153

Does your default bucket have a password?

Steve

Top