Couchbase
  • Why NoSQL?
  • Couchbase Server
  • Download
  • Resources
  • Careers
Home | Forums | Membase | Membase Server 1.6.x

'Web request failed' error repeated seen in Membase cluster

2 replies [Last post]
  • Login or register to post comments
Mon, 07/25/2011 - 03:28
theburningmonk
Offline
Joined: 06/29/2010
Groups: None

Hi, we had an outage the other day with our production cache cluster and looking at the logs the server seemed to have problem communicating with one another, we see entries that says one server sees other nodes go down and then soon after it'll see them come up and other nodes will this server go down and back up, etc. etc. and soon after that we see this error message continuously in the log:
menelaus_web:19:warning:server error during request processing - Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,error},
{what,badarith},
{trace,
[{menelaus_stats,
'-last_memcached_sample/2-fun-0-',2},
{lists,foldl,3},
{menelaus_stats,last_memcached_sample,2},
{menelaus_stats,last_bucket_stats,3},
{menelaus_stats,basic_stats,3},
{ns_storage_conf,
'-do_cluster_storage_info/1-fun-1-',3},
{lists,foldl,3},
{ns_storage_conf,do_cluster_storage_info,
1}]}]
for the next 15hrs or so until we restarted the membase process on each of the servers in the cluster.

At the first instance of this error, all our servers seemed to have lost their ability to talk to the cache but then recovered and continued to work whilst the above error continued, the only thing that wasn't working was the web console on port 8091. But then later on when another process runs and tries to persist data in the cache into SimpleDB we seemed to have picked up an old version of the objects at least for some of the players, almost as if the hashing of keys to servers had been changed..

Another curious thing we noticed is that on every one of the cache servers, everyday at 10:38AM and 10:38PM we see an entry by the 'Dhcp-Client' in the system events log that says:
"Your computer was successfully assigned an address from the network, and it can now connect to other computers."
This seems to be an AWS thing, but the private and public IP address of the instance doesn't seem to be affected, at least not usually.

Sorry that's a bit long winded, but ultimately we have a couple of questions:
- what could cause those errors to happen continuously
- would the mapping from virtual buckets to cache server change if these errors occur?

Thanks,

Top
  • Login or register to post comments
Mon, 07/25/2011 - 15:17
perry
Offline
Joined: 10/11/2010
Groups:

That particular error "shouldn't" be anything to be concerned about...you might see some weird stats but the functioning of the system shouldn't be affected.

What version of Membase are you running?

If the nodes have problems communicating with each other, I would be looking at the available network bandwidth...assuming you don't see any of the servers actually crashing.m

To answer your questions, I don't know yet and no ;-)

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!

Top
  • Login or register to post comments
Mon, 07/25/2011 - 16:22
perry
Offline
Joined: 10/11/2010
Groups:

Also, do you have the logs available from those nodes?

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!

Top
  • Login or register to post comments
  • Login or register to post comments
  • Login
  • Register

Company

  • About Us
  • Leadership
  • Customers
  • Partners
  • Contact Us

Product

  • Couchbase Server
  • Couchbase SDKs
  • Use Cases
  • Documentation
  • Forums

Open Source

  • Couchbase Project
  • Couchbase vs. CouchDB

Commercial

  • Subscriptions & Support
  • Training & Services

News

  • Blog
  • Newsletter
  • Press Releases
  • Buzz

Follow Us

    
  • Customer Login
  • Terms of Service
  • Privacy Policy
  • Trademark Policy
  • Site Map

© 2013 COUCHBASE All rights reserved.

Sign in to Couchbase Community

close
  • Create new account
  • Request new password
You are logging into the Forums, Wiki and Issue Tracker