Uneven network traffic between one Couchbase server and webservers

We are seeing strange behavior with one of our servers in Couchbase cluster. Network traffic on that server is 10 times greater than on the remaining servers. We were thinking that this can be caused by replication but we’ve crossed that out because the traffic is between cache servers and web servers.

This problems leads to situation when we are getting timeouts from the couchbase client library when our site is under heavy load (network path to affected server seems to be overloaded).

What’s even more strange is fact that we didn’t have this problem in the past.

We are using PHP SDK and our connection string contains all servers.

Server access is primarily determined by the type operations you’re performing and the items you’re accessing. If your data access pattern is uneven (that is, you’re accessing some keys more often than others), then the access pattern to your servers will be biased towards those frequently used keys.

The servers in the connection string only affect which servers are initially contacted to get the cluster information. Once the client is bootstrapped, the nodes in the connection string are discarded.

Just to add to mnunberg’s response - you can look at the Top Keys section on the Bucket Details page in the GUI to see the keys of the most frequently-accessed documents, which may assist you in understanding the applications’ behaviour.

There was one key which was requested very often and it was large. We solved the issue by splitting into smaller items.