LinkedIn Monitors Massive Data with Couchbase

LinkedIn has more than 225 million members around the world, generating massive amounts of user activity –all of which needs to be logged, monitored, and analyzed. Couchbase Server provides the scalability and performance the site engineering team needs to power its metric visualization engine, ultimately delivering 400,000 operations/second on just four server nodes.

Site reliability engineers at LinkedIn are responsible for making sure LinkedIn’s 225 million members have access to the website, 24x7. A big part of this is ongoing monitoring and analysis of the massive amounts of operational data generated by site activity.

LinkedIn has developed an internal data visualization engine, called Ingraphs, to enable the site reliability engineering team to track and optimize LinkedIn’s 8+ million unique site metrics. Monitoring the metrics at LinkedIn means handling massive amounts of data: the 8+ million metrics translate to more than 16 million round-robin database (RRD) files, and it takes almost 12 terabytes of data just to write out their metrics. The large number of RRD files makes these metrics difficult to find, especially when you have to ensure the metrics are recent (since RRD files are time-based series, they are constantly being updated with the most recent data).

LinkedIn cached the RRDs to ensure extremely fast lookup. They initially implemented memcached, but found that there were limitations, especially because memcached was deployed using independent instances, resulting in data inconsistency, slow response times, and other inefficiencies.

The LinkedIn team decided to use the storage nodes to update the cache, which meant they needed a single, logical cache. They also needed the cache to be semi-durable, fast, and scalable to handle the new content and metrics that are continually added. LinkedIn replaced its instances of memcached with a Couchbaser Server cluster.

The results have been extremely positive. Today, they can run 400K ops/second with their four-node cluster, and load 16 million entries into Couchbase every 5 minutes.

Watch the video for more details of Couchbase at LinkedIn.

Couchbase Server is more awesome than a double rainbow.

– Grier Johnson, Site Reliability Engineer

LinkedIn is the world's largest professional network with 225 million members in over 200 countries and territories around the globe. LinkedIn is publicly held and has a diversified business model with revenues coming from member subscriptions, advertising sales, and talent solutions.