Two weeks ago McGraw Hill presented at CouchConf SF and our users expressed so much interest that I thought I’d share more details in a blog. Earlier this year McGraw Hill and Couchbase teamed up to build a proof-of-concept application showing off the power of using Couchbase and ElasticSearch together.<\/p>\n\n\n\n
The goal of the project was to build a self-adapting learning portal that delivers personalized results. Specifically that meant:<\/p>\n\n\n\n
Couchbase Server was used to store all of the content meta-data, as well as the full-text source of the text articles. This gives the application sub-millisecond latency access to the primary data set.<\/p>\n\n\n\n
ElasticSearch was chosen to handle the full-text search requirements for the application. ElasticSearch combines rich querying capabilities with excellent clustering capabilities, making it a great match with Couchbase. Integration between Couchbase Server and ElasticSearch was provided by the\u00a0Couchbase Transport plug-in<\/a>. This transport uses the Cross Data Center Replication feature of Couchbase Server 2.0 to reliably transfer all document mutations to the ElasticSearch index (Learn more about this here<\/a>).<\/p>\n\n\n\n On the front-end, the decision was made to build the application using Ruby on Rails. Our primary objective in the code being to clearly document the best practices when using Couchbase and ElasticSearch together.<\/p>\n\n\n\n Here is what a user sees when they first log in to the application.<\/p>\n\n\n\n When a user selects on a particular piece of content, the data is loaded directly from Couchbase Server by its key. Here’s a sample document in Couchbase:<\/p>\n\n\n\n {<\/span> \u00a0<\/p>\n\n\n\n Users of the system can browse content by exploring the systems top contributors and top tags.<\/p>\n\n\n\n \u00a0<\/p>\n\n\n\n Let’s take a closer look at how the top tags are determined.<\/p>\n\n\n\n First, here is the map function we’re using:<\/p>\n\n\n\n function<\/span>(<\/span>doc)<\/span>{<\/span> And we’re using the built-in reduce\u00a0_count<\/strong><\/p>\n\n\n\n When we access this view with a group_level of 1, we see each tag, and the number of times it has been used to describe a document.<\/p>\n\n\n\n \u00a0<\/p>\n\n\n\n Couchbase Views are sorted by the key, so we cannot directly query for the top 8 tags. Instead, we have a job that runs every 10 minutes, queries this view, sorts the results, and stores the top 8 results into another document in Couchbase. Here is what that document looks like:<\/p>\n\n\n\n {<\/span> Now, we have very fast access to the top-tags updated every 10 minutes.<\/p>\n\n\n\n Users of the system can also perform complex search queries.<\/p>\n\n\n\n \u00a0<\/p>\n\n\n\n Using this interface a user could search for the term “water”. This results in search query sent to ElasticSearch:<\/p>\n\n\n\n {<\/span> This particular query matches 42 documents, below is a subset of the response showing 1 document:<\/p>\n\n\n\n {<\/span>Learning Portal<\/h3>\n\n\n\n
Fast Access to Documents using Couchbase Client SDK<\/h5>\n\n\n\n
\n“title”<\/span>:<\/span> “Vince Shields”<\/span>,<\/span>
\n“url”<\/span>:<\/span> “https:\/\/en.wikipedia.org\/wiki\/Vince_Shields”<\/span>,<\/span>
\n“type”<\/span>:<\/span> “text”<\/span>,<\/span>
\n“is_text”<\/span>:<\/span> 1<\/span>,<\/span>
\n“is_video”<\/span>:<\/span> 0<\/span>,<\/span>
\n“is_image”<\/span>:<\/span> 0<\/span>,<\/span>
\n“popularity”<\/span>:<\/span> 0<\/span>,<\/span>
\n“views”<\/span>:<\/span> 0<\/span>,<\/span>
\n“categories”<\/span>:<\/span> [<\/span>
\n“1900 births”<\/span>,<\/span>
\n“1952 deaths”<\/span>,<\/span>
\n“Baseball people from New Brunswick”<\/span>,<\/span>
\n“Canadian baseball pitcher stubs”<\/span>,<\/span>
\n“Fort Smith Twins players”<\/span>,<\/span>
\n“Independence Producers players”<\/span>,<\/span>
\n“Major League Baseball pitchers”<\/span>,<\/span>
\n“Major League Baseball players from Canada”<\/span>,<\/span>
\n“People from Fredericton”<\/span>,<\/span>
\n“St. Louis Cardinals players”<\/span>
\n]<\/span>,<\/span>
\n“timestamp”<\/span>:<\/span> “2012-01-06T02:27:11Z”<\/span>,<\/span>
\n“content”<\/span>:<\/span> “{{Infobox MLB playern<\/span>|name=Vince Shields…”<\/span>,<\/span>
\n“authors”<\/span>:<\/span> [<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Chris the speller”<\/span>
\n}<\/span>
\n]<\/span>,<\/span>
\n“contributors”<\/span>:<\/span> [<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Chris the speller”<\/span>,<\/span>
\n“timestamp”<\/span>:<\/span> “2012-01-06T02:27:11Z”<\/span>
\n}<\/span>
\n]<\/span>
\n}<\/span><\/p>\n\n\n\nTop Contributors and Top Tags using Couchbase Map Reduce Views<\/h5>\n\n\n\n
\nif<\/span> (<\/span>doc.type<\/span>)<\/span>{<\/span>
\ndoc.categories<\/span>.forEach<\/span>(<\/span>function<\/span>(<\/span>category)<\/span>{<\/span>
\nemit(<\/span>category,<\/span> null<\/span>)<\/span>;<\/span>
\n}<\/span>)<\/span>;<\/span>
\n}<\/span>
\n}<\/span><\/p>\n\n\n\n
\n“tags”<\/span>:<\/span> [<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Living people”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 27554<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Persondata templates without short description parameter”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 20971<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “All articles with unsourced statements”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 13509<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Article Feedback Blacklist”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 9205<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Articles with hCards”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 9028<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Disambiguation pages”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 5912<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Articles lacking sources from December 2009”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 4158<\/span>
\n}<\/span>,<\/span>
\n{<\/span>
\n“name”<\/span>:<\/span> “Commons category template with no category set”<\/span>,<\/span>
\n“count”<\/span>:<\/span> 2904<\/span>
\n}<\/span>
\n]<\/span>
\n}<\/span><\/p>\n\n\n\n<\/h5>\n\n\n\n
Full-Text Search<\/h5>\n\n\n\n
\n“query”<\/span>:<\/span> {<\/span>
\n“query_string”<\/span>:<\/span> {<\/span>
\n“query”<\/span>:<\/span> “water”<\/span>
\n}<\/span>
\n}<\/span>
\n}<\/span><\/p>\n\n\n\n
\n“took”<\/span>:<\/span> 5<\/span>,<\/span>
\n“timed_out”<\/span>:<\/span> false<\/span>,<\/span>
\n“_shards”<\/span>:<\/span> {<\/span>
\n“total”<\/span>:<\/span> 5<\/span>,<\/span>
\n“successful”<\/span>:<\/span> 5<\/span>,<\/span>
\n“failed”<\/span>:<\/span> 0<\/span>
\n}<\/span>,<\/span>
\n“hits”<\/span>:<\/span> {<\/span>
\n“total”<\/span>