{"id":5308,"date":"2018-06-13T07:15:51","date_gmt":"2018-06-13T14:15:51","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=5308"},"modified":"2025-06-13T19:28:08","modified_gmt":"2025-06-14T02:28:08","slug":"data-visualization-couchbase-knowi","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/data-visualization-couchbase-knowi\/","title":{"rendered":"Data Visualization with Couchbase and Knowi"},"content":{"rendered":"
\n

Note:<\/strong> This post uses the the Couchbase Analytics Data Definition Language as of the version 5.5 preview release.\u00a0 For updates and information on breaking changes in newer versions, please refer to\u00a0Changes to the Couchbase Analytics Service<\/a>.<\/p>\n

Data visualization and reporting can help decision makers to identify patterns and concepts in analytics. Couchbase Analytics (currently in developer preview) provides a seamless NoETL experience (video)<\/a> to analyze data with powerful queries. Knowi<\/a> is a cloud-based reporting and visualization tool that integrates with Couchbase. These tools can work together to provide powerful and easy data visualization.<\/p>\n<\/div>\n

\n

As I blogged about previously, the Cincinnati Reds (a Couchbase customer) are using Apache NiFi<\/a> to process data from their enterprise into a Couchbase Cluster. One of the many use cases they are exploring with Couchbase is to create a real-time dashboard. This dashboard would, among other things, show a data visualization of the ballpark after the gates open and tickets are starting to be scanned.<\/p>\n<\/div>\n

\n

This real-time data visualization can help show patterns at the ballpark, and give the Reds an idea of when ticket holders are showing up and where they are going.<\/p>\n<\/div>\n

\n

In this blog post, I\u2019m going to show you an example of Couchbase and Knowi in action to provide a data visualization of Great American Ball Park (where the Cincinnati Reds play their home games). What I\u2019m showing in this post is not exactly what the Reds are using: it has been simplified.<\/em><\/p>\n<\/div>\n

\n

Data in Couchbase<\/h2>\n
\n
\n

The Reds have a sophisticated mapping system to match a section, row, and seat number to a specific spot on a map graphic. For this blog post, however, I\u2019m going to use latitude and longitude with Google Maps.<\/p>\n<\/div>\n

\n

The data I\u2019m working with is a simplified document consisting of a ticket holder\u2019s name, geo-coordinates, and a type<\/code> field. An example document:<\/p>\n<\/div>\n

\n
\n
{\r\n\t\"name\": \"Sparky Anderson\",\r\n\t\"geo\": {\r\n\t\t\"lat\": 39.097303,\r\n\t\t\"lon\": -84.507277\r\n\t},\r\n\t\"type\": \"ticketScan\"\r\n}<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
I would normally expect mapping data to be separate, and possibly joined on-the-fly with a ticket scan. But let\u2019s assume that\u2019s been done already. We have all we need in a single document.<\/p>\n<\/div>\n
\n
I\u2019ve created a bucket called “tickets” which will contain documents like the above. In production, data will continually flow into this bucket (probably through NiFi) during a baseball game. In fact, I recently attended a game and helped test this system as I was walking through the gates and getting my ticket scanned during peak times at a weekend game.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
Couchbase Analytics<\/h2>\n
\n
\n
Knowi could connect to this bucket directly and start reporting on it. However, it might be better to use Couchbase Analytics in this situation. Using Couchbase Analytics means we don\u2019t have to worry about writing indexes for more complex queries, and it also means we will reduce any performance impact on the operational bucket.<\/p>\n<\/div>\n
\n
To get started with Couchbase Analytics, make sure you\u2019ve enabled the Analytics service when you installed Couchbase. The analytics service should appear in the “Servers” tab:<\/p>\n<\/div>\n
\n
\"Analytics<\/span><\/p>\n<\/div>\n
\n
Next, go to the “Analytics” tab. You will see a screen that looks very similar to the normal N1QL Query Workbench.<\/p>\n<\/div>\n
\n
Setting up analytics<\/h3>\n
\n
There are three simple steps for getting started.<\/p>\n<\/div>\n
\n
First, create an analytics bucket. This bucket will correspond to a regular Couchbase bucket. Simply enter a command like this: CREATE BUCKET ticketsAnalytics WITH {\"name\":\"tickets\"};<\/code><\/p>\n<\/div>\n
\n
Next, define a dataset from within that bucket. You can specify which documents that should go into that dataset by using a WHERE<\/code> clause. Example:<\/p>\n<\/div>\n
\n
\n
CREATE SHADOW DATASET ticketScans ON ticketsAnalytics WHERE `type` = \"ticketScan\";<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
This is a “shadow”: a real-time copy of (a subset of) a Couchbase bucket.<\/p>\n<\/div>\n
\n
Finally, kick off the shadowing with one more command: CONNECT BUCKET ticketsAnalytics;<\/code><\/p>\n<\/div>\n
\n
At this point, ticketScan documents from the Couchbase bucket “tickets” will start to be copied over into the Analytics bucket “ticketsAnalytics”.<\/p>\n<\/div>\n
\n
It will start happening immediately. You can run a query like select * from ticketScans;<\/code> to see the data in Analytics.<\/p>\n<\/div>\n
\n
You should also see a visual representation of this setup under “Bucket Insights” at the top right of the page.<\/p>\n<\/div>\n
\n
\"Bucket<\/span><\/p>\n<\/div>\n
\n
Note that Analytics is still in developer preview at this time, and the above steps may change after general release (currently planned for later this year).<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n
Setting up the Knowi Agent (optional)<\/h2>\n
\n
\n
You can create a free 14-day trial of Knowi<\/a>.<\/p>\n<\/div>\n
\n
Knowi is cloud-based, so you don\u2019t necessarily need to download anything. You can build reports and visualizations right in your browser.<\/p>\n<\/div>\n
\n
However, if your Couchbase cluster isn\u2019t exposed to the internet, or you don\u2019t have a VPN tunnel set up, you can still use Knowi by running the Knowi Agent on your network. This is an easy way for Knowi to get access to your data. My local Couchbase cluster is not exposed to the internet, so I simply downloaded the Knowi Agent:<\/p>\n<\/div>\n
\n
\"Download<\/span><\/p>\n<\/div>\n
\n
You\u2019ll need to create a datasource JSON file in the \/config folder. This is really easy to do, because there are a ton of example files in the \/example folder. For Couchbase, check out datasource_example_couchbase.json<\/code>. For my local cluster, I created a file with these contents:<\/p>\n<\/div>\n
\n
\n
[\r\n    {\r\n        \"name\": \"demoCouchbase\",\r\n        \"host\": \"localhost\",\r\n        \"dbName\": \"matt\",\r\n        \"password\": \"password\",\r\n        \"datasource\": \"couchbase\"\r\n    }\r\n]<\/code><\/pre>\n<\/div>\n<\/div>\n
\n
This assumed you created a user named “matt” in Couchbase with permission to use Analytics.<\/em><\/p>\n<\/div>\n
\n
The Agent comes with some handy shell scripts and batch files. It can run as a service, or you can just do what I did and run run.bat<\/code> to kick off the agent. At this point, Knowi will be able to communicate with your Couchbase cluster through the agent.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
Setting up a data source in Knowi<\/h2>\n
\n
\n
Now go back to the Knowi.com UI. Create a new datasource. You can give it whatever name you\u2019d like. If you\u2019re using the agent like me, set “localhost” to be the data source. I also specified a bucket name of “tickets”. I entered the user credentials. For Analytics, you need to specify a “Dataverse”. Out of the box, there is already a “Default” dataverse available.<\/p>\n<\/div>\n
\n
Finally, check the box for “Internal Datasource”. If the agent is working properly, you will see a dropdown and be able to select the agent. If you aren\u2019t using an Agent, then don\u2019t check that box.<\/p>\n<\/div>\n
\n
\"New<\/span><\/p>\n<\/div>\n
\n
At this point, you can use this data source to create 1 or more Knowi data feeds.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
Knowi Data Feeds<\/h2>\n
\n
\n
A data feed \/ data set in Knowi is analogous to a query. Here is the (very simple) data feed that I created to get the ticket information from Couchbase.<\/p>\n<\/div>\n
\n
\"Knowi<\/span><\/p>\n<\/div>\n
\n
Notice that Knowi supports N1QL natively. All the power of Couchbase\u2019s N1QL language is available to you when creating a data feed. My query is a very simple one just for demonstration.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
Knowi widget<\/h2>\n
\n
\n
Finally, I can create a Knowi widget. Widgets are individual reports or data visualizations. There are a variety of widgets available, and you can even customize and combine widgets. I\u2019m going to keep it simple and create a single widget. Click “New Widget+” and then select the data set that you created in the previous section.<\/p>\n<\/div>\n
\n
Next, click the “settings” cog next to the widget. There are a lot of options for reporting and data visualization. Definitely check them out and experiment with them. I\u2019m going right for “Geo-Heatmaps”.<\/p>\n<\/div>\n
\n
\"Geo<\/span><\/p>\n<\/div>\n
\n
I\u2019ve chosen the middle of Great American Ball Park for the Center Latitude and Center Longitude (39.097208,-84.506530) and a high zoom level. The radius represents how big you want the spots on the heatmap to be. Mine correspond to seats in a baseball stadium, so I\u2019ve entered a relatively small value of 5.<\/p>\n<\/div>\n
\n
That\u2019s it. Save the widget. Click the eyeball icon to see a preview. Here\u2019s a preview with just 6 tickets total in my Couchbase bucket.<\/p>\n<\/div>\n
\n
\"Geo<\/span><\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
Data Visualization Dashboard<\/h2>\n
\n
\n
Finally, I\u2019m going to create a dashboard with this data visualization on it. Go to the Knowi dashboard, click “Add Widget to Dashboard” and select the widget. The widget can then be resized and moved around. You can add other widgets for a more complete dashboard. This is what the Reds are working on: just seeing a visualization by itself isn\u2019t enough. They also want to report on the overall trends: what sections are the fullest, how many total tickets have been scanned over a time period, etc.<\/p>\n<\/div>\n
\n
I created a .NET program (source code available here)<\/a> to start randomly adding tickets. I wanted to see a real-time update of this data visualization. My example is not entirely accurate: I didn\u2019t want to spend the time mapping every seat to latitude and longitude myself. But it\u2019s close enough for you to get the idea.<\/p>\n<\/div>\n
\n
\"Realtime<\/span><\/p>\n<\/div>\n
\n
(This clip represents a 5-minute period of my sample program running, compressed down into 30 seconds).<\/em><\/p>\n<\/div>\n
\n
The distribution is random, and it\u2019s within two arbitrary boxed areas. This is the part where the Cincinnati Reds have more expertise in actually mapping their data to a more efficient seat map, but hopefully you get the idea.<\/p>\n<\/div>\n
\n
Note that for a data visualization to be real-time (like above), you can\u2019t use the agent. I had to give direct access to Knowi for a “Direct Query” checkbox to appear when I was creating a data feed (so I used Couchbase on Microsoft Azure<\/a>).<\/p>\n<\/div>\n
\n
This particular visualization might also be useful for a real-time outage map for a utility company. You could create a dashboard to report internet, power, gas, or phone outages on a map.<\/p>\n<\/div>\n<\/div>\n<\/div>\n
\n
What\u2019s next?<\/h2>\n
\n
\n
In this post, you saw how easy it was to use Knowi and Couchbase Analytics to start building real-time data visualizations.<\/p>\n<\/div>\n
\n
Couchbase provides the flexibility of N1QL and the performance of a memory-first architecture, and Knowi provides the data visualization and dashboard capabilities.<\/p>\n<\/div>\n
\n
Remember that Knowi has a free trial. I\u2019ve worked with Knowi on this Reds project only for a short time, but I\u2019ve found their support to be helpful and responsive and their documentation to be complete and useful.<\/p>\n<\/div>\n
\n
Some next steps for you:<\/p>\n<\/div>\n
\n