Elastic Data and the Cloud

(Also see video blog)
Today we announced a very exciting relationship with Heroku: Heroku’s Memcached Add-On is powered by NorthScale Memcached Server and NorthScale is operating the add-on service on Heroku’s behalf. For those who haven’t checked out Heroku, they are the leading Ruby on Rails Platform as a Service (PaaS) cloud service provider, with over 55,000 applications running on their platform. In just a few weeks since opening the add-on for public beta, we’ve added over 300 new companies to the NorthScale Memcached Server customer roster; with adoption accelerating. This week the NorthScale add-on is generally available from Heroku with pricing announced by Heroku today.

One of the unique features of NorthScale Memcached Server is its ability to support application multi-tenancy – critically important for a successful cloud service offering. In a cloud environment, many applications are able to share a virtualized infrastructure. Applications have widely varying needs. Some applications will be very busy, require a lot of memory and generate a substantial transaction load on a memcached instance. Some applications will be less resource intensive. In the case of our customer base on the Heroku platform, some customers run just a few transactions per second a large percentage of the time while others are consistently generating thousands of memcached operations per second.

Rather than require every application to deploy its own server (or virtual server), with its own operating system, each with its own administration requirements and overhead, NorthScale Memcached Server is itself virtualized – carving up memcached into virtual memcached instances – allowing many applications to securely and efficiently share an underlying memcached service. Rather than spin up hundreds of servers, each running NorthScale Memcached Server, we are able to manage a handful of servers that are, in the aggregate, more than capable of meeting the needs of the over 300 applications currently running on the platform. That’s the beauty of cloud computing – one cluster to manage and administer, extremely rapid and easy provisioning of new “buckets” to onboard new customers, and very efficient utilization of the underlying hardware. Of course, if a customer desires their own dedicated resources and machines, they can choose to deploy their own instance of NorthScale Memcached Server.

Beyond the coolness of adding a few hundred more customers to the fold and getting to work with the leading PaaS cloud provider, this announcement underscores something that has not really received a lot of attention: the need for elastic data infrastructure software to enable public and private cloud computing environments.

I spent a couple years at VMware before NorthScale. VMware clearly has its sights set on becoming the dominant provider of cloud infrastructure software: providing software enabling IT organizations to build cloud computing environments in their data centers.

I lost count of the number of times customers indicated they believed there was a big piece missing from our cloud offerings -namely the data management piece. We’d talk about how virtualization technology (VMware vSphere), combined with management and automation technology (VMware vCenter) and an application framework (like SpringSource, now part of VMware) enabled the construction of a cloud computing environment, much like the one at Heroku: “By spinning up new virtual machines, pre-configured with a Spring application framework, dropping application logic in the container, and updating a load balancer (all orchestrated by vCenter) an application can expand and contract very easily to increase or reduce application resources to match demand. We call this elasticity.”

But, inevitably, the conversation would turn to the data: “that’s great for the application, but what about my data? It is still stored in a complex, refrigerator-sized box, with proprietary and complex server technology, and heavyweight relational database management software. It doesn’t even approximate the model you’ve just described. And we know that Google, Amazon and others don’t manage their data that way. Where’s the rest of the solution”

NorthScale was formed explicitly to provide the rest of the solution – elastic data management software optimized for building cloud computing environments and storing the data behind the web applications that typically run in those environments. Ultimately, our announcement with Heroku is a very visible rendering of our vision. NorthScale technology is providing elastic data caching services to hundreds, going to thousands, of applications on Heroku – one of the most successful public clouds today. The same technology that powers data management at Heroku is packaged in the form of NorthScale Memcached Server and NorthScale Membase Server, allowing enterprise organizations to build private cloud computing environments in their own data centers. It can be freely downloaded and easily installed in your own environment.