Introduction to offline data storage and sync with PouchBD and Couchbase

Peter Mbanugo, is a Software Developer with experience working with the Microsoft .NET stack. He’s passionate about building quality software, with interest area around Offline Apps and software architecture.

Offline-first is an approach to software development that is different from the traditional approach – where lack of network connection is treated as an error, thereby affecting the overall user experience. With the offline-first approach, you start with the most constrained environment and later respond to the user experience of your application by progressively enhancing it as the functionalities become available. For offline-first, it is assumed that lack of network connection creates an unsatisfactory user experience; therefore, offline-first seeks to provide the best possible user experience in all conditions.

The offline-first approach comes with some concerns such as, how to store data or manage transactions when offline and sync it with the server, how to keep the offline data secure, or how to resolve data conflicts if two users make changes to the same record while offline, etc. For this post, we’ll look at offline data storage and synchronization by building a web phonebook app to store contacts.

Building the sample app

Our sample app will be a web app that’ll be built with Bootstrap, jQuery, PouchDB, and Couchbase Sync Gateway. To start, we’ll lay out the page, which will include a form to enter the contact’s name, email and phone, and also display a list of saved contacts.

Laying out the page

Create a folder for the application, then download bootstrap and jQuery. Unzip the files and put it in a new folder called asset. Add a new file called index.html to your root folder and copy the below snippet to it.

What we have is a page with a form to enter and save contacts, and also display list of saved contacts

Offline Data Access/Storage

Data becomes one of the key concerns of an Offline-First app. Any application that works offline has to deal with offline data access and storage, which will be handled by storing data on the client, and dealing with the issue of syncing data reliably. There are various client-side databases for mobile and web apps, including Couchbase Lite and Cloudant Sync for mobile and desktop clients, and IndexedDB and Web SQL for browsers. For our sample we’ll use PouchDB, a JavaScript client-side database API modelled after the CouchDB API. PouchDB abstracts away the different browser supported database and their different programming interface. It was created to help build applications that work as well offline as they do online, by storing data locally while offline and synchronizing to the server and other connected clients when online.

I’ll also use hoodie store-client, a PouchDB plugin for data persistence and offline sync. I prefer to work with this library because of the API available for working with data and synchronisation. It automatically adds timestamps when I add new data. I’ll add this plugin as well as PouchDB using the npm with the following command or download from this store-client & PouchBD. Note that I’m using PouchDB 6.1.2 and hoodie-store-client 7.0.1.

Now include these files to the page.

With that done, we add a new file called index.js in the asset folder and add a link to this file in the page. Inside this file, the first thing we do is initialize a Store object with the name of the database and URL to the server for synchronization.

Add the following code to save the value entered in the form and also add it to the list of contact on the page.

‘ $(“#contactList tbody”).append(newContact); } //when a new entry is added to the database, run the corresponding function store.on(‘add’, addNewContactToList); function loadContacts() { store.findAll().then(function(contacts) { var tbody = ”; $.each(contacts, function (i, contact) { var row = ‘

‘; tbody += row; }); $(“#contactList tbody”).html(”).html(tbody); }); } // when the site loads in the browser, // we load all previously saved contacts from hoodie loadContacts();

I used store.add to insert to the local database, and then listen to events using store.on, particularly the add event, and this will be useful when we start synchronizing data and will display new data after a complete synchronization with remote databases. I also added function to display local data when the page loads or refreshes. We can check to see that we can save data locally.

We now have offline data access/storage working. With what we’ve done so far, you should begin to see the shift in mindset or thinking process. This being storing data locally first, and afterwards pushing changes to the server when online. Our next step is to make it sync this data with a Couchbase server. To make this work, we need Couchbase Sync Gateway.

Sync Gateway

Sync Gateway is a secure web gateway application with synchronization, REST, stream, batch and event APIs for accessing and synchronizing data over the web. Sync Gateway enables, among other things, secure data replication between Couchbase Server and
Couchbase Lite and/or PouchDB. For a quick guide on installing it, check this guide. If Sync Gateway is installed, it’ll be accessible from the http://localhost:4984 when started. If you remember, we had a similar url when initializing the Store object to use PouchDB but with an extra /example appended to it. The extra appendix specifies the name of the database we sync with. To get our app using offline data storage and synchronization, let’s start the Sync Gateway service and update the page’s JavaScript for automatic sync. I’ll use the configuration setting for Sync Gateway.

We have a basic configuration which will establish a connection to the example database and use the in-memory storage option (walrus), which in production we should change to point to a Couchbase server. We also added setting to allow cross origin resource sharing for our app which is on a different port. We now have to start the Sync Gateway service running the following command in the terminal

$ ./bin/sync_gateway my-config/phonebook-config.json

With Sync Gateway started we have to update the code for continuous synchronization. Add the following code to index.js

The code above uses the hoodie-store connect method to tell PouchDB to start a continuous replication with the remote database. Once we reload our database, the data we added in the previous step will automatically be synced to the server. You can see this using the Sync Gateway admin URL http://localhost:4985/_admin/db/example.

Wrap Up

With all we have now, the app works well with offline data storage and real-time cross-device synchronization. Since we have a web app and users would likely open the page when offline or on a flaky connection, we can make the page load during this situations and even load fast using Service Worker, Cache API, and Fetch API by caching the app’s asset and intercepting request to these and returning the cached responses. This is beyond the scope of this blog post, but I believe I’ve showed how easy it to make data available when offline and keeping all connected browser clients in-sync, while making the app load data fast, and also the mind shift of developing Offline-First.

Below are GIFs to see it working

You can grab the sourcecode here and give it a spin!