I recognize that countering with, “Couchbase can<\/em> be your XML database” is pretty bold, especially for a JSON-oriented document database. Some folks in the Couchbase community might even do a double take, but I hope you see what I mean by the end of this post.<\/p>\n It’s a fact of life that many legacy applications still rely on XML, but is Couchbase really the right solution for storing and processing your XML data? Don’t worry: with respect to Couchbase and XML you can have your cake and eat it too.<\/p>\n After introducing a few concepts, this article shows you how to automatically and instantly convert XML into JSON equivalents within Couchbase with almost zero effort \u2013 all via an Eventing function. If you just think of JSON as the intermediate computer format that’s used to store your XML, you might already see where I’m headed.<\/p>\n When you transform your XML data into a native JSON representation, you take advantage of the rich ecosystem of Couchbase’s services including N1QL<\/a>, Indexing, Full-Text Search<\/a>, Eventing<\/a>, and Analytics<\/a>.<\/p>\n Couchbase<\/a> is a purpose-built database you assemble to your needs. Just need a fast cache? You only need the Data Service or key-value store. Want SQL-like access? Add Query and Index services. Or maybe you need Analytics or Full-Text Search? Just add some nodes of the proper type.<\/p>\n Each of the above services independently scale so you end up building \u2013 and more importantly, paying for<\/em> \u2013 only what you need. In the article below, I use Eventing to transform Couchbase into an XML-capable database. I know this sounds too good to be true, but trust me it really does work.<\/p>\n If you are familiar with Couchbase, XML, and JSON please feel free to skip ahead to the Prerequisites section<\/a>.<\/em><\/p>\n Couchbase Server<\/a> is an open source, distributed data platform. It stores data as items<\/em>, each of which has a key<\/em> and a value<\/em>. Sub-millisecond data operations are provided by powerful services for querying and indexing, as well as a feature-rich, document-oriented query language, N1QL<\/em>. Multiple instances of Couchbase Server can be combined into a single cluster<\/em>.<\/p>\n Each value (binary or JSON) is identified by a unique key, defined by the user or application when the item is saved. The key is immutable: once the item is saved, the key cannot be changed. Note that Couchbase also refers to an item’s key as its id<\/em>.<\/p>\n Each key:<\/p>\n The maximum size of a value is 20 MiB. A value can be either:<\/p>\n Let’s talk more about JSON. You can parse, index, query and manipulate JSON documents. In fact, Couchbase introduced the N1QL query language (pronounced “nickel”) to meet the query needs of distributed document-oriented databases. N1QL is used for manipulating the JSON data in Couchbase, just like SQL manipulates data in a relational database (RDBMS). It has You could store XML as a string or just a binary blob, but where’s the fun \u2013 or the utility \u2013 in that? Why not automatically convert your XML into JSON instead? I’m so glad you asked.<\/p>\n XML is a markup language introduced in 1996 that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable. It is a World Wide Web Consortium (W3C) recommendation.<\/p>\n The design goals of XML emphasize simplicity, generality and usability across the Internet. It is a textual data format with strong support via Unicode for different human languages. Although the design of XML focuses on documents, the language is widely used for the representation of arbitrary data structures such as those used in web services.<\/p>\n Here’s an example of XML:<\/p>\n Created circa 2001, JSON is a fairly lightweight open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs and arrays (or other serializable values).<\/p>\n JSON is a very common data format, with a diverse range of applications, one example being web applications that communicate with a server.<\/p>\n Here’s a sample of JSON:<\/p>\n Today, when any two applications communicate with each other across the internet, odds are they do so using JSON, especially if they communicate in human-readable text.<\/p>\n If you’re an XML fan, don’t shoot the messenger here. I’m sure XML will never go away given HTML as a markup language, but I like to think statistics don’t lie when it comes to developing internet applications that need to communicate with one another.<\/p>\n In this article we will be using the latest version of Couchbase \u2013 version 6.6.2 \u2013 however, it should work just fine in prior versions as well.<\/p>\n If you are not familiar with Couchbase or the Eventing service please walk through the following resources, including at least one Eventing example:<\/p>\n Now you should have an application that loads data, or you may have used In the code below, I load my first document with a key<\/em> We can’t do too much with the above XML string. Sure, we could index the property We might pass it to the Full-Text Search (FTS) product. In this case, you would get more utility, but you’d have to index the entire XML payload (not just something important).<\/p>\n What we really want is the JSON representation of the XML string property Using the new property With these preliminary steps behind us, you might think to just parse your data and transform it to JSON before putting it in Couchbase. But the downside is that we now have more baggage to carry around and more infrastructure (outside of Couchbase) to deploy as we scale out by adding more nodes.<\/p>\n If we can avoid it, we don’t want to write a custom parser. That isn’t fun, and it’s hard to maintain as your requirements and data formats change over time.<\/p>\n What you ideally want is an in-server solution that generically converts XML to JSON in real time with no impact on data loaders.<\/p>\n Enter Couchbase’s Eventing Service.<\/p>\n Just think of Eventing as a “post trigger” on a document that responds to every change (or mutation) in your data. The main idea is that whenever you insert or update your document (refer to In this case, we want to convert a document like the one below:<\/p>\n In real time, we want to convert the above document into an enriched document like the one below:<\/p>\n The enriched document shown above is searchable either via key-value or via an index on any given field using N1QL.<\/p>\n As before, we see individual JSON properties in a proper structure but under the Eventing allows you to write pure business logic and the Eventing service takes care of all the infrastructure needed to manage and scale your function (horizontally and vertically) across multiple nodes in a performant and reliable fashion.<\/p>\n All Eventing functions have two entry points When a document changes or mutates (insert, upsert, replace, etc.) a copy of the document and some metadata about the document will be passed to a small JavaScript entry point The Here’s the core logic for the XML-to-JSON conversion:<\/p>\n Thanks to user The Couchbase Data Model<\/h2>\n
Keys<\/h3>\n
\n
\n
(<\/code>, %<\/code>, \/<\/code>, \"<\/code> and _<\/code> are acceptable.<\/li>\nValues<\/h3>\n
\n
\n
JSON Is the Database<\/h3>\n
SELECT<\/code>, INSERT<\/code>, UPDATE<\/code>, DELETE<\/code> and MERGE<\/code> statements to operate on JSON data.<\/p>\nXML: Extensible Markup Language<\/h3>\n
<CD>\r\n <TITLE>EmpireBurlesque<\/TITLE>\r\n <ARTIST>BobDylan<\/ARTIST>\r\n <COUNTRY>USA<\/COUNTRY>\r\n <COMPANY>Columbia<\/COMPANY>\r\n <PRICE>10.90<\/PRICE>\r\n <YEAR>1985<\/YEAR>\r\n<\/CD>\r\n<\/pre>\n
JSON: JavaScript Object Notation<\/h3>\n
{\r\n \"email\": \"testme@example.org\",\r\n \"friends\": [\r\n {\"name\": \"rick\"},\r\n {\"name\": \"cate\"}\r\n ]\r\n}\r\n<\/pre>\nJSON Has Taken over the World<\/h3>\n
![\"JSON](\"https:\/\/twobithistory.org\/images\/json.svg\" "\\"JSON")
<\/p>\nPrerequisites: Learning about Eventing<\/h2>\n
\n
\n
Inserting XML into Couchbase<\/h2>\n
cbimport<\/code> to load your XML data into your Couchbase cluster.<\/p>\nxml::1<\/code> and a body<\/em> consisting of a type<\/code> and the id<\/code> used in the key (these values are optional but useful) and finally a property in_xml<\/code> which contains the XML string itself.<\/p>\n{\r\n \"type\": \"xml\",\r\n \"id\": 1,\r\n \"in_xml\": \"<CD><TITLE>EmpireBurlesque<\/TITLE><ARTIST>BobDylan<\/ARTIST><COUNTRY>USA<\/COUNTRY><COMPANY>Columbia<\/COMPANY><PRICE>10.90<\/PRICE><YEAR>1985<\/YEAR><\/CD>\"\r\n}\r\n<\/pre>\nin_xml<\/code> as a string, but we won’t get a lot of performance out of doing this unless we use a prefix search, which in my humble opinion, is most likely worthless.<\/p>\nin_xml<\/code>, perhaps something like:<\/p>\n\"out_json\": {\r\n \"CD\": {\r\n \"TITLE\": \"EmpireBurlesque\",\r\n \"ARTIST\": \"BobDylan\",\r\n \"COUNTRY\": \"USA\",\r\n \"COMPANY\": \"Columbia\",\r\n \"PRICE\": \"10.90\",\r\n \"YEAR\": \"1985\"\r\n }\r\n}\r\n<\/pre>\nout_json<\/code>, we now see individual JSON properties in a proper structure: something that N1QL, Indexing, Full-Text Search (FTS), Eventing and Analytics all natively and efficiently work with.<\/p>\nConverting XML to JSON on the Fly<\/h2>\n
xml::1<\/code> above) a small JavaScript fragment, or lambda, is executed against the document and encapsulates the needed business logic to manipulate the document to ensure that the document is in the format you want.<\/p>\n{\r\n \"type\": \"xml\",\r\n \"id\": 1,\r\n \"in_xml\": \"<CD><TITLE>EmpireBurlesque<\/TITLE><ARTIST>BobDylan<\/ARTIST><COUNTRY>USA<\/COUNTRY><COMPANY>Columbia<\/COMPANY><PRICE>10.90<\/PRICE><YEAR>1985<\/YEAR><\/CD>\"\r\n}\r\n<\/pre>\n{\r\n \"type\": \"xml\",\r\n \"id\": 2,\r\n \"in_xml\": \"<CD><TITLE>EmpireBurlesque<\/TITLE><ARTIST>BobDylan<\/ARTIST><COUNTRY>USA<\/COUNTRY><COMPANY>Columbia<\/COMPANY><PRICE>10.90<\/PRICE><YEAR>1985<\/YEAR><\/CD>\",\r\n \"out_json\": {\r\n \"CD\": {\r\n \"TITLE\": \"EmpireBurlesque\",\r\n \"ARTIST\": \"BobDylan\",\r\n \"COUNTRY\": \"USA\",\r\n \"COMPANY\": \"Columbia\",\r\n \"PRICE\": \"10.90\",\r\n \"YEAR\": \"1985\"\r\n }\r\n }\r\n}\r\n<\/pre>\nout_json<\/code> property, which enables the efficient use of Full-Text Search (FTS), Analytics, and even other Eventing functions.<\/p>\nEventing Function: convertXMLtoJSON<\/h3>\n
OnUpdate(doc,meta)<\/code> and OnDelete(meta,options)<\/code>. Note that we’re not worried about the latter entry point in this example.<\/p>\nOnUpdate(doc,meta)<\/code>:<\/p>\nfunction OnUpdate(doc, meta) {\r\n \/\/ filter out non XML\r\n if (!meta.id.startsWith(\"xml:\")) return;\r\n \/\/ The KEY started with \"xml\" try to process it\r\n \/\/ ===========================================================\r\n \/\/ *** Do other work required here on non .in_xml changes ***\r\n \/\/ ===========================================================\r\n var jsonDoc = parseXmlToJson(doc.in_xml);\r\n log(meta.id, \"1. INPUT xml doc.in_xml :\", doc.in_xml);\r\n log(meta.id, \"2. OUTPUT doc.out_json :\", jsonDoc);\r\n doc.out_json = jsonDoc;\r\n \/\/ ===========================================================\r\n \/\/ enrich the source bucket with .out_json\r\n src_bkt[meta.id] = doc;\r\n}\r\n<\/pre>\nOnUpdate(doc,meta)<\/code> logic above performs four steps on any mutation.<\/p>\n\n
\n
xml:<\/code>, we don’t do anything else. Note that meta.id<\/code> is the key<\/em> of the doc.<\/li>\nparseXmlToJson(doc.in_xml)<\/code> where we just pass the XML string into the function.<\/li>\nout_json<\/code>.<\/li>\nThe Core XML-to-JSON Conversion Logic<\/h3>\n
\/\/ 6.6.0 version no String.matchAll need our own MatchAll function\r\nfunction* MatchAll(str, regExp) {\r\n if (!regExp.global) {\r\n throw new TypeError('Flag \/g must be set!');\r\n }\r\n const localCopy = new RegExp(regExp, regExp.flags);\r\n let match;\r\n while (match = localCopy.exec(str)) {\r\n yield match;\r\n }\r\n}\r\n\r\n\/\/ A simple XML to JSON parser\r\nfunction parseXmlToJson(xml) {\r\n const json = {};\r\n for (const res of MatchAll(xml,\/(?:<(\\w*)(?:\\s[^>]*)*>)((?:(?!<\\1).)*)(?:<\\\/\\1>)|<(\\w*)(?:\\s*)*\\\/>\/gm)) {\r\n const key = res[1] || res[3];\r\n const value = res[2] && parseXmlToJson(res[2]);\r\n json[key] = ((value && Object.keys(value).length) ? value : res[2]) || null;\r\n }\r\n return json;\r\n}\r\n<\/pre>\nMasterPiece<\/code> on Stack Overflow for sharing the method parseXmlToJson(xml)<\/code> with the world<\/a>. The method uses a cool regular expression<\/a> on the XML string and returns a first-class JSON object. (Yes, it works and is both elegant and compact<\/em>.)<\/p>\n
![\"The](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_1_buckets-2-1024x290.png\")
<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n![\"Couchbase](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_2_addfunc-1011x1024.png\")
<\/li>\n![\"convert](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_3_addfunc-1024x297.png\")
<\/li>\n![\"The](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_4_addfunc-1.png\")
<\/li>\n![\"Eventing](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_5_deploy-1024x288.png\")
<\/li>\n![\"Confirm](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2021\/06\/XML2JSON_6_deployconf.png\")
<\/li>\n