{"id":1855,"date":"2014-12-24T00:25:53","date_gmt":"2014-12-24T00:25:52","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=1855"},"modified":"2014-12-24T00:25:53","modified_gmt":"2014-12-24T00:25:52","slug":"nosql-proof-concept-data-creation","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/","title":{"rendered":"NoSQL Proof of Concept: Data Creation"},"content":{"rendered":"<p class=\"p1\">Reliable performance at scale is\u00a0paramount\u00a0for every data-centric mission critical system. As technology professionals we put a lot of work into providing data platforms that will help ensure successful system deployments. Architecture discussions, concept presentations and product training is a great place to start in order to facilitate the vision and help define needs. However, we need (and want) to get our hands dirty to better understand and architect a reliable data platform. Conducting a formal Proof-of-Concept (POC) in an environment similar to which the system will run with similar data sizes and user workloads is often times the solution.\u00a0<\/p>\n<p class=\"p1\">A POC style engagement model should help:<\/p>\n<ul>\n<li class=\"li1\">Stakeholders understand what the implementation will feel like for the organization.<\/li>\n<li class=\"li1\">Define benefits the new system will provide.\u00a0<\/li>\n<li class=\"li1\">Give insight to the business impact.<\/li>\n<li class=\"li1\">Define the developer experience.<\/li>\n<li class=\"li1\">Give operations teams an understanding of supporting the system.\u00a0<\/li>\n<\/ul>\n<p class=\"p1\">In many cases, in order to meet all these objectives for a Couchbase POC we need data business owners understand and can be a big hurdle to jump in getting a proof-of-concept off the ground. Couchbase provides sample datasets but we often are faced with how to create datasets that make more sense to our customers.\u00a0<\/p>\n<h4 class=\"p1\">Data Creation Option 1<\/h4>\n<p class=\"p1\">If we have existing systems we can take the time to export data and create JSON structures that are sourced from existing production systems in place. Here I was able to take a sample dataset stocks.json and import into Couchbase. Each line of the file contains a JSON structure representing stock information.<\/p>\n<ul>\n<li class=\"p3\"><span class=\"s1\">The code leverages line-reader =>\u00a0<a href=\"https:\/\/github.com\/nickewing\/line-reader\"><span class=\"s2\">https:\/\/github.com\/nickewing\/line-reader<\/span><\/a><\/span><\/li>\n<li class=\"p3\"><span class=\"s1\">Data File of Stock Information =>\u00a0<a href=\"https:\/\/jsonstudio.com\/wp-content\/uploads\/2014\/02\/stocks.zip\"><span class=\"s2\">https:\/\/jsonstudio.com\/wp-content\/uploads\/2014\/02\/stocks.zip<\/span><\/a><\/span><\/li>\n<li class=\"p3\"><span class=\"s1\">Sample Import => <a href=\"https:\/\/github.com\/justinmichaels006\/datasample\/blob\/master\/appOG2.js\"><span class=\"s2\">https:\/\/github.com\/justinmichaels006\/datasample\/blob\/master\/appOG2.js<\/span><\/a><\/span><\/li>\n<\/ul>\n<h4 class=\"p1\">Data Creation Option 2<\/h4>\n<p class=\"p1\">As often is the case we don\u2019t have time or maybe don\u2019t have access or need to garner involvement from other teams to use existing data. There are online tools available for generating sample data. Although often times they do not provide the ability to create datasets that are production size a very robust utility is available here \u2026 <a href=\"https:\/\/www.generatedata.com\/\"><span class=\"s3\">https:\/\/www.generatedata.com<\/span><\/a>. Alternatively, we could create the structure we\u2019re after and iterate through to create the size we need. Here we define a data structure that mirrors the data we would eventually have in production.\u00a0<\/p>\n<ul>\n<li class=\"p3\"><span class=\"s1\">Sample Import =>\u00a0<a href=\"https:\/\/github.com\/justinmichaels006\/datasample\/blob\/master\/appOG.js\"><span class=\"s2\">https:\/\/github.com\/justinmichaels006\/datasample\/blob\/master\/appOG.js<\/span><\/a><\/span><\/li>\n<\/ul>\n<p class=\"p1\">We need to be careful how large a dataset is created at a given time because Node.js spawns everything in the loop concurrently. Nevertheless I am able to generate a useful dataset for supporting a successful POC. Whatever the approach, using the options above we are able to provide a system that houses familiar data and a size that mirrors what we expect in production.<\/p>\n<hr \/>\n<p class=\"p2\"><span style=\"font-family: inherit; font-size: 1em; line-height: 1.4375em;\">Next time, we\u2019ll use these datasets to understand what we can do with Couchbase leveraging views and how our up coming query language nickel (N1QL) will dramatically simplify our development efforts.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Reliable performance at scale is\u00a0paramount\u00a0for every data-centric mission critical system. As technology professionals we put a lot of work into providing data platforms that will help ensure successful system deployments. Architecture discussions, concept presentations and product training is a great [&hellip;]<\/p>\n","protected":false},"author":47,"featured_media":13873,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[9021],"class_list":["post-1855","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.0 (Yoast SEO v26.0) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>NoSQL Proof of Concept: Data Creation - The Couchbase Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"NoSQL Proof of Concept: Data Creation\" \/>\n<meta property=\"og:description\" content=\"Reliable performance at scale is\u00a0paramount\u00a0for every data-centric mission critical system. As technology professionals we put a lot of work into providing data platforms that will help ensure successful system deployments. Architecture discussions, concept presentations and product training is a great [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-12-24T00:25:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2022\/11\/couchbase-nosql-dbaas.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Justin Michaels, Solutions Engineer, Couchbase\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Justin Michaels, Solutions Engineer, Couchbase\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\"},\"author\":{\"name\":\"Justin Michaels, Solutions Engineer, Couchbase\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/3953b54b90c8d9474f4d88b142db9fc5\"},\"headline\":\"NoSQL Proof of Concept: Data Creation\",\"datePublished\":\"2014-12-24T00:25:52+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\"},\"wordCount\":506,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\",\"name\":\"NoSQL Proof of Concept: Data Creation - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"datePublished\":\"2014-12-24T00:25:52+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"width\":1800,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"NoSQL Proof of Concept: Data Creation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/3953b54b90c8d9474f4d88b142db9fc5\",\"name\":\"Justin Michaels, Solutions Engineer, Couchbase\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/19d2966da4f60b608ab69fc089913195\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/2fac3118ab1e1f4f03a854b0d78d7746cc0bc8af09381abd362a6d1071a10e54?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/2fac3118ab1e1f4f03a854b0d78d7746cc0bc8af09381abd362a6d1071a10e54?s=96&d=mm&r=g\",\"caption\":\"Justin Michaels, Solutions Engineer, Couchbase\"},\"description\":\"Justin Michaels is a Solutions Engineer at Couchbase. His primary responsibilities involve supporting the Couchbase user community help drives customer success.\",\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/justin-michaels\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"NoSQL Proof of Concept: Data Creation - The Couchbase Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/","og_locale":"en_US","og_type":"article","og_title":"NoSQL Proof of Concept: Data Creation","og_description":"Reliable performance at scale is\u00a0paramount\u00a0for every data-centric mission critical system. As technology professionals we put a lot of work into providing data platforms that will help ensure successful system deployments. Architecture discussions, concept presentations and product training is a great [&hellip;]","og_url":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/","og_site_name":"The Couchbase Blog","article_published_time":"2014-12-24T00:25:52+00:00","og_image":[{"width":1800,"height":630,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2022\/11\/couchbase-nosql-dbaas.png","type":"image\/png"}],"author":"Justin Michaels, Solutions Engineer, Couchbase","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Justin Michaels, Solutions Engineer, Couchbase","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/"},"author":{"name":"Justin Michaels, Solutions Engineer, Couchbase","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/3953b54b90c8d9474f4d88b142db9fc5"},"headline":"NoSQL Proof of Concept: Data Creation","datePublished":"2014-12-24T00:25:52+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/"},"wordCount":506,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/","url":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/","name":"NoSQL Proof of Concept: Data Creation - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","datePublished":"2014-12-24T00:25:52+00:00","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","width":1800,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/nosql-proof-concept-data-creation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"NoSQL Proof of Concept: Data Creation"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/3953b54b90c8d9474f4d88b142db9fc5","name":"Justin Michaels, Solutions Engineer, Couchbase","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/19d2966da4f60b608ab69fc089913195","url":"https:\/\/secure.gravatar.com\/avatar\/2fac3118ab1e1f4f03a854b0d78d7746cc0bc8af09381abd362a6d1071a10e54?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/2fac3118ab1e1f4f03a854b0d78d7746cc0bc8af09381abd362a6d1071a10e54?s=96&d=mm&r=g","caption":"Justin Michaels, Solutions Engineer, Couchbase"},"description":"Justin Michaels is a Solutions Engineer at Couchbase. His primary responsibilities involve supporting the Couchbase user community help drives customer success.","url":"https:\/\/www.couchbase.com\/blog\/author\/justin-michaels\/"}]}},"authors":[{"term_id":9021,"user_id":47,"is_guest":0,"slug":"justin-michaels","display_name":"Justin Michaels, Solutions Engineer, Couchbase","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/2fac3118ab1e1f4f03a854b0d78d7746cc0bc8af09381abd362a6d1071a10e54?s=96&d=mm&r=g","author_category":"","last_name":"Michaels","first_name":"Justin","job_title":"","user_url":"","description":"Justin Michaels is a Solutions Engineer at Couchbase. His primary responsibilities involve supporting the Couchbase user community help drives customer success."}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1855","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/47"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=1855"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1855\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/13873"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=1855"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=1855"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=1855"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1855"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}