Let’s continue this “walk through a house” analogy and try to make the new time series knowledge stick.<\/p>\n
The Green<\/span> Room<\/h2>\nGreen<\/span> is a happy color: nature, renewal, growth, fresh ideas.\u00a0 Our definition of time series belongs in the green room.<\/p>\nWhen your child has a fever, and you take his or her body temperature every hour<\/strong>, you have a series<\/strong> of measurements over time.\u00a0 When you clandestinely attach a GPS tracker to a criminal’s car, you get a series<\/strong> of locations where the car stops over weeks<\/strong> or months<\/strong>.\u00a0 How about a\u00a0 Raspberry Pi<\/a> project to collect temperature and humidity readings<\/strong> once a second<\/strong> that your kid built for a school science fair?\u00a0 Have it installed in your cannabis farm to stream<\/strong> this data to your Couchbase cluster!\u00a0 You can then use the data with a Couchbase Mobile<\/a> app to assure the optimal conditions for the best harvest of this important crop.<\/p>\nAs you can see, the time series use cases extend well beyond the old boring stock ticker and server logs examples.\u00a0 They still boil down to a list<\/strong> (sequence, series) of values<\/strong> (readings, measurements, data points) collected<\/strong> over time<\/strong>.\u00a0 Time series use cases are common nowadays in consumer products, industrial technologies, and business services.\u00a0 This popularity inspired the appearance of time series databases<\/a>, which include different optimizations for working specifically with time series data.\u00a0 E.g., we provide Prometheus exporter<\/a> (written in Python) to convert Couchbase Server metrics into Prometheus<\/a> time series format, so you can build your monitoring dashboards with Grafana<\/a>.<\/p>\nWe are leaving our green<\/span> room with a melted clock<\/a> hanging from the ceiling (it needs to be weird, right?) and a melting white board with a series of distorted time-stamped measurements as an example of the time series <\/strong>data from a summer day in Arizona (in degrees Fahrenheit):<\/p>\n2019-07-27 07:11:22, temperature=90, humidity=\"what humidity?\", feeling=\"sweaty\"\r\n2019-07-27 10:21:22, temperature=98, humidity=\"this word again...\", feeling=\"the pain\"\r\n2019-07-27 12:34:56, temperature=107, humidity=\"gotta look up this word in a dictionary\", feeling=\"desperate\"\r\n2019-07-27 15:00:27, temperature=120, feeling=\"boiling inside, fried on the outside\"\r\n2019-07-27 15:32:15, temperature=, feeling=\"confused\", thermometer-status=\"melted\"\r\n<\/pre>\nThe Orange<\/span> Room<\/h2>\nOrange<\/span> is the color that attracts our attention; it usually brings the feelings of enthusiasm and warmth.\u00a0 So, it is a good place to put the details of how we work with time series data in Couchbase Server.<\/p>\nW<\/span>rite!<\/h3>\nBe prepared to handle a lot of writes!\u00a0 Real-life use cases involving time series data produce thousands of readings – per hour, per minute, per second.\u00a0 When you multiply it by the number of devices or applications generating the data, you will quickly reach millions of new writes per day.\u00a0 Couchbase Server optimizes data ingestion with its memory-first<\/a>, asynchronous architecture.\u00a0 JSON documents are compressed in memory, on disk, and on the wire.<\/p>\nCouchbase Server handles all types of writes – inserts, updates, and upserts (insert if the key does not exist; otherwise, update) – with equal efficiency.\u00a0 You can further optimize your updates with our sub-document operations available for every supportable SDK (e.g., Java SDK sub-document API<\/a>).<\/p>\nA<\/span>ggregate!<\/h3>\nWhen you take multiple collected measurements and roll them up into a single JSON document, you aggregate your data.\u00a0 It is a typical step to optimize the storage of time series data.\u00a0 This can happen before<\/strong> inserting the measurements in a database (pre-aggregation): your application or your data ingestion software, like StreamSets Data Collector, can do the job.\u00a0 Couchbase Server Eventing service<\/a> can roll up time series values after<\/strong> they have been added to a bucket.\u00a0 In this case, a JavaScript function executes as a database OnUpdate trigger to save values from multiple JSON documents<\/strong> into an array of values in a single JSON document<\/strong>.\u00a0 Either approach will get you from, for example, 60 documents<\/strong> of per-second readings (see below)<\/p>\nkey = \"sensor::temp-press::2020-01-02T12:34:00\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:00\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 112.4,\r\n \"pressure\": 21.7\r\n}\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:01\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:01\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 110.8,\r\n \"pressure\": 21.2\r\n}\r\n\r\n...\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:58\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:58\",\r\n \"sensor\": \"tps-001\",\r\n \"pressure\": 22.8\r\n}\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:59\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:59\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 113.1,\r\n \"pressure\": 22.5\r\n}<\/pre>\nto a single<\/strong> document with all the readings, like the one below:<\/p>\nkey = \"sensor::tps-001::1577968440\"\r\n{\r\n \"values\":\r\n {\r\n \"t\": [112.4, 110.8, ... null, 113.1],\r\n \"p\": [21.7, 21.2, ... 22.8, 22.5]\r\n },\r\n \"type\": \"temp-press\",\r\n \"schema\": \"1.3.5\"\r\n}<\/pre>\nApart from eliminating duplicate data (timestamps, long JSON field\/attribute names), we have also made the following changes:<\/p>\n
\n- replaced ISO timestamp with epoch value in the document key.\u00a0 The epoch value corresponds to the minute (2020-01-02 12:34), for which we collected per-second readings.<\/li>\n
- added sensor name to the document key.\u00a0 This way we can query the values for the specific sensor using key-value operations, which is always the fastest way to work with documents.<\/li>\n
- listed temperature and pressure data in 60-element arrays, one measurement per second;<\/li>\n
- added JSON schema version of the document.<\/li>\n<\/ul>\n
The flexibility of JSON format makes it easy to get your first version out quickly (in the spirit of “Worse is better<\/a>” principle) and evolve your document schema as your application matures.\u00a0 Specialized time series databases are much more rigid in this respect.\u00a0 However, it is a best practice to keep track of the schema version in your documents.\u00a0 This helps to ensure backward compatibility in your applications.\u00a0 It also allows you to update the documents to a new schema by running a N1QL UPDATE<\/a> query.<\/p>\nR<\/span>ead!<\/h3>\nIt is time to do something useful with our time series data.\u00a0 Here are the options that Couchbase data platform offers:<\/p>\n
\n- Key-value reads in less than a millisecond is what Couchbase Server does great, all day long.\u00a0 After all, one of the main reasons we aggregated our data was to get it easily in a single read operation.<\/li>\n
- SQL queries<\/a> for your JSON data.\u00a0 N1QL language purposely inherited SQL syntax to make it easy for you to learn.<\/li>\n
- Full text search<\/a> is another way to work with data that’s available with Couchbase Server.\u00a0 Depending on your use case, the search can be a better alternative for language-aware, numeric range, date range, and geospatial queries.\u00a0 Better yet, you can combine search and N1QL<\/a> in a single query.<\/li>\n
- Analytics is a natural fit for time series data.\u00a0 As part of our data platform, we offer Analytics service<\/a> that allows you to obtain a wide variety of business insights from your data.\u00a0 Couchbase Analytics runs as part of the same cluster where your data resides, so there is no ETL (Extract, Transform, Load) operations necessary.\u00a0 You execute efficient parallel queries against up-to-date shadow copies of the data.<\/li>\n
- Couchbase also plays nice with others.\u00a0 We offer supported big data connectors for popular systems like Spark<\/a>, Kafka<\/a>, and Elasticsearch<\/a>.<\/li>\n<\/ul>\n
M<\/span>ove On!<\/h3>\nSooner or later, it will be time to move on.\u00a0 How long do you need to keep your data?\u00a0 We always ask this question when we help our customers size their clusters.\u00a0 With Couchbase, it is easy to scale up or down.\u00a0 An online gaming company may want to start with a few extra nodes when launching a new game.\u00a0 On the other hand, a startup on a tight budget may have to be extra vigilant about how much data they keep around.<\/p>\n
How can we delete data in Couchbase?<\/p>\n
\n- Set document expiration (also known as TTL, time to live).\u00a0 TTL values are part of the document metadata: if TTL is zero, document will not expire.\u00a0 A value greater than zero is the number of seconds, after which the document will be marked as deleted.\u00a0 TTL can be set and updated via SDK methods or (as of Couchbase Server 6.5.1) via N1QL query<\/a>.<\/li>\n
- Set TTL on a bucket<\/a>.\u00a0 All new documents entering the bucket will get the TTL assigned, unless it’s already set.<\/li>\n
- Delete documents via N1QL query or Eventing function.\u00a0 Eventing functions can also be executed by timers<\/a> .<\/li>\n
- Flush or drop the entire bucket.<\/li>\n<\/ul>\n
Prior to deleting data from Couchbase cluster, you may want to archive or share it for some of the reasons below:<\/p>\n
When your child has a fever, and you take his or her body temperature every hour<\/strong>, you have a series<\/strong> of measurements over time.\u00a0 When you clandestinely attach a GPS tracker to a criminal’s car, you get a series<\/strong> of locations where the car stops over weeks<\/strong> or months<\/strong>.\u00a0 How about a\u00a0 Raspberry Pi<\/a> project to collect temperature and humidity readings<\/strong> once a second<\/strong> that your kid built for a school science fair?\u00a0 Have it installed in your cannabis farm to stream<\/strong> this data to your Couchbase cluster!\u00a0 You can then use the data with a Couchbase Mobile<\/a> app to assure the optimal conditions for the best harvest of this important crop.<\/p>\n As you can see, the time series use cases extend well beyond the old boring stock ticker and server logs examples.\u00a0 They still boil down to a list<\/strong> (sequence, series) of values<\/strong> (readings, measurements, data points) collected<\/strong> over time<\/strong>.\u00a0 Time series use cases are common nowadays in consumer products, industrial technologies, and business services.\u00a0 This popularity inspired the appearance of time series databases<\/a>, which include different optimizations for working specifically with time series data.\u00a0 E.g., we provide Prometheus exporter<\/a> (written in Python) to convert Couchbase Server metrics into Prometheus<\/a> time series format, so you can build your monitoring dashboards with Grafana<\/a>.<\/p>\n We are leaving our green<\/span> room with a melted clock<\/a> hanging from the ceiling (it needs to be weird, right?) and a melting white board with a series of distorted time-stamped measurements as an example of the time series <\/strong>data from a summer day in Arizona (in degrees Fahrenheit):<\/p>\n Orange<\/span> is the color that attracts our attention; it usually brings the feelings of enthusiasm and warmth.\u00a0 So, it is a good place to put the details of how we work with time series data in Couchbase Server.<\/p>\n Be prepared to handle a lot of writes!\u00a0 Real-life use cases involving time series data produce thousands of readings – per hour, per minute, per second.\u00a0 When you multiply it by the number of devices or applications generating the data, you will quickly reach millions of new writes per day.\u00a0 Couchbase Server optimizes data ingestion with its memory-first<\/a>, asynchronous architecture.\u00a0 JSON documents are compressed in memory, on disk, and on the wire.<\/p>\n Couchbase Server handles all types of writes – inserts, updates, and upserts (insert if the key does not exist; otherwise, update) – with equal efficiency.\u00a0 You can further optimize your updates with our sub-document operations available for every supportable SDK (e.g., Java SDK sub-document API<\/a>).<\/p>\n When you take multiple collected measurements and roll them up into a single JSON document, you aggregate your data.\u00a0 It is a typical step to optimize the storage of time series data.\u00a0 This can happen before<\/strong> inserting the measurements in a database (pre-aggregation): your application or your data ingestion software, like StreamSets Data Collector, can do the job.\u00a0 Couchbase Server Eventing service<\/a> can roll up time series values after<\/strong> they have been added to a bucket.\u00a0 In this case, a JavaScript function executes as a database OnUpdate trigger to save values from multiple JSON documents<\/strong> into an array of values in a single JSON document<\/strong>.\u00a0 Either approach will get you from, for example, 60 documents<\/strong> of per-second readings (see below)<\/p>\n to a single<\/strong> document with all the readings, like the one below:<\/p>\n Apart from eliminating duplicate data (timestamps, long JSON field\/attribute names), we have also made the following changes:<\/p>\n The flexibility of JSON format makes it easy to get your first version out quickly (in the spirit of “Worse is better<\/a>” principle) and evolve your document schema as your application matures.\u00a0 Specialized time series databases are much more rigid in this respect.\u00a0 However, it is a best practice to keep track of the schema version in your documents.\u00a0 This helps to ensure backward compatibility in your applications.\u00a0 It also allows you to update the documents to a new schema by running a N1QL UPDATE<\/a> query.<\/p>\n It is time to do something useful with our time series data.\u00a0 Here are the options that Couchbase data platform offers:<\/p>\n Sooner or later, it will be time to move on.\u00a0 How long do you need to keep your data?\u00a0 We always ask this question when we help our customers size their clusters.\u00a0 With Couchbase, it is easy to scale up or down.\u00a0 An online gaming company may want to start with a few extra nodes when launching a new game.\u00a0 On the other hand, a startup on a tight budget may have to be extra vigilant about how much data they keep around.<\/p>\n How can we delete data in Couchbase?<\/p>\n Prior to deleting data from Couchbase cluster, you may want to archive or share it for some of the reasons below:<\/p>\n2019-07-27 07:11:22, temperature=90, humidity=\"what humidity?\", feeling=\"sweaty\"\r\n2019-07-27 10:21:22, temperature=98, humidity=\"this word again...\", feeling=\"the pain\"\r\n2019-07-27 12:34:56, temperature=107, humidity=\"gotta look up this word in a dictionary\", feeling=\"desperate\"\r\n2019-07-27 15:00:27, temperature=120, feeling=\"boiling inside, fried on the outside\"\r\n2019-07-27 15:32:15, temperature=, feeling=\"confused\", thermometer-status=\"melted\"\r\n<\/pre>\n
The Orange<\/span> Room<\/h2>\n
W<\/span>rite!<\/h3>\n
A<\/span>ggregate!<\/h3>\n
key = \"sensor::temp-press::2020-01-02T12:34:00\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:00\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 112.4,\r\n \"pressure\": 21.7\r\n}\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:01\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:01\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 110.8,\r\n \"pressure\": 21.2\r\n}\r\n\r\n...\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:58\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:58\",\r\n \"sensor\": \"tps-001\",\r\n \"pressure\": 22.8\r\n}\r\n\r\nkey = \"sensor::temp-press::2020-01-02T12:34:59\"\r\n{\r\n \"ts\": \"2020-01-02 12:34:59\",\r\n \"sensor\": \"tps-001\",\r\n \"temperature\": 113.1,\r\n \"pressure\": 22.5\r\n}<\/pre>\nkey = \"sensor::tps-001::1577968440\"\r\n{\r\n \"values\":\r\n {\r\n \"t\": [112.4, 110.8, ... null, 113.1],\r\n \"p\": [21.7, 21.2, ... 22.8, 22.5]\r\n },\r\n \"type\": \"temp-press\",\r\n \"schema\": \"1.3.5\"\r\n}<\/pre>\n\n
R<\/span>ead!<\/h3>\n
\n
M<\/span>ove On!<\/h3>\n
\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/arrow-squiggly.png\"){kind=icon}
<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/04\/arrow-heart.png\"){kind=icon}
<\/p>\n","protected":false},"excerpt":{"rendered":"