{"id":13147,"date":"2022-04-26T20:13:45","date_gmt":"2022-04-27T03:13:45","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=13147"},"modified":"2025-06-13T23:04:15","modified_gmt":"2025-06-14T06:04:15","slug":"build-a-python-microservice-with-couchbase-part-3","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/build-a-python-microservice-with-couchbase-part-3\/","title":{"rendered":"Build A Python Microservice With Couchbase – Part 3"},"content":{"rendered":"

Recap<\/span><\/h2>\n

In the previous two installments of this series<\/a>, we discussed the drivers behind creating microservices. We also looked at why Couchbase is the perfect datastore to use with such an architecture. We also looked at three variants of a sample user profile microservice. Two were written in Python, and one was written in Node.js for comparison. In this last part of the series, we will discuss one method for generating test data for the microservice schema. <\/span><\/p>\n

Why use test data?<\/span><\/h2>\n

Rigorous testing is critical to releasing stable software. Many elements go into a good test strategy. In the past decade, the term \u201cDevOps\u201d is often used to reference both the toolsets used in software development and testing and the culture and processes and procedures that must be in place to achieve rapid, agile development. Regardless of the terminology, there is one common thread \u2013 automation and orchestration. These two things can easily be their own blog series, so we won\u2019t dwell too much on them except to say that test data generation is an essential aspect of DevOps and agile software development and testing.<\/span><\/p>\n

I personally have fallen into the trap of not having a good development data set. For expedience, I manually created some data to fit the desired schema so that I could do unit tests on code. But the next day, I realized that the code that worked fine with my small data set didn\u2019t work too well when it was run against a test database with thousands of records and more realistic data.<\/span><\/p>\n

Generating JSON data to test the Python microservice<\/span><\/h2>\n

There are many ways to get test data into a test or development database. One method is to copy and sanitize production data if possible. It may not be feasible to have full production copies for multiple development and test environments depending on available resources. Database platforms must be sized appropriately, and care must be taken to remove sensitive data for compliance and security reasons. Complete production copies should be a requirement for regression and pre-prod test environments. Generating synthetic and randomized data with a minimal dataset for unit testing and development is usually suitable.<\/span><\/p>\n

With a relational database, it can be challenging to create a database instance with a subset of tables because relational models typically require all the data structures to be present. However, with Couchbase\u2019s JSON document format, it is much easier to do. Looking at the microservice sample, the schema was built around Couchbase\u2019s Scopes and Collections. These could easily be part of a larger schema with more collections and scopes. But all we need are the collections to access keys, user profiles, and the user images, as that is all our service example needs to access to function.<\/span><\/p>\n

In the second part of the blog series, we used JMeter to do performance testing. To accomplish performance testing, the database should have at least thousands of records to give JMeter a reasonable dataset for test generation. The microservice simply gets profile elements and returns them via the REST API. The assumption is other application components consume the data and perform business logic using the results.<\/span><\/p>\n

Revisiting the three profile collections, the following is an example of a document in the <\/span>user_data<\/span> collection. The service does not enforce whether the picture field has an ID referencing a picture record. The service just returns the profile data or picture data. The assumption is that whatever consumes it upstream will handle that logic.<\/span><\/p>\n

{\r\n\u00a0\u00a0\"record_id\": 1,\r\n\u00a0\u00a0\"name\": \"Michael Jones\",\r\n\u00a0\u00a0\"nickname\": \"mjones\",\r\n\u00a0\u00a0\"picture\": \"\",\r\n\u00a0\u00a0\"user_id\": \"michaeljones2104\",\r\n\u00a0\u00a0\"email\": \"michael.jones@example.com\",\r\n\u00a0\u00a0\"email_verified\": \"True\",\r\n\u00a0\u00a0\"first_name\": \"Michael\",\r\n\u00a0\u00a0\"last_name\": \"Jones\",\r\n\u00a0\u00a0\"address\": \"0208 River Parkway\",\r\n\u00a0\u00a0\"city\": \"Strongdol\",\r\n\u00a0\u00a0\"state\": \"IL\",\r\n\u00a0\u00a0\"zip_code\": \"29954\",\r\n\u00a0\u00a0\"phone\": \"363-555-9036\",\r\n\u00a0\u00a0\"date_of_birth\": \"10\/27\/1958\"\r\n}<\/pre>\n
The user_images collection is straightforward with a record ID (that can be referenced by a user profile), an image type field so multiple image codecs can be supported and the Base64 encoded image itself. As noted in the first post in the series, Couchbase supports binary documents, but we used JSON for its portability and extensibility in the sample application. This comes at the expense of some extra network bandwidth to get the data and processor cycles to do the decoding.<\/p>\n
{\r\n\u00a0\u00a0\"record_id\": 1,\r\n\u00a0\u00a0\"type\": \"jpeg\",\r\n\u00a0\u00a0\"image\": \"AAAAD\u2026\"\r\n}<\/pre>\n
The basic <\/span>service_auth<\/span><\/em> collection contains the auth token but can be easily extended to include other authentication fields.<\/span><\/p>\n
{\r\n\u00a0\u00a0\"record_id\": 1,\r\n\u00a0\u00a0\"token\": \"6j6nW3KD0ZXodBv1\"\r\n}<\/pre>\n
Generating randomized sample JSON data<\/span><\/h3>\n
I wrote a utility called <\/span>cb_perf<\/span> to generate randomized data and insert it into Couchbase. The utility also does some performance tests to gauge the Python capabilities on a platform. Tools like YCSB use an unrealistic schema as the only goal is to gauge raw performance. I wanted the ability to take arbitrary JSON and insert it with randomized synthetic data.<\/span><\/p>\n
To test the sample microservice, I added the schema to <\/span>cb_perf<\/span> so that I could generate test data sets in any Couchbase cluster. The <\/span>cb_perf<\/span> utility uses a JSON schema definition file. For each collection, you define the JSON data structure.\u00a0<\/span><\/p>\n
You can specify Jinja2 expressions with variables that map to randomizer data types for each JSON value. That way, each record will be different. You can also define which indexes to create. In this case, we create a primary index and secondary indexes on a few fields.\u00a0<\/span><\/p>\n
It specifies that <\/span>record_id<\/span><\/em> is an ID field and that the overall record count requested should not be overwritten (so we can create as many records as we want).<\/span><\/p>\n
{\r\n\"name\": \"user_data\",\r\n\"schema\": {\r\n\u00a0\u00a0\"record_id\": \"record_id\",\r\n\u00a0\u00a0\"name\": \"{{ rand_first }} {{ rand_last }}\",\r\n\u00a0\u00a0\"nickname\": \"{{ rand_nickname }}\",\r\n\u00a0\u00a0\"picture\": \"\",\r\n\u00a0\u00a0\"user_id\": \"{{ rand_username }}\",\r\n\u00a0\u00a0\"email\": \"{{ rand_email }}\",\r\n\u00a0\u00a0\"email_verified\": \"{{ rand_bool }}\",\r\n\u00a0\u00a0\"first_name\": \"{{ rand_first }}\",\r\n\u00a0\u00a0\"last_name\": \"{{ rand_last }}\",\r\n\u00a0\u00a0\"address\": \"{{ rand_address }}\",\r\n\u00a0\u00a0\"city\": \"{{ rand_city }}\",\r\n\u00a0\u00a0\"state\": \"{{ rand_state }}\",\r\n\u00a0\u00a0\"zip_code\": \"{{ rand_zip_code }}\",\r\n\u00a0\u00a0\"phone\": \"{{ rand_phone }}\",\r\n\u00a0\u00a0\"date_of_birth\": \"{{ rand_dob_1 }}\"\r\n\u00a0\u00a0},\r\n\"idkey\": \"record_id\",\r\n\"primary_index\": true,\r\n\"override_count\": false,\r\n\"indexes\": [\r\n\u00a0\u00a0\"record_id\",\r\n\u00a0\u00a0\"nickname\",\r\n\u00a0\u00a0\"user_id\"\r\n\u00a0\u00a0]\r\n},<\/pre>\n
We leverage the cb_perf<\/em> randomizer option for the images collection to create a random JPEG image (it creates a random map of RGB values to make an image of random colors). We define the following:<\/p>\n