{"id":2135,"date":"2022-03-22T08:54:20","date_gmt":"2022-03-22T15:54:20","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/validate-json-documents-in-python-using-pydantic\/"},"modified":"2022-03-22T08:54:20","modified_gmt":"2022-03-22T15:54:20","slug":"validate-json-documents-in-python-using-pydantic","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/es\/validate-json-documents-in-python-using-pydantic\/","title":{"rendered":"Validate JSON Documents in Python using Pydantic"},"content":{"rendered":"\n

One of the main attractions of <\/span>document databases<\/b> is the flexibility of the document structure or schema. For the most part, the flexibility in the document schema is beneficial. However, there might be some situations where having some structure to the document might be helpful. This article shows you how to validate your JSON documents against a specified schema using the popular Python library <\/span>pydantic<\/span><\/a>.<\/span><\/p>\n\n\n\n

When do you need to validate documents?<\/span><\/h4>\n\n\n\n

A common misconception about using NoSQL databases is that no structures or document schemas are required. In most cases, applications tend to have some constraints for the data even though they may not specifically validate it. For example, there might be some fields in the document that the application depends on for functionality. An application might simply not operate correctly when some of these pydantic JSON fields are missing.<\/span><\/p>\n\n\n\n

One real-world example of this problem could be an application that reads data from another unreliable application that periodically sends bad data. It would be prudent to highlight any documents that could break the application in such cases. Developers can do this either <\/span>in the application <\/span><\/i>or <\/span>at the document level<\/span><\/i>.\u00a0<\/span><\/p>\n\n\n\n

In this approach, we specify a JSON to pydantic schema for the documents to help identify those that do not match the application specifications at the document level.<\/span><\/p>\n\n\n\n

Generate JSON testing data<\/span><\/h4>\n\n\n\n

We use the open-source library, <\/span>Faker<\/span><\/a>, to generate some fake user profiles for this tutorial.<\/span><\/p>\n\n\n\n

This is the structure of a single document:<\/span><\/p>\n\n\n

[crayon decode-attributes=”false” lang=”js” decode=”true”]{
\n “job”: “Radiographer, diagnostic”,
\n “company”: “Klein, Dillon and Neal”,
\n “residence”: “2232 Jackson ForksnLake Teresa, CO 46959”,
\n “website”: [
\n “https:\/\/bishop-torres.net\/”
\n ],
\n “username”: “aabbott”,
\n “name”: “Madison Mitchell”,
\n “address”: “1782 Moore Hill Apt. 717nWest Stephaniestad, NM 75293”,
\n “mail”: “amberrodriguez@hotmail.com”,
\n “phone”: {
\n “home”: “677-197-4239×889”,
\n “mobile”: “001-594-038-9255×99138”
\n }
\n}[\/crayon]<\/p>\n\n\n\n

To simulate the broken documents, I modify a small portion of the user profiles by deleting some of the mobile phone and mail fields. We aim to identify these records that, in the real world, would be stored in a document database like Couchbase.<\/span><\/p>\n\n\n\n

I load the generated pydantic data from JSON into a bucket on our hosted <\/span>Couchbase Capella<\/span><\/a> cluster using the <\/span>import <\/span><\/i>functionality in the built-in web console UI for our testing. I specify the <\/span>username <\/span><\/i>field as the key to uniquely identify each document.<\/span><\/p>\n\n\n\n

How to specify a schema for JSON documents?<\/span><\/h4>\n\n\n\n

In the user profile data, I expect the documents to conform to the following structure in my applications:<\/span><\/p>\n\n\n\n