Is this possible to generate document id with all lower case letters using cbimport

Couchbase Server
1

I am using couchbase latest 5.x and trying to import data using cbimport tool. I need to import document id with all lower case letters. But I couldn’t create, If I specify name, it will create key::Name::1.

I need key::name::1 instead of “key::Name::1.”

cbimport json -c localhost:8091 -u Administrator -p password -b name -d file:///name.json -f list --generate-key key::%name%::#MONO_INCR#

Kindly provide your inputs.

2

Hello @ssakthi.t,

In this case cbimport is using the value of name field, so the data provided has the first letter capitalised. Unfortunately cbimport currently can’t do any transformation on the field(s) used to generated the key. I have opened MB-29336 for this feature.

In the mean time the name.json file could be preprocess to covert the name field to lower case.

1 Like
3

Ok. Thank you for your reply @pvarley!

4

I found this post while creating a question. You mentioned that cbimport can not do any transformations on the field(s) from the document used to generate the key. That is exactly what I am trying to do but I believe you answered my question.

I’m trying to grab a substring of a field to use in a key.

This currently works as intended:

--generate-key "%propCode%:%roomNumber%:%date%"

but I would only like to get a substring from the date field. I tried using a bash regex for the first 7 characters but it can not parse the expression after the first “%”.

--generate-key "%propCode%:%roomNumber%:%${date::7}%"

This results in a key that looks like

MEMQP:1203:%

I currently see the open feature ticket

It has been some time since this original question was posed. I’m using server version 7.1.5. Is this limitation still the case?

5

I do not find any possibility for bash regex or anything else. You could use the operating system jq command to transform the json before processing with cbimport.

-g,--generate-key <key_expr>

Specifies a key expression used for generating a key for each document imported. This parameter is required for list and lines formats, but not for the sample format. See the KEY GENERATION section below for more information on specifying key generators. If the resulting key is not unique the values will be overridden, resulting in fewer documents than expected being imported. To ensure that the key is unique add #MONO_INCR# or #UUID# to the key generator expression.

KEY GENERATORS
Key generators are used in order to generate a key for each document loaded. Keys can be generated by using a combination of characters, the values of a given field in a document, and custom generators. Field substitutions are done by wrapping the field name in "%" and custom generators are wrapped in "#". Below is an example of a key generation expression.

Given the document:

{
  "name": "alice",
  "age": 40,
  "address": {
    "street": "Broadway",
    "country": "USA"
  }
}
Key Generator Expression:

  --generate-key key::%name%::#MONO_INCR#
The following key would be generated:

  key::alice::1
In the example above we generate a key using both the value of a field in the document and a custom generator. We use the "name" field to use the value of the name field as part of the key. This is specified by "%name%" which tells the key generator to substitute the value of the field "name" into the key. To reference a nested field we would use "parent.child" syntax. For example to reference the country we would use '%address.country%'. To reference a field that contains a dot in the name we escape the string using `` . For example '%`address.country`%' refers to a field named "address.country".

This example also contains a generator function MONO_INCR which will increment by 1 each time the key generator is called. Since this is the first time this key generator was executed it returns 1. If we executed the key generator again it would return 2 and so on. The starting value of the MONO_INCR generator is 1 by default, but it can be changed by specifying a number in brackets after the MONO_INCR generator name. To start generating monotonically incrementing values starting at 100 for example, the generator MONO_INCR[100] would be specified. The cbimport command current contains a monotonic increment generator (MONO_INCR) and a UUID generator (UUID).

Any text that isn’t wrapped in "%" or "#" is static text and will be in the result of all generated keys. If a key needs to contain a "%" or "#" in static text then they need to be escaped by providing a double "%" or "#" (ex. "%%" or "##"). The delimiter characters can be changed to avoid having to escape them by using the --field-delimiter and --generator-delimiter flags.

If a key cannot be generated because the field specified in the key generator is not present in the document then the key will be skipped. To see a list of document that were not imported due to failed key generation users can specify the --errors-log <path> parameter to dump a list of all documents that could not be imported to a file.