Continuous Integration and Continuous Deployment are now common software development practices. In the world of databases, this translates into needing on-demand, stateful, ephemeral environments. Â
Provisioning a stateless environment is not tied to any particular source of data. All that is needed is to run the code you want to test in your CI environment. This is the basis of most CI/CD tools and won’t be covered in this article.Â
The slightly harder part comes from the dependencies the application needs to be tested properly, which is often referred to as external services. Couchbase being one of them. There are different ways to get those, through Docker containers for instance, or hosted in your test infrastructure, or some external as a Service solution. It does not really matter as long as they are available while running your test. Good practices would be to use Environment Variables to refer to those instances.Â
Assuming these services are running, like a Couchbase Free Tier instance or a Docker container, the next step is to make sure that they are configured correctly, and seeded with the data needed for the test.
A while ago, I posted about using Couchbase Shell in GitHub actions. This will tell you the basics about using Couchbase Shell with GitHub Actions, but this can be applied to most CI/CD solutions as well. Today, I want to go further and show you some useful scripts to clone a cluster or elements of a cluster for your on demand environments.
Using Couchbase Shell to clone environments
When using Couchbase Shell, the first thing that comes to mind when wanting to do something is, is there a function for that? As of now we don’t have a function to clone something. Most of the available functions reflect our APIs capabilities and we have no cloning APIs today. But, we have the ability to write scripts, which means we can make our own!
The first thing that comes to mind when managing databases is often to recreate the structure and schemas. As Couchbase is Schemaless, this will only consist of the existing buckets, scopes, collections, and indexes in the source cluster. The first step is to export that structure so it can be reimported later. This function will list every bucket, then inner scopes and collections, and add them to an array. Then it will list all indexes and add them to the output JSON.Â
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Exports all buckets, scopes, collections and indexes # for the given cluster def export-cluster-struct [     source: string # The cluster to export ] {     mut export = []    let buckets = buckets --clusters $source # List the buckets of the given cluster     for bucket in $buckets {         mut scope_structs = []         let scopes = scopes --clusters $source --bucket $bucket.name         for scope in $scopes {             let collections = (collections --clusters $source --bucket $bucket.name --scope $scope.scope | reject -i cluster)             $scope_structs ++= [{                 scope: $scope.scope,                collections: $collections             }]         }         # Merge the scopes with the bucket object and add it to the export array         let buc = ( $bucket | merge {scopes: $scope_structs } )         $export ++= [ $buc ]     }     let indexes = query indexes --definitions --disable-context --clusters $source     let output = {         buckets: $export,         indexes: $indexes     }     return $output } |
This works because under the hood, Couchbase Shell is using Nushell, a new type of shell that is portable (meaning it works the same way on Linux, Windows, or OS X, which is great for CI/CD scripts having to support different OS), and that considers any structure data as a DataFrame, making the manipulation of JSON extremely easy.
To try it out, run cbsh, then source the file containing the function. For me it’s ci_scripts.nu. I have a cluster already configured in my cbsh config, called local
|
1 2 3 4 |
Laurent Doguin at local in travel-sample.inventory._default > source ci-scripts.nu Laurent Doguin at local in travel-sample.inventory._default > export-cluster-struct local | save local-cluster-export.json |
Now if you open local-cluster-export.json, you will get the structure of your cluster:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 |
{ Â Â "buckets": [ Â Â Â Â { Â Â Â Â Â Â "cluster": "local", Â Â Â Â Â Â "name": "travel-sample", Â Â Â Â Â Â "type": "couchbase", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "min_durability_level": "none", Â Â Â Â Â Â "ram_quota": 209715200, Â Â Â Â Â Â "flush_enabled": false, Â Â Â Â Â Â "cloud": false, Â Â Â Â Â Â "max_expiry": 0, Â Â Â Â Â Â "scopes": [ Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "airport", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "airline", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "route", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "landmark", Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "hotel", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "tenant_agent_00", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "users", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "bookings", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "tenant_agent_01", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "users", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "bookings", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "tenant_agent_02", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "users", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "bookings", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "tenant_agent_03", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "users", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "bookings", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "tenant_agent_04", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "users", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "bookings", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "inherited" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â "scope": "_system", Â Â Â Â Â Â Â Â Â Â "collections": [ Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "_query", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "" Â Â Â Â Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â Â Â "collection": "_mobile", Â Â Â Â Â Â Â Â Â Â Â Â Â Â "max_expiry": "" Â Â Â Â Â Â Â Â Â Â Â Â } Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â } Â Â Â Â Â Â ] Â Â Â Â } Â Â ], Â Â "indexes": [ Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_system", Â Â Â Â Â Â "collection": "_query", Â Â Â Â Â Â "name": "#primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `#primary` ON `travel-sample`.`_system`.`_query`", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_airportname", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_airportname` ON `travel-sample`(`airportname`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_city", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_city` ON `travel-sample`(`city`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_faa", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_faa` ON `travel-sample`(`faa`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_icao", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_icao` ON `travel-sample`(`icao`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "airline", Â Â Â Â Â Â "name": "def_inventory_airline_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_inventory_airline_primary` ON `travel-sample`.`inventory`.`airline` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "airport", Â Â Â Â Â Â "name": "def_inventory_airport_airportname", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_airport_airportname` ON `travel-sample`.`inventory`.`airport`(`airportname`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "airport", Â Â Â Â Â Â "name": "def_inventory_airport_city", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_airport_city` ON `travel-sample`.`inventory`.`airport`(`city`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "airport", Â Â Â Â Â Â "name": "def_inventory_airport_faa", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_airport_faa` ON `travel-sample`.`inventory`.`airport`(`faa`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "airport", Â Â Â Â Â Â "name": "def_inventory_airport_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_inventory_airport_primary` ON `travel-sample`.`inventory`.`airport` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "hotel", Â Â Â Â Â Â "name": "def_inventory_hotel_city", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_hotel_city` ON `travel-sample`.`inventory`.`hotel`(`city`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "hotel", Â Â Â Â Â Â "name": "def_inventory_hotel_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_inventory_hotel_primary` ON `travel-sample`.`inventory`.`hotel` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "landmark", Â Â Â Â Â Â "name": "def_inventory_landmark_city", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_landmark_city` ON `travel-sample`.`inventory`.`landmark`(`city`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "landmark", Â Â Â Â Â Â "name": "def_inventory_landmark_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_inventory_landmark_primary` ON `travel-sample`.`inventory`.`landmark` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "route", Â Â Â Â Â Â "name": "def_inventory_route_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_inventory_route_primary` ON `travel-sample`.`inventory`.`route` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "route", Â Â Â Â Â Â "name": "def_inventory_route_route_src_dst_day", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_route_route_src_dst_day` ON `travel-sample`.`inventory`.`route`(`sourceairport`,`destinationairport`,(distinct (array (`v`.`day`) for `v` in `schedule` end))) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "route", Â Â Â Â Â Â "name": "def_inventory_route_schedule_utc", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_route_schedule_utc` ON `travel-sample`.`inventory`.`route`(array (`s`.`utc`) for `s` in `schedule` end) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "inventory", Â Â Â Â Â Â "collection": "route", Â Â Â Â Â Â "name": "def_inventory_route_sourceairport", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_inventory_route_sourceairport` ON `travel-sample`.`inventory`.`route`(`sourceairport`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_name_type", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_name_type` ON `travel-sample`(`name`) WHERE (`_type` = \"User\") WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_primary", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE PRIMARY INDEX `def_primary` ON `travel-sample` WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_route_src_dst_day", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_route_src_dst_day` ON `travel-sample`(`sourceairport`,`destinationairport`,(distinct (array (`v`.`day`) for `v` in `schedule` end))) WHERE (`type` = \"route\") WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_schedule_utc", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_schedule_utc` ON `travel-sample`(array (`s`.`utc`) for `s` in `schedule` end) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_sourceairport", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_sourceairport` ON `travel-sample`(`sourceairport`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â }, Â Â Â Â { Â Â Â Â Â Â "bucket": "travel-sample", Â Â Â Â Â Â "scope": "_default", Â Â Â Â Â Â "collection": "_default", Â Â Â Â Â Â "name": "def_type", Â Â Â Â Â Â "status": "Ready", Â Â Â Â Â Â "storage_mode": "memory_optimized", Â Â Â Â Â Â "replicas": 0, Â Â Â Â Â Â "definition": "CREATE INDEX `def_type` ON `travel-sample`(`type`) WITH {Â \"defer_build\":true }", Â Â Â Â Â Â "cluster": "local" Â Â Â Â } Â Â ] } |
I have deleted that bucket for the purpose of this test, to reimport it later: buckets drop travel-sample.
The next logical step is to have a function that takes this file as input and recreate the complete structure in another cluster:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# Import all buckets, scopes and collections structure # in the given cluster def import-cluster-struct [     destination: string # The cluster to import ] {     let structure = $in # Assigning the piped structure to a variable     let buckets = $structure.buckets     for bucket in $buckets {         $bucket | _create-bucket-definition $destination         for scope in ($bucket.scopes | where not ( $it.scope | str starts-with "_" ) ) {             print $"Create scope ($destination)_($bucket.name)_($scope.scope)"             scopes create --clusters $destination --bucket $bucket.name $scope.scope             for col in $scope.collections {                 print $"Create collection ($destination)_($bucket.name)_($scope.scope)_($col.collection)"                 collections create --clusters $destination --bucket $bucket.name --scope $scope.scope $col.collection             }         }     }     let indexes = $structure.indexes     $indexes | _create-indexes $destination # Nushell allows you to use other functions you created } def _create-indexes [     destination: string # the cluster where to create indexes ] {     let indexes = $in     for index in $indexes {         print $"Recreating index ($index.name) on cluster ($destination) with: "         print $index.definition         query $index.definition --disable-context --clusters $destination     } } |
Now to run that function:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
Laurent Doguin at local in travel-sample.inventory._default > open local-cluster-export.json | import-cluster-struct capella Laurent Doguin at local in travel-sample.inventory._default > open local-cluster-export.json | import-cluster-struct local Create Bucket local_travel-sample with 200 quota, type couchbase, 0 replicas, none durability, 0 expiry Create scope local_travel-sample_inventory Create collection local_travel-sample_inventory_airport Create collection local_travel-sample_inventory_airline Create collection local_travel-sample_inventory_route Create collection local_travel-sample_inventory_landmark Create collection local_travel-sample_inventory_hotel Create scope local_travel-sample_tenant_agent_00 Create collection local_travel-sample_tenant_agent_00_users Create collection local_travel-sample_tenant_agent_00_bookings Create scope local_travel-sample_tenant_agent_01 Create collection local_travel-sample_tenant_agent_01_users Create collection local_travel-sample_tenant_agent_01_bookings Create scope local_travel-sample_tenant_agent_02 Create collection local_travel-sample_tenant_agent_02_users Create collection local_travel-sample_tenant_agent_02_bookings Create scope local_travel-sample_tenant_agent_03 Create collection local_travel-sample_tenant_agent_03_users Create collection local_travel-sample_tenant_agent_03_bookings Create scope local_travel-sample_tenant_agent_04 Create collection local_travel-sample_tenant_agent_04_users Create collection local_travel-sample_tenant_agent_04_bookings Recreating index #primary on cluster local with:Â CREATE PRIMARY INDEX `#primary` ON `travel-sample`.`_system`.`_query` Recreating index def_airportname on cluster local with: CREATE INDEX `def_airportname` ON `travel-sample`(`airportname`) WITH {Â "defer_build":true } Recreating index def_city on cluster local with: CREATE INDEX `def_city` ON `travel-sample`(`city`) WITH {Â "defer_build":true } Recreating index def_faa on cluster local with: CREATE INDEX `def_faa` ON `travel-sample`(`faa`) WITH {Â "defer_build":true } Recreating index def_icao on cluster local with: CREATE INDEX `def_icao` ON `travel-sample`(`icao`) WITH {Â "defer_build":true } Recreating index def_inventory_airline_primary on cluster local with: CREATE PRIMARY INDEX `def_inventory_airline_primary` ON `travel-sample`.`inventory`.`airline` WITH {Â "defer_build":true } Recreating index def_inventory_airport_airportname on cluster local with: CREATE INDEX `def_inventory_airport_airportname` ON `travel-sample`.`inventory`.`airport`(`airportname`) WITH {Â "defer_build":true }Recreating index def_inventory_airport_city on cluster local with: CREATE INDEX `def_inventory_airport_city` ON `travel-sample`.`inventory`.`airport`(`city`) WITH {Â "defer_build":true } Recreating index def_inventory_airport_faa on cluster local with: CREATE INDEX `def_inventory_airport_faa` ON `travel-sample`.`inventory`.`airport`(`faa`) WITH {Â "defer_build":true } Recreating index def_inventory_airport_primary on cluster local with:Â CREATE PRIMARY INDEX `def_inventory_airport_primary` ON `travel-sample`.`inventory`.`airport` WITH {Â "defer_build":true } Recreating index def_inventory_hotel_city on cluster local with:Â CREATE INDEX `def_inventory_hotel_city` ON `travel-sample`.`inventory`.`hotel`(`city`) WITH {Â "defer_build":true } Recreating index def_inventory_hotel_primary on cluster local with:Â CREATE PRIMARY INDEX `def_inventory_hotel_primary` ON `travel-sample`.`inventory`.`hotel` WITH {Â "defer_build":true } Recreating index def_inventory_landmark_city on cluster local with:Â CREATE INDEX `def_inventory_landmark_city` ON `travel-sample`.`inventory`.`landmark`(`city`) WITH {Â "defer_build":true } Recreating index def_inventory_landmark_primary on cluster local with:Â CREATE PRIMARY INDEX `def_inventory_landmark_primary` ON `travel-sample`.`inventory`.`landmark` WITH {Â "defer_build":true } Recreating index def_inventory_route_primary on cluster local with:Â CREATE PRIMARY INDEX `def_inventory_route_primary` ON `travel-sample`.`inventory`.`route` WITH {Â "defer_build":true } Recreating index def_inventory_route_route_src_dst_day on cluster local with:Â CREATE INDEX `def_inventory_route_route_src_dst_day` ON `travel-sample`.`inventory`.`route`(`sourceairport`,`destinationairport`,(distinct (array (`v`.`day`) for `v` in `schedule` end))) WITH {Â "defer_build":true } Recreating index def_inventory_route_schedule_utc on cluster local with:Â CREATE INDEX `def_inventory_route_schedule_utc` ON `travel-sample`.`inventory`.`route`(array (`s`.`utc`) for `s` in `schedule` end) WITH {Â "defer_build":true } Recreating index def_inventory_route_sourceairport on cluster local with:Â CREATE INDEX `def_inventory_route_sourceairport` ON `travel-sample`.`inventory`.`route`(`sourceairport`) WITH {Â "defer_build":true } Recreating index def_name_type on cluster local with:Â CREATE INDEX `def_name_type` ON `travel-sample`(`name`) WHERE (`_type` = "User") WITH {Â "defer_build":true } Recreating index def_primary on cluster local with:Â CREATE PRIMARY INDEX `def_primary` ON `travel-sample` WITH {Â "defer_build":true } Recreating index def_route_src_dst_day on cluster local with:Â CREATE INDEX `def_route_src_dst_day` ON `travel-sample`(`sourceairport`,`destinationairport`,(distinct (array (`v`.`day`) for `v` in `schedule` end))) WHERE (`type` = "route") WITH {Â "defer_build":true } Recreating index def_schedule_utc on cluster local with:Â CREATE INDEX `def_schedule_utc` ON `travel-sample`(array (`s`.`utc`) for `s` in `schedule` end) WITH {Â "defer_build":true } Recreating index def_sourceairport on cluster local with:Â CREATE INDEX `def_sourceairport` ON `travel-sample`(`sourceairport`) WITH {Â "defer_build":true } Recreating index def_type on cluster local with:Â CREATE INDEX `def_type` ON `travel-sample`(`type`) WITH {Â "defer_build":true } |
And there you have it, functions that allow you to export and import the data structure from one cluster to another. While this is a good starting point, there are still questions about how to reimport data, or about granularity. Also, you may not want to export and import a complete cluster.
Filtering buckets to import is fairly easy as Nushell allows you to filter dataframes:
|
1 2 |
Laurent Doguin at local in travel-sample.inventory._default > open local-cluster-export.json | { buckets: ( $in.buckets | where name == 'travel-sample'), indexes :( $in.indexes | where bucket == 'travel-sample') } |
This will recreate a JSON object containing only a bucket named travel-sample and indexes for this bucket.
From there you should be all set to manage basic cluster structure. What about the data? There are different ways you can import data with cbsh, as it covers most key/value operations as well as any INSERT/UPSERT queries. And then we have the doc import command. Its usage is fairly straightforward, all you need is a list of rows with an identified id field. This can be anything that can be turned into a dataframe for Nushell (XML, CSV, TSV, Parquet, and more). And of course, it can be a JSON file from a Couchbase SQL++ query. This is an example that will save a query result to a file and import that file back to a collection:Â
|
1 2 3 4 5 6 7 8 |
# Save file content to filename let filename = $"temp_($src_bucket)_($src_scope)_($src_collection).json" let query = "SELECT meta().id as meta_id, meta().expiration as expiration, c.* FROM `" + $src_bucket + "`." + $src_scope + "." + $src_collection + " c" query --disable-context --clusters $p.src $query | save -f $filename # Import the file content and print the results print $"Import collection content from ($src)_($src_bucket)_($src_scope)_($src_collection) to ($dest)_($dest_bucket)_($dest_scope)_($dest_collection)" print ( doc import --bucket $p.dest_bucket --scope $p.dest_scope --collection $p.dest_collection --clusters $p.dest --id-column meta_id $filename ) |
That’s one particular example but the whole point of using scripting language is to make them your own. You will find a more complete example in this GitHub Gist. It has support for environment variables for source and destination and you can decide to either clone all buckets of a cluster, a specific bucket, scope, or collection.
Don’t hesitate to drop us a comment here or on Discord, we are always looking for suggestions to improve the global Couchbase experience.
