Full Text Search using the Go (Golang) SDK: Fields: unexpected end of JSON input

Hello.

Does anyone have an idea on how to use Fields in a full text search?

if rows, err = cluster.SearchQuery(
	"ft", search.NewMatchQuery("hello"), &gocb.SearchOptions{
		Fields: []string{ "*", }, Limit: 100, },); err != nil { (...) }

	for rows.Next() {
		var l interface{}
		row = rows.Row()
		if err = row.Fields(&l); err != nil { (...) }

        (...)
    } 
}

has search results for hello, but the last line results in unexpected end of JSON input.

fmt.Printf("%s\n", row) shows e.g.

{ft_666ad278618091aa_... b305da1a88f1e... %!s(float64=0.9210534410919049) <nil> map[] map[] }

ft_... seems to be the id of the row, b305da... is my document id, but I don’t see any fields. I tried to replace * with a field name that exists in the document → same result.

Help! :wink:

Best regards

Stefan

From the examples here Search | Couchbase Docs, it looks like the Fields option should be omitted to return only the matching documents (?)

Thank you for your quick response!

The problem is that I need the Fields-feature to render a preview of the documents.
In my case I have a title and a description that I want to present to the user, the document id wouldn’t work.

Hi @beyer-stefan,

It’s difficult to work with just a fragment and no data, so hopefully this will help

  • Load the travel-sample test set

  • Make an index on hello and indexed and stored three top level fields title, name, and id. I will provide the JSON at the end with a curl command if you want to skip the UI search index creation.

  • After you make your index in the UI use the UI to search for Giverny in the UI you should get two (2) hits.

Here is the FTS code you need to do the same, of course update your USERNAME and PASSWORD in the code.

package main

import (
  "fmt"
  "time"
  "log"
  "github.com/couchbase/gocb/v2"
  "github.com/couchbase/gocb/v2/search"
)

func main() {
  cluster, err := gocb.Connect("couchbase://127.0.0.1", gocb.ClusterOptions{
    Authenticator: gocb.PasswordAuthenticator{
      Username: "some_user",
      Password: "some_password",
    },
  })
  if err != nil {
    log.Fatal(err)
  }

   bucket := cluster.Bucket("travel-sample")
   err = bucket.WaitUntilReady(5*time.Second, nil)
   if err != nil {
     log.Fatal(err)
   }

   // In the future you might need to use a new scoped index 
   // like  "travel-sample.inventory.hello" but for pre 7.6 just use the
   // legacy global index name of "hello"
 
   matchResult, err := cluster.SearchQuery(
     "hello",
     search.NewMatchQuery("Giverny"),
     &gocb.SearchOptions{
       Limit:  3,
       Fields: []string{"*"},
     },
   )
   if err != nil {
     log.Fatal(err)
   }

   for matchResult.Next() {
     row := matchResult.Row()
     docID := row.ID

     var fields interface{}
     err := row.Fields(&fields)
     if err != nil {
       log.Fatal(err)
     }

     fmt.Printf("Document ID: %s fields %v\n", docID, fields)
   }

   // always check for errors after iterating
   err = matchResult.Err()
   if err != nil {
     log.Fatal(err)
   }
}

The output of the program is as follows:

$ go run main.go
Document ID: hotel_10063 fields map[name:The Robins title:Giverny]
Document ID: hotel_10064 fields map[name:Le Clos Fleuri title:Giverny]

The index I used is as follows:

{
 "name": "hello",
 "type": "fulltext-index",
 "params": {
  "doc_config": {
   "docid_prefix_delim": "",
   "docid_regexp": "",
   "mode": "scope.collection.type_field",
   "type_field": "type"
  },
  "mapping": {
   "default_analyzer": "standard",
   "default_datetime_parser": "dateTimeOptional",
   "default_field": "_all",
   "default_mapping": {
    "dynamic": true,
    "enabled": false
   },
   "default_type": "_default",
   "docvalues_dynamic": false,
   "index_dynamic": true,
   "store_dynamic": false,
   "type_field": "_type",
   "types": {
    "inventory.hotel": {
     "dynamic": false,
     "enabled": true,
     "properties": {
      "id": {
       "enabled": true,
       "dynamic": false,
       "fields": [
        {
         "include_in_all": true,
         "index": true,
         "name": "id",
         "store": true,
         "type": "text"
        }
       ]
      },
      "name": {
       "enabled": true,
       "dynamic": false,
       "fields": [
        {
         "include_in_all": true,
         "index": true,
         "name": "name",
         "store": true,
         "type": "text"
        }
       ]
      },
      "title": {
       "enabled": true,
       "dynamic": false,
       "fields": [
        {
         "include_in_all": true,
         "index": true,
         "name": "title",
         "store": true,
         "type": "text"
        }
       ]
      }
     }
    }
   }
  },
  "store": {
   "indexType": "scorch",
   "segmentVersion": 15
  }
 },
 "sourceType": "gocbcore",
 "sourceName": "travel-sample",
 "sourceParams": {},
 "planParams": {
  "maxPartitionsPerPIndex": 1024,
  "indexPartitions": 1,
  "numReplicas": 0
 }
}

If you don’t want to use the UI put the above in legacy_def.json and load import it via the following cURL command:

curl -XPUT -H "Content-type:application/json" -u USERNAME:PASSWORD http://SERVER_IP:8094/api/index/hello -d @./legacy_def.json

Best

Jon Strabala
Principal Product Manager - Server‌

1 Like

Hi @jon.strabala ,

thank you for posting the full example!

This is what I did:

  1. Tried your code with your index on the travel-sample. It worked like a charm. This was crucial to me because it confirmed that Fields are working in the Go library, so the problem had to be my code and/or my full text index.
  2. Compared you code with mine. There were no obvious differences in the way we handled searching and Fields. This was no surprise to me, because my code was based on the code in the official documentation, the main functionality can be found within a few lines of code.
  3. Dropped my full text search index and re-created it. Did some testing. Dropped the index again. Re-created it with the same fields.
  4. All of a sudden my code worked! There were no code changes necessary, so the problem definitely was my index!

At this point it also became obvious to me that I made a silly mistake: I didn’t save the definition of the initial index (or take at least a screenshot). Now I can no longer reproduce the problem, which is a bit disappointing to me.
I still think there is a problem in the Go library, because I had valid search results, but empty Fields, which feels odd to me. But as I said, I can no longer reproduce the problem so I can only guess what the problem was (potentially a typo in the index?)

Bottom line: THANK YOU FOR YOUR HELP!

btw: the search is live on https://cve.threatint.com . I LOVE the performance and promise to keep developing with Couchbase :wink:

Best regards | Με εκτίμηση | Mit freundlichen Grüßen

Stefan Beyer

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.