Indexer crash in HA setup but requests still routed to it

SQL++
1

Setup:

  • CB 4.6.3
  • 4 CB data nodes, 2 N1QL index/query nodes the n1ql have the same index (but different names)

On Node A, we got this error

 ** Last message in was {#Port<0.3627>,{exit_status,1}}
 ** When Server state == {state,#Port<0.3627>,indexer,
                                {["[goport] 2018/02/07 00:13:52 /opt/couchbase/bin/indexer terminated: signal: aborted (core dumped)",
                                  "\t/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:145 +0x166",
                                  "created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection",
                                  "\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6.3/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8238b9f88 sp=0xc8238b9f80"],
                                 ["runtime.goexit()"]},
                                indexer,undefined,[],0}```

On the client we got 
```Indexer In Bootstrap State. Please retry the request later. ```

and we had a number of requests that failed, Why were the requests still getting routed to Node A instead of B?
2

We tested with 4.6.3 and 5.0, we have 30M docs, and it seem if query did not match an index and used had use a the primary index, the table scan will cause the indexer to crash, it seems the indexer ran out of RAM.

Here is the CB 5.0 Crash

os.(*File).read(0xc42000c040, 0xc4200e6000, 0x10000, 0x10000, 0xc42031bee8, 0x0, 0x0)
    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/os/file_unix.go:165 +0x4f fp=0xc42031be60 sp=0xc42031be18
os.(*File).Read(0xc42000c040, 0xc4200e6000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/os/file.go:101 +0x76 fp=0xc42031bec8 sp=0xc42031be60
bufio.(*Reader).Read(0xc4200146c0, 0xc4200e6000, 0x10000, 0x10000, 0x53, 0x0, 0x0)
    /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/bufio/bufio.go:199 +0x198 fp=0xc42031bf58 sp=0xc42031bec8
main.(*childReader).loop.func1(0xc42000e570, 0xc4200e6000, 0x10000, 0x10000, 0xc4202a8180)
    /home/couchbase/jenkins/workspace/couchbase-server-unix/ns_server/deps/gocode/src/goport/main.go:73 +0x50 fp=0xc42031bfb8 ```
3

@dipen_patel, can you please share the indexer.log or the full stack at the time of crash. This crash trace is not complete.

Do you also see the crash in CB 4.6.3? What is the GSI storage mode(Standard or Memory Optimized) being used?