User_views causes Sync Gateway crash on Linux

Mobile Sync Gateway
1

Hi,

Crash seen on Linux with:

==== Couchbase Sync Gateway/1.3.1(16;f18e833) ====
Linux 4.4.0-36-generic #55~14.04.1-Ubuntu SMP Fri Aug 12 11:49:30 UTC 2016

Using the above OS and sync gateway version, when enabling the unsupported->user_views option in my sync gateway config file, sync gateway crashes with
panic: runtime error: invalid memory address or nil pointer dereference
I’ve pasted the full crash log below.

Running with the same config file on MacOS there is no crash and everything (including the view) works perfectly. The versions on Mac are:

==== Couchbase Sync Gateway/1.3.0(274;8c3ee28) ====
15.6.0 Darwin Kernel Version 15.6.0: Mon Aug 29 20:21:34 PDT 2016; root:xnu-3248.60.11~1/RELEASE_X86_64

Here is the offending sync-gateway config file. I’ve removed the SSL certs, sync function and CORS setup which is in my real file, none of which helped.

    {
        "log":["CRUD+", "REST+", "Changes+", "Attach+"],
        "databases": {
            "yoyottaid": {
                "server": "http://localhost:8091",
                "unsupported": {
                    "user_views": {
                        "enabled": true
                    }
                },
                "users": {
                    "GUEST": {
                        "disabled": false,
                        "admin_channels": ["public"]
                    }
                }
            }
        }
    }

If I take out this bit:

    "unsupported": {
                    "user_views": {
                        "enabled": true
                    }
                },

It works perfectly, but of course I can’t use my views from the user port.

I haven’t tried SG v1.3.0 on Linux, or SG 1.3.1 on Mac to see whether the behaviour is related to the sync gateway version. I can try this if it’s likely to be the issue.

Thanks for anyone that can shed any light on this,
Giles

Here is the log when the crash occurs, using the config file above:

node@ubuntu-2gb-nyc1-Main:/opt/couchbase-sync-gateway/bin$ ./sync_gateway /home/sync_gateway/sync_gateway_view.json 
2016-10-14T10:20:35.278Z Enabling logging: [CRUD+ REST+ Changes+ Attach+]
2016-10-14T10:20:35.278Z ==== Couchbase Sync Gateway/1.3.1(16;f18e833) ====
2016-10-14T10:20:35.278Z requestedSoftFDLimit >= currentHardFdLimit (5000 >= 4096) capping at 4096
2016-10-14T10:20:35.278Z Configured process to allow 4096 open file descriptors
2016-10-14T10:20:35.278Z Opening db /yoyottaid as bucket "yoyottaid", pool "default", server <http://localhost:8091>
2016-10-14T10:20:35.278Z Opening Couchbase database yoyottaid on <http://localhost:8091>
_time=2016-10-14T10:20:35.296+00:00 _level=INFO _msg= Trying with selected node 0 
_time=2016-10-14T10:20:35.296+00:00 _level=INFO _msg= Trying with http://10.136.24.210:8091/pools/default/bucketsStreaming/yoyottaid 
_time=2016-10-14T10:20:35.311+00:00 _level=INFO _msg=Got new configuration for bucket yoyottaid 
_time=2016-10-14T10:20:35.333+00:00 _level=INFO _msg= Trying with selected node 0 
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x4e78ea]

goroutine 1 [running]:
github.com/couchbase/sync_gateway/rest.(*ServerContext)._getOrAddDatabaseFromConfig(0xc820194900, 0xc8201661c0, 0xc820194900, 0x15acc20, 0x0, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:473 +0x1f9a
github.com/couchbase/sync_gateway/rest.(*ServerContext).getOrAddDatabaseFromConfig(0xc820194900, 0xc8201661c0, 0x406a00, 0x0, 0x0, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:254 +0xa4
github.com/couchbase/sync_gateway/rest.(*ServerContext).AddDatabaseFromConfig(0xc820194900, 0xc8201661c0, 0xc820141e40, 0x0, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:650 +0x3c
github.com/couchbase/sync_gateway/rest.RunServer(0xc82015f7a0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:676 +0x328
github.com/couchbase/sync_gateway/rest.ServerMain(0xc800000000)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:735 +0x3a
main.main()
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/main.go:34 +0x133

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 7 [IO wait]:
net.runtime_pollWait(0x7fef64068a20, 0x72, 0xc82000e160)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201a15d0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201a15d0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201a1570, 0xc8201b8000, 0x1000, 0x1000, 0x0, 0x7fef65eb4050, 0xc82000e160)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820028048, 0xc8201b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fef65eb8790, 0xc820028048, 0xc820160478, 0xc8201b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc82000a720, 0xc8201b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8200141e0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8200141e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820160420)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 18 [select, locked to thread]:
runtime.gopark(0xf842b8, 0xc820020728, 0xd49e40, 0x6, 0x42e518, 0x2)
    /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc820020728, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc820020728)
    /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 19 [chan receive]:
main.main.func1(0xc8201709c0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/main.go:28 +0x51
created by main.main
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/main.go:32 +0x12a

goroutine 8 [select]:
net/http.(*persistConn).writeLoop(0xc820160420)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 21 [IO wait]:
net.runtime_pollWait(0x7fef640688a0, 0x72, 0xc82000e160)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820056840, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820056840, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200567e0, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x7fef65eb4050, 0xc82000e160)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820028090, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fef65eb8790, 0xc820028090, 0xc8200cb7b8, 0xc82027a000, 0x1000, 0x1000, 0x9, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820227700, 0xc82027a000, 0x1000, 0x1000, 0x9, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc820014b40)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc820014b40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:328 +0x21a
net/http/internal.readLine(0xc820014b40, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:110 +0x4b
net/http/internal.(*chunkedReader).beginChunk(0xc82019fbf0)
    /usr/local/go/src/net/http/internal/chunked.go:47 +0x39
net/http/internal.(*chunkedReader).Read(0xc82019fbf0, 0xc820280004, 0x1dfc, 0x1dfc, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:77 +0xb7
net/http.(*body).readLocked(0xc820195340, 0xc820280004, 0x1dfc, 0x1dfc, 0xc800000000, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:632 +0x98
net/http.(*body).Read(0xc820195340, 0xc820280004, 0x1dfc, 0x1dfc, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:624 +0x110
net/http.(*bodyEOFSignal).Read(0xc820195380, 0xc820280004, 0x1dfc, 0x1dfc, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1296 +0x26a
encoding/json.(*Decoder).refill(0xc82017dba0, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:152 +0x287
encoding/json.(*Decoder).readValue(0xc82017dba0, 0x1, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:128 +0x41b
encoding/json.(*Decoder).Decode(0xc82017dba0, 0xa8d620, 0xc8200f8298, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:57 +0x159
github.com/couchbase/go-couchbase.(*Bucket).UpdateBucket(0xc820228540, 0x0, 0x0)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/streaming.go:139 +0xd0f
github.com/couchbase/go-couchbase.(*Bucket).RunBucketUpdater.func1(0xc820228540, 0xc820255b50)
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/streaming.go:61 +0x25
created by github.com/couchbase/go-couchbase.(*Bucket).RunBucketUpdater
    /home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/streaming.go:68 +0x3f

goroutine 15 [select]:
net/http.(*persistConn).writeLoop(0xc8200cbb80)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 14 [runnable]:
syscall.Syscall(0x3, 0x7, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Close(0x7, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:296 +0x4a
net.(*netFD).destroy(0xc820056bd0)
    /usr/local/go/src/net/fd_unix.go:135 +0x46
net.(*netFD).decref(0xc820056bd0)
    /usr/local/go/src/net/fd_unix.go:153 +0x40
net.(*netFD).Close(0xc820056bd0, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:199 +0x87
net.(*conn).Close(0xc8200280b0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:196 +0xab
net/http.(*persistConn).closeLocked(0xc8200cbb80)
    /usr/local/go/src/net/http/transport.go:1248 +0x4e
net/http.(*persistConn).close(0xc8200cbb80)
    /usr/local/go/src/net/http/transport.go:1242 +0x7a
net/http.(*persistConn).readLoop(0xc8200cbb80)
    /usr/local/go/src/net/http/transport.go:1004 +0x954
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 11 [select]:
net/http.(*persistConn).readLoop(0xc8200cb760)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 12 [select]:
net/http.(*persistConn).writeLoop(0xc8200cb760)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d
2

Hi, it looks like no-one is able to help with resolving this, but can anyone tell me if they have successfully used the “Unsupported->user_views” option through SG 1.3.1 on Linux?

If so, I am obviously doing something wrong and so will try to work out what. However, as my example is so simple, I can’t see what this could be. Therefore I think that I should report this as a bug through issues.couchbase.com - just wanted to check first

Thanks,
Giles

3

@giles did you take a look at some of the other threads on user_views? Nothing stands out as identical to your issue, but there is some info that might help you diagnose. Check, e.g. Getting Forbidden when trying to access a view from public URL of sync_gateway

That’s not the only one, though.

Hod

4

Hi Hod,

Thanks for the reply. Yes, I’ve already read that thread and all others that I can find, and tried all the tips that they suggest. Unfortunately they don’t help. The main suggestion from that thread in particular is to make sure that user_views is enabled, which in my case is what is causing the crash.

Also, I can get this to work on MacOS. It’s only on Linux where I see the crash.

Thanks,
Giles

5

Hi Giles,

This sounds like issue https://github.com/couchbase/sync_gateway/issues/2080. There are details on the ticket about how to work around this issue.

Thanks,
Adam

1 Like
6

Thank you Adam! The suggestion in that thread fixed my crash, and I have now restarted sync_gateway and added my view. This is a definite advancement and fixes the issue in my first post in this thread.

----- Unfortunately I can’t get the view to work on the public port - it returns 0 rows… -----
(ignore the above - now resolved as a channel access issue).

One thing - the fix in the thread that you linked was to add the following to the config file:

“oidc_test_provider”: {
“enabled”:false
}

This fixes the issue but I haven’t got a clue what it’s doing and what oidc_test_provider is supposed to be used for. The thread doesn’t give any details, Do you (or anyone else on here) have any idea?

Thanks,
Giles

7

Actually don’t worry about that. I’ve realised there is documentation on the oidc_test_provider option. It’s to do with OpenID Connect which I don’t fully understand yet (I need to do some reading) but it does state that this option should not be enabled for production code, so it looks like setting this to false is the correct thing to do.

Thanks,
Giles