I am not sure what happened to the server and it’s not responding.
Somehow, after months of operating, the server decided to become non-responsive in some function.
Couchbase Server 4.5, is inside a docker container. I have tried restarting everything, but nothing helped.
- Views are working fine (I wrote a new one in-between actually, everything is returning beautifully).
- N1QL server is not responding to me, but some other colleague somehow can get queries, both failing and successful. The weird thing is he is using my scripts to fetch queries.
- Sync_gateway stopped accepting new documents (according to my colleague), and I have some evidence to provide that (after some time, the bucket activity was showing 0, requests to 8091 port balanced [0, 1].
What is more weird about the N1QL queries, is that server activity is a constant [400, 600] queries per second. I believe it showed [4k, 6k]. Also, queries run for 4h30m and then they completely failing, without any kind of results, with or without indexes.
I get these kind of responses: (With error codes being red)
SELECT did, ARRAY_AGG({batteryCharge, batteryStatus, bcn, stepCounterDelta, t, type}) AS entries
FROM agora_bucket USE INDEX(`#primary`)
WHERE type="bcn_scan"
GROUP BY did ORDER BY did ASC LIMIT 3;
Connected to : http://localhost:8091/. Type Ctrl-D or \QUIT to exit.
Path to history file for the shell : /root/.cbq_history
ERROR 174 : N1QL: Query nodes not responding
SELECT did, ARRAY_AGG({batteryCharge, batteryStatus, bcn, stepCounterDelta, t, type}) AS entries
FROM agora_bucket
WHERE type="bcn_scan"
GROUP BY did ORDER BY did ASC LIMIT 3;
ERROR 100 : Get http://localhost:8093/admin/clusters/default/nodes: dial tcp 127.0.0.1:8093: connection refused
Path to history file for the shell : /root/.cbq_history
ERROR 107 : Not connected to any cluster. Use \CONNECT command.
SELECT did, ARRAY_AGG({batteryCharge, batteryStatus, bcn, stepCounterDelta, t, type}) AS entries
FROM agora_bucket
WHERE type="bcn_scan"
GROUP BY did ORDER BY did ASC;
ERROR 100 : Get http://localhost:8093/admin/clusters/default/nodes: dial tcp 127.0.0.1:8093: connection refused
Path to history file for the shell : /root/.cbq_history
ERROR 107 : Not connected to any cluster. Use \CONNECT command.
SELECT did, ARRAY_AGG({batteryCharge, batteryStatus, bcn, stepCounterDelta, t, type}) AS entries
FROM agora_bucket USE INDEX(`#primary`)
WHERE type="bcn_scan"
GROUP BY did ORDER BY did ASC;
ERROR 100 : Get http://localhost:8093/admin/clusters/default/nodes: dial tcp 127.0.0.1:8093: connection refused
Path to history file for the shell : /root/.cbq_history
ERROR 107 : Not connected to any cluster. Use \CONNECT command.
The server has limited resources, but a) it’s gonna be hard to intervene, b) the need for that server is almost over. So I just need it to work for a month or two (or just dump the whole data in JSON) and the server will go out of commission.
Prompt responces are appreciated and thank you for wasting your time with me <3
Edit: have also been getting messages like:
Date: Mon, 27 Mar 2017 22:55:47 +0000
From: couchbase@silent.cs.abo.fi
To: ntentos@ntentos.abo.local
Subject: Couchbase Server alert: IP address changed
IP address seems to have changed. Unable to listen on 'ns_1@127.0.0.1'. (Underlaying POSIX error code: 'nxdomain')