No data returned when using 2 or more nodes via gui as well as server crash

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 2.0
Fix Version/s: 2.0.2
Component/s: couchbase-bucket
Security Level: Public
Labels:
None
Environment:
running version 2.0.0 enterprise edition (build-1976) on 2 Servers Ubuntu 11.10

Description

Installed 2 Servers with latest version, created initial server and added some basic docs, index like post_picture_1 and so on, docs are in jason format.
As long its only single server all works fine. When i add 2nd Server and finish rebalance i see the docs are distributed among both servers. When i go
to buckets it shows activity indicator and comes back with "There are currently no documents in this bucket." if i search for the doc by index it either
returns nothing or sometimes reports it lost communication to server. When going back to the server the gui was run on in the os, i can verify status
that couchbase no longer runs and i need to start manually again.
So i am not certain if the 2 issues are related to each other why there is no data in the document screen in gui once i use cluster. All works normal
with single server. I also was able to create same issues with 3 server cluster

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

ns-diag-20130103182922.rar

03/Jan/13 12:56 PM

12.43 MB

makeawish

screenshot-1.jpg

64 kB

03/Jan/13 1:12 PM
screenshot-2.jpg

50 kB

03/Jan/13 1:14 PM
screenshot-3.jpg

52 kB

03/Jan/13 1:16 PM
screenshot-4.jpg

46 kB

03/Jan/13 1:16 PM

Activity

Ascending order - Click to sort in descending order

Mike Wiederhold made changes - 02/Jan/13 5:04 PM

Field	Original Value	New Value
Assignee	Chiyoung Seo [ chiyoung ]	Mike Wiederhold [ mikew ]

Hide

Permalink

Mike Wiederhold added a comment - 02/Jan/13 5:08 PM

Can you attach the server logs so I can take a look? They can generated by running /opt/coucbase/bin/collectinfo. You can also get them by clicking on the logs tab in the web console and then clicking the link that says "generate diagnostic report".

Show

Mike Wiederhold added a comment - 02/Jan/13 5:08 PM Can you attach the server logs so I can take a look? They can generated by running /opt/coucbase/bin/collectinfo. You can also get them by clicking on the logs tab in the web console and then clicking the link that says "generate diagnostic report".

makeawish made changes - 03/Jan/13 12:56 PM

Attachment

ns-diag-20130103182922.rar [ 16096 ]

Hide

Permalink

makeawish added a comment - 03/Jan/13 12:57 PM - edited

Uploaded diag file
If you need anything else, let me know

Show

makeawish added a comment - 03/Jan/13 12:57 PM - edited Uploaded diag file If you need anything else, let me know

Hide

Permalink

makeawish added a comment - 03/Jan/13 1:12 PM

Shows both servers and 5 docs

Show

makeawish added a comment - 03/Jan/13 1:12 PM Shows both servers and 5 docs

makeawish made changes - 03/Jan/13 1:12 PM

Attachment

screenshot-1.jpg [ 16097 ]

Hide

Permalink

makeawish added a comment - 03/Jan/13 1:14 PM

Data Bucket shows also 5 docs in default

Show

makeawish added a comment - 03/Jan/13 1:14 PM Data Bucket shows also 5 docs in default

makeawish made changes - 03/Jan/13 1:14 PM

Attachment

screenshot-2.jpg [ 16098 ]

Hide

Permalink

makeawish added a comment - 03/Jan/13 1:16 PM

Screen shows searching for docs for about 10 to 15 sec

Show

makeawish added a comment - 03/Jan/13 1:16 PM Screen shows searching for docs for about 10 to 15 sec

makeawish made changes - 03/Jan/13 1:16 PM

Attachment

screenshot-3.jpg [ 16099 ]

Hide

Permalink

makeawish added a comment - 03/Jan/13 1:16 PM

Returns no Docs in bucket even so there are docs in the bucket

Show

makeawish added a comment - 03/Jan/13 1:16 PM Returns no Docs in bucket even so there are docs in the bucket

makeawish made changes - 03/Jan/13 1:16 PM

Attachment

screenshot-4.jpg [ 16100 ]

Hide

Permalink

Mike Wiederhold added a comment - 04/Jan/13 12:29 AM

How much memory does each server contain? Can you also run top on one of the machines and paste the output here? From looking at the logs it appears that one of the processes in the server is being kill by the OS. This information will help me to understand if this is the case.

Show

Mike Wiederhold added a comment - 04/Jan/13 12:29 AM How much memory does each server contain? Can you also run top on one of the machines and paste the output here? From looking at the logs it appears that one of the processes in the server is being kill by the OS. This information will help me to understand if this is the case.

Hide

Permalink

makeawish added a comment - 09/Jan/13 4:31 PM

Right now they ron with 512 MB, but i thought it was the mem issue and moved the VM's to GB and it behaved still same way. The single server (non cluster ) behaves just fine with 512 MB or 8 GB

Show

makeawish added a comment - 09/Jan/13 4:31 PM Right now they ron with 512 MB, but i thought it was the mem issue and moved the VM's to GB and it behaved still same way. The single server (non cluster ) behaves just fine with 512 MB or 8 GB

Mike Wiederhold made changes - 13/Jan/13 3:54 PM

Fix Version/s

2.0.2 [ 10418 ]

Mike Wiederhold made changes - 18/Jan/13 3:09 PM

Assignee

Mike Wiederhold [ mikew ]

Chiyoung Seo [ chiyoung ]

Chiyoung Seo made changes - 15/Feb/13 7:12 PM

Sprint Status

Current Sprint [ 10027 ]

Hide

Permalink

Chiyoung Seo added a comment - 15/Feb/13 7:17 PM

For the bug distributions in the engine team.

Show

Chiyoung Seo added a comment - 15/Feb/13 7:17 PM For the bug distributions in the engine team.

Chiyoung Seo made changes - 15/Feb/13 7:17 PM

Assignee

Chiyoung Seo [ chiyoung ]

Xiaoqin Ma [ xiaoqin ]

Chiyoung Seo made changes - 15/Feb/13 7:17 PM

Planned End

(re-schedule end date based on new assignee)

Hide

Permalink

Xiaoqin Ma added a comment - 20/Feb/13 5:16 PM

@makawish, what is the size of your document? Do you have log files? Thanks!

Show

Xiaoqin Ma added a comment - 20/Feb/13 5:16 PM @makawish, what is the size of your document? Do you have log files? Thanks!

Hide

Permalink

Mike Wiederhold added a comment - 20/Feb/13 5:29 PM

Xiaoqin,

The logs for this issue are already attached.

Show

Mike Wiederhold added a comment - 20/Feb/13 5:29 PM Xiaoqin, The logs for this issue are already attached.

Hide

Permalink

Mike Wiederhold added a comment - 20/Feb/13 6:26 PM

I looked through the logs and memcached looks like it is continually killed by the OS for some reason. See the message below where sigkill (137) is reported. I don't think there is anything we can do about this since a non-Couchbase process is killing us.

=========================CRASH REPORT=========================
  crasher:
    initial call: ns_port_server:init/1
    pid: <0.881.0>
    registered_name: ns_port_memcached
    exception exit: {abnormal,137}
      in function gen_server:terminate/6
    ancestors: [<0.879.0>,ns_port_sup,ns_server_sup,ns_server_cluster_sup,
                  <0.58.0>]
    messages: [{'EXIT',#Port<0.7456>,normal}]
    links: [<0.879.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 75025
    stack_size: 24
    reductions: 1342314
  neighbours:

[error_logger:error,2012-12-20T0:58:59.107,ns_1@10.181.2.115:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.879.0> terminating
** Last message in was {die,{abnormal,137}}
** When Server state == {state,memcached,5000,
                               {1355,962940,896183},
                               undefined,infinity}
** Reason for termination ==
** {abnormal,137}

Show

Mike Wiederhold added a comment - 20/Feb/13 6:26 PM I looked through the logs and memcached looks like it is continually killed by the OS for some reason. See the message below where sigkill (137) is reported. I don't think there is anything we can do about this since a non-Couchbase process is killing us. =========================CRASH REPORT========================= crasher: initial call: ns_port_server:init/1 pid: <0.881.0> registered_name: ns_port_memcached exception exit: {abnormal,137} in function gen_server:terminate/6 ancestors: [<0.879.0>,ns_port_sup,ns_server_sup,ns_server_cluster_sup, <0.58.0>] messages: [{'EXIT',#Port<0.7456>,normal}] links: [<0.879.0>] dictionary: [] trap_exit: true status: running heap_size: 75025 stack_size: 24 reductions: 1342314 neighbours: [error_logger:error,2012-12-20T0:58:59.107, ns_1@10.181.2.115 :error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server <0.879.0> terminating ** Last message in was {die,{abnormal,137}} ** When Server state == {state,memcached,5000, {1355,962940,896183}, undefined,infinity} ** Reason for termination == ** {abnormal,137}

Hide

Permalink

Mike Wiederhold added a comment - 27/Feb/13 5:11 PM

I'm closing this issue since the logs show that the memcached process appears to be killed by the os.

Show

Mike Wiederhold added a comment - 27/Feb/13 5:11 PM I'm closing this issue since the logs show that the memcached process appears to be killed by the os.

Mike Wiederhold made changes - 27/Feb/13 5:11 PM

Status	Open [ 1 ]	Resolved [ 5 ]
Resolution		Fixed [ 1 ]

Hide

Permalink

Maria McDuff added a comment - 05/Apr/13 2:35 AM

sigkill

Show

Maria McDuff added a comment - 05/Apr/13 2:35 AM sigkill

Maria McDuff made changes - 05/Apr/13 2:35 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

People

Assignee:

Xiaoqin Ma

Reporter:

makeawish

Vote (0)

Watch (4)

Dates

Created:

02/Jan/13 2:43 PM

Updated:

05/Apr/13 2:35 AM

Resolved:

27/Feb/13 5:11 PM

Agile

View on Board