Home | Forums | Couchbase | Couchbase Server 2.0

Does Indexing Ever Stop?

6 replies [Last post]
Wed, 02/27/2013 - 10:51
Michael Rickey
Offline
Joined: 02/15/2013
Groups: None

I installed Couchbase (CB) on my development box to see how it would work for our application.

I started with about 50,000 records, created a design doc and two views. Each of the views emits one field and uses the built-in "_count" reduce function. Once I got the views working correctly, I started pushing more data into CB.

I put about 26 million records in CB. This was last Thursday, I let it index over night. On Friday I spent some time looking at the data and writing a program to extract the view results so I can report on them. I turned the machine off for the weekend.

This morning I started up and my hard drive never settled down after the OS loaded and stabilized. After some sleuthing I found CB indexing. It has been going non-stop since, about eight hours. It appears to have restarted the indexing process several times today. I've seen it as high as 28% complete.

Why is it indexing the entire database again? Is there a shutdown procedure I should be following? Or, is there something else going on here? I haven't been able to work with CB at all today because of the indexing.

This installation is on a Windows 7 machine, if it matters.

__________________

--
Michael Rickey

Top
Wed, 02/27/2013 - 11:53
Michael Rickey
Offline
Joined: 02/15/2013
Groups: None

I have learned something new about this. The Couchbase log shows the server starting numerous times during the day. I think it's restarting before it finishes indexing.

For example, it has started at 9:38:10, 8:34:49, 7:20:03, 5:43:35, 3:15:01, 1:37:41, 1:12:58, 1:01:32, and 0:51:47. And, that's just today.

There are also Windows Event Log messages that correspond to the restarts, here is an example:

Faulting application name: erl.exe, version: 0.0.0.0, time stamp: 0x4e8a0dd4
Faulting module name: beam.smp.dll, version: 0.0.0.0, time stamp: 0x4e8a0cbe
Exception code: 0xc0000005
Fault offset: 0x0008db3d
Faulting process id: 0x2714
Faulting application start time: 0x01ce15082f97815c
Faulting application path: D:\Program Files\Couchbase\Server\erts-5.8.5\bin\erl.exe
Faulting module path: D:\PROGRA~2\COUCHB~1\Server\ERTS-5~1.5\bin\beam.smp.dll
Report Id: 4c8c292c-8104-11e2-9c59-1c6f6591c5c7

How can I find out what causes erl.exe to fail?

__________________

--
Michael Rickey

Top
Wed, 02/27/2013 - 12:07
tgrall
Offline
Joined: 09/05/2012
Groups: None

Hello Michael,

It looks like your problem is not related to Indexing itself but more a Windows/Couchbase/Erlang issue. (we are working on fix for this)

Could you please give us some information about your environment:
- OS
- Hardware
- HD
...

Have you configure the IP address of your Couchbase Internal configuration manually as documented here:
http://tugdualgrall.blogspot.dk/2012/12/what-to-do-if-your-couchbase-ser...

Regards

__________________

Tug
@tgrall

Top
Wed, 02/27/2013 - 12:22
Michael Rickey
Offline
Joined: 02/15/2013
Groups: None

I;m running Windows 7 SP1 64 bit. The computer has an i7-920, 20GB RAM, 256GB SSD and 2TB regular hard drive. Couchbase and it's buckets are on the regular HD.

No, I have not setup my environment according to the article you referenced. I didn't have the problems described in the article.

__________________

--
Michael Rickey

Top
Wed, 02/27/2013 - 20:27
ingenthr
Offline
Joined: 03/16/2010
Groups:

There is a regular background job that will update the view, but it should consume minimal resources if the index is complete.

Did the indexing complete in the first place? Are you making lots of changes to the documents or inserting many more? If so, this could explain the additional indexing.

Also, note that if your "start up" was having the machine return from hibernation or the like, there is a known issue which will consume a lot of resources on startup. If this is troublesome, we recommend shutting down Couchbase Server before suspending and starting it up after resuming. However, with this amount of data, the warmup time may be long.

Again, once the index is built, subsequent updates shouldn't consume many resources. There may be something else going on here.

Top
Wed, 02/27/2013 - 20:28
ingenthr
Offline
Joined: 03/16/2010
Groups:

Regular restarts are one symptom of the problem Tug describes in his article.

Top
Thu, 02/28/2013 - 10:45
Michael Rickey
Offline
Joined: 02/15/2013
Groups: None

I performed the configuration change as recommended. I'm able to connect to the management console, but I get an error back with I try to click "Next" on "Step 1 of 5".

On the browser window it says "Attention - Request returned error."

I captured the request using Fiddler and the return data is actually

The socket connection to rmr-d4.mpc.com failed. 
ErrorCode: 10061. 
No connection could be made because the target machine actively refused it 127.0.0.1:8091

There are actually two posts made when I click "Next". The are to /nodes/self/controller/settings and /pools/default.

Both return a 502 status and the message above.

I'll do some sleuthing in the forums for an answer...

__________________

--
Michael Rickey

Top