couchdb terminates unexpectedly
Today I wrote a performance testing program to insert data and add attachments to couchdb. The server software is couchbase(which is a wrapper of couchdb) and the operating system is Windows 2003 server. The program is developed in C# and the couchdb driver is LoveSeat(which encapsulates some http methods to deal with couchdb). The data is quite simple, but the attachments are not small, about 70kb each. There were about 200 attachments and I was attaching them repeatedly. The client program is developed in C# and I started 5 threads to insert data and add attachments.
Everything looked all right until couchdb server terminated unexpectedly. Actually it was not the first time it crashed. At first I thought it was because of the client program. But at last I found couchdb crashed. I don't think it was because of the data or attachment. because I was inserting the same data and adding the same attachment repeatedly. The program run for about an hour without problem until it crashed.
[Tue, 05 Jul 2011 11:00:19 GMT] [info] [<0.142.0>] 192.168.1.135 - - 'GET' /test/67366 200
[Tue, 05 Jul 2011 11:00:19 GMT] [info] [<0.108.0>] 192.168.1.135 - - 'GET' /test/7136 200
[Tue, 05 Jul 2011 11:00:19 GMT] [info] [<0.108.0>] 192.168.1.135 - - 'GET' /test/47306 200
[Tue, 05 Jul 2011 11:00:19 GMT] [info] [<0.108.0>] 192.168.1.135 - - 'GET' /test/27257 200
[Tue, 05 Jul 2011 11:00:19 GMT] [info] [<0.108.0>] 192.168.1.135 - - 'PUT' /test/7136/f?rev=1-334efd144dcdc52fd3a3a981dce4472f 201
[Tue, 05 Jul 2011 11:00:25 GMT] [error] [<0.145.0>] ** Generic server <0.145.0> terminating
** Last message in was {pread_iolist,4294342003}
** When Server state == {file,{file_descriptor,prim_file,{#Port<0.3143>,1464}},
0,4295164786}
** Reason for termination ==
** {{badmatch,{ok,<<183,92,29,219,169,127,153,2,50,217,252,186,178,175,202,
144,215,209,191,69,109,230,227,154,114,174,173,157,231,
153,246,124,105,239,174,51,143,24,108,175,101,215,175,
221,35,99,53,124,108,109,249,112,202,29,85,87,81,176,94,
219,11,103,129,231,25,111,242,108,246,207,107,72,173,172,
57,246,195,16,236,79,243,134,211,93,131,218,180,93,240,
173,213,199,226,175,176,217,250,154,89,39,237,157,250,77,
173,151,156,139,248,106,85,21,134,253,85,234,108,85,208,
67,177,130,124,247,161,98,77,173,126,170,111,80,84,45,
212,201,72,149,90,138,252,89,23,85,165,252,105,187,191,
41,86,125,148,106,149,175,252,78,185,198,154,207,172,142,
148,101,83,140,99,222,102,26,41,131,206,132,221,31,74,3,
172,176,158,236,136,71,120,169,63,35,161,251,208,86,202,
1,95,208,25,51,76,250,100,182,177,122,31,91,230,249,214,
245,229,250,212,118,86,167,120,116,6,173,78,113,18,171,
143,215,191,38,207,51,92,150,10,10,83,164,98,154,181,157,
......... a loooooot of numbers.
I think I just found the cause: the file size reached 4GB. couchbase doesn't support file size more than 4GB?
Hi Peter,
Double-checking -- were you using a 32-bit O/S?
Steve
Yes I am using 32-bit windows 2003 server. But windows 2003 server ntfs file system support files larger than 4gb.
I tried Couchbase Single Server 2.0 Developer Preview, it is 5 times slower than 1.0. So I am still wait for the file size to exceed 4GB. I don't think such a slow product could be used in production.
2.0 version does not crash when the file exceeds 4GB. But the speed is 10 times slower than 1.0 version. Now it takes 1.4 second to add an attachment.
Verrry interesting. This is why we love Developer Preview releases. Is there any chance you could publish your client test app somewhere? I know most of the core couch guys aren't necessarily C# folk, but at least they would be able to read it to get a sense of what it's doing and attempt to replicate.
Cheers,
Steve
the main part of the code, I just started 5 threads and call the method SaveAttachment() repeatedly:
private CouchDatabase cdb = null;
///
/// 新建一个存储附件的server实例
///
/// 主机地址
/// 端口
/// 数据库名
public AttachmentServer(string host, int port, string database)
{
this.cdb = new CouchClient(host, port, null, null).GetDatabase(database);
}
///
/// 向couchdb存储附件. 更新附件也可以用这个方法.
///
/// 包含后缀的文档名称, 比如10000202_123.docx
/// 附件的byte数组
/// 其它需要存储的参数
public void SaveAttachment(string fileNameWithExtension, byte[] attachment, Dictionary parameters)
{
var id = Path.GetFileNameWithoutExtension(fileNameWithExtension);
var extension = Path.GetExtension(fileNameWithExtension);
if (!parameters.ContainsKey("_id"))
parameters["_id"] = id;
if (!parameters.ContainsKey("ext"))
parameters["ext"] = extension;
var doc = new Document(JsonConvert.SerializeObject(parameters));
var result = cdb.SaveDocument(doc);
var contentType = GetContentType(fileNameWithExtension);
cdb.AddAttachment(id, attachment, "f", contentType);
}
Thanks -- also double-checking. When you bring up the Futon UI, what version of couch does it report in the lower-right-hand corner of the page?
Cheers,
Steve
As far as I could remember. At first it was 1.02. Later I changed to couchbase 2.0 preview version it became 1.2.
And now I am using couchdb from this link and it is 1.1: https://github.com/dch/couchdb/downloads.
Hi Peter,
The binaries from github.com/dch/couchdb/downloads are from Dave Cottlehuber, a long-time Apache CouchDB contributor who's been very kindly providing pre-built Windows binaries. However, those are different than the Couchbase pre-built Windows binaries from http://www.couchbase.com/downloads. (Likely different compiler settings, fairy dust, and a different path through the cross-platform compilation minefield. I'm no Windows expert, but have overheard from colleagues much about the fun of building something from unix heritage on Windows.)
That said, another colleague mentioned that using tcp nodelay on Windows might help make a difference. If not already, can you edit your local.ini ($INSTALL_DIR\etc\couchdb\local.ini) to define the httpd socket_options of nodelay as true...
[httpd]
socket_options = [{nodelay, true}]In the source, btw, that option is shown commented out here:
https://github.com/couchbase/couchdb/blob/preview/etc/couchdb/default.in...
Cheers,
Steve
Hi Peter,
I'm wondering if you could retry your test but using the latest "Couchbase Single Server 2.0 Developer Preview"? There were some changes/fixes to the storage system that may have addressed this. Please find it available for download at: http://files.couchbase.org/developer-previews/
Cheers,
Steve