Couchbase Server

cbtransfer crashes with Python 2.4 and couchstore-files as source

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.2
  • Component/s: build, tools
  • Security Level: Public
  • Labels:
    None
  • Environment:
    build 2.0.1-170-rel

Description



Steps to reproduce (build 170, VM to reproduce - 10.3.3.95):
1. Create default bucket and load 100k items using mcsoda.
2. Use the below command to transfer data to file:
/opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
(there is sufficient disk space in /tmp)

Getting the below error:
[root@caper-012 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
2013-03-01 03:19:31,359: mt cbtransfer...
2013-03-01 03:19:31,360: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/
2013-03-01 03:19:31,360: mt sink : /tmp/backup
2013-03-01 03:19:31,360: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'bucket_destination': None, 'destination_operation': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': None}
2013-03-01 03:19:31,361: mt source_class: <class 'pump_sfd.SFDSource'>
2013-03-01 03:19:31,395: mt sink_class: <class 'pump_bfd.BFDSink'>
2013-03-01 03:19:31,395: mt source_buckets: default
2013-03-01 03:19:31,395: mt bucket: default
2013-03-01 03:19:31,396: mt source_nodes: N/A
2013-03-01 03:19:31,411: mt enqueueing node: N/A
2013-03-01 03:19:31,411: w0 node: N/A
2013-03-01 03:19:31,509: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
2013-03-01 03:19:31,509: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
....................python: Objects/obmalloc.c:765: PyObject_Malloc: Assertion `bp != ((void *)0)' failed.
Aborted

With less number of items e.g. 10k, this works fine.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Deepkaran Salooja added a comment -
With build 1976(2.0), I am hitting the original issue filed in MB-7832. So this issue is not a regression. These tests were not run with python 2.4 before.
Show
Deepkaran Salooja added a comment - With build 1976(2.0), I am hitting the original issue filed in MB-7832 . So this issue is not a regression. These tests were not run with python 2.4 before.
Hide
Permalink
Bin Cui added a comment -
You can modify parameter cbb_max_mb to limit batch size. The default is 100000 MB. A small number may do it. Looks like python SDK fails to allocate memory for sqlite db operation.

 /opt/couchbase/bin/cbtransfer -v -v -v -x cbb_max_mb=1000 couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
Show
Bin Cui added a comment - You can modify parameter cbb_max_mb to limit batch size. The default is 100000 MB. A small number may do it. Looks like python SDK fails to allocate memory for sqlite db operation.  /opt/couchbase/bin/cbtransfer -v -v -v -x cbb_max_mb=1000 couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
Hide
Permalink
Bin Cui added a comment -
[root@caper-012 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///
opt/couchbase/var/lib/couchbase/data/ /tmp/backup -x cbb_max_mb=100
2013-03-01 10:55:35,071: mt cbtransfer...
2013-03-01 10:55:35,072: mt source : couchstore-files:///opt/couchbase/var/lib/
couchbase/data/
2013-03-01 10:55:35,072: mt sink : /tmp/backup
2013-03-01 10:55:35,072: mt opts : {'username': None, 'source_vbucket_state':
 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False
, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 10
0.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_
max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}, 'single_node': Fals
e, 'bucket_destination': None, 'destination_operation': None, 'threads': 4, 'key
': None, 'password': None, 'id': None, 'bucket_source': None}
2013-03-01 10:55:35,073: mt source_class: <class 'pump_sfd.SFDSource'>
2013-03-01 10:55:35,107: mt sink_class: <class 'pump_bfd.BFDSink'>
2013-03-01 10:55:35,108: mt source_buckets: default
2013-03-01 10:55:35,108: mt bucket: default
2013-03-01 10:55:35,108: mt source_nodes: N/A
2013-03-01 10:55:35,124: mt enqueueing node: N/A
2013-03-01 10:55:35,125: w0 node: N/A
2013-03-01 10:55:35,226: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/d
ata-0000.cbb
2013-03-01 10:55:35,227: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/
data-0000.cbb
...............Traceback (most recent call last):
  File "source/callbacks.c", line 206, in 'calling callback function'
  File "/opt/couchbase/lib/python/couchstore.py", line 358, in callback
    fn(DocumentInfo._fromStruct(docInfoPtr.contents, self))
  File "/opt/couchbase/lib/python/pump_sfd.py", line 211, in change_callback
    cas, exp, flg = struct.unpack(SFD_REV_META, doc_info.revMeta)
TypeError: unpack() argument 2 must be string or read-only buffer, not CArgObjec
t
Segmentation fault
Show
Bin Cui added a comment - [ root@caper-012 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:/// opt/couchbase/var/lib/couchbase/data/ /tmp/backup -x cbb_max_mb=100 2013-03-01 10:55:35,071: mt cbtransfer... 2013-03-01 10:55:35,072: mt source : couchstore-files:///opt/couchbase/var/lib/ couchbase/data/ 2013-03-01 10:55:35,072: mt sink : /tmp/backup 2013-03-01 10:55:35,072: mt opts : {'username': None, 'source_vbucket_state':  'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False , 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 10 0.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_ max_size': 1000.0, 'report': 5.0, 'recv_min_bytes': 4096.0}, 'single_node': Fals e, 'bucket_destination': None, 'destination_operation': None, 'threads': 4, 'key ': None, 'password': None, 'id': None, 'bucket_source': None} 2013-03-01 10:55:35,073: mt source_class: <class 'pump_sfd.SFDSource'> 2013-03-01 10:55:35,107: mt sink_class: <class 'pump_bfd.BFDSink'> 2013-03-01 10:55:35,108: mt source_buckets: default 2013-03-01 10:55:35,108: mt bucket: default 2013-03-01 10:55:35,108: mt source_nodes: N/A 2013-03-01 10:55:35,124: mt enqueueing node: N/A 2013-03-01 10:55:35,125: w0 node: N/A 2013-03-01 10:55:35,226: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/d ata-0000.cbb 2013-03-01 10:55:35,227: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/ data-0000.cbb ...............Traceback (most recent call last):   File "source/callbacks.c", line 206, in 'calling callback function'   File "/opt/couchbase/lib/python/couchstore.py", line 358, in callback     fn(DocumentInfo._fromStruct(docInfoPtr.contents, self))   File "/opt/couchbase/lib/python/pump_sfd.py", line 211, in change_callback     cas, exp, flg = struct.unpack(SFD_REV_META, doc_info.revMeta) TypeError: unpack() argument 2 must be string or read-only buffer, not CArgObjec t Segmentation fault
Hide
Permalink
Jin Lim added a comment -
Bin this isn't a regression from 2.0.1 but can you please advise on how likely (often) users may run into this? We need to figure out whether to push this to 2.0.2 or not based on your input. Thanks!
Show
Jin Lim added a comment - Bin this isn't a regression from 2.0.1 but can you please advise on how likely (often) users may run into this? We need to figure out whether to push this to 2.0.2 or not based on your input. Thanks!
Hide
Permalink
Jin Lim added a comment -
From Bin:
It’s a use case that we never test before. The good thing is the bug doesn’t sit in the critical path for backup/restore. So we can defer the fix to 2.0.2 .
 
Moving this to 2.0.2.
Show
Jin Lim added a comment - From Bin: It’s a use case that we never test before. The good thing is the bug doesn’t sit in the critical path for backup/restore. So we can defer the fix to 2.0.2 .   Moving this to 2.0.2.
Hide
Permalink
Maria McDuff added a comment -
bug scrub: would be good to fix in 2.0.2 release.
Show
Maria McDuff added a comment - bug scrub: would be good to fix in 2.0.2 release.
Hide
Permalink
Steve Yen added a comment - - edited
Following up on Jin's comment, this is an edge case of usage; but from Bin's analysis, it exposes a bug that regular backup might also trigger. So leaving in 2.0.2 as Bin would like to address this for 2.0.2.
Show
Steve Yen added a comment - - edited Following up on Jin's comment, this is an edge case of usage; but from Bin's analysis, it exposes a bug that regular backup might also trigger. So leaving in 2.0.2 as Bin would like to address this for 2.0.2.
Hide
Permalink
Maria McDuff added a comment -
per bug committee, this is critical for 2.0.2 release. must fix due to bkup that can affect customer.
Show
Maria McDuff added a comment - per bug committee, this is critical for 2.0.2 release. must fix due to bkup that can affect customer.
Hide
Permalink
Pavel Paulau added a comment -
1. 100K isn't a magic number. The issue is occasional and large number of items increases probability.

2. I analyzed several core dumps, cbtransfer usually crashes because of segfault or because of:

    Objects/obmalloc.c:765: PyObject_Malloc: Assertion `bp != ((void *)0)' failed.

    Objects/obmalloc.c:953: PyObject_Free: Assertion `pool->ref.count > 0' failed

      File "source/callbacks.c", line 206, in 'calling callback function'
      File "/opt/couchbase/lib/python/couchstore.py", line 358, in callback
        fn(DocumentInfo._fromStruct(docInfoPtr.contents, self))
      File "/opt/couchbase/lib/python/couchstore.py", line 128, in _fromStruct
        self = DocumentInfo(str(info.id))
    TypeError: __str__ returned non-string (type buffer)

backtraces in turn vary as well:

#0 0x00002b8a11230e23 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0
#1 0x00002b8a112301cd in _PyObject_New () from /usr/lib64/libpython2.4.so.1.0
#2 0x00002b8a1630b950 in new_CArgObject () at source/callproc.c:288
#3 0x00002b8a16305a39 in PointerType_paramfunc (self=0x38) at source/_ctypes.c:564
#4 0x00002b8a1630b3de in ConvParam (obj=0x38, index=1, pa=0x2b8a1c157620) at source/callproc.c:477
#5 0x00002b8a1630bafd in _CallProc (pProc=0x2b8a163075b0 <string_at>, argtuple=0x1f152b48, flags=4097, argtypes=0x1ef82cb0, restype=0x1ef16790, checker=0x0)
    at source/callproc.c:959
#6 0x00002b8a16306bf3 in CFuncPtr_call (self=0x1eebf710, inargs=0x2, kwds=0x0) at source/_ctypes.c:3362
#7 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#8 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#9 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#10 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#11 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#12 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#13 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#14 0x00002b8a1120b90f in ?? () from /usr/lib64/libpython2.4.so.1.0
#15 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#16 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#17 0x00002b8a112445e8 in ?? () from /usr/lib64/libpython2.4.so.1.0
#18 0x00002b8a1122fe45 in PyObject_Str () from /usr/lib64/libpython2.4.so.1.0
#19 0x00002b8a1123a617 in ?? () from /usr/lib64/libpython2.4.so.1.0
#20 0x00002b8a1123ff53 in ?? () from /usr/lib64/libpython2.4.so.1.0
#21 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#22 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#23 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#24 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#25 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#26 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#27 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#28 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#29 0x00002b8a1630aacc in _CallPythonObject (cif=<value optimized out>, resp=0x2b8a1c158af0, args=0x2b8a1c158960, userdata=<value optimized out>)
    at source/callbacks.c:206
#30 closure_fcn (cif=<value optimized out>, resp=0x2b8a1c158af0, args=0x2b8a1c158960, userdata=<value optimized out>) at source/callbacks.c:252
#31 0x00002b8a1631054b in ffi_closure_unix64_inner (closure=0x2b8a1c15bab0, rvalue=0x2b8a1c158af0, reg_args=0x2b8a1c158a40, argp=0x2b8a1c158b10 "j\003")
    at /tmp/ctypes/source/libffi/src/x86/ffi64.c:563
#32 0x00002b8a16310800 in ffi_closure_unix64 () at /tmp/ctypes/source/libffi/src/x86/unix64.S:228
#33 0x00002b8a1651dc3d in lookup_callback (rq=<value optimized out>, k=0x2b8a1c158bb0, v=<value optimized out>) at src/couch_db.c:623
#34 0x00002b8a1651b714 in btree_lookup_inner (rq=0x2b8a1c158d10, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78
#35 0x00002b8a1651b608 in btree_lookup_inner (rq=0x2b8a1c158d10, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
#36 0x00002b8a1651c08c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>,
    callback=<value optimized out>, ctx=<value optimized out>) at src/couch_db.c:667
#37 0x00002b8a163106d4 in ffi_call_unix64 () at /tmp/ctypes/source/libffi/src/x86/unix64.S:73
#38 0x00002b8a16310244 in ffi_call (cif=0x2b8a1c159090, fn=0x2b8a1651bf40 <couchstore_changes_since>, rvalue=0x2b8a1c158f90, avalue=0x2b8a1c158f50)
    at /tmp/ctypes/source/libffi/src/x86/ffi64.c:428
#39 0x00002b8a1630bce1 in _call_function_pointer (pProc=0x2b8a1651bf40 <couchstore_changes_since>, argtuple=0x1f1a2ef0, flags=<value optimized out>, argtypes=0x0,
    restype=0x1ef9cce0, checker=0x0) at source/callproc.c:668
#40 _CallProc (pProc=0x2b8a1651bf40 <couchstore_changes_since>, argtuple=0x1f1a2ef0, flags=<value optimized out>, argtypes=0x0, restype=0x1ef9cce0, checker=0x0)
    at source/callproc.c:991
#41 0x00002b8a16306bf3 in CFuncPtr_call (self=0x1f073e90, inargs=0x1f1a2ef0, kwds=0x0) at source/_ctypes.c:3362
#42 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#43 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#44 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#45 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#46 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#47 0x00002b8a1121bb9a in ?? () from /usr/lib64/libpython2.4.so.1.0
#48 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#49 0x00002b8a11263c1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#50 0x00002b8a11265256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#51 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#52 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#53 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#54 0x00002b8a1120b90f in ?? () from /usr/lib64/libpython2.4.so.1.0
#55 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#56 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#57 0x00002b8a1128c96d in ?? () from /usr/lib64/libpython2.4.so.1.0
#58 0x00002b8a1150a83d in start_thread () from /lib64/libpthread.so.0
#59 0x00002b8a11e7ffad in clone () from /lib64/libc.so.6

#0 0x00002b7dd51b0285 in raise () from /lib64/libc.so.6
#1 0x00002b7dd51b1d30 in abort () from /lib64/libc.so.6
#2 0x00002b7dd51a9716 in __assert_fail () from /lib64/libc.so.6
#3 0x00002b7dd4606204 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0
#4 0x00002b7dd45decbe in ?? () from /usr/lib64/libpython2.4.so.1.0
#5 0x00002b7dd45def36 in ?? () from /usr/lib64/libpython2.4.so.1.0
#6 0x00002b7dd4614f53 in ?? () from /usr/lib64/libpython2.4.so.1.0
#7 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#8 0x00002b7dd4637fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#9 0x00002b7dd463b6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#10 0x00002b7dd45f0b9a in ?? () from /usr/lib64/libpython2.4.so.1.0
#11 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#12 0x00002b7dd4638c1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#13 0x00002b7dd463a256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#14 0x00002b7dd463b6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#15 0x00002b7dd45f0aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#16 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#17 0x00002b7dd45e090f in ?? () from /usr/lib64/libpython2.4.so.1.0
#18 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#19 0x00002b7dd463532d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#20 0x00002b7dd466196d in ?? () from /usr/lib64/libpython2.4.so.1.0
#21 0x00002b7dd48df83d in start_thread () from /lib64/libpthread.so.0
#22 0x00002b7dd5254fad in clone () from /lib64/libc.so.6

#0 0x00002b00a0fa2285 in raise () from /lib64/libc.so.6
#1 0x00002b00a0fa3d30 in abort () from /lib64/libc.so.6
#2 0x00002b00a0f9b716 in __assert_fail () from /lib64/libc.so.6
#3 0x00002b00a03f8204 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0
#4 0x00002b00a040139f in PyString_FromString () from /usr/lib64/libpython2.4.so.1.0
#5 0x00002b00a04014d9 in PyString_InternFromString () from /usr/lib64/libpython2.4.so.1.0
#6 0x00002b00a03f58b6 in PyObject_GetAttrString () from /usr/lib64/libpython2.4.so.1.0
#7 0x00002b00a54d1b62 in ConvParam (obj=0x126d5510, index=1, pa=0x2b00ab31ec90) at source/callproc.c:562
#8 0x00002b00a54d1f7a in _CallProc (pProc=0x2b00a56e3300 <couchstore_open_doc_with_docinfo>, argtuple=0x124ec680, flags=4097, argtypes=0x0, restype=0x1231a980,
    checker=0x0) at source/callproc.c:966
#9 0x00002b00a54ccd0a in CFuncPtr_call (self=<value optimized out>, inargs=<value optimized out>, kwds=0x0) at source/_ctypes.c:3362
#10 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#11 0x00002b00a0429fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#12 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#13 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#14 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#15 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#16 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#17 0x00002b00a03e2aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#18 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#19 0x00002b00a042732d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#20 0x00002b00a54d0ef3 in _CallPythonObject (cif=<value optimized out>, resp=0x2b00ab31fac0, args=<value optimized out>, userdata=<value optimized out>)
    at source/callbacks.c:206
#21 closure_fcn (cif=<value optimized out>, resp=0x2b00ab31fac0, args=<value optimized out>, userdata=<value optimized out>) at source/callbacks.c:252
#22 0x00002b00a54d6639 in ffi_closure_unix64_inner (closure=0x2b00ab322e10, rvalue=0x2b00ab31fac0, reg_args=0x2b00ab31fa10, argp=0x2b00ab31fae0 "\021\005")
    at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/ffi64.c:563
#23 0x00002b00a54d6fd4 in ffi_closure_unix64 () at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/unix64.S:228
#24 0x00002b00a56e4c3d in lookup_callback (rq=<value optimized out>, k=0x2b00ab31fb80, v=<value optimized out>) at src/couch_db.c:623
#25 0x00002b00a56e2714 in btree_lookup_inner (rq=0x2b00ab31fce0, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78
#26 0x00002b00a56e2608 in btree_lookup_inner (rq=0x2b00ab31fce0, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52
#27 0x00002b00a56e308c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>,
    callback=<value optimized out>, ctx=<value optimized out>) at src/couch_db.c:667
#28 0x00002b00a54d6ea8 in ffi_call_unix64 () at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/unix64.S:73
#29 0x00002b00a54d6c35 in ffi_call (cif=0x2b00ab320080, fn=0x2b00a56e2f40 <couchstore_changes_since>, rvalue=<value optimized out>, avalue=<value optimized out>)
    at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/ffi64.c:428
#30 0x00002b00a54d215d in _call_function_pointer (pProc=0x2b00a56e2f40 <couchstore_changes_since>, argtuple=0x124ba7d0, flags=4097, argtypes=0x0, restype=0x1231a980,
    checker=0x0) at source/callproc.c:668
#31 _CallProc (pProc=0x2b00a56e2f40 <couchstore_changes_since>, argtuple=0x124ba7d0, flags=4097, argtypes=0x0, restype=0x1231a980, checker=0x0)
    at source/callproc.c:991
#32 0x00002b00a54ccd0a in CFuncPtr_call (self=<value optimized out>, inargs=<value optimized out>, kwds=0x0) at source/_ctypes.c:3362
#33 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#34 0x00002b00a0429fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#35 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#36 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#37 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#38 0x00002b00a03e2b9a in ?? () from /usr/lib64/libpython2.4.so.1.0
#39 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#40 0x00002b00a042ac1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#41 0x00002b00a042c256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0
#42 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0
#43 0x00002b00a03e2aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0
#44 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#45 0x00002b00a03d290f in ?? () from /usr/lib64/libpython2.4.so.1.0
#46 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0
#47 0x00002b00a042732d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0
#48 0x00002b00a045396d in ?? () from /usr/lib64/libpython2.4.so.1.0
#49 0x00002b00a06d183d in start_thread () from /lib64/libpthread.so.0
#50 0x00002b00a1046fad in clone () from /lib64/libc.so.6

3. Installing Python 2.6 on this machine fixed the problem. However I'd not blame Python 2.4 - as I mentioned before absolutely identical setup worked pretty well on other machine.

4. ctypes that we ship is another suspect. That was a workaround, not necessarily it addresses all edge cases. And couchstore + Python 2.4 is one of them, in fact all other sources work fine.
Show
Pavel Paulau added a comment - 1. 100K isn't a magic number. The issue is occasional and large number of items increases probability. 2. I analyzed several core dumps, cbtransfer usually crashes because of segfault or because of:     Objects/obmalloc.c:765: PyObject_Malloc: Assertion `bp != ((void *)0)' failed.     Objects/obmalloc.c:953: PyObject_Free: Assertion `pool->ref.count > 0' failed       File "source/callbacks.c", line 206, in 'calling callback function'       File "/opt/couchbase/lib/python/couchstore.py", line 358, in callback         fn(DocumentInfo._fromStruct(docInfoPtr.contents, self))       File "/opt/couchbase/lib/python/couchstore.py", line 128, in _fromStruct         self = DocumentInfo(str(info.id))     TypeError: __str__ returned non-string (type buffer) backtraces in turn vary as well: #0 0x00002b8a11230e23 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0 #1 0x00002b8a112301cd in _PyObject_New () from /usr/lib64/libpython2.4.so.1.0 #2 0x00002b8a1630b950 in new_CArgObject () at source/callproc.c:288 #3 0x00002b8a16305a39 in PointerType_paramfunc (self=0x38) at source/_ctypes.c:564 #4 0x00002b8a1630b3de in ConvParam (obj=0x38, index=1, pa=0x2b8a1c157620) at source/callproc.c:477 #5 0x00002b8a1630bafd in _CallProc (pProc=0x2b8a163075b0 <string_at>, argtuple=0x1f152b48, flags=4097, argtypes=0x1ef82cb0, restype=0x1ef16790, checker=0x0)     at source/callproc.c:959 #6 0x00002b8a16306bf3 in CFuncPtr_call (self=0x1eebf710, inargs=0x2, kwds=0x0) at source/_ctypes.c:3362 #7 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #8 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #9 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #10 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #11 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #12 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #13 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #14 0x00002b8a1120b90f in ?? () from /usr/lib64/libpython2.4.so.1.0 #15 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #16 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #17 0x00002b8a112445e8 in ?? () from /usr/lib64/libpython2.4.so.1.0 #18 0x00002b8a1122fe45 in PyObject_Str () from /usr/lib64/libpython2.4.so.1.0 #19 0x00002b8a1123a617 in ?? () from /usr/lib64/libpython2.4.so.1.0 #20 0x00002b8a1123ff53 in ?? () from /usr/lib64/libpython2.4.so.1.0 #21 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #22 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #23 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #24 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #25 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #26 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #27 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #28 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #29 0x00002b8a1630aacc in _CallPythonObject (cif=<value optimized out>, resp=0x2b8a1c158af0, args=0x2b8a1c158960, userdata=<value optimized out>)     at source/callbacks.c:206 #30 closure_fcn (cif=<value optimized out>, resp=0x2b8a1c158af0, args=0x2b8a1c158960, userdata=<value optimized out>) at source/callbacks.c:252 #31 0x00002b8a1631054b in ffi_closure_unix64_inner (closure=0x2b8a1c15bab0, rvalue=0x2b8a1c158af0, reg_args=0x2b8a1c158a40, argp=0x2b8a1c158b10 "j\003")     at /tmp/ctypes/source/libffi/src/x86/ffi64.c:563 #32 0x00002b8a16310800 in ffi_closure_unix64 () at /tmp/ctypes/source/libffi/src/x86/unix64.S:228 #33 0x00002b8a1651dc3d in lookup_callback (rq=<value optimized out>, k=0x2b8a1c158bb0, v=<value optimized out>) at src/couch_db.c:623 #34 0x00002b8a1651b714 in btree_lookup_inner (rq=0x2b8a1c158d10, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78 #35 0x00002b8a1651b608 in btree_lookup_inner (rq=0x2b8a1c158d10, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52 #36 0x00002b8a1651c08c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>,     callback=<value optimized out>, ctx=<value optimized out>) at src/couch_db.c:667 #37 0x00002b8a163106d4 in ffi_call_unix64 () at /tmp/ctypes/source/libffi/src/x86/unix64.S:73 #38 0x00002b8a16310244 in ffi_call (cif=0x2b8a1c159090, fn=0x2b8a1651bf40 <couchstore_changes_since>, rvalue=0x2b8a1c158f90, avalue=0x2b8a1c158f50)     at /tmp/ctypes/source/libffi/src/x86/ffi64.c:428 #39 0x00002b8a1630bce1 in _call_function_pointer (pProc=0x2b8a1651bf40 <couchstore_changes_since>, argtuple=0x1f1a2ef0, flags=<value optimized out>, argtypes=0x0,     restype=0x1ef9cce0, checker=0x0) at source/callproc.c:668 #40 _CallProc (pProc=0x2b8a1651bf40 <couchstore_changes_since>, argtuple=0x1f1a2ef0, flags=<value optimized out>, argtypes=0x0, restype=0x1ef9cce0, checker=0x0)     at source/callproc.c:991 #41 0x00002b8a16306bf3 in CFuncPtr_call (self=0x1f073e90, inargs=0x1f1a2ef0, kwds=0x0) at source/_ctypes.c:3362 #42 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #43 0x00002b8a11262fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #44 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #45 0x00002b8a11264e2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #46 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #47 0x00002b8a1121bb9a in ?? () from /usr/lib64/libpython2.4.so.1.0 #48 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #49 0x00002b8a11263c1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #50 0x00002b8a11265256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #51 0x00002b8a112666d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #52 0x00002b8a1121baa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #53 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #54 0x00002b8a1120b90f in ?? () from /usr/lib64/libpython2.4.so.1.0 #55 0x00002b8a112057e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #56 0x00002b8a1126032d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #57 0x00002b8a1128c96d in ?? () from /usr/lib64/libpython2.4.so.1.0 #58 0x00002b8a1150a83d in start_thread () from /lib64/libpthread.so.0 #59 0x00002b8a11e7ffad in clone () from /lib64/libc.so.6 #0 0x00002b7dd51b0285 in raise () from /lib64/libc.so.6 #1 0x00002b7dd51b1d30 in abort () from /lib64/libc.so.6 #2 0x00002b7dd51a9716 in __assert_fail () from /lib64/libc.so.6 #3 0x00002b7dd4606204 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0 #4 0x00002b7dd45decbe in ?? () from /usr/lib64/libpython2.4.so.1.0 #5 0x00002b7dd45def36 in ?? () from /usr/lib64/libpython2.4.so.1.0 #6 0x00002b7dd4614f53 in ?? () from /usr/lib64/libpython2.4.so.1.0 #7 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #8 0x00002b7dd4637fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #9 0x00002b7dd463b6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #10 0x00002b7dd45f0b9a in ?? () from /usr/lib64/libpython2.4.so.1.0 #11 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #12 0x00002b7dd4638c1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #13 0x00002b7dd463a256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #14 0x00002b7dd463b6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #15 0x00002b7dd45f0aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #16 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #17 0x00002b7dd45e090f in ?? () from /usr/lib64/libpython2.4.so.1.0 #18 0x00002b7dd45da7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #19 0x00002b7dd463532d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #20 0x00002b7dd466196d in ?? () from /usr/lib64/libpython2.4.so.1.0 #21 0x00002b7dd48df83d in start_thread () from /lib64/libpthread.so.0 #22 0x00002b7dd5254fad in clone () from /lib64/libc.so.6 #0 0x00002b00a0fa2285 in raise () from /lib64/libc.so.6 #1 0x00002b00a0fa3d30 in abort () from /lib64/libc.so.6 #2 0x00002b00a0f9b716 in __assert_fail () from /lib64/libc.so.6 #3 0x00002b00a03f8204 in PyObject_Malloc () from /usr/lib64/libpython2.4.so.1.0 #4 0x00002b00a040139f in PyString_FromString () from /usr/lib64/libpython2.4.so.1.0 #5 0x00002b00a04014d9 in PyString_InternFromString () from /usr/lib64/libpython2.4.so.1.0 #6 0x00002b00a03f58b6 in PyObject_GetAttrString () from /usr/lib64/libpython2.4.so.1.0 #7 0x00002b00a54d1b62 in ConvParam (obj=0x126d5510, index=1, pa=0x2b00ab31ec90) at source/callproc.c:562 #8 0x00002b00a54d1f7a in _CallProc (pProc=0x2b00a56e3300 <couchstore_open_doc_with_docinfo>, argtuple=0x124ec680, flags=4097, argtypes=0x0, restype=0x1231a980,     checker=0x0) at source/callproc.c:966 #9 0x00002b00a54ccd0a in CFuncPtr_call (self=<value optimized out>, inargs=<value optimized out>, kwds=0x0) at source/_ctypes.c:3362 #10 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #11 0x00002b00a0429fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #12 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #13 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #14 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #15 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #16 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #17 0x00002b00a03e2aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #18 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #19 0x00002b00a042732d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #20 0x00002b00a54d0ef3 in _CallPythonObject (cif=<value optimized out>, resp=0x2b00ab31fac0, args=<value optimized out>, userdata=<value optimized out>)     at source/callbacks.c:206 #21 closure_fcn (cif=<value optimized out>, resp=0x2b00ab31fac0, args=<value optimized out>, userdata=<value optimized out>) at source/callbacks.c:252 #22 0x00002b00a54d6639 in ffi_closure_unix64_inner (closure=0x2b00ab322e10, rvalue=0x2b00ab31fac0, reg_args=0x2b00ab31fa10, argp=0x2b00ab31fae0 "\021\005")     at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/ffi64.c:563 #23 0x00002b00a54d6fd4 in ffi_closure_unix64 () at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/unix64.S:228 #24 0x00002b00a56e4c3d in lookup_callback (rq=<value optimized out>, k=0x2b00ab31fb80, v=<value optimized out>) at src/couch_db.c:623 #25 0x00002b00a56e2714 in btree_lookup_inner (rq=0x2b00ab31fce0, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:78 #26 0x00002b00a56e2608 in btree_lookup_inner (rq=0x2b00ab31fce0, diskpos=<value optimized out>, current=0, end=1) at src/btree_read.c:52 #27 0x00002b00a56e308c in couchstore_changes_since (db=<value optimized out>, since=<value optimized out>, options=<value optimized out>,     callback=<value optimized out>, ctx=<value optimized out>) at src/couch_db.c:667 #28 0x00002b00a54d6ea8 in ffi_call_unix64 () at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/unix64.S:73 #29 0x00002b00a54d6c35 in ffi_call (cif=0x2b00ab320080, fn=0x2b00a56e2f40 <couchstore_changes_since>, rvalue=<value optimized out>, avalue=<value optimized out>)     at /home/buildbot/centos-x64-201-builder/build/build/ctypes/source/libffi/src/x86/ffi64.c:428 #30 0x00002b00a54d215d in _call_function_pointer (pProc=0x2b00a56e2f40 <couchstore_changes_since>, argtuple=0x124ba7d0, flags=4097, argtypes=0x0, restype=0x1231a980,     checker=0x0) at source/callproc.c:668 #31 _CallProc (pProc=0x2b00a56e2f40 <couchstore_changes_since>, argtuple=0x124ba7d0, flags=4097, argtypes=0x0, restype=0x1231a980, checker=0x0)     at source/callproc.c:991 #32 0x00002b00a54ccd0a in CFuncPtr_call (self=<value optimized out>, inargs=<value optimized out>, kwds=0x0) at source/_ctypes.c:3362 #33 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #34 0x00002b00a0429fbe in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #35 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #36 0x00002b00a042be2f in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #37 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #38 0x00002b00a03e2b9a in ?? () from /usr/lib64/libpython2.4.so.1.0 #39 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #40 0x00002b00a042ac1c in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #41 0x00002b00a042c256 in PyEval_EvalFrame () from /usr/lib64/libpython2.4.so.1.0 #42 0x00002b00a042d6d5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.4.so.1.0 #43 0x00002b00a03e2aa7 in ?? () from /usr/lib64/libpython2.4.so.1.0 #44 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #45 0x00002b00a03d290f in ?? () from /usr/lib64/libpython2.4.so.1.0 #46 0x00002b00a03cc7e0 in PyObject_Call () from /usr/lib64/libpython2.4.so.1.0 #47 0x00002b00a042732d in PyEval_CallObjectWithKeywords () from /usr/lib64/libpython2.4.so.1.0 #48 0x00002b00a045396d in ?? () from /usr/lib64/libpython2.4.so.1.0 #49 0x00002b00a06d183d in start_thread () from /lib64/libpthread.so.0 #50 0x00002b00a1046fad in clone () from /lib64/libc.so.6 3. Installing Python 2.6 on this machine fixed the problem. However I'd not blame Python 2.4 - as I mentioned before absolutely identical setup worked pretty well on other machine. 4. ctypes that we ship is another suspect. That was a workaround, not necessarily it addresses all edge cases. And couchstore + Python 2.4 is one of them, in fact all other sources work fine.
Hide
Permalink
Pavel Paulau added a comment -
Deep,

May I ask you reproduce it on any other machine? This is very confusing issue...
Show
Pavel Paulau added a comment - Deep, May I ask you reproduce it on any other machine? This is very confusing issue...
Hide
Permalink
Deepkaran Salooja added a comment -
Reproduced on VM 10.3.3.104. 100k items loaded with mcsoda on default bucket. Build 2.0.2-781-rel. Crash is reproducible.

[root@caper-016 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup
2013-05-06 10:41:23,838: mt cbtransfer...
2013-05-06 10:41:23,838: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/
2013-05-06 10:41:23,839: mt sink : /tmp/backup
2013-05-06 10:41:23,839: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'bucket_destination': None, 'destination_operation': None, 'vbucket_list': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': None}
2013-05-06 10:41:23,840: mt source_class: <class 'pump_sfd.SFDSource'>
2013-05-06 10:41:24,094: mt sink_class: <class 'pump_bfd.BFDSink'>
2013-05-06 10:41:24,095: mt source_buckets: default
2013-05-06 10:41:24,095: mt bucket: default
2013-05-06 10:41:24,096: mt source_nodes: N/A
2013-05-06 10:41:24,108: mt enqueueing node: N/A
2013-05-06 10:41:24,108: w0 node: N/A
2013-05-06 10:41:24,217: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
2013-05-06 10:41:24,217: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb
...Segmentation fault
Show
Deepkaran Salooja added a comment - Reproduced on VM 10.3.3.104. 100k items loaded with mcsoda on default bucket. Build 2.0.2-781-rel. Crash is reproducible. [ root@caper-016 ~]# /opt/couchbase/bin/cbtransfer -v -v -v couchstore-files:///opt/couchbase/var/lib/couchbase/data/ /tmp/backup 2013-05-06 10:41:23,838: mt cbtransfer... 2013-05-06 10:41:23,838: mt source : couchstore-files:///opt/couchbase/var/lib/couchbase/data/ 2013-05-06 10:41:23,839: mt sink : /tmp/backup 2013-05-06 10:41:23,839: mt opts : {'username': None, 'source_vbucket_state': 'active', 'destination_vbucket_state': 'active', 'verbose': 3, 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': 0.0, 'nmv_retry': 1.0, 'cbb_max_mb': 100000.0, 'try_xwm': 1.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, 'batch_max_size': 1000.0, 'report': 5.0, 'design_doc_only': 0.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'bucket_destination': None, 'destination_operation': None, 'vbucket_list': None, 'threads': 4, 'key': None, 'password': None, 'id': None, 'bucket_source': None} 2013-05-06 10:41:23,840: mt source_class: <class 'pump_sfd.SFDSource'> 2013-05-06 10:41:24,094: mt sink_class: <class 'pump_bfd.BFDSink'> 2013-05-06 10:41:24,095: mt source_buckets: default 2013-05-06 10:41:24,095: mt bucket: default 2013-05-06 10:41:24,096: mt source_nodes: N/A 2013-05-06 10:41:24,108: mt enqueueing node: N/A 2013-05-06 10:41:24,108: w0 node: N/A 2013-05-06 10:41:24,217: s0 create_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb 2013-05-06 10:41:24,217: s0 connect_db: /tmp/backup/bucket-default/node-N%2FA/data-0000.cbb ...Segmentation fault
Hide
Permalink
Pavel Paulau added a comment -
In fact 10.3.3.104 is a virtual copy of 10.3.3.95.
Show
Pavel Paulau added a comment - In fact 10.3.3.104 is a virtual copy of 10.3.3.95.
Hide
Permalink
Pavel Paulau added a comment -
http://review.couchbase.org/#/c/26260/
Show
Pavel Paulau added a comment - http://review.couchbase.org/#/c/26260/
Hide
Permalink
Maria McDuff added a comment -
pls verify / close.
Show
Maria McDuff added a comment - pls verify / close.
Hide
Permalink
Pavel Paulau added a comment -
Actually it wasn't resolved. We still ship old version of ctypes, without direct access to builders I can't fix that.
Show
Pavel Paulau added a comment - Actually it wasn't resolved. We still ship old version of ctypes, without direct access to builders I can't fix that.
Hide
Permalink
Phil Labee added a comment -
tlm commit af7b66f4c81b890adfcb8b7520a89436f2e4d0cd adds a 'clean-all' target and a 'clean-grommit' target

The 'clean-grommit' target will remove directories named:

    ctypes* curl* google-perftools* gperftools* libevent* sqlite*

Buildbot master.cfg now uses the 'clean all' target for "make clean".
Show
Phil Labee added a comment - tlm commit af7b66f4c81b890adfcb8b7520a89436f2e4d0cd adds a 'clean-all' target and a 'clean-grommit' target The 'clean-grommit' target will remove directories named:     ctypes* curl* google-perftools* gperftools* libevent* sqlite* Buildbot master.cfg now uses the 'clean all' target for "make clean".
Hide
Permalink
Maria McDuff added a comment -
Phil,

Is this fixed? can you confirm.
pavel's last comment states it's not resolved....
Show
Maria McDuff added a comment - Phil, Is this fixed? can you confirm. pavel's last comment states it's not resolved....
Hide
Permalink
Pavel Paulau added a comment -
It must be fixed. At least build 805 includes required version of ctypes.
Show
Pavel Paulau added a comment - It must be fixed. At least build 805 includes required version of ctypes.
Hide
Permalink
Deepkaran Salooja added a comment -
verified with 2.0.2-807-rel
Show
Deepkaran Salooja added a comment - verified with 2.0.2-807-rel

People

  • Assignee:
    Deepkaran Salooja
    Reporter:
    Deepkaran Salooja
Vote (0)
Watch (8)

Dates

  • Created:
    Updated:
    Resolved:

Agile