Couchbase Server

memcached crash when i restart the set/get load on a 2 node after successful rebalance ( 1.2x dgm 1M items)

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: None
  • Security Level: Public
  • Labels:
    None

Description

backtrace :

Thread 12 (Thread 0x7f59e0147700 (LWP 4126)):
#0 0x00007f59df5a4937 in HashTable::getLockedBucket (this=0x7f59e0145ea0,
    s=..., bucket=0x7f59e0145ea0) at stored-value.hh:1381
#1 0x00007f59df596212 in EventuallyPersistentStore::get (this=0x2539000,
    key=..., vbucket=222, cookie=0x29b0c8c8, queueBG=false, honorStates=false)
    at ep.cc:1275
#2 0x00007f59df5dc303 in EventuallyPersistentEngine::doWalkTapQueue (
    this=0x2508000, cookie=0x29b0c8c8, itm=0x7f59e0146528, es=0x7f59e0146520,
    nes=0x7f59e014653c, ttl=0x7f59e014653f "\377", flags=0x7f59e014653a,
    seqno=0x7f59e0146534, vbucket=0x7f59e0146538, connection=0x1900bc00,
    retry=@0x7f59e0146381) at ep_engine.cc:1603
#3 0x00007f59df5bf8c3 in EventuallyPersistentEngine::walkTapQueue (
    this=0x2508000, cookie=0x29b0c8c8, itm=0x7f59e0146528, es=0x7f59e0146520,
    nes=0x7f59e014653c, ttl=0x7f59e014653f "\377", flags=0x7f59e014653a,
    seqno=0x7f59e0146534, vbucket=0x7f59e0146538) at ep_engine.cc:1762
#4 0x00007f59df5c1f90 in EvpTapIterator (handle=0x2508000, cookie=0x29b0c8c8,
    itm=0x7f59e0146528, es=0x7f59e0146520, nes=0x7f59e014653c,
    ttl=0x7f59e014653f "\377", flags=0x7f59e014653a, seqno=0x7f59e0146534,
    vbucket=0x7f59e0146538) at ep_engine.cc:969
#5 0x00007f59e2150660 in bucket_tap_iterator_shim (handle=0x7f59e2354ba0,
    cookie=0x29b0c8c8, itm=0x7f59e0146528, engine_specific=0x7f59e0146520,
    nengine_specific=0x7f59e014653c, ttl=0x7f59e014653f "\377",
---Type <return> to continue, or q <return> to quit---
    flags=0x7f59e014653a, seqno=0x7f59e0146534, vbucket=0x7f59e0146538)
    at bucket_engine.c:1941
#6 0x000000000040fad2 in ship_tap_log (c=0x29b0c8c8)
    at daemon/memcached.c:2378
#7 conn_ship_log (c=0x29b0c8c8) at daemon/memcached.c:5166
#8 0x0000000000407556 in event_handler (fd=<value optimized out>,
    which=<value optimized out>, arg=0x29b0c8c8) at daemon/memcached.c:5620
#9 0x00007f59e48efdf9 in event_process_active_single_queue (base=0x2430780,
    flags=0) at event.c:1308
#10 event_process_active (base=0x2430780, flags=0) at event.c:1375
#11 event_base_loop (base=0x2430780, flags=0) at event.c:1572
#12 0x00000000004134a4 in worker_libevent (arg=0x2412900)
    at daemon/thread.c:304
#13 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#14 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f59e214b700 (LWP 4122)):
#0 0x00007f59e3b564a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f59e4900c28 in epoll_dispatch (base=0x2430500,
    tv=<value optimized out>) at epoll.c:404
#2 0x00007f59e48efa4c in event_base_loop (base=0x2430500, flags=0)
    at event.c:1558
#3 0x00000000004134a4 in worker_libevent (arg=0x2412500)
---Type <return> to continue, or q <return> to quit---
    at daemon/thread.c:304
#4 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f59e2b55700 (LWP 4121)):
#0 0x00007f59e3b1b46d in nanosleep () from /lib64/libc.so.6
#1 0x00007f59e3b1b2e0 in sleep () from /lib64/libc.so.6
#2 0x0000000000415068 in check_isasl_db_thread (arg=<value optimized out>)
    at daemon/isasl.c:233
#3 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f59e3356700 (LWP 4120)):
#0 0x00007f59e3b48b5d in read () from /lib64/libc.so.6
#1 0x00007f59e3ae6228 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x00007f59e3ae7d2e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x00007f59e3ae324b in getc () from /lib64/libc.so.6
#4 0x00007f59e3357875 in check_stdin_thread (arg=0x403610)
    at extensions/daemon/stdin_check.c:19
#5 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f59e0948700 (LWP 4125)):
---Type <return> to continue, or q <return> to quit---
#0 0x00007f59e3b564a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f59e4900c28 in epoll_dispatch (base=0x2430a00,
    tv=<value optimized out>) at epoll.c:404
#2 0x00007f59e48efa4c in event_base_loop (base=0x2430a00, flags=0)
    at event.c:1558
#3 0x00000000004134a4 in worker_libevent (arg=0x2412800)
    at daemon/thread.c:304
#4 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f59e4f35720 (LWP 4111)):
#0 0x00007f59e3b564a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f59e4900c28 in epoll_dispatch (base=0x2430000,
    tv=<value optimized out>) at epoll.c:404
#2 0x00007f59e48efa4c in event_base_loop (base=0x2430000, flags=0)
    at event.c:1558
#3 0x000000000040930b in main (argc=15, argv=0x7fff81476b88)
    at daemon/memcached.c:7531

Thread 6 (Thread 0x7f59e1149700 (LWP 4124)):
#0 0x00007f59e3df590c in recv () from /lib64/libpthread.so.0
#1 0x000000000040dd70 in try_read_network (c=0x216b12c8)
    at daemon/memcached.c:4923
---Type <return> to continue, or q <return> to quit---
#2 conn_read (c=0x216b12c8) at daemon/memcached.c:5201
#3 0x0000000000407556 in event_handler (fd=<value optimized out>,
    which=<value optimized out>, arg=0x216b12c8) at daemon/memcached.c:5620
#4 0x00007f59e48efdf9 in event_process_active_single_queue (base=0x2430c80,
    flags=0) at event.c:1308
#5 event_process_active (base=0x2430c80, flags=0) at event.c:1375
#6 event_base_loop (base=0x2430c80, flags=0) at event.c:1572
#7 0x00000000004134a4 in worker_libevent (arg=0x2412700)
    at daemon/thread.c:304
#8 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f59dda64700 (LWP 4165)):
#0 0x00007f59e3df27a9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00007f59df58be5c in SyncObject::wait (this=0x2432200, tv=...)
    at syncobject.hh:42
#2 0x00007f59df5d5a9c in SyncObject::wait (this=0x2432200, secs=1)
    at syncobject.hh:58
#3 0x00007f59df5d5ad7 in TapConnMap::wait (this=0x2432200, howlong=1)
    at tapconnmap.hh:209
#4 0x00007f59df5b904f in EventuallyPersistentEngine::notifyTapIoThread (
    this=0x2508000) at ep_engine.cc:3393
---Type <return> to continue, or q <return> to quit---
#5 0x00007f59df5c2224 in EvpNotifyTapIo (arg=0x2508000) at ep_engine.cc:1037
#6 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f59df267700 (LWP 4162)):
#0 0x00007f59e3b4c7a3 in poll () from /lib64/libc.so.6
#1 0x00007f59df628700 in MemcachedEngine::waitForReadable (this=0x2427100)
    at mc-kvstore/mc-engine.cc:999
#2 0x00007f59df628a9a in MemcachedEngine::wait (this=0x2427100)
    at mc-kvstore/mc-engine.cc:1026
#3 0x00007f59df6295b6 in MemcachedEngine::tap (this=0x2427100, vbids=...,
    full=true, cb=...) at mc-kvstore/mc-engine.cc:1324

#4 0x00007f59df62e4f6 in MCKVStore::dump (this=0x2411b80, vb=176, cb=...)
    at mc-kvstore/mc-kvstore.cc:156

#5 0x00007f59df573544 in BackfillDiskLoad::callback (this=0x37d66e40, d=...,
    t=...) at backfill.cc:56
#6 0x00007f59df58a8ee in Task::run (this=0x33043800, d=..., t=...)
    at dispatcher.hh:139
#7 0x00007f59df58949c in Dispatcher::run (this=0x241b000) at dispatcher.cc:119
#8 0x00007f59df5899fc in launch_dispatcher_thread (arg=0x241b000)
    at dispatcher.cc:28
#9 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f59e3b55ead in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---

Thread 3 (Thread 0x7f59de265700 (LWP 4164)):
#0 0x00007f59e3df27a9 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00007f59df58be5c in SyncObject::wait (this=0x2552e10, tv=...)
    at syncobject.hh:42
#2 0x00007f59df588397 in IdleTask::run (this=0x2444100, d=...)
    at dispatcher.cc:244
#3 0x00007f59df58949c in Dispatcher::run (this=0x2552e00) at dispatcher.cc:119
#4 0x00007f59df5899fc in launch_dispatcher_thread (arg=0x2552e00)
    at dispatcher.cc:28
#5 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f59e194a700 (LWP 4123)):
#0 0x00007f59e3b564a3 in epoll_wait () from /lib64/libc.so.6
#1 0x00007f59e4900c28 in epoll_dispatch (base=0x2430280,
    tv=<value optimized out>) at epoll.c:404
#2 0x00007f59e48efa4c in event_base_loop (base=0x2430280, flags=0)
    at event.c:1558
#3 0x00000000004134a4 in worker_libevent (arg=0x2412600)
    at daemon/thread.c:304
#4 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f59dea66700 (LWP 4163)):
#0 0x00007f59e3aa76b5 in raise () from /lib64/libc.so.6
#1 0x00007f59e3aa8e95 in abort () from /lib64/libc.so.6
#2 0x00007f59e3aa0645 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f59df62e096 in MCKVStore::commit (this=0x24115c0)
    at mc-kvstore/mc-kvstore.cc:187
#4 0x00007f59df58e23e in TransactionContext::commit (this=0x25391c8)
    at ep.cc:2631
#5 0x00007f59df592ef8 in EventuallyPersistentStore::flushSome (
    this=0x2539000, q=0x2539130, rejectQueue=0x5902910) at ep.cc:1936
#6 0x00007f59df5e1f94 in Flusher::doFlush (this=0x2505240) at flusher.cc:325
#7 0x00007f59df5e2626 in Flusher::step (this=0x2505240, d=..., tid=...)
    at flusher.cc:239
#8 0x00007f59df5e2abf in FlusherStepper::callback (this=0x24103b0, d=...,
    t=...) at flusher.cc:23
#9 0x00007f59df58a8ee in Task::run (this=0x2444300, d=..., t=...)
    at dispatcher.hh:139
#10 0x00007f59df58949c in Dispatcher::run (this=0x241b1c0) at dispatcher.cc:119
#11 0x00007f59df5899fc in launch_dispatcher_thread (arg=0x241b1c0)
    at dispatcher.cc:28
#12 0x00007f59e3dee7e1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#13 0x00007f59e3b55ead in clone () from /lib64/libc.so.6

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
memcached stats on that node

[root@domU-12-31-39-15-19-AC ~]# /opt/couchbase/bin/mbstats localhost:11210 all
 accepting_conns: 1
 auth_cmds: 2
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 6
 bytes_read: 471455298
 bytes_written: 7233470
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 0
 cmd_set: 0
 conn_yields: 12133
 connection_structures: 15
 curr_connections: 15
 curr_items: 788262
 curr_items_tot: 1355561
 daemon_connections: 10
 decr_hits: 0
 decr_misses: 0
 delete_hits: 0
 delete_misses: 0
 ep_allow_data_loss_during_shutdown: 1
 ep_backend: couchdb
 ep_bg_fetch_delay: 0
 ep_bg_fetched: 0
 ep_cache_size: 0
 ep_chk_max_items: 30000
 ep_chk_period: 3600
 ep_chk_remover_stime: 5
 ep_commit_num: 0
 ep_commit_time: 0
 ep_commit_time_total: 0
 ep_concurrentDB: 1
 ep_config_file:
 ep_couch_bucket: default
 ep_couch_default_batch_size: 500
 ep_couch_host: localhost
 ep_couch_port: 11213
 ep_couch_reconnect_sleeptime: 250
 ep_couch_vbucket_batch_count: 4
 ep_data_age: 0
 ep_data_age_highwat: 0
 ep_db_cleaner_status: running
 ep_db_shards: 4
 ep_db_strategy: multiDB
 ep_dbinit: 0
 ep_dbname: /tmp/test.db
 ep_degraded_mode: 0
 ep_diskqueue_drain: 0
 ep_diskqueue_fill: 428518
 ep_diskqueue_items: 428518
 ep_diskqueue_memory: 47994016
 ep_diskqueue_pending: 497913960
 ep_exp_pager_stime: 3600
 ep_expired: 0
 ep_expiry_window: 3
 ep_failpartialwarmup: 0
 ep_flush_all: false
 ep_flush_duration: 0
 ep_flush_duration_highwat: 0
 ep_flush_duration_total: 0
 ep_flush_preempts: 0
 ep_flusher_state: loading data
 ep_flusher_todo: 0
 ep_getl_default_timeout: 15
 ep_getl_max_timeout: 30
 ep_ht_locks: 5
 ep_ht_size: 3079
 ep_inconsistent_slave_chk: 0
 ep_initfile:
 ep_io_num_read: 0
 ep_io_num_write: 0
 ep_io_read_bytes: 0
 ep_io_write_bytes: 0
 ep_item_begin_failed: 0
 ep_item_commit_failed: 0
 ep_item_flush_expired: 0
 ep_item_flush_failed: 0
 ep_item_num_based_new_chk: 0
 ep_items_rm_from_checkpoints: 0
 ep_kv_size: 746748643
 ep_latency_arith_cmd: 0
 ep_latency_get_cmd: 0
 ep_latency_store_cmd: 0
 ep_max_checkpoints: 2
 ep_max_data_size: 1887436800
 ep_max_item_size: 20971520
 ep_max_size: 1887436800
 ep_max_txn_size: 10000
 ep_max_vbuckets: 1024
 ep_mem_high_wat: 1415577600
 ep_mem_low_wat: 1132462080
 ep_min_data_age: 0
 ep_mutation_mem_threshold: 0
 ep_num_active_non_resident: 788262
 ep_num_checkpoint_remover_runs: 51
 ep_num_eject_failures: 0
 ep_num_eject_replicas: 0
 ep_num_expiry_pager_runs: 0
 ep_num_non_resident: 788262
 ep_num_not_my_vbuckets: 0
 ep_num_pager_runs: 0
 ep_num_value_ejects: 1576405
 ep_obs_reg_clean_job: 5
 ep_observe_calls: 0
 ep_observe_errors: 0
 ep_observe_registry_size: 0
 ep_onlineupdate: false
 ep_onlineupdate_revert_add: 0
 ep_onlineupdate_revert_delete: 0
 ep_onlineupdate_revert_update: 0
 ep_oom_errors: 0
 ep_overhead: 71952846
 ep_pending_ops: 0
 ep_pending_ops_max: 0
 ep_pending_ops_max_duration: 0
 ep_pending_ops_total: 0
 ep_postInitfile:
 ep_queue_age_cap: 900
 ep_queue_size: 428518
 ep_restore_mode: 0
 ep_shardpattern: %d/%b-%i.sqlite
 ep_stats_observe_polls: 0
 ep_storage_age: 0
 ep_storage_age_highwat: 0
 ep_storage_type: featured
 ep_store_max_concurrency: 10
 ep_store_max_readers: 9
 ep_store_max_readwrite: 1
 ep_stored_val_type:
 ep_sync_cmd_timeout: 2500
 ep_sync_on_persist: 1
 ep_tap_ack_grace_period: 300
 ep_tap_ack_initial_sequence_number: 1
 ep_tap_ack_interval: 1000
 ep_tap_ack_window_size: 10
 ep_tap_backfill_resident: 0.9
 ep_tap_backlog_limit: 5000
 ep_tap_backoff_period: 5
 ep_tap_bg_fetch_requeued: 0
 ep_tap_bg_fetched: 0
 ep_tap_bg_max_pending: 500
 ep_tap_conn_map_notifications: 0
 ep_tap_keepalive: 300
 ep_tap_noop_interval: 20
 ep_tap_requeue_sleep_time: 0.1
 ep_tmp_oom_errors: 0
 ep_too_old: 0
 ep_too_young: 0
 ep_total_cache_size: 730411863
 ep_total_del_items: 0
 ep_total_enqueued: 428518
 ep_total_new_items: 0
 ep_total_observe_sets: 0
 ep_total_persisted: 0
 ep_uncommitted_items: 0
 ep_unobserve_calls: 0
 ep_value_size: 601419692
 ep_vb0: 0
 ep_vb_chunk_del_time: 500
 ep_vb_del_chunk_size: 100
 ep_vb_total: 256
 ep_vbucket_del: 0
 ep_vbucket_del_fail: 0
 ep_version: 1.7.1.1_205_ge1a9877
 ep_waitforwarmup: 0
 ep_warmed_up: 160405
 ep_warmup: 1
 ep_warmup_dups: 12343
 ep_warmup_keys_time: 187874018
 ep_warmup_oom: 0
 ep_warmup_thread: complete
 ep_warmup_time: 0
 get_hits: 0
 get_misses: 0
 incr_hits: 0
 incr_misses: 0
 libevent: 2.0.11-stable
 limit_maxbytes: 67108864
 listen_disabled_num: 0
 mem_used: 818701489
 pid: 4425
 pointer_size: 64
 rejected_conns: 0
 rusage_system: 69.768360
 rusage_user: 39.990499
 tap_checkpoint_start_sent: 128
 tap_connect_received: 1
 tap_mutation_received: 428586
 tap_opaque_received: 130
 tap_opaque_sent: 2
 threads: 4
 time: 1320626663
 total_connections: 15
 uptime: 271
 vb_active_curr_items: 788262
 vb_active_eject: 788262
 vb_active_ht_memory: 6346752
 vb_active_itm_memory: 87496538
 vb_active_num: 128
 vb_active_num_non_resident: 788262
 vb_active_ops_create: 0
 vb_active_ops_delete: 0
 vb_active_ops_reject: 0
 vb_active_ops_update: 0
 vb_active_perc_mem_resident: 0
 vb_active_queue_age: 0
 vb_active_queue_drain: 0
 vb_active_queue_fill: 0
 vb_active_queue_memory: 0
 vb_active_queue_pending: 0
 vb_active_queue_size: 0
 vb_dead_num: 0
 vb_pending_curr_items: 0
 vb_pending_eject: 0
 vb_pending_ht_memory: 0
 vb_pending_itm_memory: 0
 vb_pending_num: 0
 vb_pending_num_non_resident: 0
 vb_pending_ops_create: 0
 vb_pending_ops_delete: 0
 vb_pending_ops_reject: 0
 vb_pending_ops_update: 0
 vb_pending_perc_mem_resident: 0
 vb_pending_queue_age: 0
 vb_pending_queue_drain: 0
 vb_pending_queue_fill: 0
 vb_pending_queue_memory: 0
 vb_pending_queue_pending: 0
 vb_pending_queue_size: 0
 vb_replica_curr_items: 567299
 vb_replica_eject: 788143
 vb_replica_ht_memory: 5219104
 vb_replica_itm_memory: 642915325
 vb_replica_num: 128
 vb_replica_num_non_resident: 0
 vb_replica_ops_create: 0
 vb_replica_ops_delete: 0
 vb_replica_ops_reject: 0
 vb_replica_ops_update: 0
 vb_replica_perc_mem_resident: 100
 vb_replica_queue_age: 16055352000
 vb_replica_queue_drain: 0
 vb_replica_queue_fill: 428518
 vb_replica_queue_memory: 47994016
 vb_replica_queue_pending: 497913960
 vb_replica_queue_size: 428518
Show
Farshid Ghods added a comment - memcached stats on that node [ root@domU-12-31-39-15-19-AC ~]# /opt/couchbase/bin/mbstats localhost:11210 all  accepting_conns: 1  auth_cmds: 2  auth_errors: 0  bucket_active_conns: 1  bucket_conns: 6  bytes_read: 471455298  bytes_written: 7233470  cas_badval: 0  cas_hits: 0  cas_misses: 0  cmd_flush: 0  cmd_get: 0  cmd_set: 0  conn_yields: 12133  connection_structures: 15  curr_connections: 15  curr_items: 788262  curr_items_tot: 1355561  daemon_connections: 10  decr_hits: 0  decr_misses: 0  delete_hits: 0  delete_misses: 0  ep_allow_data_loss_during_shutdown: 1  ep_backend: couchdb  ep_bg_fetch_delay: 0  ep_bg_fetched: 0  ep_cache_size: 0  ep_chk_max_items: 30000  ep_chk_period: 3600  ep_chk_remover_stime: 5  ep_commit_num: 0  ep_commit_time: 0  ep_commit_time_total: 0  ep_concurrentDB: 1  ep_config_file:  ep_couch_bucket: default  ep_couch_default_batch_size: 500  ep_couch_host: localhost  ep_couch_port: 11213  ep_couch_reconnect_sleeptime: 250  ep_couch_vbucket_batch_count: 4  ep_data_age: 0  ep_data_age_highwat: 0  ep_db_cleaner_status: running  ep_db_shards: 4  ep_db_strategy: multiDB  ep_dbinit: 0  ep_dbname: /tmp/test.db  ep_degraded_mode: 0  ep_diskqueue_drain: 0  ep_diskqueue_fill: 428518  ep_diskqueue_items: 428518  ep_diskqueue_memory: 47994016  ep_diskqueue_pending: 497913960  ep_exp_pager_stime: 3600  ep_expired: 0  ep_expiry_window: 3  ep_failpartialwarmup: 0  ep_flush_all: false  ep_flush_duration: 0  ep_flush_duration_highwat: 0  ep_flush_duration_total: 0  ep_flush_preempts: 0  ep_flusher_state: loading data  ep_flusher_todo: 0  ep_getl_default_timeout: 15  ep_getl_max_timeout: 30  ep_ht_locks: 5  ep_ht_size: 3079  ep_inconsistent_slave_chk: 0  ep_initfile:  ep_io_num_read: 0  ep_io_num_write: 0  ep_io_read_bytes: 0  ep_io_write_bytes: 0  ep_item_begin_failed: 0  ep_item_commit_failed: 0  ep_item_flush_expired: 0  ep_item_flush_failed: 0  ep_item_num_based_new_chk: 0  ep_items_rm_from_checkpoints: 0  ep_kv_size: 746748643  ep_latency_arith_cmd: 0  ep_latency_get_cmd: 0  ep_latency_store_cmd: 0  ep_max_checkpoints: 2  ep_max_data_size: 1887436800  ep_max_item_size: 20971520  ep_max_size: 1887436800  ep_max_txn_size: 10000  ep_max_vbuckets: 1024  ep_mem_high_wat: 1415577600  ep_mem_low_wat: 1132462080  ep_min_data_age: 0  ep_mutation_mem_threshold: 0  ep_num_active_non_resident: 788262  ep_num_checkpoint_remover_runs: 51  ep_num_eject_failures: 0  ep_num_eject_replicas: 0  ep_num_expiry_pager_runs: 0  ep_num_non_resident: 788262  ep_num_not_my_vbuckets: 0  ep_num_pager_runs: 0  ep_num_value_ejects: 1576405  ep_obs_reg_clean_job: 5  ep_observe_calls: 0  ep_observe_errors: 0  ep_observe_registry_size: 0  ep_onlineupdate: false  ep_onlineupdate_revert_add: 0  ep_onlineupdate_revert_delete: 0  ep_onlineupdate_revert_update: 0  ep_oom_errors: 0  ep_overhead: 71952846  ep_pending_ops: 0  ep_pending_ops_max: 0  ep_pending_ops_max_duration: 0  ep_pending_ops_total: 0  ep_postInitfile:  ep_queue_age_cap: 900  ep_queue_size: 428518  ep_restore_mode: 0  ep_shardpattern: %d/%b-%i.sqlite  ep_stats_observe_polls: 0  ep_storage_age: 0  ep_storage_age_highwat: 0  ep_storage_type: featured  ep_store_max_concurrency: 10  ep_store_max_readers: 9  ep_store_max_readwrite: 1  ep_stored_val_type:  ep_sync_cmd_timeout: 2500  ep_sync_on_persist: 1  ep_tap_ack_grace_period: 300  ep_tap_ack_initial_sequence_number: 1  ep_tap_ack_interval: 1000  ep_tap_ack_window_size: 10  ep_tap_backfill_resident: 0.9  ep_tap_backlog_limit: 5000  ep_tap_backoff_period: 5  ep_tap_bg_fetch_requeued: 0  ep_tap_bg_fetched: 0  ep_tap_bg_max_pending: 500  ep_tap_conn_map_notifications: 0  ep_tap_keepalive: 300  ep_tap_noop_interval: 20  ep_tap_requeue_sleep_time: 0.1  ep_tmp_oom_errors: 0  ep_too_old: 0  ep_too_young: 0  ep_total_cache_size: 730411863  ep_total_del_items: 0  ep_total_enqueued: 428518  ep_total_new_items: 0  ep_total_observe_sets: 0  ep_total_persisted: 0  ep_uncommitted_items: 0  ep_unobserve_calls: 0  ep_value_size: 601419692  ep_vb0: 0  ep_vb_chunk_del_time: 500  ep_vb_del_chunk_size: 100  ep_vb_total: 256  ep_vbucket_del: 0  ep_vbucket_del_fail: 0  ep_version: 1.7.1.1_205_ge1a9877  ep_waitforwarmup: 0  ep_warmed_up: 160405  ep_warmup: 1  ep_warmup_dups: 12343  ep_warmup_keys_time: 187874018  ep_warmup_oom: 0  ep_warmup_thread: complete  ep_warmup_time: 0  get_hits: 0  get_misses: 0  incr_hits: 0  incr_misses: 0  libevent: 2.0.11-stable  limit_maxbytes: 67108864  listen_disabled_num: 0  mem_used: 818701489  pid: 4425  pointer_size: 64  rejected_conns: 0  rusage_system: 69.768360  rusage_user: 39.990499  tap_checkpoint_start_sent: 128  tap_connect_received: 1  tap_mutation_received: 428586  tap_opaque_received: 130  tap_opaque_sent: 2  threads: 4  time: 1320626663  total_connections: 15  uptime: 271  vb_active_curr_items: 788262  vb_active_eject: 788262  vb_active_ht_memory: 6346752  vb_active_itm_memory: 87496538  vb_active_num: 128  vb_active_num_non_resident: 788262  vb_active_ops_create: 0  vb_active_ops_delete: 0  vb_active_ops_reject: 0  vb_active_ops_update: 0  vb_active_perc_mem_resident: 0  vb_active_queue_age: 0  vb_active_queue_drain: 0  vb_active_queue_fill: 0  vb_active_queue_memory: 0  vb_active_queue_pending: 0  vb_active_queue_size: 0  vb_dead_num: 0  vb_pending_curr_items: 0  vb_pending_eject: 0  vb_pending_ht_memory: 0  vb_pending_itm_memory: 0  vb_pending_num: 0  vb_pending_num_non_resident: 0  vb_pending_ops_create: 0  vb_pending_ops_delete: 0  vb_pending_ops_reject: 0  vb_pending_ops_update: 0  vb_pending_perc_mem_resident: 0  vb_pending_queue_age: 0  vb_pending_queue_drain: 0  vb_pending_queue_fill: 0  vb_pending_queue_memory: 0  vb_pending_queue_pending: 0  vb_pending_queue_size: 0  vb_replica_curr_items: 567299  vb_replica_eject: 788143  vb_replica_ht_memory: 5219104  vb_replica_itm_memory: 642915325  vb_replica_num: 128  vb_replica_num_non_resident: 0  vb_replica_ops_create: 0  vb_replica_ops_delete: 0  vb_replica_ops_reject: 0  vb_replica_ops_update: 0  vb_replica_perc_mem_resident: 100  vb_replica_queue_age: 16055352000  vb_replica_queue_drain: 0  vb_replica_queue_fill: 428518  vb_replica_queue_memory: 47994016  vb_replica_queue_pending: 497913960  vb_replica_queue_size: 428518
Hide
Permalink
Farshid Ghods added a comment -
I will leave the vm in ec2 for another day or so


ip : ec2-50-19-145-133.compute-1.amazonaws.com

Show
Farshid Ghods added a comment - I will leave the vm in ec2 for another day or so ip : ec2-50-19-145-133.compute-1.amazonaws.com
Hide
Permalink
Farshid Ghods added a comment -
this is the same bug Tony has seen in the 10 node cluster and I am also seeing this after loading data into a 1 node in ec2

this is a blocker now

Control connection to memcached on 'ns_1@127.0.0.1' disconnected: {{badmatch,
{error,
closed}},
[{mc_client_binary,
stats_recv,
4},
{mc_client_binary,
stats,4},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} ns_memcached004 ns_1@127.0.0.1 15:56:13 - Mon Nov 7, 2011
Port server memcached on node 'ns_1@127.0.0.1' exited with status 134. Restarting. Messages: Trying to connect to mccouch: "localhost:11213"
Connected to mccouch: "localhost:11213"
memcached: mc-kvstore/mc-kvstore.cc:187: virtual bool MCKVStore::commit(): Assertion `intransaction' failed.
Show
Farshid Ghods added a comment - this is the same bug Tony has seen in the 10 node cluster and I am also seeing this after loading data into a 1 node in ec2 this is a blocker now Control connection to memcached on ' ns_1@127.0.0.1 ' disconnected: {{badmatch, {error, closed}}, [{mc_client_binary, stats_recv, 4}, {mc_client_binary, stats,4}, {ns_memcached, handle_call, 3}, {gen_server, handle_msg, 5}, {proc_lib, init_p_do_apply, 3}]} ns_memcached004 ns_1@127.0.0.1 15:56:13 - Mon Nov 7, 2011 Port server memcached on node ' ns_1@127.0.0.1 ' exited with status 134. Restarting. Messages: Trying to connect to mccouch: "localhost:11213" Connected to mccouch: "localhost:11213" memcached: mc-kvstore/mc-kvstore.cc:187: virtual bool MCKVStore::commit(): Assertion `intransaction' failed.
Hide
Permalink
Farshid Ghods added a comment -

Chiyoung is already looking at this.
Show
Farshid Ghods added a comment - Chiyoung is already looking at this.
Hide
Permalink
Farshid Ghods added a comment - - edited
also tried with a more recent build 213 and seeing the same issue - crash after loading 400k items
Show
Farshid Ghods added a comment - - edited also tried with a more recent build 213 and seeing the same issue - crash after loading 400k items

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Farshid Ghods
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: