Details
Description
obvious from these logs :
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
(repeated 1 times) ns_orchestrator002 ns_1@10.113.11.11 01:24:57 - Mon Jun 4, 2012
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
ns_orchestrator002 ns_1@10.113.11.11 01:19:18 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
ns_orchestrator004 ns_1@10.113.11.11 01:19:07 - Mon Jun 4, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.113.11.11 01:13:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.17' for server shutdown ns_memcached002 ns_1@10.113.11.17 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.19' for server shutdown ns_memcached002 ns_1@10.113.11.19 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.23' for server shutdown ns_memcached002 ns_1@10.113.11.23 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.11' for server shutdown ns_memcached002 ns_1@10.113.11.11 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.12' for server shutdown ns_memcached002 ns_1@10.113.11.12 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.5' for server shutdown ns_memcached002 ns_1@10.113.11.5 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.16' for server shutdown ns_memcached002 ns_1@10.113.11.16 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.24' for server shutdown ns_memcached002 ns_1@10.113.11.24 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.21' for server shutdown ns_memcached002 ns_1@10.113.11.21 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.27' for server shutdown ns_memcached002 ns_1@10.113.11.27 01:02:19 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
(repeated 1 times) ns_orchestrator002 ns_1@10.113.11.11 01:24:57 - Mon Jun 4, 2012
Rebalance exited with reason {wait_for_memcached_failed,"Authentication",
['ns_1@10.113.11.5','ns_1@10.113.11.11',
'ns_1@10.113.11.12','ns_1@10.113.11.16',
'ns_1@10.113.11.17','ns_1@10.113.11.19',
'ns_1@10.113.11.21','ns_1@10.113.11.23',
'ns_1@10.113.11.24','ns_1@10.113.11.27']}
ns_orchestrator002 ns_1@10.113.11.11 01:19:18 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
ns_orchestrator004 ns_1@10.113.11.11 01:19:07 - Mon Jun 4, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.113.11.11 01:13:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.17' for server shutdown ns_memcached002 ns_1@10.113.11.17 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.19' for server shutdown ns_memcached002 ns_1@10.113.11.19 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.23' for server shutdown ns_memcached002 ns_1@10.113.11.23 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.11' for server shutdown ns_memcached002 ns_1@10.113.11.11 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.12' for server shutdown ns_memcached002 ns_1@10.113.11.12 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.5' for server shutdown ns_memcached002 ns_1@10.113.11.5 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.16' for server shutdown ns_memcached002 ns_1@10.113.11.16 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.24' for server shutdown ns_memcached002 ns_1@10.113.11.24 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.21' for server shutdown ns_memcached002 ns_1@10.113.11.21 01:02:19 - Mon Jun 4, 2012
Shutting down bucket "Authentication" on 'ns_1@10.113.11.27' for server shutdown ns_memcached002 ns_1@10.113.11.27 01:02:19 - Mon Jun 4, 2012
Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
'ns_1@10.113.11.3','ns_1@10.113.11.4',
'ns_1@10.113.11.6','ns_1@10.113.11.7',
'ns_1@10.113.11.8','ns_1@10.113.11.9',
'ns_1@10.113.11.10','ns_1@10.113.11.13',
'ns_1@10.113.11.14','ns_1@10.113.11.15',
'ns_1@10.113.11.18','ns_1@10.113.11.20',
'ns_1@10.113.11.22','ns_1@10.113.11.25',
'ns_1@10.113.11.26','ns_1@10.113.11.28',
'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
'ns_1@10.113.11.11',
'ns_1@10.113.11.12',
'ns_1@10.113.11.16',
'ns_1@10.113.11.17',
'ns_1@10.113.11.19',
'ns_1@10.113.11.21',
'ns_1@10.113.11.23',
'ns_1@10.113.11.24',
'ns_1@10.113.11.27']
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Farshid Ghods
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Attachment | Screen Shot 2012-06-04 at 1.30.06 AM.png [ 13478 ] |
Farshid Ghods
made changes -
| Summary | rebalance waits with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt | rebalance fails with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt |
Aleksey Kondratenko
made changes -
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Resolution | Fixed [ 1 ] |
Farshid Ghods
made changes -
| Status | Resolved [ 5 ] | Closed [ 6 ] |