Couchbase Server

vbucketmigrator exits with Auth failure

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Incomplete
  • Affects Version/s: 1.6.0 beta4
  • Fix Version/s: Backlog
  • Component/s: ns_server
  • Labels:
  • Environment:
    Operating System: All
    Platform: All

Description

Created an attachment (id=590)
diag

1.6.0beta3.2b-22
centos 5.4 32bit
4 nodes

1) cluster 4 nodes
2) create 3 new buckets
 * 1 with replication count 1
 * 1 with replication count 2
 * 1 with replication count 3
3) load some data in to the buckets


Port server vbucketmigrator on node 'ns_1@10.1.4.199' exited with status 78. Restarting. Messages: Authenticating towards: {Sock 10.1.4.207:11210}
Authenticated towards: {Sock 10.1.4.207:11210}
Connecting to source: 10.1.4.199:11210
Authenticating towards: {Sock 10.1.4.199:11210}
Failed to authenticate: Authentication failed: Auth failure

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Sean Lynch (Inactive) added a comment -
The second node wasn't fully added due to a bug that's fixed in http://review.northscale.com/#change,2542, though that doesn't appear to be the cause of this problem.

It appears that the final bucket was created an extremely short period before the rebalance started - was this an automated test? Did retrying the rebalance succeed?

Memcached only checks the password file once per second, so if you create a bucket and then try to do something to it within a short period of time, you can get auth failures. It's difficult to make this synchronous, so for beta4, assuming the failure was transient, I'm inclined to say "don't do that."
Show
Sean Lynch (Inactive) added a comment - The second node wasn't fully added due to a bug that's fixed in http://review.northscale.com/#change,2542, though that doesn't appear to be the cause of this problem. It appears that the final bucket was created an extremely short period before the rebalance started - was this an automated test? Did retrying the rebalance succeed? Memcached only checks the password file once per second, so if you create a bucket and then try to do something to it within a short period of time, you can get auth failures. It's difficult to make this synchronous, so for beta4, assuming the failure was transient, I'm inclined to say "don't do that."
Hide
Permalink
Sean Lynch (Inactive) added a comment -
Actually on further examination I see there are auth failures long after the file was written, but what I do see is that the file was written several times in a row.

I suspect there's a race condition here even though we're replacing the file atomically, so several subsequent updates to the file cause the later ones to get missed. I think it'll be easy to fix: don't write to the file more than once in some specified time window.
Show
Sean Lynch (Inactive) added a comment - Actually on further examination I see there are auth failures long after the file was written, but what I do see is that the file was written several times in a row. I suspect there's a race condition here even though we're replacing the file atomically, so several subsequent updates to the file cause the later ones to get missed. I think it'll be easy to fix: don't write to the file more than once in some specified time window.
Hide
Permalink
Sean Lynch (Inactive) added a comment -
This is a pain to fix for beta4 and will only happen if you create several buckets at once. You can work around it by touching the isasl.pw file.
Show
Sean Lynch (Inactive) added a comment - This is a pain to fix for beta4 and will only happen if you create several buckets at once. You can work around it by touching the isasl.pw file.
Hide
Permalink
Sean Lynch (Inactive) added a comment -
Moving to P3 since it only seems to affect us when we create or change lots of buckets at once and I've been having trouble reproducing it.
Show
Sean Lynch (Inactive) added a comment - Moving to P3 since it only seems to affect us when we create or change lots of buckets at once and I've been having trouble reproducing it.
Hide
Permalink
Aleksey Kondratenko added a comment -
Is that still alive?
Show
Aleksey Kondratenko added a comment - Is that still alive?

People

  • Assignee:
    Keith Batten
    Reporter:
    kbatten@gmail.com
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: