vbucketmigrator exits with Auth failure

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Incomplete
Affects Version/s: 1.6.0 beta4
Fix Version/s: Backlog
Component/s: ns_server
Labels:
- need_info
Environment:
Operating System: All
Platform: All

Description

Created an attachment (id=590)
diag

1.6.0beta3.2b-22
centos 5.4 32bit
4 nodes

1) cluster 4 nodes
2) create 3 new buckets
* 1 with replication count 1
* 1 with replication count 2
* 1 with replication count 3
3) load some data in to the buckets

Port server vbucketmigrator on node 'ns_1@10.1.4.199' exited with status 78. Restarting. Messages: Authenticating towards: {Sock 10.1.4.207:11210}
Authenticated towards: {Sock 10.1.4.207:11210}
Connecting to source: 10.1.4.199:11210
Authenticating towards: {Sock 10.1.4.199:11210}
Failed to authenticate: Authentication failed: Auth failure

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

ns-diag-20100917173018.txt.gz

18/Sep/10 12:33 AM

735 kB

kbatten@gmail.com

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Sean Lynch (Inactive) added a comment - 18/Sep/10 7:01 PM

The second node wasn't fully added due to a bug that's fixed in http://review.northscale.com/#change,2542, though that doesn't appear to be the cause of this problem.

It appears that the final bucket was created an extremely short period before the rebalance started - was this an automated test? Did retrying the rebalance succeed?

Memcached only checks the password file once per second, so if you create a bucket and then try to do something to it within a short period of time, you can get auth failures. It's difficult to make this synchronous, so for beta4, assuming the failure was transient, I'm inclined to say "don't do that."

Show

Sean Lynch (Inactive) added a comment - 18/Sep/10 7:01 PM The second node wasn't fully added due to a bug that's fixed in http://review.northscale.com/#change,2542, though that doesn't appear to be the cause of this problem. It appears that the final bucket was created an extremely short period before the rebalance started - was this an automated test? Did retrying the rebalance succeed? Memcached only checks the password file once per second, so if you create a bucket and then try to do something to it within a short period of time, you can get auth failures. It's difficult to make this synchronous, so for beta4, assuming the failure was transient, I'm inclined to say "don't do that."

Hide

Permalink

Sean Lynch (Inactive) added a comment - 18/Sep/10 7:19 PM

Actually on further examination I see there are auth failures long after the file was written, but what I do see is that the file was written several times in a row.

I suspect there's a race condition here even though we're replacing the file atomically, so several subsequent updates to the file cause the later ones to get missed. I think it'll be easy to fix: don't write to the file more than once in some specified time window.

Show

Sean Lynch (Inactive) added a comment - 18/Sep/10 7:19 PM Actually on further examination I see there are auth failures long after the file was written, but what I do see is that the file was written several times in a row. I suspect there's a race condition here even though we're replacing the file atomically, so several subsequent updates to the file cause the later ones to get missed. I think it'll be easy to fix: don't write to the file more than once in some specified time window.

Hide

Permalink

Sean Lynch (Inactive) added a comment - 18/Sep/10 11:46 PM

This is a pain to fix for beta4 and will only happen if you create several buckets at once. You can work around it by touching the isasl.pw file.

Show

Sean Lynch (Inactive) added a comment - 18/Sep/10 11:46 PM This is a pain to fix for beta4 and will only happen if you create several buckets at once. You can work around it by touching the isasl.pw file.

Hide

Permalink

Sean Lynch (Inactive) added a comment - 24/Sep/10 10:37 PM

Moving to P3 since it only seems to affect us when we create or change lots of buckets at once and I've been having trouble reproducing it.

Show

Sean Lynch (Inactive) added a comment - 24/Sep/10 10:37 PM Moving to P3 since it only seems to affect us when we create or change lots of buckets at once and I've been having trouble reproducing it.

Matt Ingenthron made changes - 09/Oct/10 8:15 PM