[MB-6452] intermittent failure when loading sample data via setup wizard or the "setting" tab on mac when user starts couchbase server app for the first time due to some permission issues Created: 27/Aug/12  Updated: 09/Jan/13  Resolved: 15/Sep/12

Status: Closed
Project: Couchbase Server
Component/s: installer, tools
Affects Version/s: None
Fix Version/s: 2.0-beta
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Abhinav Dangeti Assignee: Bin Cui
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: Build 1644:
http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.0-1644-rel.zip
Environment: Mac
Per Server RAM Quota: 2048 MB

Attachments: GZip Archive 127.0.0.1-8091-diag.txt.gz     GZip Archive ns-diag-20120912194935.txt.gz     GZip Archive ns-diag-20120914002213.txt.gz    

 Description   
While using sample-loader (either beer-sample or gamesim-sample) to load on sample buckets on the couchbase server (on the web console and not using the command line), an "unexpected error" pops up. The operation fails.
*Grabdiags are attached.

 Comments   
Comment by Farshid Ghods (Inactive) [ 27/Aug/12 ]
Abhinav is still looking at the diags
Comment by Steve Yen [ 27/Aug/12 ]
Hi Abhinav,

Can you extend this jira bug with more reproduction steps info?

For example, were you using the web-based installation screen to load sample data?
Were you using the command-line tool? If so, what was the command-line?
And, what was your bucket configuration?

What O/S or environment was this? (e.g, linux? windows?)

(p.s., in general, a good bug report will have enough, relevant reproduction info in it (e.g., signal, not too much noise.))

Thanks much!
Steve
Comment by Abhinav Dangeti [ 27/Aug/12 ]
So the error seems to be the following:
 "/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/cbdocloader: line 12: /Applications/Couchbase: No such file or directory\n/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/cbdocloader: line 12: exec: /Applications/Couchbase: cannot execute: No such file or directory\n"

This could be because the space not being escaped in "Couchbase Server", it can also be fixed by enclosing the entire path with quotes.
Comment by Farshid Ghods (Inactive) [ 28/Aug/12 ]
Jens looks like there is still an issue with running docloader when there is a space in the path.

Comment by Jens Alfke [ 28/Aug/12 ]
I fixed this issue in the wrapper script in membase-cli, but there are several other copies of this wrapper in other repos that need to be fixed too...

http://review.couchbase.org/#/c/20252/
Comment by Jens Alfke [ 28/Aug/12 ]
The same fix in the workload-generator repo:
http://review.couchbase.org/#/c/20253/
Comment by Jens Alfke [ 28/Aug/12 ]
...and in ep-engine:
http://review.couchbase.org/#/c/20254/
Comment by Thuan Nguyen [ 28/Aug/12 ]
Integrated in github-ep-engine-2-0 #417 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/417/])
    MB-6452: Fix path quoting in wrapper script (Revision 3aa6e766a0657d3c021074bae5c3254eb6ae22f0)

     Result = SUCCESS
Jens Alfke :
Files :
* wrapper/wrapper.in
Comment by Abhinav Dangeti [ 10/Sep/12 ]
Verified on build 1705
Comment by Matt Ingenthron [ 12/Sep/12 ]
I see what appears to be this issue in build 1717. I also saw it in build 1700.

Will attach diags.
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
[ns_server:error,2012-09-12T19:44:25.772,ns_1@127.0.0.1:<0.895.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8636>,
                                                    {data,
                                                     "Exception in thread dispatcher-thread (most likely raised during interpreter shutdown):\nTraceback (most recent call last):\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 522, in __bootstrap_inner\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 477, in run\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 699, in _start_dispatcher\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 1034, in dispatch\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/Queue.py\", line 174, in get\n<type 'exceptions.TypeError'>: 'NoneType' object is not callable\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     "[2012-09-12 19:44:14,688] - [rest_client] [140735084821696] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=beer-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},
                                                   {#Port<0.8636>,

[ns_server:error,2012-09-12T19:44:25.772,ns_1@127.0.0.1:<0.895.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8636>,
                                                    {data,
                                                     "Exception in thread dispatcher-thread (most likely raised during interpreter shutdown):\nTraceback (most recent call last):\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 522, in __bootstrap_inner\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 477, in run\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 699, in _start_dispatcher\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 1034, in dispatch\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/Queue.py\", line 174, in get\n<type 'exceptions.TypeError'>: 'NoneType' object is not callable\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     " zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},
                                                   {#Port<0.8636>,
                                                    {data,
                                                     "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n"}},
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
not sure if this is permission issue.

Abhinav,
this is different than the centos 5.x issue exists with cbdocloader.

can you try this latest build on your mac and see if its reproducible ?
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
Matt,

what error do you get if you try to run cbdocloader manually from ~/Couchase Server.app/..../couchbase-core/bin/tools/ to load this sample data ?
Comment by Abhinav Dangeti [ 12/Sep/12 ]
Farshid,
With build 1719, on my mac I am able to load both beer sample and gamesim sample.
Comment by Farshid Ghods (Inactive) [ 12/Sep/12 ]
I think this happens if you have earlier installations can you try that scenario
Comment by Farshid Ghods (Inactive) [ 13/Sep/12 ]
this is not reproducible on new installations , so moving this to beta refresh as this could be happening if node already has previous version installed and user did not clean up the data folder entirely
Comment by Matt Ingenthron [ 14/Sep/12 ]
No, this still happens for me, even with build 1723. I'm not sure if matches what's above. I'll attach diags.
Comment by Matt Ingenthron [ 14/Sep/12 ]
Attached diags
Comment by Andrei Baranouski [ 14/Sep/12 ]
from Matt's logs:
[ns_server:error,2012-09-14T0:13:36.475,ns_1@127.0.0.1:<0.1033.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8685>,
                                                    {data,
                                                     " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},
                                                   {#Port<0.8685>,
                                                    {data,
                                                     " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},
                                                   {#Port<0.8685>,
                                                    {data,
                                                     "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},
                                                   {#Port<0.8685>,
                                                    {data,
                                                     "[2012-09-14 00:13:25,449] - [rest_client] [140735084821696] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=beer-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},
                                                   {#Port<0.8685>,
                                                    {data,
                                                     "[2012-09-14 00:13:25,444] - [rest_client] [140735084821696] - INFO - existing buckets : []\n"}}]
Comment by Farshid Ghods (Inactive) [ 14/Sep/12 ]
Looks. Like permission problem to me
Maybe its happening if you drag it to applications before starting it for the first time
Comment by Farshid Ghods (Inactive) [ 14/Sep/12 ]
Looks. Like permission problem to me
Maybe its happening if you drag it to applications before starting it for the first time
Comment by Bin Cui [ 14/Sep/12 ]
Matt, can you manually remove /tmp/_working directory and try again?
For beta refresh, i will add logic to delete any existed tmp working directory before extracting zip files to working directory.
Comment by Farshid Ghods (Inactive) [ 14/Sep/12 ]
I and Bin have tried this on multiple mac and eventually found one Mac box that has this issue.

the issue is not related to /tmp or the other temp folder that python obtains as part of the its request but mostly about the permissions that the application seems to have when it starts for the first time.

upon restarting the app we were able to load the data.

bin is currently looking into this issue but as i mentioned this is only mac specific
Comment by Jens Alfke [ 14/Sep/12 ]
The specific Python exception:
    IOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'
This is ENOTDIR. Implies that some component of that path is a file not a directory.

Also interesting is that the logs above show that cbdocloader is writing into /tmp, while when I call the same Python API that cbdocloader does — tempfile.gettempdir() — I get a per-user temp directory (/var/folders/4f/mvdftm3d5ss0ystm0m2q8q3m0000gn/T).

This may correlate to the OS version. I don't see any details above of what exactly OS version was running, but the logs refer to Python 2.6, which I believe appeared in OS X 10.6. (I'm on 10.8 which has Python 2.7.2, and IIRC 10.7 had an earlier Python 2.7 release.)
Comment by Farshid Ghods (Inactive) [ 14/Sep/12 ]
another error i got

[ns_server:error,2012-09-14T14:59:03.206,ns_1@127.0.0.1:<0.875.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8599>,
                                                    {data,
                                                     "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'gamesim-sample', 'ram_quota': 100} ['/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/gamesim-sample.zip']\nTraceback (most recent call last):\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 245, in <module>\n main()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 239, in main\n docloader.populate_docs()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 197, in populate_docs\n self.unzip_file_and_upload()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 177, in unzip_file_and_upload\n zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\nIOError: [Errno 20] Not a directory: '/tmp/_working/gamesim-sample/design_docs'\n"}},
                                                   {#Port<0.8599>,
                                                    {data,
                                                     "[2012-09-14 14:58:52,260] - [rest_client] [140735073193152] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=gamesim-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},
                                                   {#Port<0.8599>,
                                                    {data,
                                                     "[2012-09-14 14:58:52,255] - [rest_client] [140735073193152] - INFO - existing buckets : []\n"}}]
Comment by Jens Alfke [ 14/Sep/12 ]
There are several problems with the function in cbdocloader that ends up throwing the exception. I don't know if they're causing this bug, but they're worth pointing out:

        working_dir = os.path.join(tmpdir, '_working')
        if not os.path.exists(working_dir):
            try:
                os.makedirs(working_dir)
            except:
                print "Unexpected error:", sys.exc_info()[0]
                return

1. This code will fail during the zip extraction with ENOTDIR if _working already exists in tmpdir but is a file not a directory. That might actually be the cause of this bug, but I have no idea where the _working file might come from. (But note that this temp dir is shared by all processes belonging to that user, and "_working" is a pretty generic name, so it could be any other process/app creating it.)
2. The test-then-create sequence is a filesystem race condition. There's a chance the _working dir could have existed but been deleted in between the two os calls. This kind of issue has led to security holes in the past. It's safer to always call os.makedirs and then ignore a duplicate-file (EEXIST) error.
3. The exception handler effectively ignores the exception. It skips unzipping the file, but the caller has no idea anything went wrong, so it's likely to break. It's probably best to _not_ catch the exception at all but let it propagate to the top level where it can be reported.
4. If _working already exists, shouldn't it be deleted instead of reused? It might contain unrelated files that will get mixed in with the files from the zip archive.
5. As I mentioned above, "_working" is a generic name, and since this is a temp dir used by other processes, it should be something more specific like "cbdocloader_working".
Comment by Jens Alfke [ 14/Sep/12 ]
Farshid: re. your last comment, what was in /tmp/_working after that error occurred?
Comment by Farshid Ghods (Inactive) [ 14/Sep/12 ]
beer-sample gamesim-sample
macsvr-106-01:_working couchbase$ ls -alh
total 0
drwxr-xr-x 4 couchbase wheel 136B Sep 14 17:12 .
drwxrwxrwt 8 root wheel 272B Sep 14 17:12 ..
-rw-r--r-- 1 couchbase wheel 0B Sep 14 17:12 beer-sample
-rw-r--r-- 1 couchbase wheel 0B Sep 14 14:59 gamesim-sample
Comment by Bin Cui [ 14/Sep/12 ]
http://review.couchbase.org/#/c/20880/
Comment by Bin Cui [ 15/Sep/12 ]
There are several problems with the function in cbdocloader that ends up throwing the exception. I don't know if they're causing this bug, but they're worth pointing out:

        working_dir = os.path.join(tmpdir, '_working')
        if not os.path.exists(working_dir):
            try:
                os.makedirs(working_dir)
            except:
                print "Unexpected error:", sys.exc_info()[0]
                return

Jens Alfke commented on MB-6452:
--------------------------------

1. This code will fail during the zip extraction with ENOTDIR if _working already exists in tmpdir but is a file not a directory. That might actually be the cause of this bug, but I have no idea where the _working file might come from. (But note that this temp dir is shared by all processes belonging to that user, and "_working" is a pretty generic name, so it could be any other process/app creating it.) 2. The test-then-create sequence is a filesystem race condition. There's a chance the _working dir could have existed but been deleted in between the two os calls. This kind of issue has led to security holes in the past. It's safer to always call os.makedirs and then ignore a duplicate-file (EEXIST) error.
3. The exception handler effectively ignores the exception. It skips unzipping the file, but the caller has no idea anything went wrong, so it's likely to break. It's probably best to _not_ catch the exception at all but let it propagate to the top level where it can be reported.
4. If _working already exists, shouldn't it be deleted instead of reused? It might contain unrelated files that will get mixed in with the files from the zip archive.
5. As I mentioned above, "_working" is a generic name, and since this is a temp dir used by other processes, it should be something more specific like "cbdocloader_working".
Generated at Sat Aug 23 01:06:07 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.