intermittent failure when loading sample data via setup wizard or the "setting" tab on mac when user starts couchbase server app for the first time due to some permission issues

Farshid Ghods added a comment - 27/Aug/12 5:08 PM Abhinav is still looking at the diags

Steve Yen added a comment - 27/Aug/12 5:17 PM Hi Abhinav, Can you extend this jira bug with more reproduction steps info? For example, were you using the web-based installation screen to load sample data? Were you using the command-line tool? If so, what was the command-line? And, what was your bucket configuration? What O/S or environment was this? (e.g, linux? windows?) (p.s., in general, a good bug report will have enough, relevant reproduction info in it (e.g., signal, not too much noise.)) Thanks much! Steve

Abhinav Dangeti added a comment - 27/Aug/12 8:36 PM So the error seems to be the following:  "/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/cbdocloader: line 12: /Applications/Couchbase: No such file or directory\n/Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/cbdocloader: line 12: exec: /Applications/Couchbase: cannot execute: No such file or directory\n" This could be because the space not being escaped in "Couchbase Server", it can also be fixed by enclosing the entire path with quotes.

Farshid Ghods added a comment - 28/Aug/12 11:18 AM Jens looks like there is still an issue with running docloader when there is a space in the path.

Jens Alfke added a comment - 28/Aug/12 11:47 AM I fixed this issue in the wrapper script in membase-cli, but there are several other copies of this wrapper in other repos that need to be fixed too... http://review.couchbase.org/#/c/20252/

Jens Alfke added a comment - 28/Aug/12 11:50 AM The same fix in the workload-generator repo: http://review.couchbase.org/#/c/20253/

Jens Alfke added a comment - 28/Aug/12 11:53 AM ...and in ep-engine: http://review.couchbase.org/#/c/20254/

Thuan Nguyen added a comment - 28/Aug/12 11:28 PM Integrated in github-ep-engine-2-0 #417 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/417/ ])      MB-6452 : Fix path quoting in wrapper script (Revision 3aa6e766a0657d3c021074bae5c3254eb6ae22f0)      Result = SUCCESS Jens Alfke : Files : * wrapper/wrapper.in

Abhinav Dangeti added a comment - 10/Sep/12 2:40 PM Verified on build 1705

Matt Ingenthron added a comment - 12/Sep/12 9:52 PM I see what appears to be this issue in build 1717. I also saw it in build 1700. Will attach diags.

Farshid Ghods added a comment - 12/Sep/12 10:07 PM [ns_server:error,2012-09-12T19:44:25.772, ns_1@127.0.0.1 :<0.895.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8636>,                                                     {data,                                                      "Exception in thread dispatcher-thread (most likely raised during interpreter shutdown):\nTraceback (most recent call last):\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 522, in __bootstrap_inner\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 477, in run\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 699, in _start_dispatcher\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 1034, in dispatch\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/Queue.py\", line 174, in get\n<type 'exceptions.TypeError'>: 'NoneType' object is not callable\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      "[2012-09-12 19:44:14,688] - [rest_client] [140735084821696] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=beer-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},                                                    {#Port<0.8636>, [ns_server:error,2012-09-12T19:44:25.772, ns_1@127.0.0.1 :<0.895.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8636>,                                                     {data,                                                      "Exception in thread dispatcher-thread (most likely raised during interpreter shutdown):\nTraceback (most recent call last):\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 522, in __bootstrap_inner\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py\", line 477, in run\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 699, in _start_dispatcher\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/lib/python/couchbase/couchbaseclient.py\", line 1034, in dispatch\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/Queue.py\", line 174, in get\n<type 'exceptions.TypeError'>: 'NoneType' object is not callable\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      " zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},                                                    {#Port<0.8636>,                                                     {data,                                                      "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1717/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n"}},

Farshid Ghods added a comment - 12/Sep/12 10:08 PM not sure if this is permission issue. Abhinav, this is different than the centos 5.x issue exists with cbdocloader. can you try this latest build on your mac and see if its reproducible ?

Farshid Ghods added a comment - 12/Sep/12 10:09 PM Matt, what error do you get if you try to run cbdocloader manually from ~/Couchase Server.app/..../couchbase-core/bin/tools/ to load this sample data ?

Abhinav Dangeti added a comment - 12/Sep/12 11:38 PM Farshid, With build 1719, on my mac I am able to load both beer sample and gamesim sample.

Farshid Ghods added a comment - 12/Sep/12 11:44 PM I think this happens if you have earlier installations can you try that scenario

Farshid Ghods added a comment - 13/Sep/12 10:18 AM this is not reproducible on new installations , so moving this to beta refresh as this could be happening if node already has previous version installed and user did not clean up the data folder entirely

Matt Ingenthron added a comment - 14/Sep/12 2:22 AM No, this still happens for me, even with build 1723. I'm not sure if matches what's above. I'll attach diags.

Matt Ingenthron added a comment - 14/Sep/12 2:23 AM Attached diags

Andrei Baranouski added a comment - 14/Sep/12 10:53 AM from Matt's logs: [ns_server:error,2012-09-14T0:13:36.475, ns_1@127.0.0.1 :<0.1033.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8685>,                                                     {data,                                                      " return self._extract_member(member, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\n target = file(targetpath, \"wb\")\nIOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs'\n"}},                                                    {#Port<0.8685>,                                                     {data,                                                      " self.extract(zipinfo, path, pwd)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n"}},                                                    {#Port<0.8685>,                                                     {data,                                                      "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'beer-sample', 'ram_quota': 100} ['/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/beer-sample.zip']\nTraceback (most recent call last):\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 192, in <module>\n main()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 186, in main\n docloader.populate_docs()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 144, in populate_docs\n self.unzip_file_and_upload()\n File \"/Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 127, in unzip_file_and_upload\n zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n"}},                                                    {#Port<0.8685>,                                                     {data,                                                      "[2012-09-14 00:13:25,449] - [rest_client] [140735084821696] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=beer-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},                                                    {#Port<0.8685>,                                                     {data,                                                      "[2012-09-14 00:13:25,444] - [rest_client] [140735084821696] - INFO - existing buckets : []\n"}}]

Farshid Ghods added a comment - 14/Sep/12 11:26 AM Looks. Like permission problem to me Maybe its happening if you drag it to applications before starting it for the first time

Farshid Ghods added a comment - 14/Sep/12 11:26 AM Looks. Like permission problem to me Maybe its happening if you drag it to applications before starting it for the first time

Bin Cui added a comment - 14/Sep/12 11:36 AM Matt, can you manually remove /tmp/_working directory and try again? For beta refresh, i will add logic to delete any existed tmp working directory before extracting zip files to working directory.

Farshid Ghods added a comment - 14/Sep/12 5:39 PM I and Bin have tried this on multiple mac and eventually found one Mac box that has this issue. the issue is not related to /tmp or the other temp folder that python obtains as part of the its request but mostly about the permissions that the application seems to have when it starts for the first time. upon restarting the app we were able to load the data. bin is currently looking into this issue but as i mentioned this is only mac specific

Jens Alfke added a comment - 14/Sep/12 7:17 PM The specific Python exception:     IOError: [Errno 20] Not a directory: '/tmp/_working/beer-sample/design_docs' This is ENOTDIR. Implies that some component of that path is a file not a directory. Also interesting is that the logs above show that cbdocloader is writing into /tmp, while when I call the same Python API that cbdocloader does — tempfile.gettempdir() — I get a per-user temp directory (/var/folders/4f/mvdftm3d5ss0ystm0m2q8q3m0000gn/T). This may correlate to the OS version. I don't see any details above of what exactly OS version was running, but the logs refer to Python 2.6, which I believe appeared in OS X 10.6. (I'm on 10.8 which has Python 2.7.2, and IIRC 10.7 had an earlier Python 2.7 release.)

Farshid Ghods added a comment - 14/Sep/12 7:23 PM another error i got [ns_server:error,2012-09-14T14:59:03.206, ns_1@127.0.0.1 :<0.875.0>:menelaus_web:install_samples:549]cbdocloader failed unexpectedly: status: 1, msgs: [{#Port<0.8599>,                                                     {data,                                                      "{'username': None, 'node': '127.0.0.1:8091', 'password': None, 'bucket': 'gamesim-sample', 'ram_quota': 100} ['/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/../samples/gamesim-sample.zip']\nTraceback (most recent call last):\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 245, in <module>\n main()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 239, in main\n docloader.populate_docs()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 197, in populate_docs\n self.unzip_file_and_upload()\n File \"/Users/couchbase/Downloads/couchbase-server-community_x86_64_2-1/Couchbase Server.app/Contents/Resources/couchbase-core/bin/tools/../../lib/python/cbdocloader\", line 177, in unzip_file_and_upload\n zfobj.extractall(working_dir)\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 940, in extractall\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 928, in extract\n File \"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/zipfile.py\", line 965, in _extract_member\nIOError: [Errno 20] Not a directory: '/tmp/_working/gamesim-sample/design_docs'\n"}},                                                    {#Port<0.8599>,                                                     {data,                                                      "[2012-09-14 14:58:52,260] - [rest_client] [140735073193152] - INFO - http://127.0.0.1:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&name=gamesim-sample&replicaNumber=1&saslPassword=&ramQuotaMB=100\n"}},                                                    {#Port<0.8599>,                                                     {data,                                                      "[2012-09-14 14:58:52,255] - [rest_client] [140735073193152] - INFO - existing buckets : []\n"}}]

Jens Alfke added a comment - 14/Sep/12 7:27 PM There are several problems with the function in cbdocloader that ends up throwing the exception. I don't know if they're causing this bug, but they're worth pointing out:         working_dir = os.path.join(tmpdir, '_working')         if not os.path.exists(working_dir):             try:                 os.makedirs(working_dir)             except:                 print "Unexpected error:", sys.exc_info()[0]                 return 1. This code will fail during the zip extraction with ENOTDIR if _working already exists in tmpdir but is a file not a directory. That might actually be the cause of this bug, but I have no idea where the _working file might come from. (But note that this temp dir is shared by all processes belonging to that user, and "_working" is a pretty generic name, so it could be any other process/app creating it.) 2. The test-then-create sequence is a filesystem race condition. There's a chance the _working dir could have existed but been deleted in between the two os calls. This kind of issue has led to security holes in the past. It's safer to always call os.makedirs and then ignore a duplicate-file (EEXIST) error. 3. The exception handler effectively ignores the exception. It skips unzipping the file, but the caller has no idea anything went wrong, so it's likely to break. It's probably best to _not_ catch the exception at all but let it propagate to the top level where it can be reported. 4. If _working already exists, shouldn't it be deleted instead of reused? It might contain unrelated files that will get mixed in with the files from the zip archive. 5. As I mentioned above, "_working" is a generic name, and since this is a temp dir used by other processes, it should be something more specific like "cbdocloader_working".

Jens Alfke added a comment - 14/Sep/12 7:31 PM Farshid: re. your last comment, what was in /tmp/_working after that error occurred?

Farshid Ghods added a comment - 14/Sep/12 7:33 PM beer-sample gamesim-sample macsvr-106-01:_working couchbase$ ls -alh total 0 drwxr-xr-x 4 couchbase wheel 136B Sep 14 17:12 . drwxrwxrwt 8 root wheel 272B Sep 14 17:12 .. -rw-r--r-- 1 couchbase wheel 0B Sep 14 17:12 beer-sample -rw-r--r-- 1 couchbase wheel 0B Sep 14 14:59 gamesim-sample

Bin Cui added a comment - 14/Sep/12 8:40 PM http://review.couchbase.org/#/c/20880/

Bin Cui added a comment - 15/Sep/12 8:36 AM There are several problems with the function in cbdocloader that ends up throwing the exception. I don't know if they're causing this bug, but they're worth pointing out:         working_dir = os.path.join(tmpdir, '_working')         if not os.path.exists(working_dir):             try:                 os.makedirs(working_dir)             except:                 print "Unexpected error:", sys.exc_info()[0]                 return Jens Alfke commented on MB-6452 : -------------------------------- 1. This code will fail during the zip extraction with ENOTDIR if _working already exists in tmpdir but is a file not a directory. That might actually be the cause of this bug, but I have no idea where the _working file might come from. (But note that this temp dir is shared by all processes belonging to that user, and "_working" is a pretty generic name, so it could be any other process/app creating it.) 2. The test-then-create sequence is a filesystem race condition. There's a chance the _working dir could have existed but been deleted in between the two os calls. This kind of issue has led to security holes in the past. It's safer to always call os.makedirs and then ignore a duplicate-file (EEXIST) error. 3. The exception handler effectively ignores the exception. It skips unzipping the file, but the caller has no idea anything went wrong, so it's likely to break. It's probably best to _not_ catch the exception at all but let it propagate to the top level where it can be reported. 4. If _working already exists, shouldn't it be deleted instead of reused? It might contain unrelated files that will get mixed in with the files from the zip archive. 5. As I mentioned above, "_working" is a generic name, and since this is a temp dir used by other processes, it should be something more specific like "cbdocloader_working".