[doc + RN] cbtransfer CSV

Steve Yen added a comment - 05/Nov/12 8:40 PM http://review.couchbase.org/#/c/22293/1

Steve Yen added a comment - 26/Nov/12 2:24 PM to 2.0.2 per bug-scrub

Steve Yen added a comment - 31/Jan/13 12:47 PM Bin, I think these might be in gerrit already, but probably on the wrong branch. Can you help rescue them? Thanks, Steve

Bin Cui added a comment - 19/Feb/13 5:43 PM Merge Steve's change into 2.0.2

Maria McDuff added a comment - 05/Apr/13 12:12 PM pls verify in 2.0.2 build.

Bin Cui added a comment - 10/Apr/13 4:55 PM http://review.couchbase.org/#/c/25603/ Export as csv file:   cbtransfer http://HOST:8091 csv: /tmp/data.csv

Anil Kumar added a comment - 11/Apr/13 1:46 PM Looks like the code is already fixed only needs to code-reviewed and merged.

Maria McDuff added a comment - 18/Apr/13 1:18 AM pls verify / close.

Shashank Gupta added a comment - 22/Apr/13 7:58 AM Imported the data as CSV. But got an error during exporting the csv data using the command : ./cbtransfer /tmp/data.csv http://localhost:8091 -B def -u Administrator -p password First it transfers some of the messages to destination bucket and then gives the following error: .Exception in thread w0: Traceback (most recent call last):   File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap     self.run()   File "/usr/lib64/python2.4/threading.py", line 422, in run     self.__target(*self.__args, **self.__kwargs)   File "/opt/couchbase/lib/python/pump.py", line 271, in run_worker     source_map, sink_map, self.ctl, curx).run()   File "/opt/couchbase/lib/python/pump.py", line 328, in run     rv_batch, batch = self.source.provide_batch()   File "/opt/couchbase/lib/python/pump_csv.py", line 78, in provide_batch     vals = self.r.next() Error: newline inside string

Bin Cui added a comment - 23/Apr/13 1:08 PM Please attach the generated csv file and assign it back to me. I tested on centos and it works fine. So maybe it is data specific.

Shashank Gupta added a comment - 24/Apr/13 4:59 AM - edited csv file attached

Bin Cui added a comment - 24/Apr/13 12:55 PM http://review.couchbase.org/#/c/25879/ The csv file is not well formatted. Looks like the last line ends prematurely. Anyway, we need to handle such cases when processing csv file.

Maria McDuff added a comment - 29/Apr/13 2:26 PM pls verify / close.

Shashank Gupta added a comment - 30/Apr/13 12:13 AM Tried with 2.0.2-779-rel and gave the same command as I mentioned above, but it is still not working. I am attaching screenshot of the error. Please see.

Maria McDuff added a comment - 06/May/13 3:01 PM pls see shashank's comment

Bin Cui added a comment - 08/May/13 8:01 PM The problem is on the csv file. Please describe the detail steps how you populate data and generate csv file. And please attach the csv files that cause the problem.

Shashank Gupta added a comment - 09/May/13 1:34 AM Please find the required info below: [ root@caper-003 bin]# pwd /opt/couchbase/bin Used the following command to transfer the data into csv format from bucket to disk [ root@caper-003 bin]# ./cbtransfer http://localhost:8091 csv: > /data.csv -b default -u Administrator -p password   [####################] 100.0% (10000/10000 msgs) bucket: default, msgs transferred...        : total | last | per sec  batch : 1053 | 1053 | 550.8  byte : 4783385 | 4783385 | 2502156.4  msg : 10000 | 10000 | 5230.9 2013-05-08 23:26:45,107: mt warning: cannot save bucket design on a CSV destination done Then I gave the following command to store the csv data back to a different bucket: ./cbtransfer /data.csv http://localhost:8091 -B def -u Administrator -p password And after 9K out of 10K data items got transferred to bucket 'def', it gives the error as described in the screen-shot. Also attaching the csv file by the name 'new_data.csv' . Please verify.

Maria McDuff added a comment - 10/May/13 1:59 PM if stdout shld error out. Bin to fix and handle with error.

Bin Cui added a comment - 10/May/13 5:40 PM The right approach is: /cbtransfer http://localhost:8091 csv:./data.csv -b default -u Administrator -p password If you want to generate a csv output, you need to specify a filename with .csv as suffix. This will make sure that all data transferred completely to the external file. csv: > will use standard output and redirect to a file. Due to file buffering, no such final flush to put last line to the redirected file. And file content can be incomplete. We still keep standout as possible output though.

Maria McDuff added a comment - 13/May/13 11:13 PM shashank, pls update with how it is now being handled in the latest build.

Shashank Gupta added a comment - 14/May/13 1:27 AM Working fine if I don't use the > operator. Verified. Build used : 2.0.2-788

Anil Kumar added a comment - 20/May/13 12:35 PM Assigning this to Karen for documentation.