Hi ,
We have web scraping application exists to scrap the data (through mutual agreement) using Perl and so far we are storing this data to MySQL Database and pushing this data to Oracle table for transformations and ETL jobs and finally preparing for reporting .
Is there any way if we can ingest the data directly from scrap programming Perl to Couchbase through any method . For realtime access of this data we are currently using REST API to call Perl program to scrap this data and put into MYSQL and Oracle but anyway to get this data directly to Couchbase from scrap ? The reason is, we can ask RESTAPI to return the data in JSON which could be directly used to store in CB . Trying to avoid layers for the ETL and if possible we can do some data transformations inside CB buckets before preparing it ready for UI to consume.
Any suggestions and advice would be appreciated .
thanks