Have you considered storing the meta-data which would include the location, id, and access information necessary to retrieve and stream this video in Couchbase yet store and maintain the video file itself in a streaming-optimized cloud-service?
Let Couchbase store and maintain how to fetch the video resource and let an AWS or Azure cloud-based service manage the streaming. Best of both worlds.
The max files size remains 20 mb. If you are merely storing the meta-data and identifying information in Couchbase and the actual media data in a media-streaming cloud service, the max file size is not an issue.
There are a number of considerations one weighs when modeling data stored in a distributed architecture like Couchbase. A recommended minimum node count in a cluster is 3. This is to establish a quorum to resolve conflicts in the case of simultaneous updates. The 3rd node serves as a tie-breaker when deciding which request is considered “last in” and stored.
At any rate, each document is stored and replicated across the network, node-to-node and memory first. Unnecessarily large data-objects could really hurt your responsiveness and performance.
My recommendation is NOT to store large objects of any kind in Couchbase or any database technology, regardless of whether they will fit or not.
An old project which might show some of what you’re looking for is in cbfs. It’s not officially supported in any way. IIRC, Couchbase was mainly a metadata store for it, but it has some of the same ideas in chunking files.
Was curious about this and tested with some data sets. It appears like couchbase accepts files larger than 20MB and logs a warning with OverThresholdRequestsRecordedEvent tag. At what size will couchbase reject the input?
I’m surprised to hear that. I ran a little test just now and confirmed Couchbase Server 7.0.0 rejects values larger than 20 mebibytes (20,971,520 bytes). Can you double-check the results of your experiment?
The OverThresholdRequestsRecordedEvent message is about some requests taking longer to execute than others; it’s referring to a time threshold, not a size threshold.