Couchbase is fetching all data when using Stream From as from Beginning in structured streaming is there any way we can filter ids or provide query instead of loading all data.
Spark Structured Streaming is built atop our DCP Protocol, which is designed to give you a stream of deduplicated mutations. It doesn’t support pushing down filters or queries, though you’re free to filter in your code, of course.
Spark Structured Streaming is really only intended for a few use-cases. Most users are going to want to use Spark SQL, or possibly raw RDDs. This will give you the option to e.g. run SQL++ queries, or fetch some specific keys via KV.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.