Issue with couchbaseTemplate.removeByQuery

RosarioB · May 24, 2023, 10:32pm

Hi everyone,
I have an issue with the method couchbaseTemplate.deleteByQuery.

TransactionOptions transactionOptions = TransactionOptions.transactionOptions().durabilityLevel(DurabilityLevel.NONE);
Transaction transactions = couchbaseTemplate.getCouchbaseClientFactory().getCluster().transactions();
        transactions.run(ctx -> couchbaseTemplate.removeByQuery(Customer.class)
                .withConsistency(QueryScanConsistency.REQUEST_PLUS)
                .all(),  transactionOptions);
    }

This is the error I get :
The server reported an issue with the underlying index "Keyspace not found in CB datastore: default:customers - cause: No bucket named customers

I don’t understand why he is looking for a customer bucket, my bucket is bucket1. If i simply use repository.deleteAll() without a transaction it works. If I call couchbaseTemplate.getBucketName() it yields correctly bucket1. Thank you

mreiche · May 24, 2023, 10:47pm

Hi Rosario -

First make sure you are running the latest version of spring-data-couchbase. The issue you have looks exactly like scope and collection annotations on repository are not considered during execution. · Issue #1441 · spring-projects/spring-data-couchbase · GitHub

It’s difficult to determine exactly what occurred without your configuration class, the repository class and the entity class (Customer). There are a number of possibilities - they are outlined in the various links I provided in your other post - especially those on scopes and collections.

A ‘keyspace’ as mentioned in the error message is either a bucket (when not using scopes and collections), or a collection (when using scopes and collections).

At the bottom of the issue is a query like “delete from customer …” being sent to the server without a query_context (which would specify the bucket and collection). So it appears that somewhere you have defined an @Collection(“customer”) , and the query is being constructed with that keyspace - but the corresponding query_context (which specifies the bucket and scope) is not being sent - because your Configuration does not specify a scope (@Override getScopeName())

RosarioB · May 28, 2023, 5:25pm

Hi Michael,

thank you very much for your reply. I have uploaded the project on github. You can find the class CustomerServiceImpl here.

In the entity class Customer I have set the scope and the collection with the annotations @Scope and @Collection. All methods of the class CustomerServiceImpl are working fine except deleteAll, which raises this exception:

RosarioB · May 28, 2023, 5:27pm

Transaction has failed with cause ‘com.couchbase.client.core.error.IndexFailureException: The server reported an issue with the underlying index {“completed”:true,“coreId”:“0x38b17e9300000001”,“errors”:[{“code”:12003,“message”:“Keyspace not found in CB datastore: default:customers - cause: No bucket named customers”,“retry”:false}]’

Unfortunately I cannot upload the whole stack trace because I have limitations on the characters in each post.

Clearly i am doing something wrong but I could not be able to find the issue here.

mreiche · May 30, 2023, 4:29am

I’ll look at it tomorrow. Today’s a holiday.

graham.pople · May 30, 2023, 10:13am

Hi @RosarioB , this is unrelated to your problem but just to comment on this part:

TransactionOptions transactionOptions = TransactionOptions.transactionOptions().durabilityLevel(DurabilityLevel.NONE);

DurabilityLevel.NONE is an unsupported transactions config that won’t provide ACID guarantees. If you’re applying it because you hit a DurabilityImpossibleException against a single node cluster, please see the notice near the top of Using Couchbase Transactions | Couchbase Docs for how to solve this.

mreiche · May 30, 2023, 5:42pm

public class CustomerRepositoryImpl implements CustomerRepository {

Maybe not related to the issue, but do not implement the repository class. Spring Data will provide the implementation. Please refer to any of the Spring Data Couchbase (or even generic Spring Data) examples.

Also - in the test, the application doesn’t load -

java.lang.IllegalStateException: Failed to load ApplicationContext for [MergedContextConfiguration@5b051a5c testClass = com.rosariob.crud.couchbase.service.CustomerServiceTest, locations = [], classes = [com.rosariob.crud.couchbase.service.CustomerServiceTestConfig], contextInitializerClasses = [], activeProfiles = [], propertySourceLocations = [], propertySourceProperties = [], contextCustomizers = [org.springframework.boot.test.context.filter.ExcludeFilterContextCustomizer@1f75a668, org.springframework.boot.test.json.DuplicateJsonObjectContextCustomizerFactory$DuplicateJsonObjectContextCustomizer@4e31276e, org.springframework.boot.test.mock.mockito.MockitoContextCustomizer@88cf1c9d, org.springframework.boot.test.autoconfigure.actuate.observability.ObservabilityContextCustomizerFactory$DisableObservabilityContextCustomizer@9da1, org.springframework.boot.test.autoconfigure.properties.PropertyMappingContextCustomizer@0, org.springframework.boot.test.autoconfigure.web.servlet.WebDriverContextCustomizerFactory$Customizer@3148f668], contextLoader = org.springframework.test.context.support.DelegatingSmartContextLoader, parent = null]

	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:142)
	at org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:127)
	at org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener.postProcessFields(MockitoTestExecutionListener.java:110)
	at org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener.injectFields(MockitoTestExecutionListener.java:94)
	at org.springframework.boot.test.mock.mockito.MockitoTestExecutionListener.prepareTestInstance(MockitoTestExecutionListener.java:61)
	at org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:241)
	at org.springframework.test.context.junit.jupiter.SpringExtension.postProcessTestInstance(SpringExtension.java:138)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeTestInstancePostProcessors$10(ClassBasedTestDescriptor.java:377)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.executeAndMaskThrowable(ClassBasedTestDescriptor.java:382)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeTestInstancePostProcessors$11(ClassBasedTestDescriptor.java:377)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeTestInstancePostProcessors(ClassBasedTestDescriptor.java:376)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$instantiateAndPostProcessTestInstance$6(ClassBasedTestDescriptor.java:289)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.instantiateAndPostProcessTestInstance(ClassBasedTestDescriptor.java:288)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$4(ClassBasedTestDescriptor.java:278)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$testInstancesProvider$5(ClassBasedTestDescriptor.java:277)
	at org.junit.jupiter.engine.execution.TestInstancesProvider.getTestInstances(TestInstancesProvider.java:31)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$prepare$0(TestMethodTestDescriptor.java:105)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:104)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.prepare(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$prepare$2(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.prepare(NodeTestTask.java:123)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:90)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'customerService': Unsatisfied dependency expressed through field 'couchbaseTemplate': No qualifying bean of type 'org.springframework.data.couchbase.core.CouchbaseTemplate' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:713)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:693)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:133)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:482)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1416)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:597)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:917)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:584)
	at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:221)
	at org.springframework.test.context.support.AbstractGenericContextLoader.loadContext(AbstractGenericContextLoader.java:110)
	at org.springframework.test.context.support.AbstractDelegatingSmartContextLoader.loadContext(AbstractDelegatingSmartContextLoader.java:212)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:184)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:118)
	... 73 more
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.data.couchbase.core.CouchbaseTemplate' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1824)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1383)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1337)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:710)
	... 91 more

Don’t mix-and-match the Couchbase Java SDK with Spring Data Couchbase. Spring Data Couchbase augments the document being saved with a _class property, and then augments the queries with a corresponding predicate (_class = <classname>) When you insert with the Couchbase Java SDK, the stored documents do not have the _class property, and Spring Data Couchbase queries do not find them.
```
 Transactions transactions = couchbaseTemplate.getCouchbaseClientFactory().getCluster().transactions();
 customerList.forEach(customer -> transactions.run(ctx ->
         ctx.insert(collection, customer.getId(), customer))
 );
 customerRepository.deleteAll(); // not going to find the customers from customerList.
```

Fix: user the template or repository methods to insert data

When documents are inserted into couchbase, they are indexed synchronously. So there will be a short period following the insert that the documents will not be found by a query. There is a query Option QueryScanConsistency.REQUEST_PLUS) that will block the query until the indexing of prior changes have been complete.

Fix: use couchbaseTemplate.deleteByQuery(Customer.class).withConsistency(QueryScanConsistency.REQUEST_ALL).all()

mreiche · May 30, 2023, 6:43pm

I see the issue. Opened removeByQuery - possible others - don't use queryContext for _default scope · Issue #1745 · spring-projects/spring-data-couchbase · GitHub

RosarioB · May 30, 2023, 10:13pm

Thank you very much guys!

I have updated the modified repository here . I have extended ListCrudRepository and fixed the CustomerRepositoryTest by using only the repository methods.

Unfortunately all repository methods seem to work except deleteById and deleteAll. The tests work fine but if I try the API with postman I have some problems.

If I try DELETE on http://localhost:8080/api/customers it gives me the same error as before:

Keyspace not found in CB datastore: default:customers - cause: No bucket named customers",“retry”:false

If I try DELETE on http://localhost:8080/api/customers/id_customer I get

Transaction has failed with cause 'java.lang.IllegalArgumentException: cas must be supplied for tx remove

I haven’t figured out how to pass the cas in this case…

mreiche · May 30, 2023, 11:19pm

deleteAll() won’t work in a transaction because of issue #1745.

deleteById() needs a cas. So you need to pass the whole cusomter entity since it contains the cas. (at least it should - in an @Version property) then -

deleteById(Customer.class).oneEntity(customer).

So when you call in Postman, you first have to fetch the customer so you have the cas. (it’s actually in the metadata of the doc, but spring data couchbase copies it into the @Version property in the entity).