Couchbase Server

View query on Rebalance.Out fails with Reason: A view spec can not consist of merges exclusively.

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    4 2.0.0 Machines, running on Linux.

Description

I've been running SDKQE stester runs for the Java SDK, specifically testing the following scenario:

/stester -C 127.0.0.1:8050 -i 20devcluster.ini -c rebalance.Once --vdsw_dvname ddoc/vquery --rbcount 2 --dsw_timeres 1 --hdsw_mc_threads 10 --workload dsw.Hybrid --mode out --hdsw_http_threads 5 --hdsw_cb_threads 10 --rebound 90 -d -o rebalance-once.log

My cluster is a 4 node cluster, and this scenario rebalances 2 nodes out of the cluster. During the rebalance, view queries are issued.

Now it happens that during this rebalance (I assume this happens at the end of the rebalance) I get the following Exception:

java.lang.RuntimeException: Failed to access the view
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:838)
at com.couchbase.sdkd.cbclient.ViewQueryCommandContext.execIter(ViewQueryCommandContext.java:252)
at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: error Reason: A view spec can not consist of merges exclusively.
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:89)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:73)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:63)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:834)
... 5 more
Caused by: OperationException: SERVER: error Reason: A view spec can not consist of merges exclusively.
at com.couchbase.client.protocol.views.NoDocsOperationImpl.parseError(NoDocsOperationImpl.java:106)
at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:68)
at com.couchbase.client.ViewNode$MyHttpRequestExecutionHandler.handleResponse(ViewNode.java:199)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.processResponse(AsyncNHttpClientHandler.java:417)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.inputReady(AsyncNHttpClientHandler.java:242)
at com.couchbase.client.http.AsyncConnectionManager$ManagedClientHandler.inputReady(AsyncConnectionManager.java:244)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:172)
at org.apache.http.impl.nio.DefaultClientIOEventDispatch.inputReady(DefaultClientIOEventDispatch.java:155)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:161)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:335)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:275)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:542)
at java.lang.Thread.run(Thread.java:680)
Feb 1, 2013 1:04:30 PM com.couchbase.sdkd.cbclient.CommandResult warnAbout
WARNING: Unknown exception encountered (for operation) future warnings will be suppressed
java.lang.RuntimeException: Failed to access the view
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:838)
at com.couchbase.sdkd.cbclient.ViewQueryCommandContext.execIter(ViewQueryCommandContext.java:252)
at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311)
at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135)
at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156)
at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212)
Caused by: java.util.concurrent.ExecutionException: OperationException: SERVER: no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets
at com.couchbase.client.internal.HttpFuture.waitForAndCheckOperation(HttpFuture.java:89)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:73)
at com.couchbase.client.internal.HttpFuture.get(HttpFuture.java:63)
at com.couchbase.client.CouchbaseClient.query(CouchbaseClient.java:834)
... 5 more
Caused by: OperationException: SERVER: no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets
at com.couchbase.client.protocol.views.NoDocsOperationImpl.parseError(NoDocsOperationImpl.java:106)
at com.couchbase.client.protocol.views.ViewOperationImpl.handleResponse(ViewOperationImpl.java:68)
at com.couchbase.client.ViewNode$MyHttpRequestExecutionHandler.handleResponse(ViewNode.java:199)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.processResponse(AsyncNHttpClientHandler.java:417)
at org.apache.http.nio.protocol.AsyncNHttpClientHandler.inputReady(AsyncNHttpClientHandler.java:242)
at com.couchbase.client.http.AsyncConnectionManager$ManagedClientHandler.inputReady(AsyncConnectionManager.java:244)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:172)
at org.apache.http.impl.nio.DefaultClientIOEventDispatch.inputReady(DefaultClientIOEventDispatch.java:155)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:161)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:335)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:275)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:542)

Note that the Java stuff is not of particular interest for this ticket, but the Error messages are. First, I get

error Reason: A view spec can not consist of merges exclusively.
and then
 no_active_vbuckets Reason: Cannot execute view query since the node has no active vbuckets

The way the Java client currently implements is that it will remove the ViewNode during rebalance when it vanishes from the couchNodes list. I'll attach a rebalance log from one of the nodes that is getting rebalanced out, but as expected this happens at the very last step, its still in the list when it has no vbuckets anymore.

Should the server not be able to handle view requests, even when it has no vbuckets attached?

Thanks,
Michael

Issue Links

blocks

Bug - A problem which impairs or prevents the functions of the product. JCBC-189 Views having odd timeout issues on some clusters

  • Major - Major loss of function.
  • In Progress - This issue is being actively worked on at the moment by the assignee.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Michael Nitschinger added a comment -
For reference, IP-Addresses 192.168.56.101 - 104, 103 and 104 get removed. The log is from .104.
Show
Michael Nitschinger added a comment - For reference, IP-Addresses 192.168.56.101 - 104, 103 and 104 get removed. The log is from .104.
Hide
Permalink
Aleksey Kondratenko added a comment - - edited
This is known problem. Some duplicate ticket should exist somewhere. We cannot easily fix it. Perhaps best thing we can do is send you back redirect to a node that likely works.
Show
Aleksey Kondratenko added a comment - - edited This is known problem. Some duplicate ticket should exist somewhere. We cannot easily fix it. Perhaps best thing we can do is send you back redirect to a node that likely works.
Hide
Permalink
Matt Ingenthron added a comment -
Alk: but do we currently redirect, or do we currently error? It looks like we currently error.
Show
Matt Ingenthron added a comment - Alk: but do we currently redirect, or do we currently error? It looks like we currently error.
Hide
Permalink
Aleksey Kondratenko added a comment -
No. We don't redirect AFAIK.

You can try to mitigate on your side by either trying different node on any error, or by trying to detect this particular error.

I cannot promise you yet any particular release when we'll start redirecting.
Show
Aleksey Kondratenko added a comment - No. We don't redirect AFAIK. You can try to mitigate on your side by either trying different node on any error, or by trying to detect this particular error. I cannot promise you yet any particular release when we'll start redirecting.
Hide
Permalink
Aleksey Kondratenko added a comment -
Too late for 2.0.2 but IMHO must have for 2.1
Show
Aleksey Kondratenko added a comment - Too late for 2.0.2 but IMHO must have for 2.1

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Michael Nitschinger
Vote (0)
Watch (4)

Dates

  • Created:
    Updated: