[JCBC-198] Using ReplicateTo.ONE after node-failover leads to index out of bounds

Wed, 27 Feb 2013 13:46:03 -0600

using couchbase client library version 1.1.0, server version 2.0.0 community edition (build-1976-rel).
The cluster configuration remains same as in my initial email - three servers with "Enable Failover" property ON, a dedicated port "couchbase" bucket with 1 replica enabled.

When stopping a server the client fails to communicate with the cluster with several errors.

Client initializing -

List<URI> serverList = parseConnectionProperties(port, hosts);
CouchbaseConnectionFactory cf = new CouchbaseConnectionFactory(serverList, cacheName, "");
client = new CouchbaseClient(cf);

Usage of the client to write data -

public void put(final String key, final String value) {
client.set(key, expirationSeconds, value, PersistTo.MASTER, ReplicateTo.ONE);
}

During the tests, when cluster is not available, I did a thread dump to the application , see the print below (ajp--127.0.0.1-8020-1@8796). Is it possible that node that is down is in some way a "master" of the data, and since the client.set() method uses PersistTo.MASTER parameter the things do not work?

"ajp--127.0.0.1-8020-1@8796" daemon prio=5 tid=0x8c nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1650)
at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1750)
at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1199)
at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
at java.lang.Thread.run(Thread.java:722)

The errors i get during the test after a server was failed over -

17:45:41,911 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-4] Verify Sso Login failed. sessionId - n2b+0EZEzTaM7a64gKKS7k+L.undefined [com.liveperson.liveEngage.ssoIdpLogin.LoginVerification]
17:45:41,911 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.RuntimeException: Timed out waiting for operation
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:134)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1188)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
17:45:41,912 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:45:41,913 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
17:45:41,914 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
17:45:41,915 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
17:45:41,916 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
17:45:41,917 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)
17:45:41,917 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) Caused by: net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: tlv-le-couchbase-int1.tlv.lpnet.com/192.168.24.184:11210
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:159)
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:132)
17:45:41,918 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) ... 28 more

OR

19:21:47,997 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-3] Verify Sso Login failed. sessionId - 6tJElgOsTUcr-7lWgJlWh2HV.undefined [com.liveperson.liveEngage.ssoIdpLogin.LoginVerification]
19:21:47,997 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.ArrayIndexOutOfBoundsException: -1
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.elementData(ArrayList.java:371)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.get(ArrayList.java:384)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.config.DefaultConfig.getServer(DefaultConfig.java:81)
19:21:47,998 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.VBucketNodeLocator.getServerByIndex(VBucketNodeLocator.java:112)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1621)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1750)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1199)
19:21:47,999 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:40)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.put(CouchbaseReadWriteCache.java:17)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.setSessionDataCache(LEUsersCacheManager.java:121)
19:21:48,000 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.writeLoginDataToJbossCache(LoginVerification.java:246)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.ssoIdpLogin.LoginVerification.verifySsoLogin(LoginVerification.java:87)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:71)
19:21:48,001 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
19:21:48,002 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
19:21:48,003 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
19:21:48,004 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
19:21:48,005 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
19:21:48,006 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)

Hopefully it's the same one, but just in case, here's another one:
17:41:34,231 ERROR [stderr] (Couchbase View Thread for node tlv-le-couchbase-int2.tlv.lpnet.com/192.168.24.185:8092) 2012-12-10 17:41:34.231 INFO com.couchbase.client.ViewNode: I/O reactor terminated for tlv-le-couchbase-int2.tlv.lpnet.com
17:42:23,120 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) [ERROR] [ajp--127.0.0.1-8020-2] Exception while trying to remove data from cache for session id: y6mpopojZkaJDcWeRDL4J0xp.undefined [org.apache.jsp.views.logout_jsp]
17:42:23,121 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
17:42:23,121 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.rangeCheck(ArrayList.java:604)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.util.ArrayList.get(ArrayList.java:382)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.config.DefaultConfig.getServer(DefaultConfig.java:81)
17:42:23,122 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.vbucket.VBucketNodeLocator.getServerByIndex(VBucketNodeLocator.java:112)
17:42:23,123 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observe(CouchbaseClient.java:1624)
17:42:23,123 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1753)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.couchbase.client.CouchbaseClient.delete(CouchbaseClient.java:1113)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.remove(CouchbaseReadWriteCache.java:28)
17:42:23,124 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.CouchbaseReadWriteCache.remove(CouchbaseReadWriteCache.java:17)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.cache.LEUsersCacheManager.removeDataFromCache(LEUsersCacheManager.java:132)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jsp.views.logout_jsp._jspService(logout_jsp.java:82)
17:42:23,125 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
17:42:23,126 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:326)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:253)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
17:42:23,127 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.ExceptionFilter.doFilter(ExceptionFilter.java:26)
17:42:23,128 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,129 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,129 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.SSOAuthorizationFilterChain.doFilter(SSOAuthorizationFilterChain.java:30)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.AuthFilter.doFilter(AuthFilter.java:65)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveengage.authentication.filters.AuthenticationByProtocolFilter.doFilter(AuthenticationByProtocolFilter.java:71)
17:42:23,130 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,131 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,131 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at com.liveperson.liveEngage.filters.CommonFilter.doFilter(CommonFilter.java:35)
17:42:23,132 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280)
17:42:23,132 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
17:42:23,133 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:125)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
17:42:23,134 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.session.LockingValve.invoke(LockingValve.java:56)
17:42:23,135 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
17:42:23,136 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:490)
17:42:23,137 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:480)
17:42:23,138 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
17:42:23,140 INFO [stdout] (AsyncAppender-Dispatcher-Thread-56) at java.lang.Thread.run(Thread.java:722)