Java Client Library - strange error under high loader
Wed, 01/09/2013 - 13:33
I've tried to bulk insert some data into my single node Couchbase Development Environement and after ~ 350k inserts I do always get the following error message:
(unfortunately I've to include it as an image, because of non-printable characters in error message)
Java-Code Snippet:
OperationFuture<Boolean> setOp = client.set(key, 0, this.serializer.serialize(entity)); if (!setOp.get().booleanValue()) { throw new RuntimeException("Set failed for key '"+key+"': " + setOp.getStatus().getMessage()); }
When I try to work with multiple CouchbaseClients in Parallel (one per Thread) I do get these strange error messages on all Threads at (nearly) the same time. Any help would be appreciated.
Fri, 01/11/2013 - 11:17
As the application in which I tried this has a pretty complex structure I'll try to extract into a more simple example as good as I can. Regarding your advice to only use one CouchbaseClient and share it amongst threads - I've tried both - using a single client and share and also using a Pool of Clients, but had the exact same issue with both approaches.
Fri, 01/11/2013 - 11:24
Okay please do so!
Just to note: when you have n clients you also have n-times the cluster management overhead which is not what you want! You open lots of more connections than actually needed - that may not be obvious right now but in larger deployments you may run into issues.
So just use one and share it. For more info on your implementation we'll see.
Thanks!
Fri, 01/11/2013 - 11:39
Here you go (I know the mult-threading I built in this sample is probably not the best way of doing it, but it's working and reproduces the issue) - below I'm also including the maven build file (it has way more dependencies than needed for this example, but for simplicity reasons I did copy paste it from my application)
I'm using the following Couchbase Version
Version: 2.0.0 community edition (build-1976)
import java.io.IOException; import java.net.URI; import java.text.DecimalFormat; import java.text.SimpleDateFormat; import java.util.LinkedList; import java.util.List; import java.util.concurrent.ExecutionException; import net.spy.memcached.internal.OperationFuture; import org.apache.log4j.Logger; import org.joda.time.DateMidnight; import com.couchbase.client.CouchbaseClient; public class Test { private static final Logger log = Logger.getLogger(Test.class); private static int cnt = 0; public static void main(String[] args) throws IOException { final int tenants = 26; final int per_tenant = 1000; final int mult = 250; final String json = "{\"@class\":\"com.rainer.hms.entity.ts.TimeSlice\",\"propertyIdentifier\":{\"@class\":\"com.rainer.hms.entity.PropertyIdentifier\",\"tenantId\":\"A000\",\"propertyId\":\"SZG\"},\"start\":[\"java.util.Date\",1388962800000],\"duration\":1440,\"details\":{\"@class\":\"java.util.HashMap\",\"COMP_TS\":{\"@class\":\"com.rainer.hms.entity.ts.TimeSliceDetail\",\"name\":\"COMP_TS\",\"qualifier\":null,\"details\":{\"@class\":\"java.util.HashMap\",\"R_SOLD\":0}},\"RP_RT_TS_FLEX_A1K\":{\"@class\":\"com.rainer.hms.entity.ts.TimeSliceDetail\",\"name\":\"RP_RT_TS\",\"qualifier\":{\"@class\":\"com.rainer.hms.entity.ts.id.RatePlanRoomTypeTimeSliceIdentifier\",\"ratePlan\":\"FLEX\",\"roomType\":\"A1K\"},\"details\":{\"@class\":\"java.util.HashMap\",\"CLOSED\":false,\"REQ_DOW\":\"1111111\",\"ARR_DOW\":\"1111111\",\"FPLOS\":\"1110111011\",\"RATE\":{\"@class\":\"com.rainer.hms.entity.ts.Rate\",\"rateDetails\":[\"java.util.ArrayList\",[{\"@class\":\"com.rainer.hms.entity.ts.RateDetail\",\"minLOS\":1,\"rateMode\":\"PerRoom\",\"rateAmount\":[\"java.math.BigDecimal\",89.00],\"rateTimeUnit\":\"Day\",\"rateTimeCount\":1,\"occupancyRateAmount\":null}]]},\"AVL_DOW\":\"1111111\",\"R_SOLD\":0,\"DEP_DOW\":\"1111111\"}},\"RT_TS_A1K\":{\"@class\":\"com.rainer.hms.entity.ts.TimeSliceDetail\",\"name\":\"RT_TS\",\"qualifier\":{\"@class\":\"com.rainer.hms.entity.ts.id.RoomTypeTimeSliceIdentifier\",\"roomType\":\"A1K\"},\"details\":{\"@class\":\"java.util.HashMap\",\"R_SOLD\":0,\"R_EXIST\":228}},\"RP_TS_FLEX\":{\"@class\":\"com.rainer.hms.entity.ts.TimeSliceDetail\",\"name\":\"RP_TS\",\"qualifier\":{\"@class\":\"com.rainer.hms.entity.ts.id.RatePlanTimeSliceIdentifier\",\"ratePlan\":\"FLEX\"},\"details\":{\"@class\":\"java.util.HashMap\",\"CLOSED\":false,\"FPLOS\":\"1110111011\",\"RATE\":{\"@class\":\"com.rainer.hms.entity.ts.Rate\",\"rateDetails\":[\"java.util.ArrayList\",[{\"@class\":\"com.rainer.hms.entity.ts.RateDetail\",\"minLOS\":1,\"rateMode\":\"PerRoom\",\"rateAmount\":[\"java.math.BigDecimal\",89.00],\"rateTimeUnit\":\"Day\",\"rateTimeCount\":1,\"occupancyRateAmount\":null}]]},\"R_SOLD\":0}}},\"key\":\"A000SZG20140106\"}"; // Set the URIs and get a client List<URI> uris = new LinkedList<URI>(); // Connect to localhost or to the appropriate URI(s) uris.add(URI.create("http://192.168.153.1:8091/pools")); final CouchbaseClient client = new CouchbaseClient(uris, "default", "");; for(int x=0;x<tenants;x++) { final int z=x; for(int v=0;v<(per_tenant/mult);v++) { final int offset = v*mult; Thread t = new Thread() { public void run() { SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd"); for(int y=offset;y<(offset+mult);y++) { String tenantId = getTenantId(z, y); System.out.println(tenantId); DateMidnight d = new DateMidnight(); for (int i = 0; i <= 365; i++) { String key = tenantId+"SZG"+sdf.format(d.toDate()); try { OperationFuture<Boolean> setOp = client.set(key, 0, json); if (!setOp.get().booleanValue()) { throw new RuntimeException("Set failed for key '"+key+"': " + setOp.getStatus().getMessage()); } } catch (InterruptedException e) { log.error("InterruptedException while doing set '"+key+"'",e); throw new RuntimeException(e); } catch (ExecutionException e) { log.error("ExecutionException while doing set '"+key+"'",e); throw new RuntimeException(e); } d = d.plusDays(1); } } synchronized (this) { cnt++; } } }; t.start(); } } while(cnt < (tenants*(per_tenant/mult))) { try { Thread.sleep(500); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } private static String getTenantId(int x, int r) { char start = (char) ('A'+x); DecimalFormat formatter = new java.text.DecimalFormat("000"); return start+formatter.format(r); } }
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 <a href="http://maven.apache.org/xsd/maven-4.0.0.xsd"><br />
" title="http://maven.apache.org/xsd/maven-4.0.0.xsd"><br />
">http://maven.apache.org/xsd/maven-4.0.0.xsd"><br />
</a> <modelVersion>4.0.0</modelVersion>
<groupId>org.rainer</groupId>
<artifactId>CouchbaseTest</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>war</packaging>
<properties>
<maven-datanucleus-plugin.version>3.0.1</maven-datanucleus-plugin.version>
<datanucleus.version>3.0.1</datanucleus.version>
<datanucleus-jpa.version>3.0.1</datanucleus-jpa.version>
<datanucleus-enhancer.version>3.0.1</datanucleus-enhancer.version>
<asm.version>4.1</asm.version>
</properties>
<build>
<extensions>
<extension>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>0.11.6</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.0</version>
<extensions>true</extensions>
<configuration>
<source>1.6</source>
<target>1.6</target>
<showDeprecation>true</showDeprecation>
<showWarnings>true</showWarnings>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>java.net</id>
<url>http://download.java.net/maven/2/</url>
</repository>
<repository>
<id>jboss-public-repository</id>
<url>http://repository.jboss.org/nexus/content/groups/public-jboss/</url>
</repository>
<repository>
<id>prime-repo</id>
<name>PrimeFaces Maven Repository</name>
<url>http://repository.primefaces.org</url>
<layout>default</layout>
</repository>
<repository>
<id>cludera-cdh4</id>
<name>CDH4 Maven Repository</name>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>ow2</id>
<name>OW2 Releases Maven Repository</name>
<url>http://repository.ow2.org/nexus/content/repositories/releases/</url>
</repository>
<repository>
<id>sonatype-nexus</id>
<name>Kundera Public Repository</name>
<url>https://oss.sonatype.org/content/repositories/releases</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>riptano</id>
<name>riptano</name>
<url>http://mvn.riptano.com/content/repositories/public</url>
</repository>
<repository>
<id>couchbase</id>
<name>Couchbase Maven Reposiotry</name>
<url>http://files.couchbase.com/maven2/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.jibx.schema.org.opentravel._2012A</groupId>
<artifactId>org.jibx.schema.org.opentravel._2012A.hotel</artifactId>
<version>1.0.8</version>
</dependency>
<dependency>
<groupId>org.jibx.schema.org.opentravel._2012A</groupId>
<artifactId>org.jibx.schema.org.opentravel._2012A.fastrez</artifactId>
<version>1.0.8</version>
</dependency>
<dependency>
<groupId>couchbase</groupId>
<artifactId>couchbase-client</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>cglib</groupId>
<artifactId>cglib</artifactId>
<version>2.2.2</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.1</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>0.20.2</version>
</dependency>
<dependency>
<groupId>com.spatial4j</groupId>
<artifactId>spatial4j</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>com.vividsolutions</groupId>
<artifactId>jts</artifactId>
<version>1.13</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>0.11.6</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.21</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-core</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-beans</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-aspects</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-tx</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-orm</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-expression</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-asm</artifactId>
<version>3.1.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>1.6.11</version>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjweaver</artifactId>
<version>1.6.11</version>
</dependency>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces</artifactId>
<version>3.4.2</version>
</dependency>
<dependency>
<groupId>org.primefaces</groupId>
<artifactId>primefaces-mobile</artifactId>
<version>0.9.3</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase</artifactId>
<version>0.92.1-cdh4.1.2</version>
<exclusions>
<exclusion>
<artifactId>jersey-server</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
<exclusion>
<artifactId>servlet-api-2.5</artifactId>
<groupId>org.mortbay.jetty</groupId>
</exclusion>
<exclusion>
<artifactId>jsp-api-2.1</artifactId>
<groupId>org.mortbay.jetty</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-cdh4.1.2</version>
<exclusions>
<exclusion>
<artifactId>
jersey-test-framework-grizzly2
</artifactId>
<groupId>
com.sun.jersey.jersey-test-framework
</groupId>
</exclusion>
<exclusion>
<artifactId>jersey-server</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
<exclusion>
<artifactId>jsp-api</artifactId>
<groupId>javax.servlet.jsp</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>2.0.0-cdh4.1.2</version>
<exclusions>
<exclusion>
<artifactId>jersey-server</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
<exclusion>
<artifactId>jsp-api</artifactId>
<groupId>javax.servlet.jsp</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.6.6</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.6.6</version>
</dependency>
<dependency>
<groupId>org.apache.solr</groupId>
<artifactId>solr-solrj</artifactId>
<version>3.5.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
<version>1.6</version>
<scope>system</scope>
<systemPath>${JAVA_HOME}/lib/tools.jar</systemPath>
</dependency>
<dependency>
<groupId>org.codehaus.groovy</groupId>
<artifactId>groovy-all</artifactId>
<version>2.0.6</version>
</dependency>
<dependency>
<groupId>hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo2-api</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>javax.persistence</groupId>
<artifactId>persistence-api</artifactId>
<version>LATEST</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-validator</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.jboss.seam.international</groupId>
<artifactId>seam-international</artifactId>
<version>3.1.0.Final</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.jboss.seam.international</groupId>
<artifactId>seam-international-api</artifactId>
<version>3.1.0.Final</version>
</dependency>
<dependency>
<groupId>org.jboss.seam</groupId>
<artifactId>seam-bom</artifactId>
<version>3.1.0.Final</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>javax.enterprise</groupId>
<artifactId>cdi-api</artifactId>
<version>LATEST</version>
<exclusions>
<exclusion>
<artifactId>el-api</artifactId>
<groupId>javax.el</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>javax.faces</groupId>
<artifactId>jsf-api</artifactId>
<version>2.1</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>javax.faces</artifactId>
<version>2.1.13</version>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>jstl</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>javax.transaction</groupId>
<artifactId>jta</artifactId>
<version>1.0.1B</version>
</dependency>
<dependency>
<groupId>javax.annotation</groupId>
<artifactId>jsr250-api</artifactId>
<version>1.0</version>
</dependency>
<dependency>
<groupId>javax</groupId>
<artifactId>javaee-api</artifactId>
<version>6.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
</project>
Tue, 01/15/2013 - 12:20
@daschl - I was wondering if you were able to reproduce the issue with the code snipped I've posted?
Can you please post your full code including the multithreaded part? Also make sure to use only one CouchbaseClient and share it amongst threads!