At Couchbase we take performance very seriously, and with the launch of our new product, Couchbase Autonomous Operator 1.0, we wanted to make sure it’s Enterprise-grade and production ready for customers.
In this blog post we will discuss the detailed performance results from running YCSB Performance Benchmark tests on Couchbase Server 5.5 using the Autonomous Operator to deploy on Kubernetes platform. One of the big concerns for Enterprises planning to run database on Kubernetes is ‘performance’ .
This document gives a quick comparison of two workloads, namely YCSB A & E with Couchbase Server 5.5 on Kubernetes vs bare metal.
YCSB Workload A: This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.
Workload E: Short ranges: In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).
In general, we observed no significant performance degradation in running Couchbase Cluster on Kubernetes, Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation.
Configuração:
For the setup, Couchbase was installed using the Operator deployment as stated below. For more details on the setup, please refer aqui
Files:
Operator deployment: deployment.yaml (See Appendix)
Couchbase deployment: couchbase-cluster-simple-selector.yaml (See Appendix)
Client / workload generator deployment: pillowfight-ycsb.yaml (See Appendix) (Official pillowfight docker image from dockerhub and installed java and YCSB manually on top of it)
Hardware:
7 servers
24 CPU x 64GB RAM per server
Couchbase setup
4 servers: 2 data nodes, 2 index+query nodes
40GB RAM quota for data service
40GB RAM quota for index services
1 data/bucket replica
1 primary index replica
Testes:
YCSB WorkloadA and WorkloadE
10M docs
Workflow after new empty k8s cluster is initialized on 7 servers:
|
1 2 3 4 5 6 7 8 9 |
# assign labels to the nodes so all services/pods will be assigned to right servers: kubectl rótulo nós arke06-sa09 tipo=power kubectl rótulo nós arke07-sa10 tipo=cliente kubectl rótulo nós ark08-sa11 tipo=cliente kubectl rótulo nós arke01-sa04 tipo=kv kubectl rótulo nós arke00-sa03 tipo=kv kubectl rótulo nós arke02-sa05 tipo=kv kubectl rótulo nós arke03-sa06 tipo=kv |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
#deploy Operator: kubectl criar -f implantação.yaml #deploy Couchbase kubectl criar -f couchbase-agrupamento-simples-seletor.yaml #deploy Client(s): kubectl criar -f briga de travesseiros-ycsb.yaml I correu meu testes diretamente de o cliente nó por registro em o doca imagem de o cliente cápsula: doca executar -ele --usuário raiz <briga de travesseiros-yscb contêiner id> bash E installing YCSB ambiente lá manualmente: apto-obter atualização apto-obter atualização apto-obter instalar -y software-propriedades-comum apto-obter instalar python sudo apto-adicionar-repositório ppa:webupd8team/java sudo apto-obter atualização sudo apto-obter instalar oráculo-java8-instalador exportação JAVA_HOME=/usr/lib/jvm/java-8-oráculo cd /optar wget http://download.nextag.com/apache/mentor/mentor-3/3.5.4/binários/apache-mentor-3.5.4-caixa.tar.gz sudo alcatrão -xvzf apache-mentor-3.5.4-caixa.tar.gz exportação M2_HOME="/opt/apache-maven-3.5.4" exportação PATH=$PATH:/optar/apache-mentor-3.5.4/caixa sudo atualização-alternatives --instalar "/usr/bin/mvn" "mvn" "/opt/apache-maven-3.5.4/bin/mvn" 0 sudo atualização-alternatives --definir mvn /optar/apache-mentor-3.5.4/caixa/mvn git clone http://github.com/couchbaselabs/YCSB |
Running the workloads:
|
1 2 3 4 5 6 7 8 9 10 11 |
Exemplos de YCSB comandos usado em este exercise: Carga de trabalho A Carga: ./caixa/ycsb carregar couchbase2 -P cargas de trabalho/workloade -p couchbase.password=senha -p couchbase.host=10.44.0.2 -p couchbase.bucket=padrão -p couchbase.upsert=verdadeiro -p couchbase.epoll=verdadeiro -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=nenhum -p writeallfields=verdadeiro -p recordcount=10000000 -fios 50 -p maxexecutiontime=3600 -p operationcount=1000000000 Executar: ./caixa/ycsb executar couchbase2 -P cargas de trabalho/workloada -p couchbase.password=senha -p couchbase.host=10.44.0.2 -p couchbase.bucket=padrão -p couchbase.upsert=verdadeiro -p couchbase.epoll=verdadeiro -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicateTo=0 -p couchbase.sslMode=nenhum -p writeallfields=verdadeiro -p recordcount=10000000 -fios 50 -p operationcount=1000000000 -p maxexecutiontime=600 -p exportfile=ycsb_workloadA_22vCPU.log |
Test results:
| Env | Direct setup | Kubernetes pod resources | Teste | Bare metal | Kubernetes | Delta |
| Env 1 | 22 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 22000m = ~22vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadA
50/50 get/upsert |
Throughput: 194,158 req/sec
CPU usage avg: 86% of all 22 cores |
Throughput: 192,190 req/sec
CPU usage avg: 94% of the cpu quota |
– 1% |
| Env 2 | 16 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 16000m = ~16vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadA
50/50 get/upsert |
Throughput: 141,909 req/sec
CPU usage avg: 89% of all 16 cores |
Throughput: 145,430 req/sec
CPU usage avg: 100% of the cpu quota |
+ 2.5% |
|
1 2 3 4 5 6 7 8 9 |
Carga de trabalho E: Carga: ./caixa/ycsb carregar couchbase2 -P cargas de trabalho/workloade -p couchbase.senha=senha -p couchbase.hospedeiro=10.44.0.2 -p couchbase.balde=padrão -p couchbase.upsert=verdadeiro -p couchbase.epoll=verdadeiro -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicarPara=0 -p couchbase.sslMode=nenhum -p writeallfields=verdadeiro -p recordcount=10000000 -fios 50 -p maxexecutiontime=3600 -p operationcount=1000000000 Executar: ./caixa/ycsb executar couchbase2 -P cargas de trabalho/workloade -p couchbase.senha=senha -p couchbase.hospedeiro=10.44.0.2 -p couchbase.balde=padrão -p couchbase.upsert=verdadeiro -p couchbase.epoll=verdadeiro -p couchbase.boost=48 -p couchbase.persistTo=0 -p couchbase.replicarPara=0 -p couchbase.sslMode=nenhum -p writeallfields=verdadeiro -p recordcount=10000000 -fios 50 -p operationcount=1000000000 -p maxexecutiontime=600 -p exportfile=ycsb_workloadE_22vCPU.registro |
| Env | Direct setup | Kubernetes pod resources | Teste | Bare metal | Kubernetes | Delta |
| Env 1 | 22 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 22000m = ~22vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadE
95/5 scan/insert |
Throughput: 15,823 req/sec
CPU usage avg: 85% of all 22 cores |
Throughput: 14,281 req/sec
CPU usage avg: 87% of the cpu quota |
– 9.7% |
| Env 2 | 16 vCPU, 48 GB RAM
(cpu cores and RAM available are set on OS core level) |
Limit to:
cpu: 16000m = ~16vCPU mem: 48GB All pods are on dedicated nodes |
WorkloadE
95/5 scan/insert |
Throughput: 13,014 req/sec
CPU usage avg: 91% of all 16 cores |
Throughput: 12,579 req/sec
CPU usage avg: 100% of the cpu quota |
– 3.3% |
Conclusões:
Couchbase Server 5.5 is production ready to be deployed on Kubernetes with the Autonomous Operator. Performance of Couchbase Server 5.5 on Kubernetes comparable to running on bare metal. There is little performance penalty in running Couchbase Server on Kubernetes platform. Looking at the results Workload A had on par performance compared to bare metal and Workload E had approximately less than 10% degradation.
Referências:
- YCSB Workloads https://github.com/brianfrankcooper/YCSB/wiki/Core-Workloads
- Couchbase Kubernetes page https://www.couchbase.com/products/cloud/kubernetes
- Download Couchbase Autonomous Operator https://www.couchbase.com/downloads
- Introducing Couchbase Operator https://www.couchbase.com/blog/couchbase-autonomous-operator-1-0-for-kubernetes-and-openshift/
Apêndice
My deployment.yaml file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 |
Versão da API: extensões/v1beta1 gentil: Implantação metadados: nome: couchbase-operador especificação: réplicas: 1 modelo: metadados: rótulos: nome: couchbase-operador especificação: nodeSelector: tipo: power contêineres: - nome: couchbase-operador imagem: couchbase/couchbase-operador-interno:1.0.0-292 comando: - couchbase-operador # Remove the arguments section if you are installing the CRD manually argumentos: - -criar-crd - -ativar-atualizações=falso env: - nome: MY_POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadados.espaço de nome - nome: MY_POD_NOME valueFrom: fieldRef: fieldPath: metadados.nome portos: - nome: readiness-porto containerPort: 8080 readinessProbe: httpGet: caminho: /readyz porto: readiness-porto initialDelaySeconds: 3 periodSeconds: 3 failureThreshold: 19 |
My couchbase-cluster-simple-selector.yaml file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
Versão da API: couchbase.banco de dados.couchbase.com/v1 gentil: CouchbaseCluster metadados: nome: cb-exemplo especificação: imagem de base: couchbase/servidor versão: empresa-5.5.0 authSecret: cb-exemplo-autenticação exposeAdminConsole: verdadeiro antiAffinity: verdadeiro exposedFeatures: - xdcr agrupamento: DataServiceMemoryQuota: 40000 indexServiceMemoryQuota: 40000 searchServiceMemoryQuota: 1000 eventingServiceMemoryQuota: 1024 analyticsServiceMemoryQuota: 1024 indexStorageSetting: memória_otimizado autoFailoverTimeout: 120 autoFailoverMaxCount: 3 autoFailoverOnDataDiskIssues: verdadeiro autoFailoverOnDataDiskIssuesTimePeriod: 120 autoFailoverServerGroup: falso baldes: - nome: padrão tipo: couchbase memoryQuota: 20000 réplicas: 1 ioPrioridade: alta evictionPolicy (política de despejo): fullEviction Resolução de conflitos: seqno enableFlush: verdadeiro enableIndexReplica: falso servidores: - tamanho: 2 nome: dados serviços: - dados cápsula: nodeSelector: tipo: kv recursos: limites: CPU: 22000m memória: 48Gi solicitações: CPU: 22000m memória: 48Gi - tamanho: 2 nome: qi serviços: - índice - consulta cápsula: nodeSelector: tipo: kv recursos: limites: CPU: 22000m memória: 48Gi solicitações: CPU: 22000m memória: 48Gi |
My pillowfight-ycsb.yaml file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
Versão da API: lote/v1 gentil: Trabalho metadados: nome: briga de travesseiros especificação: modelo: metadados: nome: briga de travesseiros especificação: contêineres: - nome: briga de travesseiros imagem: sequoiatools/briga de travesseiros:v5.0.1 comando: ["sh", "-c", "tail -f /dev/null"] restartPolicy: Nunca nodeSelector: tipo: cliente |
Dear Raju,
It’s a nice blog.
Please update it a bit, so that it’s easier to follow along.
Two comments:
Comment#1:
sequoiatools/pillowfight:v5.0.1 seems to be built on coreos and hence doesn’t have a package manager specifically it doesn’t have apt-get.
Comment#2
Below steps is not working now as there’s some change in Oracle Java license agreement.
sudo apt-add-repository ppa:webupd8team/java
-Thanks
I installed JDK 11.
I got below error when using “-p couchbase.epoll=true”
Loading workload…
Starting test.
Maximum execution time specified as: 120 secs
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0 (file:/opt/ycsb-couchbase2-binding-0.15.0/lib/core-io-1.3.1.jar) to field java.nio.Buffer.address
WARNING: Please consider reporting this to the maintainers of com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0
WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Aug 23, 2019 9:15:34 AM com.couchbase.client.deps.io.netty.util.internal.PlatformDependent
INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system unstability.
com.yahoo.ycsb.DBException: Could not connect to Couchbase Bucket.
at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:208)
at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:86)
at com.yahoo.ycsb.ClientThread.run(Client.java:424)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: failed to create a child event loop
at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:68)
at com.couchbase.client.deps.io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:49)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:91)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:67)
at com.yahoo.ycsb.db.couchbase2.Couchbase2Client.init(Couchbase2Client.java:195)
… 3 more
Caused by: java.lang.NullPointerException
at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent0.allocateMemory(PlatformDependent0.java:330)
at com.couchbase.client.deps.io.netty.util.internal.PlatformDependent.allocateMemory(PlatformDependent.java:210)
at com.couchbase.client.deps.io.netty.channel.epoll.IovArray.(IovArray.java:64)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:60)
at com.couchbase.client.deps.io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:106)
at com.couchbase.client.deps.io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:64)
… 7 more