Error: failed to start container "pillowfight"

Couchbase Server Kubernetes
1

Hi all,

I’m just familiarizing myself with the new CouchBase K8s operator, and following along the instructions as per https://docs.couchbase.com/operator/1.1/install-kubernetes.html

I managed to successfully deploy the Operator and a CouchBase instance. However, I am having trouble with the sample “pillowfight” configuration for populating meaningful data.

When I roll out the pillowfight-data-loader.yaml, my containers fail as follows (pod describe output):

> Events:
>   Type     Reason     Age   From                                           Message
>   ----     ------     ----  ----                                           -------
>   Normal   Scheduled  15m   default-scheduler                              Successfully assigned default/pillowfight-cnt5z to 91920344-b3a7-4979-a100-c156db235b6d
>   Normal   Pulled     15m   kubelet, 91920344-b3a7-4979-a100-c156db235b6d  Container image "sequoiatools/pillowfight:v5.0.1" already present on machine
>   Normal   Created    15m   kubelet, 91920344-b3a7-4979-a100-c156db235b6d  Created container
>   Warning  Failed     15m   kubelet, 91920344-b3a7-4979-a100-c156db235b6d  Error: failed to start container "pillowfight": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "chdir to cwd (\"/root/libcouchbase/build/bin\") set in config.json failed: permission denied": unknown

Can anyone provide some guidance on how to address this issue?

2

Hi @cormachogan,

Did you follow the steps mentioned here Loading Data with Pillowfight to load the data?

Once you create - kubectl create -f pillowfight-data-loader.yaml you need to run

cbc-pillowfight -U couchbase://localhost/default -u Administrator -P password -J -t 4 -I 9999 -p hostname

Thanks!

3

Hi Anil,

Yes - this is embedded in the YAML, as shown here:

apiVersion: batch/v1
kind: Job
metadata:
  name: pillowfight
spec:
  template:
    metadata:
      name: pillowfight
    spec:
      containers:
      - name: pillowfight
        image: sequoiatools/pillowfight:v5.0.1
        command: ["cbc-pillowfight",
                  "-U", "couchbase://cb-example-0000.cb-example.default.svc/default?select_bucket=true",
                  "-I", "10000", "-B", "1000", "-c", "10", "-t", "1", "-P", "password"]
      restartPolicy: Never

The error…

failed to start container “pillowfight”: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused “chdir to cwd (”/root/libcouchbase/build/bin") set in config.json failed: permission denied": unknown

… is thrown by the containers as they try to run the command.

PS - this is the pillowfight YAML file included with the Couchbase operator

4

OK - I think I solved it.

Instead of using the default sequoiatools/pillowfight:v5.0.1 image that shipped with the operator, I followed the instructions from here:
https://docs.couchbase.com/operator/1.1/demo.html and noticed that it was using a different image
couchbaseutils/pillowfight:v2.9.3

Now pillowfight populates the db as expected. Not sure if there is an issue with the original sequoatools image or not.

5

@cormachogan, Perfect good to hear you were able to get pillowfight working. Yes, I would recommend using couchbaseutils/pillowfight:v2.9.3 image. I will check with our QE folks who built sequoiatools/pillowfight:v5.0.1 image.

Thanks!

Anil Kumar

6

Hi @anil – I also had the same problem. The yaml that ships with the operator package specifies the wrong image. I was able to fix that by the workaround listed above. However, even though I ran the job and it created 10,000 sample documents, something seems wrong with the documents?

See attached pic. Are these supposed to be all binary documents?

17%20PM
Screen Shot 2019-03-22 at 2.03.17 PM.png1204×719 98.8 KB

7

That’s right, pillowfight only loads binary documents. In the future only the couchbaseutils container will be used. But for now I’ve updated the sequoiatools image to work as well.