Performance issues with loading php sdk in php-fpm

hello, i have an ubuntu 14 machine with nginx, php-fpm installed,
i’m having some performence isssues with the current version of php-sdk 2.1. i’ve writen a simple php file that all it does is
echo “hello world”

i’ve tried to do some to ab testing and these are the results

this is before adding the extension of couchbase.so

ab -k -c 100 -n 10000 "http://localhost/index.php"

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.4.6
Server Hostname:        localhost
Server Port:            80

Document Path:          /index.php
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   1.710 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1560000 bytes
HTML transferred:       110000 bytes
Requests per second:    5847.30 [#/sec] (mean)
Time per request:       17.102 [ms] (mean)
Time per request:       0.171 [ms] (mean, across all concurrent requests)
Transfer rate:          890.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:     2   17   0.9     17      20
Waiting:        2   17   0.9     17      20
Total:          5   17   0.7     17      21

Percentage of the requests served within a certain time (ms)
  50%     17
  66%     17
  75%     17
  80%     17
  90%     18
  95%     18
  98%     19
  99%     19
 100%     21 (longest request)

this is the result after adding the extension of couchbase.so

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd,
Licensed to The Apache Software Foundation,

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.4.6
Server Hostname:        localhost
Server Port:            80

Document Path:          /index.php
Document Length:        11 bytes

Concurrency Level:      100
Time taken for tests:   12.643 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    0
Total transferred:      1560000 bytes
HTML transferred:       110000 bytes
Requests per second:    790.96 [#/sec] (mean)
Time per request:       126.428 [ms] (mean)
Time per request:       1.264 [ms] (mean, across all concurrent requests)
Transfer rate:          120.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       4
Processing:     3  126   6.9    126     133
Waiting:        2  126   6.9    126     133
Total:          6  126   6.8    126     136

Percentage of the requests served within a certain time (ms)
  50%    126
  66%    127
  75%    128
  80%    128
  90%    128
  95%    129
  98%    130
  99%    130
 100%    136 (longest request)

i’ve tried this on php 5.5.9 and 5.6.21 and we got pretty much the same results

i have another old machine with couchbase sdk 1.2 and the performance of the machine with the extension is great.
i’ve also tried to use the sdk with version 2.0.7 but i still get the same bad results in performance.
i would love to hear what do you think about this issues.
thank you.
Eran.

@eran, could you show what exactly are you doing with Couchbase in that index.php?

as i said the file is

<?php echo "hello"; nothing to do with couchbase its only loaded in the extensions of the php fpm

oh, interesting, I will profile it and get back here. Thank you

1 Like

great, i’m looking foward to your profiling

by the way, i’ve tried to install the beta2 of 2.2.0 on php 7.0.6 and i’ve got the same effect of slowing down requests

the benchmark before adding the extension

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.4.6
Server Hostname: localhost
Server Port: 80

Document Path: /index.php
Document Length: 11 bytes

Concurrency Level: 100
Time taken for tests: 1.516 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 0
Total transferred: 1560000 bytes
HTML transferred: 110000 bytes
Requests per second: 6597.33 [#/sec] (mean)
Time per request: 15.158 [ms] (mean)
Time per request: 0.152 [ms] (mean, across all concurrent requests)
Transfer rate: 1005.06 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.6 0 7
Processing: 3 15 1.5 15 32
Waiting: 1 15 1.6 15 32
Total: 7 15 1.2 15 33

Percentage of the requests served within a certain time (ms)
50% 15
66% 15
75% 15
80% 15
90% 16
95% 17
98% 17
99% 17
100% 33 (longest request)

and the benchmark after adding the extension

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software: nginx/1.4.6
Server Hostname: localhost
Server Port: 80

Document Path: /index.php
Document Length: 11 bytes

Concurrency Level: 100
Time taken for tests: 9.066 seconds
Complete requests: 10000
Failed requests: 0
Keep-Alive requests: 0
Total transferred: 1560000 bytes
HTML transferred: 110000 bytes
Requests per second: 1103.00 [#/sec] (mean)
Time per request: 90.662 [ms] (mean)
Time per request: 0.907 [ms] (mean, across all concurrent requests)
Transfer rate: 168.04 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.3 0 4
Processing: 6 90 5.5 90 101
Waiting: 6 90 5.6 90 101
Total: 10 90 5.4 90 101

Percentage of the requests served within a certain time (ms)
50% 90
66% 92
75% 92
80% 93
90% 94
95% 95
98% 96
99% 97
100% 101 (longest request)

So the reason of this behavior is that part of our extension API written in PHP, and we have to evaluate it to expose for the developer. At the moment the only place I know to do it is request init hook, which means that we re-evaluate our classes https://github.com/couchbase/php-couchbase/tree/master/stub for every request, which leads to this slowdown. At the moment I don’t see how we can improve it. We might get to generating real code in C and move code into module init, instead of packing everything as strings and eval-ing them for each request.

May be I don’t understand how couchbase sdk handles PHP internally, but I don’t see how those stubs slow down dramatically. Of course, C would be better, but those stubs do not look heavy. With opcache enabled, those classes will be cached in shared memory.

The slow down probably comes from the fact that PHP has to re-create a new connection to couchbase cluster whenever there is a new PHP process. Even with PHP-FPM, it has to kill itself time to time.

It would be nice to know why it happens exactly.

but the example does not use any connections, and even does not load any couchbase classes

But then how does stubs affect the speed in that case if they are not being used? I don’t believe adding an extension causes classes to be loaded unless the extension initialize those classes automatically.

As I said we do it in request init hook

Thank you for the link.

We too have experience slow down with couchbase php sdk. It would be nice if this can be improved in a way whether that init function is direct cause of it or not. I don’t have my bench # at the moment, 1.710 vs 12.63 is such a big difference.

Injecting the stubs for the developer is cool, but such an performance issue is not nice. If i understand that right. The stubs are for developers to have the possibility of getting the informations in coding ide’s or editors … or must the be injectet all the time ? If the working with the sdk is possible without the stubs loaded … maybe the following could be an option:

Load stub based on php.ini settings.

Just to put in a idea to solve this, could it be an option to provide a php.ini option ?

couchbase.inject_stub=[0|1]

read this from the config … and inject it based ob this configuration.

With that, it should be possible to have the stubs injected for the developers, and disable it on production systems, staging systems, build systems and so on.

Just if no solution is found … maybe try it on myself if i have some time :wink:

Major problem since May.
Solution ? None.

Clap clap.

And of course, like last time : https://twitter.com/oxman/status/760858154729025536
Because for you PHP is the fifth wheel.

That’s simply not true. In recent months we’ve done a ton of engineering work on PHP, including a new release that supports PHP 7 quite well, added needed features for areas like CAS, etc. You really don’t help your cause by attacking us on twitter.

This symbol loading issue is definitely something we want to improve, but it is non-trivial. There is nothing we’re doing incorrect with respect to the PHP APIs, it just turns out that the way we’d implemented the extension to work across versions of PHP and use common code in libcouchbase is an issue for PHP itself.

As @avsej mentions, the solution is probably to reimplement more of that in C. That will take some time.

As to the severity of the performance issue, here the discussion is on a microbenchmark on a PHP generated page that doesn’t actually do anything. Since it’s all localhost and apache bench is just a tight loop of requests, the throughput reported is very latency sensitive. The stub loading dominates in this microbenchmark.

A real app, one that actually uses Couchbase likely would not show this being as much a dominator. A real app that interacts with the database one or more times, does some work with the data received and sends it on, we believe will show a lot less of an impact from the stub loading.

Even then, we acknowledge the concern and want to improve it. @avsej and I were talking about how we’ll address that in the not too distant future. Lots of people are using the extension successfully today and quite happy. They’ll be even happier as we improve it and deliver additional improvements.

Thank you.
And a post like that is important.
You made absolutely no communication about this problem since May.
Communication is very important, without we can think we’re not important.

Hi, I’ve just tried to migrate extension to PCS https://github.com/flaupretre/pecl-pcs, and it seems like it fixes the performance issue, because loading everything on module init

The changeset lives here: http://review.couchbase.org/68395, I expect it to be included in 2.2.4 release