I have been load testing the beta 3 version. It has certainly improved performance. Although I still see Memcached gets 5 hits per .aspx hit, but still gives better performance. I can go over 400 req/sec, which is as good as SQL Server session performance:
However, there’s a new problem:
You can see from the below graph that suddenly cache hit rate goes down, and I can see pages are unable to retrieve their sessions. This problem wasn’t there in the previous version.
This also happens randomly, even under small load.
When the problem starts, I can see ops/sec goes down to zero. It no longer sends any request to Couchbase server.
My Jmeter tests start failing when this problem happens:
When I restart the website, it starts working again. So, I believe this is a problem on the code, not on the Couchbase server.
Another observation is, if I do not warmup the site after restart, and run the load test on a cold site, where many requests get queued, the session retrieval problem starts right from the beginning. Not a single time Home page is able to retrieve session after login. So, I believe somewhere there’s a race condition shutting down all access to couchbase server.
@oazabir -
I think the problem is that ASP.NET creates and disposes many session provider instances…which is not what was expected. I just pushed a commit that removes the code that disposed of the Cluster object when the overridden Dispose method is called by the session provider. If you can try pulling that code from master and retesting.
-Jeff
1 Like
OK I will give it a shot. Is that not going to cause resource leaks?
@oazabir -
It depends on a couple of factors…I am working on a more elegant solution ATM. The problem with using ClusterHelper was that it creates a singleton out of the Cluster object, which is nice, however in some situations customers and users have different clusters for session and caching data.
-Jeff
1 Like
@oazabir -
I just pushed a patch that should resolve the issue. LMK if you run into anything…
Thanks,
Jeff
1 Like
Just tested the master branch, so far no such failure reported. Throughput is:
Compare to SQL Server session:
SQL Server is giving me 40% more throughput than Couchbase. Moreover, Couchbase page latency is considerably higher. Login POST which creates the session is 218ms vs 85ms. Homepage GET which gets the session is 282ms vs 110ms.
I can see from couchbase monitor that there’s almost exactly 5x ops/sec reported compared to the req/sec. I get a feeling that for every page hit, couchbase is being hit 5 times. This must be the reason for the slower throughput and higher latency.
Gotta reduce the number of ops per page view.
Another observation, while load test was going on, I restarted the website. For SQL Server, it results in 0.02% and 0.04% error.
For Couchbase, it is 0.05% and 0.09% error. Slightly higher % of error. Maybe due to other factors.
An observation - I see for each session, two entries are used. So, 2 read + 2 write is happening for every page hit at least. There seems to be another one.
Redis session provider uses one entry. Is it possible to take Microsoft’s Redis session provider and just replace the storage related code with Couchbase? I would assume Microsoft has overcame these issues.
@oazabir -
The provider stores the header and body of session separately; I didn’t write the initial code, but it seems that it’s an optimization so the header state can be compared without pulling the body over the wire during a write. Note that while it takes 2x the hits to the backend, memcached is very fast and the reads/writes are in-memory.
I have considered to a complete re-write on a separate branch and storing the header/body as as single value, but I am not sure yet when that project could get started.
-Jeff
1 Like
