{"id":1756,"date":"2014-12-16T18:49:22","date_gmt":"2014-12-16T18:49:21","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=1756"},"modified":"2023-05-18T13:32:10","modified_gmt":"2023-05-18T20:32:10","slug":"paying-attention-what-time-it","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/","title":{"rendered":"Paying Attention to what Time it is"},"content":{"rendered":"

First, the news. \u00a0I’ve just posted a release candidate for spymemcached 2.6<\/a>. \u00a0The current prerelease Membase vbucket aware client<\/a> is also based on that code.<\/p>\n

So what’s new in 2.6<\/a>? \u00a0There are a number of performance, transcoder and timeout handling enhancements, but the big change is in timeout handling.<\/p>\n

There had been quite a few comments and postings on spymemcached issue number 136.\u00a0 From the bug reports in spy 136, various people are running apps and seeing a RuntimeException from the MemcachedClient get().\u00a0 This is a very straightforward method call for the consumer and thus is very common, but there is some complex work going on underneath to optimize how spymemcached is used.\u00a0 Allow me to explain a bit further.<\/p>\n

Technically speaking, nothing in spymemcached is done in a synchronous manner.\u00a0 When you make a get() method call that appears to be synchronous, we use the Java concurrency APIs to make your seemingly synchronous get by running the operation through the same asynchronous write queues for a given connection by using a countdown latch with a default timeout to let the operation flow through the system, be serviced, and responded to very quickly.\u00a0 Realistically, most of these operations are done in a matter of a few milliseconds to a microsecond.<\/p>\n

Having said that, there are still a few situations that can come up when running
\na) as a userspace application
\nb) on a busy system
\nc) on a busy or wibbily network
\nd) on a JVM, owing to garbage collection or JIT activities<\/p>\n

Many of these can be controlled to a very large degree with some effort, but they cannot be entirely eliminated.\u00a0 That means code should always be ready to receive occasional timeouts. \u00a0Equivalently, spymemcached should aim to recover as quickly as possible from the timeout condition.<\/p>\n

Having said that, when I was investigating spy issue 136, a few things came up.<\/p>\n

Mike Wiederhold had noted that he would sometimes see a TCP retransmit.\u00a0 Looking further into things, he found the default time before a retransmit is 1 second.\u00a0 Well, with our default timeout on the latch at 1 second, this meant any TCP retransmit would cause a timeout.\u00a0 With cloud and virtualized deployments these days, some of which are known to be on oversubscribed networks, it’s likely to come up occasionally. \u00a0The simple fix there is to increase the default timeout. \u00a0This is not something I did lightly, but I think it’s the right thing to do.<\/p>\n

Some of our load testing with Membase<\/a> hit this.\u00a0 From memcachetest<\/a> (a toy stress tester, not a real client) I know Membase easily handles tens of thousands to hundreds of thousands of operations per second, depending on the systems in use.<\/p>\n

I began to look about for things that would impede recovery, given that many of the bug reports were pointing to a failure to recover.\u00a0 It turns out that the mechanism which implements the synchronous timeout doesn’t do anything to prevent that item from being written to the MemcachedNode object it’s designated to.<\/p>\n

In other words, our thread dealing with the node wasn’t paying enough attention to the time for a given operation.<\/p>\n

Thinking about it in terms of just one operation in one circumstance, if the buffer for sending data to the network is full because it’s really busy or there’s a momentary disruption, there is nothing stopping other threads from heaping more operations on that MemcachedNode.\u00a0 Those operations will timeout on the calling thread, but they’ll still be sent over the network.\u00a0 This will extend the effect of that momentary disruption until that backlog of stuff that won’t be used is worked through. \u00a0With a really saturated server, it may not ever recover.<\/p>\n

With the code changes in the 2.6 release candidate, we now guard against even sending an Operation which has timed out. \u00a0I’ve had good feedback already and it’s helped to clean things up for Membase testing, but with projects as widely distributed as spymemcached, memcached and Membase, it’s good to get some real world feedback. \u00a0If you’re using existing spymemcached, please give it a shot and let me know.<\/p>\n","protected":false},"excerpt":{"rendered":"

First, the news. \u00a0I’ve just posted a release candidate for spymemcached 2.6. \u00a0The current prerelease Membase vbucket aware client is also based on that code. So what’s new in 2.6? \u00a0There are a number of performance, transcoder and timeout handling […]<\/p>\n","protected":false},"author":2,"featured_media":13873,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"ppma_author":[8968],"class_list":["post-1756","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"acf":[],"yoast_head":"\nPaying Attention to what Time it is - The Couchbase Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Paying Attention to what Time it is\" \/>\n<meta property=\"og:description\" content=\"First, the news. \u00a0I’ve just posted a release candidate for spymemcached 2.6. \u00a0The current prerelease Membase vbucket aware client is also based on that code. So what’s new in 2.6? \u00a0There are a number of performance, transcoder and timeout handling […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2014-12-16T18:49:21+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-05-18T20:32:10+00:00\" \/>\n<meta name=\"author\" content=\"The Couchbase Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"The Couchbase Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\"},\"author\":{\"name\":\"The Couchbase Team\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/764f4a6771ee19bc7af70b70a326fb93\"},\"headline\":\"Paying Attention to what Time it is\",\"datePublished\":\"2014-12-16T18:49:21+00:00\",\"dateModified\":\"2023-05-18T20:32:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\"},\"wordCount\":698,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\",\"name\":\"Paying Attention to what Time it is - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"datePublished\":\"2014-12-16T18:49:21+00:00\",\"dateModified\":\"2023-05-18T20:32:10+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png\",\"width\":1800,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Paying Attention to what Time it is\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/764f4a6771ee19bc7af70b70a326fb93\",\"name\":\"The Couchbase Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/7befc37d02226b59499817eafdec60c3\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b4c18c758421903398e84d6c9560f319f39c665798d7d23e6a6f9dff8a8f984e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b4c18c758421903398e84d6c9560f319f39c665798d7d23e6a6f9dff8a8f984e?s=96&d=mm&r=g\",\"caption\":\"The Couchbase Team\"},\"description\":\"Jennifer Garcia is a Senior Web Manager at Couchbase Inc. As the website manager, Jennifer has overall responsibility for the website properties including design, implementation, content, and performance.\",\"sameAs\":[\"https:\/\/www.couchbase.com\"],\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/jennifer-garcia\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Paying Attention to what Time it is - The Couchbase Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/","og_locale":"en_US","og_type":"article","og_title":"Paying Attention to what Time it is","og_description":"First, the news. \u00a0I’ve just posted a release candidate for spymemcached 2.6. \u00a0The current prerelease Membase vbucket aware client is also based on that code. So what’s new in 2.6? \u00a0There are a number of performance, transcoder and timeout handling […]","og_url":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/","og_site_name":"The Couchbase Blog","article_published_time":"2014-12-16T18:49:21+00:00","article_modified_time":"2023-05-18T20:32:10+00:00","author":"The Couchbase Team","twitter_card":"summary_large_image","twitter_misc":{"Written by":"The Couchbase Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/"},"author":{"name":"The Couchbase Team","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/764f4a6771ee19bc7af70b70a326fb93"},"headline":"Paying Attention to what Time it is","datePublished":"2014-12-16T18:49:21+00:00","dateModified":"2023-05-18T20:32:10+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/"},"wordCount":698,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/","url":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/","name":"Paying Attention to what Time it is - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","datePublished":"2014-12-16T18:49:21+00:00","dateModified":"2023-05-18T20:32:10+00:00","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2022\/11\/couchbase-nosql-dbaas.png","width":1800,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/paying-attention-what-time-it\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Paying Attention to what Time it is"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/764f4a6771ee19bc7af70b70a326fb93","name":"The Couchbase Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/7befc37d02226b59499817eafdec60c3","url":"https:\/\/secure.gravatar.com\/avatar\/b4c18c758421903398e84d6c9560f319f39c665798d7d23e6a6f9dff8a8f984e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b4c18c758421903398e84d6c9560f319f39c665798d7d23e6a6f9dff8a8f984e?s=96&d=mm&r=g","caption":"The Couchbase Team"},"description":"Jennifer Garcia is a Senior Web Manager at Couchbase Inc. As the website manager, Jennifer has overall responsibility for the website properties including design, implementation, content, and performance.","sameAs":["https:\/\/www.couchbase.com"],"url":"https:\/\/www.couchbase.com\/blog\/author\/jennifer-garcia\/"}]}},"authors":[{"term_id":8968,"user_id":2,"is_guest":0,"slug":"jennifer-garcia","display_name":"The Couchbase Team","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/b4c18c758421903398e84d6c9560f319f39c665798d7d23e6a6f9dff8a8f984e?s=96&d=mm&r=g","author_category":"","last_name":"Garcia","first_name":"Jennifer","job_title":"","user_url":"https:\/\/www.couchbase.com","description":"Jennifer Garcia is a Senior Web Manager at Couchbase Inc. As the website manager, Jennifer has overall responsibility for the website properties including design, implementation, content, and performance."}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1756","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=1756"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/1756\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/13873"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=1756"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=1756"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=1756"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=1756"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}