- \n
- The problem we\u2019re tackling<\/li>\n
- Why the existing options aren\u2019t enough<\/li>\n
- An introduction to reschedule hook project<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
While some familiarity with Kubernetes operators<\/a>, node draining<\/a>, admission control<\/a> and dynamic webhooks<\/a> would be helpful, I\u2019ll try and cover key concepts along the way. The <\/span>Couchbase<\/a> Autonomous Operator will also be referenced throughout as the Operator for brevity and most of the examples and demos will involve Couchbase resources of some sort.<\/p>\n
Though originally developed with the Couchbase data platform in mind, the reschedule hook project is intended to be flexible and can be configured to protect other operator-managed stateful applications.<\/p>\n
\nPart 1 – Why evictions are challenging for operator-managed stateful applications<\/h2>\n
Understanding pod evictions<\/h3>\n
In Kubernetes, evicting a pod involves triggering the pod\u2019s
PreStop<\/code> hook then sending aSIGTERM<\/code> after its completion. If the pod hasn\u2019t exited after a grace period, this is followed up by aSIGKILL<\/code>. Evictions happen both voluntarily and involuntarily for a multitude of reasons, from node pressure to autoscaling, but this project focuses on voluntary evictions triggered when draining nodes. This is the process whereby pods are removed from nodes using the Kubernetes Eviction API, which is often required to clear the way for operations like maintenance or upgrades.<\/p>\nRunning
kubectl drain<\/code> is a common way to prepare a node for maintenance in Kubernetes. It marks the node as unschedulable and sends eviction requests concurrently for all the pods running on that node.<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/nodedrains.png\")
<\/p>\n
\nWhy stateful applications are challenging<\/h3>\n
Stateful workloads add complexity to eviction handling:<\/p>\n
Unpredictable Shutdown Times:<\/strong> These workloads may involve long-running processes like in-flight queries, which need to finish before a pod can be safely terminated<\/p>\n
Application Coordination:<\/strong> The application itself often needs to be notified before pod removal to ensure data consistency and proper rebalancing<\/p>\n
Operator Management:<\/strong> Operators managing the application need time to coordinate pod removal with the operator\u2019s internal state<\/p>\n
Volume Movement:<\/strong> The migration of Kubernetes volumes from one node to another can take considerable time that exceeds the pod’s
terminationGracePeriodSeconds<\/code><\/p>\nAn Operator automates the lifecycle of complex applications by continuously reconciling their actual state with the desired state defined in custom resources.<\/p>\n
Take Couchbase as an example. A Couchbase cluster is composed of multiple pods, each running an instance of Couchbase Server, typically spread across different nodes and availability zones. The Operator ensures the cluster state matches the state defined in a
CouchbaseCluster<\/code> resource. When a pod needs to be removed, the Operator must notify the cluster to rebalance data and handle any running processes gracefully.<\/p>\nOrganizational challenges<\/h3>\n
Another common challenge is the separation of responsibilities in larger organizations:<\/p>\n
- \n
- \n
- \n
- Kubernetes administrators manage infrastructure<\/li>\n
- Application teams manage stateful applications running on that infrastructure<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
Meaning Kubernetes administrators need ways to safely drain nodes to perform maintenance without requiring constant coordination with application owners or impacting the availability or health of the underlying application.<\/p>\n
\nExisting Kubernetes protections – why they\u2019re not enough<\/h3>\n
Kubernetes provides tools like PreStop hooks<\/strong> and Pod Disruption Budgets<\/strong> to help with graceful shutdowns and application availability during evictions.<\/p>\n
PreStop Hooks<\/strong> run scripts inside the container before the
SIGTERM<\/code> signal, giving the pod a chance to gracefully shut down. For stateless applications, this is often enough. But for stateful apps, Operators often need to coordinate pod removal before termination to avoid the issues outlined above, which is tricky to do inside preStop hooks.<\/p>\nOne ingenious approach we\u2019ve seen involves adding a preStop script to the pod templates used by the Operator. This script has the pod add the reschedule annotation (discussed later) to itself in order to notify the Operator it should be moved, then loop until it has been safely ejected from the cluster.<\/p>\n
Basic outline of script, not exact:<\/strong><\/p>\n
preStop:\r\n exec:\r\n command:\r\n - \/bin\/sh\r\n \/mnt\/kubectl annotate pod $SELF_POD_NAME cao.couchbase.com\/reschedule=true \\\\\r\n while \/opt\/couchbase\/bin\/couchbase-cli server-list -c couchbases:\/\/localhost\\\\\r\n do \\\\\r\n sleep 1; \\\\\r\n done\r\n<\/pre>\n
However, this requires mounting
kubectl<\/code> andcouchbase-cli<\/code> into each pod, increasing complexity and image size. Networking or other issues could also interrupt the script and result in the pod being prematurely terminated.<\/p>\nPod Disruption Budgets (PDBs)<\/strong> ensure a minimum number of pods remain available during voluntary disruptions. But they only limit how many pods can be evicted simultaneously, meaning we\u2019re still dealing with the scenario where an operator isn\u2019t able to gracefully handle pod removal.<\/p>\n
\nWhat happens if pods are evicted today?<\/h3>\n
The Operator creates PDBs for Couchbase to limit how many pods can be evicted at once. While this ensures a minimal level of availability for the application, pods that do get evicted will still be unsafely removed from the Couchbase cluster by means of failover.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/currentDrain_terminal.gif\")
<\/p>\nWe can also see how Operator handles this by watching what happens in the Couchbase administrator UI.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/currentDrain_gui.gif\")
<\/p>\nEven when evictions happen one pod at a time, once a pod restarts on a new node, Kubernetes may allow evictions on the next pod before the new pod has been safely added to the stateful application. At Couchbase, we prevent this with a readiness gate on the pods.<\/p>\n
To help with these issues, we introduced the
cao.couchbase.com\/reschedule<\/code> annotation in CAO v2.7.0. When added to a pod, it tells the Operator it should be recreated. However, manually cordoning nodes and adding this annotation is tedious, doesn\u2019t scale well, and only affects Couchbase pods – other pods still require normal draining.<\/p>\n
\nWhy not handle this entirely inside the Operator?<\/h3>\n
We considered embedding eviction validation logic inside the Operator itself, but:<\/p>\n
- \n
- \n
- \n
- It\u2019s an anti-pattern in Kubernetes operator design. Operators work via reconciliation loops, not admission controllers<\/li>\n
- Adding admission control endpoints to operators complicates deployment and maintenance<\/li>\n
- Validating webhooks are cluster scoped resources and therefore require cluster scoped permissions. The Operator is namespace scoped and would require a large expansion of its required permissions.<\/li>\n
- Building this logic into it\u2019s own project allows open sourcing and invites community contributions for similar use cases<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\nPart 2 – leveraging Kubernetes Webhooks for intelligent pod evictions<\/h2>\n
The Eviction Reschedule Hook is an open source Kubernetes admission controller designed to improve how evictions are handled during node drains to avoid the problems outlined in part 1. Working in tandem with an Operator, it automates pod rescheduling by intercepting eviction requests before they reach the pod. These requests are selectively rejected in a way that still allows the standard
kubectl drain<\/code> command to function as expected.<\/p>\nInstead of immediately terminating pods or testing Pod Disruption Budget (PDB) limits, the admission controller notifies the Operator a pod needs to be moved using the
cao.couchbase.com\/reschedule<\/code> annotation.<\/p>\n
\nUnderstanding admission control<\/h3>\n
Admission control is a key stage in the Kubernetes API request lifecycle that occurs after a request has been authenticated and authorized, but before it’s persisted to the cluster.<\/p>\n
Admission controllers can validate or mutate these requests based on custom logic.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/nodeDrains_webhook.png\")
<\/p>\nWhen a node is drained in Kubernetes, each attempt at a pod eviction triggers a
CREATE<\/code> request for thepods\/eviction<\/code> subresource. This is where our admission controller comes into play. It intercepts these eviction requests before they reach the pods and determines whether they should be allowed. In our case, the admission logic involves signalling to the Operator that a pod should be rescheduled safely by adding the reschedule annotation.<\/p>\nWe\u2019re using a validating webhook rather than a mutating webhook because our primary goal is to evaluate whether the eviction request is allowed. While we may annotate the pod as part of the decision process, the webhook itself is performing validation, not mutation, on the eviction request.<\/p>\n
\nHandling node drains<\/h3>\n
As the title of this article makes clear, the primary goal of this project is to enable graceful handling of node drains. With the Eviction Reschedule Hook in place, the standard node drain flow is modified to support safer pod migration.<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/nodeDrains_advanced-1024x486.png\")
<\/p>\nEviction requests are intercepted before the
PreStop<\/code> hook is executed. Instead of the pod being terminated, the eviction request is rejected and the pod annotated. The Operator checks for the reschedule annotation during each reconciliation loop. If the annotation appears, it will handle safely recreating the pod.<\/p>\nAs draining a node taints it with
node.kubernetes.io\/unschedulable<\/code>, assuming there is another node in the Kubernetes cluster that matches the scheduling requirements of the pod, when the Operator creates the new pod it will exist on another node.<\/p>\nImportantly, the admission controller is designed to selectively filter only relevant pods. All other workloads on the node continue to follow the default eviction process.<\/p>\n
\nWorking with Kubectl<\/h3>\n
To integrate smoothly with
kubectl<\/code>, it’s important to understand what happens under the hood when you runkubectl drain <node><\/code>. Looking at thedrain.go<\/code> source<\/a>, after the node is cordoned, a separate goroutine is spawned for each pod on that node to handle its eviction.<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/kubectl-drain-1024x375.png\")
<\/p>\nWhether
kubectl<\/code> continues attempting to evict a pod depends on the response from the admission controller. If the eviction request succeeds (i.e., no error is returned), the loop in which eviction requests are sent is exited. There is then a follow up check wherekubectl<\/code> waits for the pod\u2019s deletion. However, for Operator managed pods, we wantkubectl<\/code> to keep retrying until they have been safely rescheduled, at which point we should end the goroutine before any further checks.<\/p>\nBy design,
kubectl<\/code> treats a429 TooManyRequests<\/code> response to the eviction request in the goroutine as a signal to pause for 5 seconds before retrying. We can leverage this behavior in our admission controller: after pod selection logic, we return a429<\/code> status if the pod is either being annotated for rescheduling or is already annotated and waiting to be moved.<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/kubectl-drain-2-1024x233.png\")
<\/p>\nAfter the pod has been successfully rescheduled, it will have a different name and therefore the admission controller will no longer be able to locate it by name and namespace. At that point, we can return a
404 NotFound<\/code> response to exit the goroutine.<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/kubectl-drain-3-1024x266.png\")
<\/p>\n
\nIn-place pod rescheduling<\/h3>\n
In some cases, the Operator may delete and recreate a pod using the same name.<\/p>\n
At Couchbase, the
upgradeProcess<\/code> field on aCouchbaseCluster<\/code> can be changed to control<\/a> how the Operator replaces pods. This defaults to SwapRebalance<\/strong>, which tells the Operator to create a new pod with a different name, rebalance it into the cluster and then delete the old pod. InPlaceUpgrade<\/strong> is a faster but less flexible alternative, whereby the Operator will delete the pod and recreate it with the same name, reusing the existing volumes. The creates a challenge for the admission controller.<\/p>\nThe eviction requests sent by the goroutine in
kubectl<\/code> include only the pod\u2019s name and namespace. Because of this limited context, the admission controller can’t assume that a pod lacking a reschedule annotation needs to be moved – it may simply be a newly recreated instance of a pod that was already rescheduled.<\/p>\nTo handle this, the admission controller maintains a lightweight tracking mechanism which is used when pods will be rescheduled with the same name. When a pod is annotated for rescheduling, we will also annotates another resource, named the tracking resource, with the
reschedule.hook\/<podNamespace>.<podName> key<\/code>.<\/p>\nIf a subsequent eviction request is intercepted for a pod without the reschedule annotation, the presence of this annotation on the tracking resource indicates that the pod has already been rescheduled. When this occurs, the webhook will clean up the tracking annotation for that pod before return a
404 NotFound<\/code> response to exit the goroutine.<\/p>\nWhile the tracking resource type will default to
CouchbaseCluster<\/code>, the webhook also supports using a pod\u2019sNamespace<\/code>. If pods will never be rescheduled with the same name, this feature can be disabled.<\/p>\n
\nRunning the Eviction Reschedule Hook<\/h3>\n
Deploying the Eviction Reschedule Hook and its supporting components is straightforward. The README<\/a> provides detailed instructions for building the image and deploying the full stack.<\/p>\n
The main components that make up the reschedule hook are:<\/p>\n
ValidatingWebhookConfiguration: <\/strong>Tells the Kubernetes API server to forward
CREATE<\/code> requests for thepods\/eviction<\/code> subresource to our webhook service<\/p>\nService: <\/strong>Routes incoming webhook requests to the admission controller pod<\/p>\n
Admission Controller: <\/strong>Runs the reschedule hook inside a container, typically deployed as a Kubernetes Deployment<\/p>\n
ServiceAccount, ClusterRole, and ClusterRoleBinding: <\/strong>These grant the admission controller the necessary permissions. At minimum,
get<\/code> andpatch<\/code> permissions for pods are needed. If using a tracking resource,get<\/code> andpatch<\/code> permissions for the tracking resource should also be added<\/p>\nOnce deployed, draining a node on which Couchbase pods reside demonstrates how the Reschedule Hook intercepts and rejects eviction requests until the Operator safely recreates and balances pods back into the cluster:<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/safeDrain_terminal.gif\")
<\/p>\nThe graceful handling of eviction requests can also be seen in the Couchbase administrator UI. Pods are no longer failed over and instead replaced with zero downtime:<\/p>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/safeDrain_gui.gif\")
<\/p>\n
\nTry it out, get involved<\/h3>\n
The Eviction Reschedule Hook makes node drains safer and more predictable for stateful applications running on Kubernetes. By coordinating eviction handling with an Operator, it enables graceful rescheduling without disrupting the underlying application.<\/p>\n
The project is open source. Check out the repository<\/a> on GitHub to get started and take a look at the the contributing<\/a> guide if you\u2019re interested in helping out. Feedback, issues and pull requests are all welcome!<\/p>\n","protected":false},"excerpt":{"rendered":"
Why and how we built an admission controller to make node drains safer when running stateful applications in Kubernetes. Running stateful applications in Kubernetes is increasingly common and these are often managed using custom resources and operators. However, the dynamic […]<\/p>\n","protected":false},"author":85555,"featured_media":17336,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[9284,2322],"tags":[],"ppma_author":[10068,9590],"class_list":["post-17325","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-couchbase-autonomous-operator","category-kubernetes"],"acf":[],"yoast_head":"\n
Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal - The Couchbase Blog<\/title>\n<meta name=\"description\" content=\"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal\" \/>\n<meta property=\"og:description\" content=\"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\" \/>\n<meta property=\"og:site_name\" content=\"The Couchbase Blog\" \/>\n<meta property=\"article:published_time\" content=\"2025-07-18T00:47:42+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-18T18:56:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2400\" \/>\n\t<meta property=\"og:image:height\" content=\"1256\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Ben Mottershead, Software Engineer, Justin Ashworth - Senior Software Engineer\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ben Mottershead, Software Engineer\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"12 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\"},\"author\":{\"name\":\"Ben Mottershead, Software Engineer\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/2e3f0eb06846472e4fb9fa87e924eb1b\"},\"headline\":\"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal\",\"datePublished\":\"2025-07-18T00:47:42+00:00\",\"dateModified\":\"2025-07-18T18:56:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\"},\"wordCount\":2256,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png\",\"articleSection\":[\"Couchbase Autonomous Operator\",\"Kubernetes\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\",\"name\":\"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal - The Couchbase Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png\",\"datePublished\":\"2025-07-18T00:47:42+00:00\",\"dateModified\":\"2025-07-18T18:56:18+00:00\",\"description\":\"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png\",\"width\":2400,\"height\":1256},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.couchbase.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#website\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"name\":\"The Couchbase Blog\",\"description\":\"Couchbase, the NoSQL Database\",\"publisher\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#organization\",\"name\":\"The Couchbase Blog\",\"url\":\"https:\/\/www.couchbase.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png\",\"width\":218,\"height\":34,\"caption\":\"The Couchbase Blog\"},\"image\":{\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/2e3f0eb06846472e4fb9fa87e924eb1b\",\"name\":\"Ben Mottershead, Software Engineer\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/e61125d073286ff4deda9445d38fa03a\",\"url\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg\",\"contentUrl\":\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg\",\"caption\":\"Ben Mottershead, Software Engineer\"},\"description\":\"Ben is a software engineer who works out of Couchbase's Manchester office. He's been a part of the autonomous operator team since August, which introduced him to the data on Kubernetes ecosystem. Ben has been developing applications for the cloud since he finished university in 2021 and has worked with a number of languages and technologies, more recently focusing on Go, K8s and Couchbase Server.\",\"url\":\"https:\/\/www.couchbase.com\/blog\/author\/benmottershead\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal - The Couchbase Blog","description":"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/","og_locale":"en_US","og_type":"article","og_title":"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal","og_description":"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.","og_url":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/","og_site_name":"The Couchbase Blog","article_published_time":"2025-07-18T00:47:42+00:00","article_modified_time":"2025-07-18T18:56:18+00:00","og_image":[{"width":2400,"height":1256,"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png","type":"image\/png"}],"author":"Ben Mottershead, Software Engineer, Justin Ashworth - Senior Software Engineer","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ben Mottershead, Software Engineer","Est. reading time":"12 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#article","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/"},"author":{"name":"Ben Mottershead, Software Engineer","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/2e3f0eb06846472e4fb9fa87e924eb1b"},"headline":"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal","datePublished":"2025-07-18T00:47:42+00:00","dateModified":"2025-07-18T18:56:18+00:00","mainEntityOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/"},"wordCount":2256,"commentCount":0,"publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png","articleSection":["Couchbase Autonomous Operator","Kubernetes"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/","url":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/","name":"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal - The Couchbase Blog","isPartOf":{"@id":"https:\/\/www.couchbase.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage"},"thumbnailUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png","datePublished":"2025-07-18T00:47:42+00:00","dateModified":"2025-07-18T18:56:18+00:00","description":"The Eviction Reschedule Hook project uses Kubernetes Admission Controllers to intercept and reject eviction requests for operator-managed pods.","breadcrumb":{"@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#primaryimage","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2025\/07\/blog-nodedrain.png","width":2400,"height":1256},{"@type":"BreadcrumbList","@id":"https:\/\/www.couchbase.com\/blog\/node-drains-webhook-pod-removal\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.couchbase.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Rethinking Node Drains: A Webhook Based Approach to Graceful Pod Removal"}]},{"@type":"WebSite","@id":"https:\/\/www.couchbase.com\/blog\/#website","url":"https:\/\/www.couchbase.com\/blog\/","name":"The Couchbase Blog","description":"Couchbase, the NoSQL Database","publisher":{"@id":"https:\/\/www.couchbase.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.couchbase.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.couchbase.com\/blog\/#organization","name":"The Couchbase Blog","url":"https:\/\/www.couchbase.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2023\/04\/admin-logo.png","width":218,"height":34,"caption":"The Couchbase Blog"},"image":{"@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/2e3f0eb06846472e4fb9fa87e924eb1b","name":"Ben Mottershead, Software Engineer","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.couchbase.com\/blog\/#\/schema\/person\/image\/e61125d073286ff4deda9445d38fa03a","url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg","contentUrl":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg","caption":"Ben Mottershead, Software Engineer"},"description":"Ben is a software engineer who works out of Couchbase's Manchester office. He's been a part of the autonomous operator team since August, which introduced him to the data on Kubernetes ecosystem. Ben has been developing applications for the cloud since he finished university in 2021 and has worked with a number of languages and technologies, more recently focusing on Go, K8s and Couchbase Server.","url":"https:\/\/www.couchbase.com\/blog\/author\/benmottershead\/"}]}},"authors":[{"term_id":10068,"user_id":85555,"is_guest":0,"slug":"benmottershead","display_name":"Ben Mottershead, Software Engineer","avatar_url":{"url":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg","url2x":"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/1\/2024\/12\/1719955373757.jpeg"},"author_category":"","last_name":"Mottershead, Software Engineer","first_name":"Ben","job_title":"Software Engineer","user_url":"","description":"Ben is a software engineer who works out of Couchbase's Manchester office. He's been a part of the autonomous operator team since August, which introduced him to the data on Kubernetes ecosystem. Ben has been developing applications for the cloud since he finished university in 2021 and has worked with a number of languages and technologies, more recently focusing on Go, K8s and Couchbase Server."},{"term_id":9590,"user_id":81274,"is_guest":0,"slug":"justin-ashworth","display_name":"Justin Ashworth - Senior Software Engineer","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/f8d9f3bd93c0cdd95cbe3e125a15ad07c4cc276956d641a724306dc519fe14fa?s=96&d=mm&r=g","author_category":"","last_name":"Ashworth","first_name":"Justin","job_title":"","user_url":"https:\/\/couchbase.com","description":""}],"_links":{"self":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/17325","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/users\/85555"}],"replies":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/comments?post=17325"}],"version-history":[{"count":0,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/posts\/17325\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media\/17336"}],"wp:attachment":[{"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/media?parent=17325"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/categories?post=17325"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/tags?post=17325"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.couchbase.com\/blog\/wp-json\/wp\/v2\/ppma_author?post=17325"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- \n
- \n