I have recently taken over the maintenance of a Spring Boot project. This project has some RateLimit errors in the logs when the app was contacting a remote REST API. Turns out that this app was also using the synchronous, blocking RestTemplate client to make the API calls, instead of the newer Spring WebClient, that happens to be using the Reactor API under the hood.
And you know what is great about Reactor and using Reactive APIs in general? It makes programming with Data Stream so easy. Which also means it makes retry strategies implementation easier.
Let’s talk about the error in the log first. What I got looked like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
org.springframework.web.client.HttpClientErrorException$TooManyRequests: 429 Too Many Requests: [{"response_type":"ERROR","message":"Number of requests has exceeded the 1 minute limit"}] at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:137) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:184) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:782) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:740) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:674) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:315) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at com.couchbase.training.couchlms.repository.LmsRepository.getCourseModules(LmsRepository.java:102) ~[classes!/:0.0.40-SNAPSHOT] at com.couchbase.training.couchlms.services.LmsProcessor.processCourseModules(LmsProcessor.java:147) [classes!/:0.0.40-SNAPSHOT] at com.couchbase.training.couchlms.services.LmsProcessor.processCourses(LmsProcessor.java:91) [classes!/:0.0.40-SNAPSHOT] at com.couchbase.training.couchlms.config.SchedulingConfig.scheduledCoursesPuller(SchedulingConfig.java:45) [classes!/:0.0.40-SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252] at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_252] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_252] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_252] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_252] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252] |
As usual, the interesting bit in stack traces is at the top. The error is 429 Too Many Requests, and the message says there is a 1 minute rate limit. Decomposing this, the HTTP status code returned is 429. It’s a Rate Limit error, meaning the API tells the caller it has sent too many request. This can usually be solved by waiting a bit, and you might even have a Retry-after header in the response telling you how long you have to wait.
Let’s see how we can get that information with Spring’s WebClient:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
.uri(ub -> ub.pathSegment(uri).queryParams(queryParams).build()) .retrieve() .onStatus( HttpStatus.TOO_MANY_REQUESTS::equals, response -> { List<String> header = response.headers().header("Retry-After"); Integer delayInSeconds; if (!header.isEmpty()) { delayInSeconds = Integer.valueOf(header.get(0)); } else { delayInSeconds = 60; } return response.bodyToMono(String.class).map(msg -> new RateLimitException(msg, delayInSeconds)); }) .bodyToMono(String.class) |
This code is sending a GET request. The WebClient allows us to take a look at the request’s response and react appropriately thanks to the onStatus method. The first parameter is a boolean used to filter on returned HTTP Status code. Here, when the status code is 429, we do something.
We take a look at the Response headers, look if there is a Retry-After header, if so, we initialize the delayInSeconds variable with it, if not, we set a 60 default value. Then we can send back the Response body mapped to a RateLimitException. The body content will be used as an error message and the delayInSeconds will be available in a separate field. Take a look at the Exception code for more details:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
import java.time.Duration; public class RateLimitException extends Throwable { private int retryAfterDelay = 60; public RateLimitException(String message) { super(message); } public RateLimitException(String message, int retryAfterDelay) { super(message); this.retryAfterDelay = retryAfterDelay; } public int getRetryAfterDelay() { return retryAfterDelay; } public Duration getRetryAfterDelayDuration() { return Duration.ofSeconds(retryAfterDelay); } } |
So what needs to be done is catching this specific error and then retrying after the given duration. Reactor makes that easy by providing Retries strategies. All you need to do is call the retryWhen method:
|
1 2 3 4 5 6 |
.bodyToMono(String.class) .retryWhen(Retry.withThrowable(throwableFlux -> { return throwableFlux.filter(t -> t instanceof RateLimitException).map(t -> { RateLimitException rle = (RateLimitException) t; return Retry.fixedDelay(1, rle.getRetryAfterDelayDuration()); }); |
There are different Retry methods, here we can use the withThrowable() builder. It gives a Flux that should contain the RateLimitException. So we start by applying a filter to make sure of that. Then we map that exception to the actual Retry object. Here it’s the Retry.fixedDelay strategy, taking a maximum attempts and duration as parameters. The duration comes from the RateLimitException that was thrown earlier.
With that, each time a request returns a 429, the client will wait the appropriate time until it retries. And it was much easier to implement with Reactor than using try/catch with Spring’s RestTemplate. I know Reactive programming might be a little intimidating at first, but it’s a great way to manage data streams like HTTP requests and responses, or to manage connections to databases that support reactive programming, like Couchbase.
Want some more help and ideas?