DataBufferLimitException: Exceeded limit on max bytes to buffer webflux error

While sending a file I receive an array of bytes. I always have a problem with webflux to receive an array. the error thrown as below :

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144 at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:101) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException 

Do you now how to resolve that in webflux ?

14 Answers

This worked for me:

1. Create a @Bean in one of your configuration classes or the main SpringBootApplication class:

   @Bean public WebClient webClient() { final int size = 16 * 1024 * 1024; final ExchangeStrategies strategies = ExchangeStrategies.builder() .codecs(codecs -> codecs.defaultCodecs().maxInMemorySize(size)) .build(); return WebClient.builder() .exchangeStrategies(strategies) .build(); } 

2. Next, go to your desired class where you want to use the WebClient:

   @Service public class TestService { @Autowired private WebClient webClient; public void test() { String out = webClient .get() .uri("/my/api/endpoint") .retrieve() .bodyToMono(String.class) .block(); System.out.println(out); } } 

I suppose this issue is about adding a new spring.codec.max-in-memory-size configuration property in Spring Boot. Add it to the application.yml file like:

spring: codec: max-in-memory-size: 10MB 

Set the maximum bytes (in megabytes) in your Spring Boot application.properties configuration file like below:

spring.codec.max-in-memory-size=20MB 

This is the official way (see documentation) to get rid of the exception:

webTestClient.mutate() .codecs(configurer -> configurer .defaultCodecs() .maxInMemorySize(16 * 1024 * 1024)) .build().get() .uri("/u/r/l") .exchange() .expectStatus() .isOk() 

i was getting this error for a simple RestController (i post a large json string).

here is how i successfully changed the maxInMemorySize

import org.springframework.context.annotation.Configuration; import org.springframework.http.codec.ServerCodecConfigurer; import org.springframework.web.reactive.config.ResourceHandlerRegistry; import org.springframework.web.reactive.config.WebFluxConfigurer; @Configuration public class WebfluxConfig implements WebFluxConfigurer { @Override public void addResourceHandlers(ResourceHandlerRegistry registry) { registry.addResourceHandler("/swagger-ui.html**") .addResourceLocations("classpath:/META-INF/resources/"); registry.addResourceHandler("/webjars/**") .addResourceLocations("classpath:/META-INF/resources/webjars/"); } @Override public void configureHttpMessageCodecs(ServerCodecConfigurer configurer) { configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024); } } 

this was surprisingly hard to find

Instead of retrieving data at once, you can stream:

Mono<String> string = webClient.get() .uri("end point of an API") .retrieve() .bodyToFlux(DataBuffer.class) .map(buffer -> { String string = buffer.toString(Charset.forName("UTF-8")); DataBufferUtils.release(buffer); return string; }); 

Alternatively convert to stream:

 .map(b -> b.asInputStream(true)) .reduce(SequenceInputStream::new) .map(stream -> { // consume stream stream.close(); return string; }); 

In most cases you don't want to really aggregate the stream, rather than processing it directly. The need to load huge amount of data in memory is mostly a sign to change the approach to more reactive one. JSON- and XML-Parsers have streaming interfaces.

This worked for me

val exchangeStrategies = ExchangeStrategies.builder() .codecs { configurer: ClientCodecConfigurer -> configurer.defaultCodecs().maxInMemorySize(16 * 1024 * 1024) }.build() return WebClient.builder().exchangeStrategies(exchangeStrategies).build() 

Another alternative could be creating a custom CodecCustomizer, which is going to be applied to both WebFlux and WebClient at the same time:

@Configuration class MyAppConfiguration { companion object { private const val MAX_MEMORY_SIZE = 50 * 1024 * 1024 // 50 MB } @Bean fun codecCustomizer(): CodecCustomizer { return CodecCustomizer { it.defaultCodecs() .maxInMemorySize(MAX_MEMORY_SIZE) } } } 

As of Spring Boot 2.3.0, there is now a dedicated configuration property for the Reactive Elasticsearch REST client.

You can use the following configuration property to set a specific memory limit for the client.

spring.data.elasticsearch.client.reactive.max-in-memory-size= The already existing spring.codec.max-in-memory-size property is separate and only affects other WebClient instances in the application.

For those who had no luck with the myriad of beans, customizers, and properties that could be added to solve this problem, check whether you have defined a bean extending WebFluxConfigurationSupport. If you have, it will disable the autoconfiguration version of the same bean (my personal experience, Boot 2.7.2), somewhere under which Spring loads properties such as the suggested spring.codec.max-in-memory-size. For this solution to work you need to have also configured this property correctly.

To test if this is the cause of your problems, remove your WebFluxConfigurationSupport implementation temporarily. The long term fix that worked for me was to use configuration beans to override attributes for the autoconfigured bean. In my case, WebFluxConfigurer had all of the same methods available and was a drop-in replacement for WebFluxConfigurationSupport. Large JSON messages are now decoding for me as configured.

If you dont want to change the default settings for webClient globally, you can use the following approach to manually merge multiple DataBuffers

webClient .method(GET) .uri("<uri>") .exchangeToMono(response -> { return response.bodyToFlux(DataBuffer.class) .switchOnFirst((firstBufferSignal, responseBody$) -> { assert firstBufferSignal.isOnNext(); return responseBody$ .collect(() -> requireNonNull(firstBufferSignal.get()).factory().allocateBuffer(), (accumulator, curr) -> { accumulator.ensureCapacity(curr.readableByteCount()); accumulator.write(curr); DataBufferUtils.release(curr); }) .map(accumulator -> { final var responseBodyAsStr = accumulator.toString(UTF_8); DataBufferUtils.release(accumulator); return responseBodyAsStr; }); }) .single(); }); 

The above code aggregates all the DataBuffers into a single DataBuffer & then converts the final DataBuffer into a string. Please note that this answer wont work as DataBuffers received might not have all the bytes to construct a character (incase of UTF-8 characters, each character can take upto 4 bytes). So we cant convert intermediate DataBuffers into String as the bytes towards the end of buffer might have only part of the bytes required to construct a valid character

Note that this loads all the response DataBuffers into memory but unlike changing global settings for the webClient across the whole application. You can use this option to read complete response only where you want i.e you can narrow down & pick this option only where you expect large responses.

As of Spring boot 2.7.x we should use below property to set the memory size to webclient which is used internally in reactive ElasticSearch

spring.elasticsearch.webclient.max-in-memory-size=512MB

Just add below code in your springboot main class.

@Bean public WebClient getWebClient() { return WebClient.builder() .baseUrl("Your_SERVICE_URL") .codecs(configurer -> configurer .defaultCodecs() .maxInMemorySize(16 * 1024 * 1024)) .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) .build(); } 

This work for me.

Note that if you're using custom decoder, you need to set the maximum in-memory size in there, for instance:

Integer CODEC_20_MB_SIZE = 20 * 1024 * 1024; WebClient.builder() .codecs(clientCodecConfigurer -> { var codec = new Jackson2JsonDecoder(); codec.setMaxInMemorySize(CODEC_20_MB_SIZE); clientCodecConfigurer.customCodecs().register(codec); clientCodecConfigurer.customCodecs().register(new Jackson2JsonEncoder()); });