[case6] using webflux to improve data export efficiency

  reactor

Order

This article mainly studies how to use webflux to improve the efficiency of data export.

Traditional export

    @GetMapping("/download-old")
    public ResponseEntity<Resource> downloadInOldWays(){
        return ResponseEntity.ok()
                .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=demo.xls")
                .header("Accept-Ranges", "bytes")
                .body(new ByteArrayResource(exportBytes(1000)));
    }
    
    public byte[] exportBytes(int dataRow){
        StringBuilder output = new StringBuilder();
        output.append(ExcelUtil.startWorkbook());
        output.append(ExcelUtil.startSheet());
        output.append(ExcelUtil.startTable());
        output.append(ExcelUtil.writeTitleRow(Sets.newHashSet("title","content")));
        IntStream.rangeClosed(1,dataRow).forEach(i -> {
            try {
                TimeUnit.MILLISECONDS.sleep(100);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            output.append(ExcelUtil.writeDataRow(Lists.newArrayList("title"+i,"content"+i)));
        });
        output.append(ExcelUtil.endTable());
        output.append(ExcelUtil.endSheet());
        output.append(ExcelUtil.endWorkbook());
        return output.toString().getBytes(StandardCharsets.UTF_8);
    }

The simulation here is to export all the data when they are ready. This speed is definitely slow. It takes almost 100 seconds for the browser to pop up the download box. If there is a gateway in front, it is easy to timeout at the gateway.

Webflux export

    @GetMapping("/download")
    public Mono<Void> downloadByWriteWith(ServerHttpResponse response) throws IOException {
        response.getHeaders().set(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=demo.xls");
        response.getHeaders().add("Accept-Ranges", "bytes");
        Flux<DataBuffer> flux = excelService.export(1000);
        return response.writeWith(flux);
    }

    public Flux<DataBuffer> export(int dataRow){
        return Flux.create(sink -> {
            sink.next(stringBuffer(ExcelUtil.startWorkbook()));
            sink.next(stringBuffer(ExcelUtil.startSheet()));
            sink.next(stringBuffer(ExcelUtil.startTable()));

            //write title row
            sink.next(stringBuffer(ExcelUtil.writeTitleRow(Sets.newHashSet("title","content"))));
            //write data row
            IntStream.rangeClosed(1,dataRow).forEach(i -> {
                try {
                    TimeUnit.MILLISECONDS.sleep(100);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                sink.next(stringBuffer(ExcelUtil.writeDataRow(Lists.newArrayList("title"+i,"content"+i))));
            });

            sink.next(stringBuffer(ExcelUtil.endTable()));
            sink.next(stringBuffer(ExcelUtil.endSheet()));
            sink.next(stringBuffer(ExcelUtil.endWorkbook()));

            sink.complete();
        });
    }

Here, use ReactiveHttpOutputMessage’s writeWith(Publisher <? Extends DataBuffer> body) method to export while preparing data
Wait for more than ten seconds to play the download box, then the server side output, browser side download, download is completed in about 100 seconds

Summary

At present, the two methods seem to take about the same time, but the latter can avoid overtime. Of course, using traditional mvc can also achieve a similar effect, that is, get the output stream of response to write and flush continuously. However, webflux can cooperate with reactive repository to realize end-to-end reactive stream, while avoiding OOM.