Spring Batch Partition for Scaling & Parallel Processing

Link: http://javasampleapproach.com/spring-framework/spring-boot/spring-batch-partition-scaling-parallel-processing

Spring Batch Partition for Scaling & Parallel Processing

For Scaling & Parallel Processing, Spring Batch provides various solutions: Multi-threaded Step, Parallel Steps, Remote Chunking of Step & Partitioning a Step. In the tutorial, JavaSampleApproach will introduce Partitioning a Step cleary by a sample project.

Related articles:
– Spring Batch Job with Parallel Steps
Link: http://javasampleapproach.com/spring-framework/spring-batch/spring-batch-job-parallel-steps
– How to use Spring Batch Late Binding – Step Scope & Job Scope
Link: http://javasampleapproach.com/spring-framework/spring-boot/use-spring-batch-late-binding-step-scope-job-scope

Spring Batch provides an solution for partitioning a Step execution by remotely or easily configuration for local processing.

The Job in left hand is executed sequentially, Master step is partitioning step that has some Slave steps. Slave steps can be remote services or local threads.

For configuring partitioning step, Spring Batch provies PartitionHandler component & Partitioner interface.

1. PartitionHandler

The componenet PartitionHandler knows about the kind of remote services(RMI remoting, EJB remoting,… or local threads) or grid numbers. PartitionHandler can send StepExecution requests to the remote Steps, in various format, like a DTO.

The gridSize defines the number of step executions, so we should to consider the size of TaskExecutor’s thread pool.

2. Partitioner

The Partitioner interface is used to build execution contexts as input parameters for step executions.

Map contains a unique name for each step execution that associated with ExecutionContext’s value.

3. How to Binding input Map to Steps

StepScope feature of Spring Batch can help us to late binding data from PartitionHandler to each step at runtime.
See more: How to use Spring Batch Late Binding – Step Scope & Job Scope

II. PRACTICE

In the tutorial, we create a Batch Job that has only one partition step with 5 slave steps for inserting data from 5 csv files to MySql database.

Technologies

– Java 1.8
– Maven 3.3.9
– Spring Tool Suite – Version 3.8.1.RELEASE
– Spring Boot: 1.5.1.RELEASE
– MySQL Database 1.4

Step to do

– Create Spring Boot project
– Create a simple model
– Create DAO class
– Create Batch Job Step

– Create Batch Job Partitioner
– Configure Partitional Batch Job
– Create JobLaunchController
– Create 5 csv files
– Run & Check results