j-easy/easy-batch

📋 Changes

• Issue #394 : Resource leak in `FileRecordReader`
• jackson-databind: `2.12.0` -> `2.12.2`
• yasson: `1.0.7` -> `1.0.8`
• univocity-parsers: `2.9.0` ->`2.9.1`
• xstream: `1.4.15` -> `1.4.16`
• spring: `5.3.2` -> `5.3.4`
• hibernate: `5.4.26.Final` -> `5.4.29.Final`

📋 Changes

• Issue #393 : Errors in reader/writer open/close methods are not handled
• jackson-databind: `2.11.2` -> `2.12.0`
• univocity-parsers: `2.8.4` ->`2.9.0`
• xstream: `1.4.12` -> `1.4.15`
• yamlbeans: `1.13` -> `1.15`
• spring: `5.2.9.RELEASE` -> `5.3.2`
• hibernate: `5.4.21.Final` -> `5.4.26.Final`
• hibernate-validator: `6.1.5.Final` -> `6.1.7.Final`

📋 Changes

• `RecordReader` (and its listener): 7ed62a5d3c0a3ddfaa89fbc0757fddd6c5591288
• `Batch` (and its listener): ba60977cd2135e3f91968ad2f95744d474e77bdc
• `RecordWriter` (and its listener): d437aa550f0c9654fde355f68261b3efd98c0aa4
• `RecordProcessor` (and its listener / sub interfaces): 279ed297d5eb60d590958a11d8971f70199d6e5f
• `Predicate`: 063fcc573d5783f48c01d3375e6d6a6459bf5d4e
• `JobBuilder`: 69299364256fade74a219553540bf04222ef62d8

📦 Migration guide from v6.1 to v7.0

• The most notable change required by this release is regarding job definitions:
• ```diff
• --Job job = new JobBuilder() // v6
• ++Job job = new JobBuilder<String, Tweet>() // v7
• // define batch components
• .build();
• ```
• Specifying input/output types is not mandatory, but not doing so will lead to a raw type usage warning.
• + 11 more

✨ New Features and enhancements

• Issue #368: Add `RetryableRecordProcessor` implementation
• Issue #381: Add default methods in record reader/writer interfaces

🐛 Bug fixes

• Issue #372: `BeanPropertiesPreparedStatementProvider` is failing to prepare statement if bean properties has `null` value

🗑️ Deprecations

• Issue #380: Deprecate the static factory method `JobBuilder#aNewJob`
• Issue #379: Deprecate `ContentBasedBlockingQueueRecordWriterBuilder`

✨ New features

• Issue #261: Add support for JSON mapping with Yasson (JSON-B RI)
• Issue #311: Add support for Java 8 date/time classes
• Issue #353: Add converters for Java 8 date/time types
• Issue #361: Make `JobExecutor` implement `java.lang.AutoCloseable`
• Issue #363: Add batch scanning feature
• Issue #366: Add formatting option in `BeanFieldExtractor`

📦 Enhancements

• Issue #351: `FileRecordWriter` should not throw an Exception at construction time
• Issue #355: Remove system properties from `JobReport#toString`
• Issue #356: Improve job duration formatting
• Issue #357: Show job duration in the logs
• Issue #364: Add default methods in listener interfaces

🐛 Bug fixes

• Issue #352: `XmlRecordReader` should close the input stream
• Issue #365: Incorrect fixed length record marshalling

✨ New tutorials

• [Reading data from a REST endpoint](https://github.com/j-easy/easy-batch/tree/master/easy-batch-tutorials/src/main/java/org/jeasy/batch/tutorials/intermediate/rest)
• [Writing a dynamic header/footer to an output file](https://github.com/j-easy/easy-batch/tree/master/easy-batch-tutorials/src/main/java/org/jeasy/batch/tutorials/intermediate/headerfooter)
• [Skipping bad records with batch scanning on write failure](https://github.com/j-easy/easy-batch/tree/master/easy-batch-tutorials/src/main/java/org/jeasy/batch/tutorials/advanced/scanning)

💥 Breaking changes

• All usages of `java.io.File` (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with `java.nio.file.Path` in v6
• Quartz support was deprecated in v5.3 and has been removed in v6. The entire module `easybatch-quartz` has been removed in v6. This module used to provide two classes that act as a bridge between Easy Batch APIs and Quartz APIs. Those two classes (`EasyBatchJob` and `EasyBatchJobFactory`) can now be copied from the [Quartz tutorial](https://github.com/j-easy/easy-batch/tree/master/easy-batch-tutorials/src/main/java/org/jeasy/batch/tutorials/advanced/quartz).
• MongoDB support has been dropped (Issue #354). Please note that this decision was made after releasing v5.3, hence you will not find any `@Deprecated` annotations on classes of this module in the v5.3 release. Sorry for any inconvenience!
• `JobListener#beforeJobStart` and `JobListener#afterJobEnd` have been renamed to `JobListener#beforeJob and JobListener#afterJob` (Issue #362)
• Jms reader/writer now work with a JMS `Destination` instead of `Queue` to support `Topics` as well (Issue #359)
• The constructor of `JmsMessageTransformer` now accepts a `javax.jms.Session` instead of `javax.jms.QueueSession` to support both queue and topic sessions (Issue #360)
• The `JmsRecordReader` now returns `null` based on a timeout and not a `JmsPoisonMessage` anymore (because `JmsPoisonMessage` has been removed, see "Removed APIs" section of the migration guide)
• The `BlockingQueueRecordReader` now returns `null` based on a timeout and not a `PoisonRecord` anymore (because `PoisonRecord ` has been removed, see "Removed APIs" section of the migration guide)
• + 1 more

📦 Migration guide from v5.3 to v6.0

• Although care has been taken to document all changes in details, a thing or two could have been missed (in which case, apologies upfront).

📦 Maven coordinates

• The group id has changed from `org.easybatch` to `org.jeasy`
• Artifact IDs have been updated like follows: `easybatch-core` -> `easy-batch-core` (same pattern for other artifacts)
• Here is the new maven dependency for the core module:
• ```xml
• <dependency>
• <groupId>org.jeasy</groupId>
• <artifactId>easy-batch-core</artifactId>
• <version>6.0.0</version>
• + 2 more

🗑️ Removed APIs

• `org.easybatch.xml.XmlWrapperTagWriter` was deprecated in v5.3 and has been removed in v6
• `org.easybatch.tools.monitoring.CliJobMonitoringListener`, `org.easybatch.tools.reporting.HtmlJobReportFormatter` and`org.easybatch.tools.reporting.JobReportEmailSender` were deprecated in v5.3 and have been removed in v6. The entire `easybatch-tools` module has been removed
• Quartz support was deprecated in v5.3 and has been removed in v6. The entire module `easybatch-quartz` has been removed in v6
• MongoDB support has been be removed in v6 (See issue #354). Please note that this decision has been made after releasing v5.3, hence you will not find any `@Deprecated` annotation on classes of this module in the v5.3 release. Sorry for any inconvenience!
• `org.easybatch.jms.JmsPoisonMessage`, `org.easybatch.jms.JmsPoisonRecord`, `org.easybatch.jms.JmsPoisonRecordBroadcaster` and `org.easybatch.jms.JmsPoisonRecordFilter` were deprecated in v5.3 and have been removed in v6.
• `org.easybatch.core.record.PoisonRecord`, `org.easybatch.core.listener.PoisonRecordBroadcaster` and `org.easybatch.core.filter.PoisonRecordFilter` were deprecated in v5.3 and have been removed in v6.
• The constructor `org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int)` has been removed in v6. Please note that this constructor was not marked with `@Deprecated` in v5.3 by mistake.
• The constructor `BlockingQueueRecordWriter(final List<BlockingQueue<Record>>` has been be removed in v6 as this writer now operates on a single blocking queue instead of multiple queues.
• + 4 more

📦 Replaced/Renamed/Moved APIs

• `org.easybatch.core.filter.FilteredRecordsSavingRecordFilter` was deprecated in v5.3 and has been removed in v6. Use the new `org.easybatch.core.filter.FilteredRecordsCollector` instead
• `org.easybatch.core.filter.StartWithStringRecordFilter` was deprecated in v5.3 and has been removed in v6. Use `org.easybatch.core.filter.StartsWithStringRecordFilter` instead
• `org.easybatch.core.filter.EmptyRecordFilter` was deprecated in v5.3 and will be removed in v6. Use `org.easybatch.core.filter.EmptyStringRecordFilter` instead
• `org.easybatch.core.filter.EndWithStringRecordFilter` was deprecated in v5.3 and will be removed in v6. Use `org.easybatch.core.filter.EndsWithStringRecordFilter ` instead
• Usages of `java.util.date` (namely in `Header` class) were deprecated in v5.3 and have been replaced with `java.time.LocalDateTime` in v6
• Usages of `java.io.File` (namely in constructors of all file based record readers/writers) were deprecated in v5.3 and have been replaced with `java.nio.file.Path` in v6
• `PayloadExtractor` was deprecated in v5.3 and has been removed in v6. Use `Utils#extractPayloads` instead
• The constructor `org.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String)` has been replaced with one that takes a `java.nio.charset.Charset` instead of a String for the charset name. Please note that this constructor was not marked with `@Deprecated` in v5.3 by mistake.
• + 4 more

✨ New features

• Issue #333: Add Header/Footer callbacks in `FileRecordWriter`
• Issue #350: Add configurable `JobReportFormatter` in `JobReportEmailSender`

📦 Enhancements

• Issue #315: Replace Java Util Logging with SLF4J
• Issue #320: Error Threshold - Zero Errors
• Issue #338: Inefficient `JobMetrics` merge loops in `DefaultJobReportMerger`
• Issue #317: Unused `XMLStreamException`

🐛 Bug fixes

• Issue #349: Incorrect writeCount when the transaction in `HibernateRecordWriter` is rolled back
• Issue #348: Incorrect writeCount when the transaction in `JpaRecordWriter` is rolled back
• Issue #347: Incorrect writeCount when the transaction in `JdbcRecordWriter` is rolled back
• Issue #314: Error while setting a date (`java.util.Date`) field in `JdbcRecordWriter`
• Issue #345: `HtmlJobReportFormatter` should not generate apache velocity logs
• Issue #337: `XmlRecordReader` throws a `ClassCastException` when reading a comment line
• Issue #319: Excel reader Skipping Empty Columns

🗑️ For removal

• `org.easybatch.xml.XmlWrapperTagWriter` is deprecated in v5.3 and will be removed in v6
• `org.easybatch.tools.monitoring.CliJobMonitoringListener`, `org.easybatch.tools.reporting.HtmlJobReportFormatter` and`org.easybatch.tools.reporting.JobReportEmailSender` are deprecated in v5.3 and will be removed in v6. The entire `easybatch-tools` module will be removed
• Quartz support is deprecated in v5.3 and will be removed in v6. All classes in `easybatch-quartz` module are deprecated and the entire module will be removed in v6
• MongoDB support will be removed in v6. Please note that this decision has been made after releasing v5.3, hence you will not find any `@Deprecated` annotation on classes of this module in the v5.3 release.
• `org.easybatch.jms.JmsPoisonMessage`, `org.easybatch.jms.JmsPoisonRecord`, `org.easybatch.jms.JmsPoisonRecordBroadcaster` and `org.easybatch.jms.JmsPoisonRecordFilter` are deprecated in v5.3 and will be removed in v6. The `JmsRecordReader` will return `null` based on a timeout and not a `JmsPoisonMessage`.
• `org.easybatch.core.record.PoisonRecord`, `org.easybatch.core.listener.PoisonRecordBroadcaster` and `org.easybatch.core.filter.PoisonRecordFilter` are deprecated in v5.3 and will be removed in v6. The `org.jeasy.batch.core.reader.BlockingQueueRecordReader` will return `null` based on a timeout and not a `PoisonRecord`
• The constructor `org.jeasy.batch.core.reader.BlockingQueueRecordReader#BlockingQueueRecordReader(java.util.concurrent.BlockingQueue<org.jeasy.batch.core.record.Record>, int)` will be removed in v6. Please note that this constructor was not marked with `@Deprecated` in v5.3 by mistake.
• The constructor `BlockingQueueRecordWriter(final List<BlockingQueue<Record>>` will be removed in v6 as this writer will operate on a single blocking queue.
• + 3 more

📦 With replacement

• `org.easybatch.core.filter.FilteredRecordsSavingRecordFilter` is deprecated in v5.3 and will be removed in v6. Use the new `org.easybatch.core.filter.FilteredRecordsCollector` instead
• `org.easybatch.core.filter.StartWithStringRecordFilter` is deprecated in v5.3 and will be removed in v6. Use `org.easybatch.core.filter.StartsWithStringRecordFilter` instead
• `org.easybatch.core.filter.EmptyRecordFilter` is deprecated in v5.3 and will be removed in v6. Use `org.easybatch.core.filter.EmptyStringRecordFilter` instead
• `org.easybatch.core.filter.EndWithStringRecordFilter` is deprecated in v5.3 and will be removed in v6. Use `org.easybatch.core.filter.EndsWithStringRecordFilter ` instead
• Usages of `java.util.date` (namely in `Header` class) are deprecated and will be replaced with `java.time.LocalDateTime` starting from v6
• Usages of `java.io.File` (namely in constrcutors of all file based record readers/writers) are deprecated and will be replaced with `java.nio.file.Path` starting from v6
• `PayloadExtractor` is deprecated in v5.3 and will be removed in v6. Use `Utils#extractPayloads` instead
• The constructor `org.jeasy.batch.extensions.yaml.YamlRecordReader#YamlRecordReader(java.io.InputStream, java.lang.String)` will be replaced with one that takes a `java.nio.charset.Charset` instead of a String for the charset name. Please note that this constructor was not marked with `@Deprecated` in v5.3 by mistake.
• + 2 more

📋 Changes

• issue #280 : Invert `PipelineListener` call order
• issue #283 : Add OSGi headers to manifest
• issue #303 : Add `FilteredRecordsSavingRecordFilter`
• issue #291 : `XmlRecordReader` does not escape `gt` and `lt`
• issue #292 : `XmlRecordReader` does not call `getLocalPart()` on `QName`
• issue #293 : `DefaultJobReportFormatter.formatReport` duration rolls over after 24 Hours
• issue #296 : `FilteredCount` metric should be renamed to `FilterCount`
• issue #301 : Add getters for `file` and `charset` in `AbstractFileRecordReader`
• + 5 more

📋 Changes

• issue #92 : Add email sending job listener
• issue #121 : Add support for univocity-parsers
• issue #122 : Add support for Yaml format
• issue #254 : Add `MultiFileRecordReader`
• issue #255 : Add `XmlFileRecordReader`
• issue #256 : Add `JsonFileRecordReader`
• issue #260 : Add parameter to control how quartz should act when certain job is delayed for a long time
• issue #269 : Add encoding parameter in `XmlRecordReader`
• + 15 more

📋 Changes

• issue #237 : MsExcel file not closed by the MsExcelRecordReader

📋 Changes

• issue #211 : Inconsistent Batch API
• issue #233 : Inconsistent RecordReader API
• issue #223 : Inconsistent RecordWriter API
• issue #227 : Inconsistent RecordDispatcher API
• issue #221 : Resource management should be handled by the framework
• issue #222 : Transaction management should be handled by the framework
• issue #230 : JobExecutor should use an ExecutorService
• issue #220 : Adapt listeners to batch processing workflow
• + 29 more

📦 JobBuilder

• Methods deprecated in v4.2 have been removed in v5:
• `JobBuilder#skip(long)`
• `JobBuilder#limit(long)`
• `JobBuilder#timeout(long)` and `JobBuilder#timeout(long, TimeUnit)`
• `JobBuilder#silentMode(boolean)`
• `JobBuilder#strictMode(boolean)`
• `JobBuilder#jmxMode(boolean)`
• `JobBuilder#call()`
• + 5 more

📦 JobExecutor

• The method `org.easybatch.core.job.JobExecutor.execute` is no more static, you need to create a `JobExecutor` instance to execute jobs
• Job executors must be shutdown explicitly

📦 Resource handling & transaction management

• The constructors of `JdbcRecordReader/JdbcRecordWriter` now take a JDBC `DataSource` as argument instead of a `Connection` and the `JdbcConnectionListener/JdbcTransactionListener` have been removed
• The constructors of `JpaRecordReader/JpaRecordWriter` now take a `EntityManagerFactory` as argument instead of a `EntityManager` and the `JpaEntityManagerListener/JpaTransactionListener` have been removed
• The constructors of `HibernateRecordReader/HibernateRecordWriter` now take a `SessionFactory` as argument instead of a `Session` and the `HibernateSessionListener/HibernateTransactionListener` have been removed

📦 Moved APIs

• `org.easybatch.core.dispatcher.Predicate` has moved to `org.easybatch.core.writer.Predicate`
• `org.easybatch.core.dispatcher.DefaultPredicate` has moved to `org.easybatch.core.writer.DefaultPredicate`
• `org.easybatch.core.dispatcher.PoisonRecordBroadcaster` has moved to `org.easybatch.core.listener.PoisonRecordBroadcaster`

🗑️ Removed APIs

• The "record dispatcher" concept has been removed. All dispatchers have been transformed into writers:
• `BroadcastRecordDispatcher` -> `BlockingQueueRecordWriter`
• `ContentBasedRecordDispatcher` -> `ContentBasedBlockingQueueRecordWriter`
• `ContentBasedRecordDispatcherBuilder` -> `ContentBasedBlockingQueueRecordWriterBuilder`
• `RoundRobinRecordDispatcher` -> `RoundRobinBlockingQueueRecordWriter`
• `RandomRecordDispatcher` -> `RandomBlockingQueueRecordWriter`
• `BroadcastJmsRecordDispatcher` -> `BroadcastJmsQueueRecordWriter`
• `ContentBasedJmsRecordDispatcher` -> `ContentBasedJmsQueueRecordWriter`
• + 16 more

📋 Other changes

• `org.easybatch.core.processor.ComputationalRecordProcessor` deprecated in v4.2 has been removed
• `JobResult` deprecated in v4.2 has been removed from `JobReport`, this is related to the removal of `ComputationalRecordProcessor`
• All constructors in `XmlWrapperTagWriter` now take a `File` instead of `FileWriter`, this is due to a resource leak
• `BeanValidationRecordValidator` is no more parametrized type
• `Predicate` is no more a parametrized type
• `BlockingQueueRecordReader` is no more a parametrized type
• `BlockingQueueRecordWriter` is no more a parametrized type
• `RecordFieldExtractor` interface has been renamed to `FieldExtractor`
• + 1 more

📋 Changes

• issue #243 : add OpenCsvRecordMarshaller
• issue #242 : add XmlRecordValidator
• issue #241 : add option to activate/deactivate recursivity in FileRecordReader
• issue #238 : FlatFileRecordReader should not throw a FileNotFoundException at construction time
• issue #236 : add constructor with java.nio.file.Path in FileRecordReader
• issue #239 : add constructor with java.nio.file.Path in FileRecordWriter
• issue #240 : add constructor with java.nio.file.Path in FlatFileRecordReader
• issue #244 : ApacheCommonCsvRecord is redundant
• + 2 more

📋 Changes

• issue #211 : Inconsistent Batch API
• issue #233 : Inconsistent RecordReader API
• issue #223 : Inconsistent RecordWriter API
• issue #227 : Inconsistent RecordDispatcher API
• issue #221 : Resource management should be handled by the framework
• issue #222 : Transaction management should be handled by the framework
• issue #230 : JobExecutor should use an ExecutorService
• issue #220 : Adapt listeners to batch processing workflow
• + 12 more

📋 Changes

• issue #205 : Add BeanPropertiesPreparedStatementProvider
• issue #208 : add StandardErrorRecordWriter
• issue #214 : Replace "strict mode" parameter with "error threshold"
• issue #202 : Incorrect duration when unable to open the record reader
• issue #203 : Incorrect data source when unable to open the record reader
• issue #206 : Incoherent log message when filtering a batch of records
• issue #207 : Incorrect log message when unable to process a batch of records
• issue #210 : java.rmi.UnmarshalException when using the JobMonitorProxy
• + 7 more

📋 Changes

• issue #123: Add support for MS Excel format using Apache POI
• issue #168: Add last error in job report
• issue #171: Add support to trim whitespace for `FixedLengthRecordMapper`
• issue #175: Add retry on failure for the record reader
• issue #187: Add host name to job report
• issue #196: Add JMX monitoring push notifications
• issue #198: `BlockingQueueRecordReader` should be configurable with the number of poison records received
• Issue #169: When Reader fails Job Listeners are not being invoked
• + 7 more

📋 Changes

• issue #136 : Redesign the workflow into a consistent pipeline
• issue #160 : Improve workflow consistency
• issue #137 : Unify Handlers and Listeners
• issue #142 : Introduce `JobParameters` and `JobMetrics`
• issue #141 : Rename `Engine` to `Job`
• issue #154 : Rename `MultiRecord` to `Batch`
• issue #148 : Migrate to Java 7
• issue #162 : Add PayloadExtractor
• + 29 more

📋 Changes

• Added the "skip" parameter (issue #77)
• Added the "limit" parameter (issue #78)
• Added record writers: OutputStream (issue #79), JPA (issue #80), MongoDB (issue #82), JDBC (issue #83), JMS (issue #85), Collection (issue #106), String (issue #109)
• Added Hibernate support: new `HibernateRecordReader` and `HibernateRecordWriter` APIs (issue #81)
• Added `XmlRecordCompactor` and `JsonRecordCompactor` to flatten hierarchical data (issue #86)
• Added `IterableRecordReader` and deprecated `ListRecordReader` (issue #88)
• Added `RecordCollector` (issue #89)
• Added record marshallers: Xstream (issue #94), Xml (issue #95), Delimited (issue #96), FixedLength (issue #97), ApacheCommonCsv (issue #98), Gson (issue #99), Jackson (issue #100)
• + 9 more