Livonia, MI 12/29/2023
We are thrilled to announce the release 2.1.0 of infoCDC – a lightweight, fast, and simple IBM i Change Data Capture (CDC) component that works with popular middleware platforms, such as MuleSoft and Kafka, and helps delivering data replication and analytics pipelines with minimal or no IBM i coding required.
With this release, infoCDC can process more data in near real time, conserve IBM i storage for analytical use cases, and simplify CDC pipeline operations. Below are the changes included into infoCDC v2.1.0:
Support for tables journaled with only *AFTER image.
CDC Tables UI improvements to display the workflow status and number of DQ messages.
Significant increase in transaction throughput and other performance improvements.
Fixed the Create Timestamp population in the message metadata.
*AFTER image support
Previously, infoCDC required that the DB2 tables added to CDC are journaled with *BOTH image option. There are multiple reasons for this requirement, including:
- Filtering changes applied to specific columns. This is a common requirement for older denormalized tables that may have hundreds or thousands of columns, while the target systems are only interested in changes applied to a small subset. In order to detect changes for specific columns, infoCDC compares the record images before and after the change and would require both to be captured in the journal.
- Sending the keys for Delete operation. When table is journaled with *AFTER image only, the keys of the deleted records are not recorded in the journal, therefore it is not possible to capture and send Delete key values.
Most data replication use cases that need to support Delete operations will still require tables to be journaled with *BOTH images captured. However, when customers build a data pipeline for analytics, AI model training and similar use cases, leveraging *AFTER images could offer a faster, simpler, and smaller-footprint solution.
UI Improvements
We have changed the Work with Tables screen to display more relevant information:
- Added Workflow status for each table. Now there is no need to go to Menu option 2 Work with Replication Flows to check whether the CDC is active for specific journals.
Show the number of messages in the data queue associated with each table. Now there is no need to use SQL queries or external tools to check how many messages are currently in each table’s DQ.
Performance improvements
We have tuned several internal programs to optimize the data retrieval for filtering and status updates. Separately, the before image is only saved and processed when there is a column-based filter defined, otherwise the before image is not checked. Combined these two changes resulted in almost 10x throughput improvements compared to the previous version.
Created Timestamp fix
InfoCDC sends additional metadata with each DQ message, including Created date / time and Updated date / time, representing the time InfoCDC received journal entry, and the time InfoCDC finished processing and placed a message into replication Data Queue. Prior to that release, the Created timestamp was erroneously set to Updated timestamp. With this fix the created and updated timestamps are different and accurately reflect the points in time when InfoCDC received and processed each transaction.
Huge thanks to our product engineering team, and of course to our customers and prospects who continuously provide us the feedback and help us prioritize the new product features.
Contact us now for more information or a free trial license.
Recent Comments