Skip to content

Hardening and scaling

This page includes more details about running IBM i change data capture reliably in production: delivery guarantees, security, throughput, recovery, and monitoring. Each section links to the product guide with the details.

Capture configuration

  • Journal tables with *BOTH record images when you need delete operations or column-level change filtering.
  • Define primary or unique keys, or user-provided composite keys for legacy files, so that every target applies changes correctly.
  • Replicate only the tables, operations, and columns that you need.
  • Set the Handle Key Changes flag to N for tables where key values never change. InfoCDC then skips the key comparison and increases throughput. See Table maintenance.

Security

Throughput and scaling

Consumer What to tune Reference
InfoConnect Hub, database targets The batch size of the Data Flow. The Hub writes each batch in one transaction and applies only the latest event for each row key in a batch. One Data Flow can replicate many tables from the same data queue. How the JDBC sink writes data, Replicating many tables with one Data Flow
InfoConnect Hub, message brokers Dynamic data publishing, which routes each table to its own queue, topic, or event hub. Dynamic data publishing
InfoConnect Hub, availability A clustered Hub behind a load balancer. Cluster configuration
InfoConnect for Kafka The number of consumers, read concurrency, poll interval, flush percentage, and data queue wait time of the source connector. Source connector configuration properties
InfoConnect for Kafka, topic routing Route messages from one data queue to multiple topics. Routing messages from a single data queue to multiple topics

Error handling and recovery

  • The Hub retries transient database errors until they succeed. For other errors it sets the Data Flow to FAILED and saves the failed batch to the failed records folder. See Errors and retries and Data Flow states and errors.
  • The Kafka sink connectors support a dead letter queue (DLQ) for records that they cannot process. See the sink connector samples.
  • A Hub Data Flow has DLQ Settings, where you enable a dead letter queue for failed messages. See Data Flows.

Observability and log aggregation

  • InfoCDC monitoring. InfoCDC can email your administrators when a workflow is in error or a data queue grows too fast. See Workflow monitoring.

  • Logs. Each Hub module writes an output log and an error log to the log folder. See Log files. The Kafka connector can write critical exceptions to a topic (as400-error-log) when you set log.exceptions=true. See Logging exceptions.

  • Log levels. The Hub and MCP log at INFO by default; the RPA Hub logs its own packages at DEBUG. To log only warnings and errors, set the InfoConnect packages to WARN; use DEBUG only while troubleshooting. See Log levels.
  • Health checks. The Hub and the MCP server provide /actuator/health. See Health check.
  • State of the flows. The Data Flow state (ACTIVE, INACTIVE, FAILED, INVALID) shows whether a flow runs. See Data Flow states and errors.
  • Aggregation tools. The log files are plain files in a known folder, so your standard log shipper can forward them to platforms such as Splunk or the Elastic stack.

Deployment

Run the Hub in containers or on Kubernetes if you want reproducible deployments. See Deploying InfoConnect Hub with Docker, Kubernetes and Cloud Run.

Delivery guarantees

  • Acknowledgements. Set the Expect Acknowledgement flag on a table in InfoCDC to get end-to-end delivery guarantees. InfoCDC then keeps each entry in ACK_PENDING status until the consumer confirms that the change reached its target. See Acknowledgements and guaranteed delivery.
  • Consumer acknowledgements. The Kafka connector acknowledges through the IBM i acknowledgement programs. See Source acknowledgement.
  • Staging table. Purge and reorganize the InfoCDC staging table regularly, so that it does not grow without limit. See Staging table maintenance.
  • Replay. The InfoCDC workflow remembers the last processed journal receiver and sequence number. You can reset the sequence number to resend historical changes. Your journal receiver management policy must keep the receivers available for the time that a consumer can be offline. See Replication flow management.