Skip to content

Data Flows

A Data Flow is a real-time data pipeline between a data queue (DQ) on your IBM i and an external system. An Outbound Data Flow sends data from the IBM i to a cloud service. An Inbound Data Flow brings data from a cloud service to the IBM i.

To create a Data Flow:

  1. In the navigation bar, click Data Flows.
  2. Click Create Data Flow.
  3. On the Choose a data flow type screen, choose Outbound (IBM i to Cloud) or Inbound (Cloud to IBM i) and select the Native Connection to your IBM i. The Help me to choose button opens the documentation in a dialog.

    Choose a data flow type and a Native Connection

  4. Fill in the form, see Data Flow form, and click Submit.

The Data Flows list shows the columns Name, Type, Status, Data Queue and Format File (as LIBRARY/NAME), Sink, Msgs 5m, DQ Depth, Min ms, Max ms, Avg ms and Last Message. Click a Data Flow to open Update Data Flow.

Monitoring a Data Flow

The metrics columns of the list show what a running Data Flow does. They are collected in memory since the Hub started and cover a rolling window of the last 5 minutes, so they start from zero after a restart.

Column Meaning
Msgs 5m Messages received from the source in the last 5 minutes
DQ Depth Entries currently waiting in the data queue. The Hub reads the value from the IBM i
Min ms, Max ms, Avg ms Fastest, slowest and average time per message in the last 5 minutes, from the read at the source to the publish at the sink
Last Message When the last message was received from the source

GET /admin/dqpollers returns the same values in the metrics object of each Data Flow. It also contains the totals since the start (messagesReceivedSinceStart), the split of the processing time into sourceTimeMs and sinkTimeMs, the age of the oldest entry in the queue (dqOldestMessageAt), the time of the last depth check (dqDepthCheckedAt) and the reason when the depth is not available (dqDepthError).

Data Flow statuses and actions

The top bar of Update Data Flow shows the status badge and the buttons Start (Stop while the Data Flow is active) and Remove. If the Data Flow has an error, its text is shown below the top bar.

Status Meaning
Active The Data Flow is running
Inactive The Data Flow is created or stopped and is not running
Invalid The sink could not be initialised. Correct the configuration and submit it again
Failed A runtime error stopped the Data Flow. The error text is shown on the page

You can start a Data Flow while its connection is ACTIVE or WARNING. To delete a Data Flow, click Remove and confirm with Yes. For how to recover from Invalid and Failed Data Flows, see Data Flow States and Errors.

Data Flow form

The form has the tabs Form and JSON. In the JSON tab (Data Flow JSON), the Form tab is updated as soon as the text is valid JSON. Sensitive fields within sinkTypeProperties, such as passwords or keys, must be encrypted values in the JSON, see Encrypting secrets.

Create Data Flow form with the JSON tab selected

Main fields

Field Notes
Data Flow Name Required, unique. Letters, digits and hyphens only (Poller Name must not contain special characters). Cannot be changed on update
Data Flow Type Source (Outbound) or Sink (Inbound)
Connection name The Native Connection
InfoCDC Source Checkbox, checked by default (Use InfoCDC as the source for this data flow)

Data Queue Settings

Field Notes
Batch size Number of messages read in one cycle. Use 1 or more; 0 or less reads one message per cycle
Thread count Minimum 1
Source data queue name Required. Labelled Sink data queue name for Inbound flows
Library name Required
Data queue key Optional
Data Search type Equal, Not equal, Less than, Less than or equal, Greater than or equal. Default Greater than or equal
Format File Name, Format File Library Optional. Leave blank to use auto: the Hub reads the format file from each InfoCDC message
Wait Time (seconds) Default 60, minimum 1
Format Exception Action Discard (default) or Place back into queue

Sink Settings

Select the Sink Type (required; default JMS). The fields below it depend on the selected type, see Sink settings reference.

DLQ Settings

Field Notes
DLQ Active -
Error Tolerance -
DLQ Name, DLQ Library, DLQ Format File Name, DLQ Format File Library DLQ Name is required by the Hub when DLQ Active is selected

Transformer

Optional. Transform the data before it is sent to the destination.

Sink types

The Sink Type list contains 15 types: JMS, SNS, SQS, ServiceBus, EventGrid, EventHub, Kafka, JDBC, GCP, GCS, S3, ADLS, FTP, SFTP and ABS. The file and object storage sinks GCS, S3, ADLS, FTP, SFTP and ABS are new in 5.0.0.

Create Data Flow form with the Sink Type list open

The UI offers all sink types for both directions, but the Hub accepts only some of them for Inbound Data Flows:

Sink Type Outbound Inbound
JMS, SQS, ServiceBus, Kafka, EventHub, GCP Yes Yes
SNS, EventGrid, JDBC, GCS, S3, ADLS, FTP, SFTP, ABS Yes No (outbound only)

If you use an outbound-only sink type in an Inbound Data Flow, the Hub rejects the Data Flow with an error, for example JDBC poller not support Sink type.

Outbound Data Flow

An Outbound Data Flow listens for changes in an IBM i data queue and sends them to an external system. Configure these settings:

  • Main fields: Set Data Flow Type to Source, select the Native Connection to your IBM i and set InfoCDC Source.
  • Data Queue Settings: Enter the Source data queue name and Library name to monitor, plus the other settings described above.
  • Sink Settings: Select the Sink Type (for example JDBC, SNS, SQS, Kafka, S3 or SFTP) and fill in the required properties for that type, such as connection URL, topic name or credentials.
  • DLQ Settings and Transformer: Optional.

Create Data Flow form for an Outbound Data Flow

Sample JSON configuration for a Data Flow
{
  "pollerName": "CDC2RDS",
  "dqName": "DQNAME",
  "libraryName": "INFOCDCDTA",
  "dqFormatFileName": "FMTFILE",
  "dqFormatFileLibrary": "INFOCDCDTA",
  "connectionName": "NATIVE",
  "infoCDCConfigured": true,
  "dqWaitTime": 30,
  "sinkType": "JDBC",
  "sinkTypeProperties": {
    "jdbcProperties": {
      "driverClassName": "com.mysql.cj.jdbc.Driver",
      "maxPoolSize": 10,
      "minPoolSize": 3,
      "jdbcUrl": "jdbc:mysql://my-database-server:3306/testdb?useSSL=false&serverTimezone=UTC&allowPublicKeyRetrieval=true",
      "username": "encrypted-user-name",
      "password": "encrypted-password"
    }
  },
  "pollerType": "Source",
  "threadCount": "5"
}

In the Admin API, a Data Flow is called a DQ poller: pollerType Source is an Outbound and Sink an Inbound Data Flow. You can also create an Outbound Data Flow with the Admin API, see API Reference.

Inbound Data Flow

An Inbound Data Flow listens for messages from an external system and places them into an IBM i data queue. The configuration is the reverse of an Outbound Data Flow:

  • Set Data Flow Type to Sink.
  • Configure the external system in Sink Settings (for example Kafka).
  • Enter the target data queue in Sink data queue name.

Inbound Data Flows support only the sink types JMS, SQS, ServiceBus, Kafka, EventHub and GCP, see Sink types. For a GCP sink, the field Subscription is shown for Inbound Data Flows and Topic for Outbound Data Flows.

Create Data Flow form for an Inbound Data Flow

You can also create an Inbound Data Flow with the Admin API, see API Reference.