Skip to content

Data Flow States and Errors

Overview

A Data Flow moves records between an IBM i data queue and an external system. This page describes the states of a Data Flow, the problems the Hub resolves on its own, and how you recover when it does not.

In the web UI the object is a Data Flow (the Data Flows page). In the Admin API it is a DQ poller (endpoints under /admin/dqpollers). This page uses "Data Flow" and keeps "DQ poller" where it is quoted from API messages. For the web UI, see Data Flows in the User Manual.


Data Flow states

A Data Flow has one of four states. The Data Flows page shows it as a badge. GET /admin/dqpollers and GET /admin/dqpollers/{pollerName} return it in the status field.

State Meaning How it is reached
INACTIVE The Data Flow exists but does not poll. Created without status (the default), stopped with POST .../stop, or updated with PUT.
ACTIVE The Data Flow polls the data queue. It stays ACTIVE while it waits for a lost network connection to return. Created with "status": "ACTIVE", or started with POST .../start.
INVALID The Hub could not initialize the Data Flow, for example because the target system rejected the credentials or cannot be reached. The reason is stored in the exception field (up to 255 characters). Initialization fails when the Data Flow is created, updated, or loaded at Hub startup.
FAILED An error stopped the Data Flow while it was running, for example the target system rejected a write with an error that is not temporary. All its threads stop, and the error message is stored in the exception field. Delivering records to the target system fails at run time.

What you can do in each state:

  • INACTIVE: start it with POST /admin/dqpollers/{pollerName}/start.
  • ACTIVE: stop it with POST /admin/dqpollers/{pollerName}/stop.
  • INVALID or FAILED: you cannot start or stop it. Correct the configuration and update it, as described in Recovering a FAILED or INVALID Data Flow. You can also delete it.

In the UI, the badges read Inactive, Active, Invalid and Failed.


Requirements for starting

A Data Flow can start only when the IBM i connection it uses is ACTIVE or WARNING.

If the connection is in any other state, the start call returns HTTP 400:

{
  "message": "Connection 'NATIVE' must be started before starting the poller"
}

Check the connection with GET /admin/connections/{connectionName}. Start it, and then start the Data Flow again:

POST /admin/connections/{connectionName}/start

The state checks come first. A FAILED or INVALID Data Flow is rejected with the messages in Start and stop responses by state, whatever the state of the connection.


Failure behavior

Automatic recovery

The Hub recovers from temporary problems without changing the state of the Data Flow:

Problem What the Hub does
The network connection to IBM i is lost while the Hub reads the data queue The Data Flow stays ACTIVE and reconnects automatically. The wait between attempts starts at 1 second and doubles up to 60 seconds. By default, the Hub retries without a limit (maxretries=-1).
A temporary error occurs while writing to the target system, such as a network timeout or an unavailable service The Hub waits network.retry.delay.interval milliseconds and repeats the write. The Data Flow stays ACTIVE, and the retries continue until the write succeeds.

Errors that are not temporary, such as rejected credentials or a missing folder, are not retried. The Data Flow becomes FAILED. The Hub does not restart a FAILED Data Flow automatically.

Set the retry values in the Hub application.properties:

Property Default Description
network.retry.delay.interval 60000 Wait time in milliseconds between retries of a temporary write error.
maxretries -1 Maximum number of retries when reading the IBM i data queue. -1 means unlimited.

Failed-record storage

When a write fails and the Data Flow becomes FAILED, the Hub saves the records that it could not deliver to a file on the Hub server. For sinks that write in batches, one entry holds the whole batch.

Files are created in the folder set by failed.records.base-path. A relative path is resolved against the working folder of the Hub.

Item Value
File name <pollerName>_<yyyy_MM_dd>_<NNN>.json, for example orders-to-sftp_2026_10_12_001.json
Format One JSON object per line
New file Created when the current file reaches the maximum size. <NNN> increases.

Each entry has these fields:

Field Description
id Unique ID of the entry.
payload The record, or the batch of records, that could not be delivered.
pollerName Name of the Data Flow.
exceptionType Java class of the error.
exception Error message.
stackTrace Full stack trace of the error.
source READ-SINK for a failed write to the target system.
timestamp Time of the failure.

Set the storage in the Hub application.properties:

Property Default Description
failed.records.base-path ${FAILED_RECORDS_PATH:failed-records} Folder for the files. The environment variable FAILED_RECORDS_PATH overrides the default folder name.
failed.records.max-file-size-mb 10 Maximum size of one file in megabytes.

Email alert for FAILED Data Flows

The Hub can send an email through an SMTP server that you configure when a Data Flow becomes FAILED. The alert is off by default. It is not a message on a queue or topic.

Item Value
Subject Poller <name> failed
Body Poller failed due to an error: <error text>. The error text is written in upper case.

The Hub connects to the SMTP server with SSL and authentication. Set these properties in the Hub application.properties:

Property Default Description
smtp.server.enable false Set to true to send alerts.
smtp.server.host smtp.gmail.com SMTP server host.
smtp.server.port 465 SMTP server port.
smtp.server.username source@example.com SMTP user name.
smtp.server.password ApplicationPassword SMTP password.
smtp.server.target target@example.com Recipient address of the alert.

The default host, user name, password and recipient are placeholders. Replace them with your own values before you enable the alert.


Recovering a FAILED or INVALID Data Flow

A FAILED or INVALID Data Flow does not return to work by itself. Update it with its complete, corrected definition. The Hub then re-creates it. The new Data Flow is INACTIVE, or INVALID again with a new exception if the Hub still cannot initialize it. Then you can start it.

In the web UI

  1. Open the Data Flows page. Point at the Failed or Invalid badge to see the error text.
  2. Click the row to open Update Data Flow. The error text is also shown above the form.
  3. Fix the cause, usually a credential, host name, folder, or other sink property, and click Submit.
  4. When the status is Inactive, click Start.

For Invalid rows, the Data Flows page also shows a restart icon next to the badge. It sends the stored definition again and starts the Data Flow. Use it when the cause was outside the Hub, for example when the target system was unavailable and is available again. If the cause is in the configuration, edit the Data Flow as described above.

Note

The restart icon exists only for Invalid rows. A Failed Data Flow has no restart icon. On the Update Data Flow page, Start returns Cannot start FAILED poller until you submit the Data Flow again.

With the Admin API

  1. Read the state and the error:

    GET /admin/dqpollers/{pollerName}
    
    {
      "pollerName": "orders-to-sftp",
      "status": "FAILED",
      "exception": "<error text>"
    }
    
  2. Fix the cause: credentials, host, folder, network access, or the target object.

  3. Send the complete Data Flow definition, which is the creation body with the corrected values. The Hub removes the old Data Flow and creates it again. Whatever status the body contains, the new Data Flow is INACTIVE, or INVALID if its initialization fails. Encrypt secrets as described in Encrypting credentials.

    PUT /admin/dqpollers/{pollerName}
    
  4. Start it:

    POST /admin/dqpollers/{pollerName}/start
    
    {
      "message": "DQ poller: 'orders-to-sftp' is running."
    }
    
Sample JSON for the update of a JDBC Data Flow
{
  "pollerName": "CDC2RDS",
  "connectionName": "NATIVE",
  "pollerType": "Source",
  "dqName": "DQNAME",
  "libraryName": "INFOCDCDTA",
  "dqFormatFileName": "FMTFILE",
  "dqFormatFileLibrary": "INFOCDCDTA",
  "infoCDCConfigured": true,
  "threadCount": 5,
  "sinkType": "JDBC",
  "sinkTypeProperties": {
    "jdbcProperties": {
      "driverClassName": "com.mysql.cj.jdbc.Driver",
      "jdbcUrl": "jdbc:mysql://my-database-server:3306/testdb",
      "username": "<encrypted-username>",
      "password": "<encrypted-password>",
      "maxPoolSize": 10,
      "minPoolSize": 3
    }
  }
}

If the new definition cannot be initialised, the update returns HTTP 200 and the Data Flow is saved as INVALID with the error in exception. Unexpected errors return HTTP 400 with DQ poller '<name>': Invalid '<sinkType>' properties.

Sink-specific settings, and the checks the Hub runs when it creates a Data Flow, are described in File Sinks for Data Flows and IBM i Data Replication.


Admin API reference

Responses of the Data Flow endpoints use the format {"message": "<text>"}.

Start and stop responses by state

A call that the current state does not allow returns HTTP 400:

{
  "message": "Cannot start FAILED poller"
}
State POST .../start POST .../stop
INACTIVE Starts the Data Flow (HTTP 200). DQ poller: '<name>' isn't running
ACTIVE DQ poller: '<name>' isn't stopped Stops the Data Flow (HTTP 200).
INVALID Cannot start INVALID poller Cannot stop INVALID poller
FAILED Cannot start FAILED poller Can not stop failed poller. Please update poller configuration

While a Data Flow is shutting down, both calls return DQ poller: '<name>' is in stopping process.

Validation errors on create and update

If the body of POST /admin/dqpollers or PUT /admin/dqpollers/{pollerName} is not valid, the Hub returns HTTP 500. The message lists every problem found, one numbered line per error:

{
  "message": "Errors: \n1: Connection name is required\n2: Name should not contain special characters\n"
}

The Hub checks these rules for every Data Flow:

Field Rule Message
pollerName Required. Only letters, digits and - are allowed. DQPoller name is required, Name should not contain special characters
connectionName Required. Connection name is required
dqName Required. Polling data queue name is required
libraryName Required. Library where data queue locates is required

The Hub also validates the sink properties. The messages are listed with each sink, for example in File Sinks for Data Flows.

If the connection does not exist, the call returns HTTP 404 Connection: '<name>' does not exist. When a Data Flow is created, the Hub also checks that the data queue and the format file exist on IBM i. A missing object is reported with HTTP 400, for example Data queue <name> does not exist in library <library>.

Common API messages

Call HTTP Message
POST /admin/dqpollers 200 DQ poller: '<name>' has been registered.
POST /admin/dqpollers/{pollerName}/start 200 DQ poller: '<name>' is running.
POST /admin/dqpollers/{pollerName}/stop 200 DQ poller: '<name>' has been stopped.
DELETE /admin/dqpollers/{pollerName} 200 DQ poller: '<name>' deleted.
POST /admin/dqpollers 400 DQ poller: '<name>' already exists
Calls with a Data Flow name in the path 404 DQ poller: '<name>' doesn't exists
POST .../start 400 Connection '<name>' must be started before starting the poller (see Requirements for starting)
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} 400 Connection '<connection>' doesn't match poller '<poller>' configuration: maxConnections '<n>' should be equal or higher then threadCount '<m>' or equal -1 (no limits to create new connection in pool)
PUT /admin/dqpollers/{pollerName} 400 DQ poller '<name>': Invalid '<sinkType>' properties
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} 404 Connection: '<name>' does not exist.
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} 500 Errors: ... (see Validation errors on create and update)

Start and stop errors that depend on the state are listed in Start and stop responses by state.