Data Flow States and Errors
Overview
A Data Flow moves records between an IBM i data queue and an external system. This page describes the states of a Data Flow, the problems the Hub resolves on its own, and how you recover when it does not.
In the web UI the object is a Data Flow (the Data Flows page). In the Admin API it is a DQ poller (endpoints under /admin/dqpollers). This page uses "Data Flow" and keeps "DQ poller" where it is quoted from API messages. For the web UI, see Data Flows in the User Manual.
Data Flow states
A Data Flow has one of four states. The Data Flows page shows it as a badge. GET /admin/dqpollers and GET /admin/dqpollers/{pollerName} return it in the status field.
| State | Meaning | How it is reached |
|---|---|---|
INACTIVE |
The Data Flow exists but does not poll. | Created without status (the default), stopped with POST .../stop, or updated with PUT. |
ACTIVE |
The Data Flow polls the data queue. It stays ACTIVE while it waits for a lost network connection to return. |
Created with "status": "ACTIVE", or started with POST .../start. |
INVALID |
The Hub could not initialize the Data Flow, for example because the target system rejected the credentials or cannot be reached. The reason is stored in the exception field (up to 255 characters). |
Initialization fails when the Data Flow is created, updated, or loaded at Hub startup. |
FAILED |
An error stopped the Data Flow while it was running, for example the target system rejected a write with an error that is not temporary. All its threads stop, and the error message is stored in the exception field. |
Delivering records to the target system fails at run time. |
What you can do in each state:
INACTIVE: start it withPOST /admin/dqpollers/{pollerName}/start.ACTIVE: stop it withPOST /admin/dqpollers/{pollerName}/stop.INVALIDorFAILED: you cannot start or stop it. Correct the configuration and update it, as described in Recovering a FAILED or INVALID Data Flow. You can also delete it.
In the UI, the badges read Inactive, Active, Invalid and Failed.
Requirements for starting
A Data Flow can start only when the IBM i connection it uses is ACTIVE or WARNING.
If the connection is in any other state, the start call returns HTTP 400:
Check the connection with GET /admin/connections/{connectionName}. Start it, and then start the Data Flow again:
The state checks come first. A FAILED or INVALID Data Flow is rejected with the messages in Start and stop responses by state, whatever the state of the connection.
Failure behavior
Automatic recovery
The Hub recovers from temporary problems without changing the state of the Data Flow:
| Problem | What the Hub does |
|---|---|
| The network connection to IBM i is lost while the Hub reads the data queue | The Data Flow stays ACTIVE and reconnects automatically. The wait between attempts starts at 1 second and doubles up to 60 seconds. By default, the Hub retries without a limit (maxretries=-1). |
| A temporary error occurs while writing to the target system, such as a network timeout or an unavailable service | The Hub waits network.retry.delay.interval milliseconds and repeats the write. The Data Flow stays ACTIVE, and the retries continue until the write succeeds. |
Errors that are not temporary, such as rejected credentials or a missing folder, are not retried. The Data Flow becomes FAILED. The Hub does not restart a FAILED Data Flow automatically.
Set the retry values in the Hub application.properties:
| Property | Default | Description |
|---|---|---|
network.retry.delay.interval |
60000 |
Wait time in milliseconds between retries of a temporary write error. |
maxretries |
-1 |
Maximum number of retries when reading the IBM i data queue. -1 means unlimited. |
Failed-record storage
When a write fails and the Data Flow becomes FAILED, the Hub saves the records that it could not deliver to a file on the Hub server. For sinks that write in batches, one entry holds the whole batch.
Files are created in the folder set by failed.records.base-path. A relative path is resolved against the working folder of the Hub.
| Item | Value |
|---|---|
| File name | <pollerName>_<yyyy_MM_dd>_<NNN>.json, for example orders-to-sftp_2026_10_12_001.json |
| Format | One JSON object per line |
| New file | Created when the current file reaches the maximum size. <NNN> increases. |
Each entry has these fields:
| Field | Description |
|---|---|
id |
Unique ID of the entry. |
payload |
The record, or the batch of records, that could not be delivered. |
pollerName |
Name of the Data Flow. |
exceptionType |
Java class of the error. |
exception |
Error message. |
stackTrace |
Full stack trace of the error. |
source |
READ-SINK for a failed write to the target system. |
timestamp |
Time of the failure. |
Set the storage in the Hub application.properties:
| Property | Default | Description |
|---|---|---|
failed.records.base-path |
${FAILED_RECORDS_PATH:failed-records} |
Folder for the files. The environment variable FAILED_RECORDS_PATH overrides the default folder name. |
failed.records.max-file-size-mb |
10 |
Maximum size of one file in megabytes. |
Email alert for FAILED Data Flows
The Hub can send an email through an SMTP server that you configure when a Data Flow becomes FAILED. The alert is off by default. It is not a message on a queue or topic.
| Item | Value |
|---|---|
| Subject | Poller <name> failed |
| Body | Poller failed due to an error: <error text>. The error text is written in upper case. |
The Hub connects to the SMTP server with SSL and authentication. Set these properties in the Hub application.properties:
| Property | Default | Description |
|---|---|---|
smtp.server.enable |
false |
Set to true to send alerts. |
smtp.server.host |
smtp.gmail.com |
SMTP server host. |
smtp.server.port |
465 |
SMTP server port. |
smtp.server.username |
source@example.com |
SMTP user name. |
smtp.server.password |
ApplicationPassword |
SMTP password. |
smtp.server.target |
target@example.com |
Recipient address of the alert. |
The default host, user name, password and recipient are placeholders. Replace them with your own values before you enable the alert.
Recovering a FAILED or INVALID Data Flow
A FAILED or INVALID Data Flow does not return to work by itself. Update it with its complete, corrected definition. The Hub then re-creates it. The new Data Flow is INACTIVE, or INVALID again with a new exception if the Hub still cannot initialize it. Then you can start it.
In the web UI
- Open the Data Flows page. Point at the Failed or Invalid badge to see the error text.
- Click the row to open Update Data Flow. The error text is also shown above the form.
- Fix the cause, usually a credential, host name, folder, or other sink property, and click Submit.
- When the status is Inactive, click Start.
For Invalid rows, the Data Flows page also shows a restart icon next to the badge. It sends the stored definition again and starts the Data Flow. Use it when the cause was outside the Hub, for example when the target system was unavailable and is available again. If the cause is in the configuration, edit the Data Flow as described above.
Note
The restart icon exists only for Invalid rows. A Failed Data Flow has no restart icon. On the Update Data Flow page, Start returns Cannot start FAILED poller until you submit the Data Flow again.
With the Admin API
-
Read the state and the error:
-
Fix the cause: credentials, host, folder, network access, or the target object.
-
Send the complete Data Flow definition, which is the creation body with the corrected values. The Hub removes the old Data Flow and creates it again. Whatever
statusthe body contains, the new Data Flow isINACTIVE, orINVALIDif its initialization fails. Encrypt secrets as described in Encrypting credentials. -
Start it:
Sample JSON for the update of a JDBC Data Flow
{
"pollerName": "CDC2RDS",
"connectionName": "NATIVE",
"pollerType": "Source",
"dqName": "DQNAME",
"libraryName": "INFOCDCDTA",
"dqFormatFileName": "FMTFILE",
"dqFormatFileLibrary": "INFOCDCDTA",
"infoCDCConfigured": true,
"threadCount": 5,
"sinkType": "JDBC",
"sinkTypeProperties": {
"jdbcProperties": {
"driverClassName": "com.mysql.cj.jdbc.Driver",
"jdbcUrl": "jdbc:mysql://my-database-server:3306/testdb",
"username": "<encrypted-username>",
"password": "<encrypted-password>",
"maxPoolSize": 10,
"minPoolSize": 3
}
}
}
If the new definition cannot be initialised, the update returns HTTP 200 and the Data Flow is saved as INVALID with the error in exception. Unexpected errors return HTTP 400 with DQ poller '<name>': Invalid '<sinkType>' properties.
Sink-specific settings, and the checks the Hub runs when it creates a Data Flow, are described in File Sinks for Data Flows and IBM i Data Replication.
Admin API reference
Responses of the Data Flow endpoints use the format {"message": "<text>"}.
Start and stop responses by state
A call that the current state does not allow returns HTTP 400:
| State | POST .../start |
POST .../stop |
|---|---|---|
INACTIVE |
Starts the Data Flow (HTTP 200). | DQ poller: '<name>' isn't running |
ACTIVE |
DQ poller: '<name>' isn't stopped |
Stops the Data Flow (HTTP 200). |
INVALID |
Cannot start INVALID poller |
Cannot stop INVALID poller |
FAILED |
Cannot start FAILED poller |
Can not stop failed poller. Please update poller configuration |
While a Data Flow is shutting down, both calls return DQ poller: '<name>' is in stopping process.
Validation errors on create and update
If the body of POST /admin/dqpollers or PUT /admin/dqpollers/{pollerName} is not valid, the Hub returns HTTP 500. The message lists every problem found, one numbered line per error:
{
"message": "Errors: \n1: Connection name is required\n2: Name should not contain special characters\n"
}
The Hub checks these rules for every Data Flow:
| Field | Rule | Message |
|---|---|---|
pollerName |
Required. Only letters, digits and - are allowed. |
DQPoller name is required, Name should not contain special characters |
connectionName |
Required. | Connection name is required |
dqName |
Required. | Polling data queue name is required |
libraryName |
Required. | Library where data queue locates is required |
The Hub also validates the sink properties. The messages are listed with each sink, for example in File Sinks for Data Flows.
If the connection does not exist, the call returns HTTP 404 Connection: '<name>' does not exist. When a Data Flow is created, the Hub also checks that the data queue and the format file exist on IBM i. A missing object is reported with HTTP 400, for example Data queue <name> does not exist in library <library>.
Common API messages
| Call | HTTP | Message |
|---|---|---|
POST /admin/dqpollers |
200 | DQ poller: '<name>' has been registered. |
POST /admin/dqpollers/{pollerName}/start |
200 | DQ poller: '<name>' is running. |
POST /admin/dqpollers/{pollerName}/stop |
200 | DQ poller: '<name>' has been stopped. |
DELETE /admin/dqpollers/{pollerName} |
200 | DQ poller: '<name>' deleted. |
POST /admin/dqpollers |
400 | DQ poller: '<name>' already exists |
| Calls with a Data Flow name in the path | 404 | DQ poller: '<name>' doesn't exists |
POST .../start |
400 | Connection '<name>' must be started before starting the poller (see Requirements for starting) |
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} |
400 | Connection '<connection>' doesn't match poller '<poller>' configuration: maxConnections '<n>' should be equal or higher then threadCount '<m>' or equal -1 (no limits to create new connection in pool) |
PUT /admin/dqpollers/{pollerName} |
400 | DQ poller '<name>': Invalid '<sinkType>' properties |
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} |
404 | Connection: '<name>' does not exist. |
POST /admin/dqpollers, PUT /admin/dqpollers/{pollerName} |
500 | Errors: ... (see Validation errors on create and update) |
Start and stop errors that depend on the state are listed in Start and stop responses by state.