You can perform a changed data transformation(CDT) from an Enterprise Database Server to a SQL Server, an Oracle database, or a Kafka distributed streaming platform. Before starting the CDT, ensure the following:
The bulk data transformation(BDT) is completed if you had decided to perform both BDT and CDT.
If you had performed a BDT from the cloned database, you must enter the new audit file number created after you had performed the audit file switch (an audit switch should have been performed at the point of cloning your transformation source).
For example, there are 10 audit files in the transformation source, and you are doing a database clone before performing a BDT. In this case, an audit file switch would close off audit file 10 and all new transactions after the point of cloning are recorded in audit file 11. Therefore, you should use audit file 11 to start a changed data transformation.
If you had performed a BDT from the original transformation source with the option With No Lock, perform an audit switch after the BDT is completed. Use the new audit file number as the starting audit file for a changed data transformation.
The keywords "AUDIT" and "INDEPENDENTTRANS" are added to OPTIONS in the DASDL file associated with the transformation source.
The keyword DATAEXCHANGE is added in the Audit Trail Specification of the DASDL or in the Database DEFAULTS or at the data set level.
Defining at the data set level reduces the number of audit records during a CDT. If you are making this change for a database that is already loaded in the DDW, you must do a DM Control Update on the MCP host, and then reload the schema.
The overall Data Exchange status appears as Stopped on the Summary tab.
Notes:
Remapped data sets are not supported for CDTs.
A CDT supports the data item size of up to 65,535 for an Enterprise Database Server .
A CDT trims the blank spaces in the Alpha data item.
It is recommended that you do not perform any CDT while any Enterprise Database Server recovery operation is in progress. It should also be noted that the results of the recovery operation, such as rollback and rebuild, may cause the transformation target to be in an inconsistent state.
The Data Exchange Runtime stops if it detects any format changes caused while running the Reorganization Utility. It awaits the deployment of the new transformation with the correct format.
To start a CDT
Click Manage.
The hosts associated with the deployed transformations appear on the Source Database Hosts pane.
On the Source Database Hosts pane, expand MCP Hosts, and then expand the required host to view the Enterprise Database Server databases hosted on it.
Select the Enterprise Database Server database from which you want to start the CDT.
Point to Start, and then click Changed Data Transformation.
The Data Exchange Transformation Session Setup appears.
On the Transformations tab, select the transformation(s) you want to run, and then click Next.
The General Settings tab appears.
On the General Settings tab, enter or change the following information, as appropriate:
The Agent TCP Port box displays the default port number (8299) on which the Agent is listening. You must change the TCP port number only if you are not using the default port number.
In the AIS Connection Name box, type the AIS Connection Name that you had defined using the AIS Configuration Utility.
If you require a secure connection from source database to Runtime Service, enable secure connection in AIS, and then enter the Connection Name for which security is enabled.
In the Login User and Password boxes, type the usercode and password, respectively.
The usercode must have the privileges to access the description file and audit files of the source database.
In the Enterprise Database Server, a usercode might be associated with either an accesscode or a chargecode, or both. An accesscode will have an associated password.
The following table displays the format in which the usercode and the associated accesscode or chargecode need to be entered in the Login User box along with their corresponding passwords. Each entry is separated from the other by a separator (/).
| User | Password |
| Usercode/Accesscode | Usercode password/Accesscode password |
| Usercode/Accesscode/Chargecode | Usercode password/Accesscode password |
| Usercode/Chargecode | Usercode password |
The Runtime Service Host box displays the host name or IP address of the Windows partition where the Runtime Service is installed.
You must change the value of the Runtime Service host if both the Runtime Service and the Administrative Service are not running on the same Windows host.
The Runtime Service TCP Port box displays the TCP port number of the Runtime Service that was configured during installation.
If you require a secured connection, turn on Secure Connection. This enables you to configure a secure connection from Runtime Service to the target database.
For Oracle as the target, also provide the Secure Connection Port number. The default secure connection port is 2484.
If your target is a Kafka streaming platform, then the Kafka target(s) box displays the list of Kafka targets. Enter the following details for each target:
Bootstrap Server(s): Enter the bootstrap server name(s) and the port number(s) in the format host1:port1,host2:port2. Only letters, numbers, hyphen(-), underline(_), and period(.) are allowed in the Bootstrap server name.
Topic: Topic is a stream of records. Enter the name of the topic. For example, MyTopic.
Acks: Select the kind of acknowledgement you require:
-1 - Acknowledgements from replicas: The leader will wait for the full set of in-sync replicas to acknowledge the record.
0 - No acknowledgement: The producer will not wait for any acknowledgment from the server.
1 - Acknowledgement from leader: The leader will write the record to its local log and respond without waiting for full acknowledgement from all the followers.
Client ID: The client ID is Data Exchange.
Secure Connection: To configure the Data Exchange Runtime service host to support an SSL connection with the Kafka target host, you can enable Secure Connection.
When you turn on Secure Connection, the Secure Connection Config window appears. In this window, browse and select the Certificate Authority file. You can also provide additional Certificate Configuration details such as certificate, key, and key password if the Kafka server requires client authentication.
Notes:
If the key file is not encrypted, you can leave the password blank. If the key file is encrypted, you must pair the file with the password.
For all the files selected in secure connection for Kafka targets, both the certificate and key files must be in PEM format.
Other Producer Config: Producers publish data to the topics of their choice. Enter the producer details in the format name1:value1, name2:value2. These configurations override the default producer configurations. For more details refer to Kafka Producer Configurations.
The Target Commit Error Handling Policy lists the errors that could occur while committing the records to the target.
On encountering some of the listed errors, you can only choose to stop the transformation or ignore the error. But there is a possibility to recover from the Deadlock Error by retrying. Hence this error has the option to retry 3 times. This recovery is faster, as the retry is internal to the Runtime Service.
Select the manner in which you want Data Exchange to respond when the specific error occurs.
Click Next.
The Transformation Settings tab appears.
On the Transformation Settings, you can start the CDT from either a particular audit file number or a particular timestamp. Select one of the following:
Audit File Number: Select this option to start CDT from the specified audit file number.
In the Audit File Number box, type the number of the audit file from where you want to begin the CDT.
Audit Time Stamp: Select this option to start CDT from the specified date and time.
In the Audit Time Stamp box, enter the date and time (of the MCP host time zone) from when you want to start the CDT.
Note: The Timestamp is accurate only till the seconds level. Hence, if in your source database there are multiple data updates within the second where you want to start CDT from, then it is recommended to use the Audit File Number to start the CDT.
Click Next.
The Start Transformations tab appears.
Verify all the details, and then click Start.
After you initiate the CDT, each change that occurs in the transformation source according to the activated transformation is continuously updated in the transformation target in near real-time. You can monitor the transformation on the Statistics page. You can also log in to the transformation target and verify the update.
Notes:
If the credentials of your transformation target have changed since you loaded the target schemas to the DDW, then the Runtime Service cannot update the transformation targets because of the mismatch in the credentials. In this case, you must use the Change Login Credentials option and update the credentials in the DDW and then redeploy the transformation to the Administrative Service. Refer to Changing Login Credentials on the Information Center for more information.
While starting a CDT and while checking the format timestamp before performing a CDT, Data Exchange calls the command LogonCheck if SecuritySupport is configured in MCP. LogonCheck captures logon attempts for all MCP usercodes. For more information on LogonCheck, refer to the *SYMBOL/SECURITYSUPPORT file.
If any of the recoverable errors configured on the Recovery subtab (of the Settings tab) occur after successfully initiating the CDT, Data Exchange automatically tries to recover from the error for the number of times specified on the Recovery subtab. To stop the automatic recovery and to perform the CDT at a later point, click Stop Recovery. The Stop Recovery button is available only when the system is in the recovery state.
If any errors not configured for recovery occur while CDT is running, you must resolve the issue and restart the CDT. Data Exchange automatically starts processing from where it stopped. You can see the error details on Events. See Common Errors and Possible Solutions for more information about possible errors and how to recover from them.
During the CDT, the overall Data Exchange status changes to Changed Data Transformation Running on the Summary tab. During the time of the automatic recovery, the overall Data Exchange status changes to Recovering.
Click Refresh to obtain the current status (stopped, running, or error) of the CDT.
On completion of the transformation to a Kafka target, the name and details of that target is displayed on the Settings > System tab.