You can perform a changed data transformation (CDT) from an RDMS schema to a SQL Server, an Oracle database, or a Kafka distributed streaming platform. Before starting a CDT from the RDMS schema, ensure the following:
The bulk data transformation (BDT) is completed if you had decided to perform both BDT and CDT.
Replication is turned ON for the schema in UREP. Refer to Enabling Runtime Data Exchange for RDMS Schemas for instructions to turn ON replication.
Note: It is recommended that you do not perform any CDT while any Application Group recovery is in progress. It should also be noted that the results of the recovery operation such as rollback and rebuild may cause the transformation target to be in an inconsistent state.
To start a CDT
Click Manage.
The hosts associated with the deployed transformations appear on the Source Database Hosts pane.
On the Source Database Hosts pane, expand OS 2200 Hosts and then expand the required host to view the various Application Groups created on the host.
Select the Application Group from which you want to start the CDT.
The Transformation Source pane appears to the right.
Point to Start, and then click Changed Data Transformation.
The Data Exchange Transformation Session Setup appears.
On the Transformations tab, select the transformation(s) you want to run, and then click Next.
The General Settings tab appears.
On the General Settings tab, enter or change the following information, as appropriate:
The Agent TCP Port box displays the default port number (5150) on which the Agent is listening. You must change the TCP port number only if you are not using the default port number.
The Runtime Service Host box displays the host name or IP address of the Windows partition where the Runtime Service is installed.
You must change the value of the Runtime Service host if both the Runtime Service and the Administrative Service are not running on the same Windows host.
The Runtime Service TCP Port box displays the TCP port number of the Runtime Service that was configured during installation.
If you require a secured connection, turn on Secure Connection. This enables you to configure a secure connection from Runtime Service to the target database.
For Oracle as the target, also provide the Secure Connection Port number. The default secure connection port is 2484.
If your target is a Kafka streaming platform, then the Kafka target(s) displays the list of Kafka targets. Enter the following details for each target:
Bootstrap Server(s): Enter the bootstrap server name(s) and the port number(s) in the format host1:port1,host2:port2. Only letters, numbers, hyphen(-), underline(_), and period(.) are allowed in the Bootstrap server name.
Topic: Topic is a stream of records. Enter the name of the topic. For example, MyTopic.
Acks: Select the kind of acknowledgement you require:
-1 - Acknowledgements from replicas: The leader will wait for the full set of in-sync replicas to acknowledge the record.
0 - No acknowledgement: The producer will not wait for any acknowledgment from the server.
1 - Acknowledgement from leader: The leader will write the record to its local log and respond without waiting for full acknowledgement from all the followers.
Client ID: The client ID is Data Exchange.
Secure Connection: To configure the Data Exchange Runtime service host to support an SSL connection with the Kafka target host, you can enable Secure Connection.
When you turn on Secure Connection, the Secure Connection Config window appears. In this window, browse and select the Certificate Authority file. You can also provide additional Certificate Configuration details such as certificate, key, and key password if the Kafka server requires client authentication.
Notes:
If the key file is not encrypted, you can leave the password blank. If the key file is encrypted, you must pair the file with the password.
For all the files selected in secure connection for Kafka targets, both the certificate and key files must be in PEM format.
Other Producer Config: Producers publish data to the topics of their choice. Enter the producer details in the format name1:value1, name2:value2. These configurations override the default producer configurations. For more details refer to Kafka Producer Configurations.
The Target Commit Error Handling Policy lists the errors that could occur while committing the records to the target.
On encountering some of the listed errors, you can only choose to stop the transformation or ignore the error. But there is a possibility to recover from the Deadlock Error by retrying. Hence this error has the option to retry 3 times. This recovery is faster, as the retry is internal to the Runtime Service.
Select the manner in which you want Data Exchange to respond when the specific error occurs.
Click Next.
The Transformation Settings tab appears.
On the Transformation Settings tab, in the Date Time box, enter the date and time (of the OS 2200 host time zone) from when the OS 2200 Agent needs to capture the changed data.
This is required only if a bulk data transformation was run with the profile setting LOCK=NONE. If it was run with the profile setting LOCK=PROTECTED , then Data Exchange automatically takes the start time for the CDT.
Note: If you have defined a primary key in the transformation target, then the transformed data is inserted as new records in the transformation target. If not, the transformed data is merged with the existing records.
Click Next.
The Start Transformations tab appears.
Verify all the details, and then click Start.
If the CDT fails to start, the error is displayed on the Changed Data Transformation Status box. Click the icon in the Details column for more information on the cause and resolution of the error. The transformation starts after you have resolved all the errors.
After you initiate the CDT, each change that occurs in the transformation source according to the activated transformation is continuously updated in the transformation target in near real-time. Click Monitor on the Summary tab to see the statistical data corresponding to the CDT. You can also log in to the transformation target and verify the update.
Note: If the credentials of your transformation target have changed since you loaded the target schemas to the DDW, then the Runtime Service cannot update the transformation targets because of the mismatch in the credentials. In this case, you must use the Change Login Credentials option, update the credentials in the DDW, and then redeploy the transformation to the Administrative Service. Refer to Changing Login Credentials on the Information Center for more information.
If any of the recoverable errors occur after successfully initiating the CDT, Data Exchange automatically tries to recover from the error. To stop the automatic recovery and to perform the CDT at a later point, click Stop Recovery on the Summary tab. The Stop Recovery button is available only when the system is in the recovery state.
If any errors not configured for recovery occur while CDT is running, you must resolve the issue and restart the CDT. Data Exchange automatically starts processing from where it stopped. You can see the error details on Events. See Common Errors and Possible Solutionsfor more information about possible errors and how to recover from them.
During the CDT, the overall Data Exchange status changes to Changed Data Transformation Running on the Summary tab. During the time of the automatic recovery, the overall Data Exchange status changes to Recovering.
Click Refresh to obtain the current status (stopped, running, or error) of the CDT.
On completion of the transformation to a Kafka target, the name and details of that target is displayed on the Settings > System tab.