Kafka Message Format for Different Operations in RDMS

INSERT Operation

These can be generated from BDT (in which case the DEKafkaMessage will not have a SourceTransactionId), or from a CDT. In the case of a BDT all the fields that are in the transformation will be present in the JSON AfterImage. For CDT if the fields in the RDMS database allow NULLS or have default values then these fields could be missing from the JSON message, it all depends on the SQL statement issued in the program and if it includes all the fields or not. All primary key fields in the target XSD will be present.

The following is from a BDT that is performing multiple Inserts per batch unit. Also note in this case the field GLB_DTIME is in BASE64 rather than being translated to HEX. Also this shows multiple MessageUnits, this can occur when the BDT batch size is > 1 or in CDT either when the CDT batch size is > 1 or there are multiple updates in a single transaction that go to the same topic.

<DEKafkaMessage>
  <Header>
    <BatchId>1</BatchId>
    <ThreadId>1</ThreadId>
    <MessageUniqueId>1-1-1</MessageUniqueId>
    <MessageType>WholeBatch</MessageType>
    <DETransmissionType>Batch</DETransmissionType>
    <TransformationId>21474be5-0d0d-424a-a498-a68f86ed3727</TransformationId>
    <ProviderId>5E309A7C-0A1F-4CD8-A201-3FF984DEFA8C[UnisysDExchTrackingDB-5030]</ProviderId>
    <MessageFormatVersion>1.0</MessageFormatVersion>
  </Header>
  <BatchUnits>
    <BatchUnit>
      <MessageUnits>
        <MessageUnit>
          <SourceOperation>Insert</SourceOperation>
          <TargetOperation>Insert</TargetOperation>
          <OrderInBatchUnit>0</OrderInBatchUnit>
          <SchemaName>APPSVN_SAMPLK_Export</SchemaName>
          <ClassifierName>CUST</ClassifierName>
          <KeysOfTarget>
            <KeyOfTarget>CUSTOMER</KeyOfTarget>
          </KeysOfTarget>
          <SchemaLastModifiedDate>2018-08-09T15:42:13.323</SchemaLastModifiedDate>
          <Schema />
          <MessagePayload>
            <BeforeImage />
            <AfterImage>{"CREDLIMIT":501,"MAINT":"C","NAM":"ZYX CORPORATION LTD","CUSTOMER":"ABC","DELADD2":"","POSTADD2":"North Ryde","DELADD3":"","POSTADD1":"115 Wicks Road","POSTADD3":"SYDNEY","GLB_DTIME":"\u0001\u0005-\u0017\u001c0%\b\u0017 \u0000\u0007","SALESREP":"AA3","CUST_TYPE":"","DELADD1":""}</AfterImage>
          </MessagePayload>
        </MessageUnit>
        <MessageUnit>
          <SourceOperation>Insert</SourceOperation>
          <TargetOperation>Insert</TargetOperation>
          <OrderInBatchUnit>0</OrderInBatchUnit>
          <SchemaName>APPSVN_SAMPLK_Export</SchemaName>
          <ClassifierName>CUST</ClassifierName>
          <KeysOfTarget>
            <KeyOfTarget>CUSTOMER</KeyOfTarget>
          </KeysOfTarget>
          <SchemaLastModifiedDate>2018-08-09T15:42:13.323</SchemaLastModifiedDate>
          <Schema />
          <MessagePayload>
            <BeforeImage />
            <AfterImage>{"CREDLIMIT":501,"MAINT":"A","NAM":"YXW CORPORATION LTD","CUSTOMER":"BCD","DELADD2":"","POSTADD2":"North Ryde","DELADD3":"","POSTADD1":"115 Wicks Road","POSTADD3":"SYDNEY","GLB_DTIME":"\u0001\u0005-\u0017\u001c0%\b\u0017 \u0000\f","SALESREP":"AA3","CUST_TYPE":"","DELADD1":""}</AfterImage>
          </MessagePayload>
        </MessageUnit>
        <MessageUnit>
          <SourceOperation>Insert</SourceOperation>
          <TargetOperation>Insert</TargetOperation>
          <OrderInBatchUnit>0</OrderInBatchUnit>
          <SchemaName>APPSVN_SAMPLK_Export</SchemaName>
          <ClassifierName>CUST</ClassifierName>
          <KeysOfTarget>
            <KeyOfTarget>CUSTOMER</KeyOfTarget>
          </KeysOfTarget>
          <SchemaLastModifiedDate>2018-08-09T15:42:13.323</SchemaLastModifiedDate>
          <Schema />
          <MessagePayload>
<BeforeImage />            <AfterImage>{"CREDLIMIT":501,"MAINT":"A","NAM":"DETEST","CUSTOMER":"CLIVE9","DELADD2":"","POSTADD2":"","DELADD3":"","POSTADD1":"MILTON KEYNES","POSTADD3":"","GLB_DTIME":"\u0014\u0016\u0003*8.<=\u0000\u0000\u0000\u0000","SALESREP":"AA2","CUST_TYPE":"","DELADD1":""}</AfterImage>
          </MessagePayload>
        </MessageUnit>

Update or Merge Operation

In case of updates or merges the BeforeImage has all the key values and the AfterImage has the values of the fields that have changed.

Note: In Enterprise Database Server the BeforeImage has all the values and AferImage has all the fields.

<MessageUnit>
<SourceOperation>Update</SourceOperation>
<TargetOperation>Update</TargetOperation>
<OrderInBatchUnit>0</OrderInBatchUnit>
<SchemaName>APPSVN_SAMPLK_Export</SchemaName>
<ClassifierName>CUST</ClassifierName>
<KeysOfTarget>
     <KeyOfTarget>CUSTOMER</KeyOfTarget>
</KeysOfTarget>
<SchemaLastModifiedDate>2018-08-09T15:42:13.323</SchemaLastModifiedDate>
<Schema />
<MessagePayload>
      <BeforeImage>{"CUSTOMER":"BCD"}</BeforeImage>
      <AfterImage>{"CREDLIMIT":"1"}</AfterImage>
</MessagePayload>
</MessageUnit>

Delete Operation

In this case the BeforeImage contains the key values and the AfterImage contains nothing.

<MessageUnit>
         <SourceOperation>Delete</SourceOperation>
         <TargetOperation>Delete</TargetOperation>
          <OrderInBatchUnit>0</OrderInBatchUnit>
          <SchemaName>APPSVN_SAMPLK_Export</SchemaName>
          <ClassifierName>INVENTORY</ClassifierName>
          <KeysOfTarget>
                <KeyOfTarget>TRANTIME</KeyOfTarget>
                <KeyOfTarget>INPUT_DATE</KeyOfTarget>
                <KeyOfTarget>PRODUCT</KeyOfTarget>
                <KeyOfTarget>ACTMTH</KeyOfTarget>
                <KeyOfTarget>GLB_DTIME</KeyOfTarget>
        </KeysOfTarget>
        <SchemaLastModifiedDate>2018-08-09T15:42:13.323</SchemaLastModifiedDate>
        <Schema />
        <MessagePayload>
<BeforeImage>{"TRANTIME":"0","INPUT_DATE":"24APR18","GLB_DTIME":"01052D171C3025081720000B","PRODUCT":"ABC","ACTMTH":"1804"}
</BeforeImage>
               <AfterImage />
         </MessagePayload>
</MessageUnit>

DeleteAll Operation

There is an exception to the general structure for the DELETE <tablename> ALL command. This command effectively removes all the records in a table. RDMS does this as a single instruction, where the individual deleted records are not audited. In this case Data Exchange issues a Message Unit as follows:

<MessageUnit>
      <SourceOperation>DeleteAll</SourceOperation>
      <TargetOperation></TargetOperation>
      <OrderInBatchUnit>X</OrderInBatchUnit>
      <SchemaName>Schema</SchemaName>
      <ClassifierName>Classifier</ClassifierName>
      <KeysOfTarget/>
      <SchemaLastModifiedDate>2019-10-28T10:26:19.98</SchemaLastModifiedDate>
      <Schema/>
      <MessagePayload/>
</MessageUnit>

Note that the Message Payload is empty in this case.