Resolving Erroneous Records Using the Data Remediation Table
Automatically identify and flag erroneous records without stopping your transformation flow with the Data Remediation table.
Data quality issues can create problems after running a transformation flow. Common errors include negative or invalid values (e.g., negative ages), null or missing values, inconsistent data (e.g., admission dates after discharge dates), and business logic violations. Enabling Support Data Validation and using the Data Remediation table help you correct these erroneous records in future runs.
-
_DRT_Status: Tracks record state:
-
0 = erroneous record.
-
1 = corrected and ready for reprocessing.
-
-
_DRT_MESSAGE: Holds a user-defined error description set at capture time.
-
Initial and Delta: Corrected records merge only when the next run detects source changes.
-
Initial: Corrected records in the Data Remediation table override incoming source data. Clear the Data Remediation table before initiating an initial load to avoid overwriting valid records with erroneous data.
-
Delta: If a corrected record in the Data Remediation table has been subsequently updated or deleted at the source, the more recent source state is preserved rather than being overridden.
Correction for Few Records
-
Click the run status Completed (with Flagged Records) to open the transformation flow in the Data Integration monitor.
-
In the Data Validation tab, select the Python operator Python1 to view flagged records. The flagged records are shown on the Data Remediation table.
-
In the Data Remediation table, you can click:
-
Resolve All to mark all flagged records as resolved without making corrections. To use only if the flagged records are valid.
-
Edit to manually correct erroneous values.
-
Save to save your changes.
-
Cancel to discard unsaved changes.
-
to refresh the data displayed in the panel.
-
to open the Settings dialog (see Choose Columns to Display).
-
-
[optional] You can remove unnecessary records from the Data Remediation table by deleting them in the Local Table (File) monitor in the Data Integration Monitor. See Delete Data From Your Local Tables (File).
-
Save changes and run the flow again. The corrected records are merged into the target table during the next run.
Mass Correction for Many Records
-
Create a new transformation flow. See Creating a Transformation Flow in a File Space.
-
Use the Data Remediation table as both source and target.
-
Add a Python operator between them. Keep the Support Data Validation toggle disabled in the Python node properties side panel.
-
Click Edit in the Script section and implement your correction code.
You must explicitly set the value of the _DRT_STATUS to 1 in the Python code to mark the records as resolved.
Corrected records are stored in the Data Remediation table and automatically included in the next main flow run.
-
Save and run the flow.
Check the transformation flow's Run Details in the Data Integration monitor to see how many records were written to the target table.
Data Remediation Table Deletion
-
Turning off the Support Data Validation toggle on a Python operator deletes the Data Remediation table during the next deployment of the transformation flow.
-
Removing the Python operator from your transformation flow deletes its associated Data Remediation table.
-
Deleting a transformation flow that has Support Data Validation enabled also deletes the Data Remediation table.
-
The table is not being used as a source or target in any other transformation flows.
-
The table is not referenced in any views or reports.
-
The flagged records have been corrected or resolved and aren't required anymore.