TRF DQ Rules Validation

Requirement: Generate the logic for data quality checks based on the rule descriptions mentioned in the attached CSV. All rules are associated with d_product_1 where src_sys_cd = 'trf'

 

The final DQ report should include the following columns: src_sys_cd, table_name, field_name, rule_id, dq_dimension, invalid_records, total_records, dq_percent, and rule_description.

 

Expected Output: Databricks PySpark, Spark SQL.

 

Unity Catalog : d_product_1