DQ_Check for Metric calculation

Requirements: Refer to the attached Excel sheet for Data Quality (DQ) checks and develop a Databricks PySpark script to execute those checks on the metric tables. Retrieve the distinct metric tables from the target_table column in the metric_config table.

 

 

Sheet Details:

 

* DQ Checks: Lists the DQ checks to be performed.

* Metric Column Datatype: Provides details about the data types of metric columns.

 

Save the results of both the DQ checks in Dataframe (row_count_validation and data_type_validation).

 

Unity Catalog Information: metric_config table

 

Expected output: Databricks PySpark code