Delta Table-Based Metric Column Validation Using MINUS Query

Requirement:

 

Develop a Databricks PySpark script to execute a MINUS query for column validation using the metric_minus_config table. Retrieve the column count of source_table and target_table from the metric_config table, then apply the validation formula:

 

(no_of_metric_col+columns in source_table)−columns in target_table=0

 

Store the result in a Dataframe, Dataframe should have columns: metric_id, no_of_metric_col, no_of_source_col, no_of_target_col, result (Pass/Fail)

 

Unity Catalog Information: metric_config table and metric_minus_config