Data profiling on metric tables

Requirement: Develop a Databricks PySpark script to generate a data profile table named metric_profile for each column in the metric tables, based on the metric tables health_insurance_claims_metric, d_product_revenue_metric and s_field_reporting_sales_source_customer_metric .

 

 

metric_profile should contain:

 

table_name

 

column_name

 

Data type

 

Null count

 

Distinct count

 

Total count

 

Minimum value

 

Maximum value

 

Mean