DQ rule for Inventory management

Information: Setting a data quality(DQ) rule is vital for Inventory management, Before the data reaches the final reporting layer, it's necessary to perform checks. If any of the rule has violated by the data then we need to be notified so that we can back track and resolve the issue.

 

Requirement: Create a Databricks Pyspark/Spark SQL code to implement data quality checks on drug_inventory_management table based on the attached Excel data_quality_mapping_file.xlsx. The DQ_rules_IM sheet of the file includes detailed rule definitions for each data quality category.

 

create a data quality dataframe with overall result of each data quality categoryin the DQ_rules_IM sheet.

 

Structure of data frame:

 

1. check_name - string
2. result - string
3. pass_%- pass percentage out of total rows

 

Unity Catalog Information: purgo_playground.drug_inventory_management table

 

Expected Output: Databricks Pyspark/Spark SQL code