Implement Dynamic Data Quality Scorecard for Weekly Metric Pipelines

Introduction: Dynamic Data Quality Scorecard that automatically evaluates weekly NBRx data against predefined validation rules and produces a summary scorecard for pipeline monitoring. The scorecard must support multiple validations such as schema completeness, null checks, duplicate week detection, date boundary checks.

 

Requirements: Execute the SQL query linked to metric_name = 'total_NBRx' from the ‘indication_config' table and retrieve the dataset. Compare SQL result set with indication_level_metric and run below quality validations: schema completeness, null validation, duplicate week validation, date continuity check (7-day window).Assign Pass / Fail for each validation rule along with aggregated score (0–100). Display the scorecard output with the following columns: Validation_Name, Validation_Status, Validation_Comments. Validation_Name should have the column of ‘schema completeness’, ‘Null Validation’, ‘Duplicate Week Validation’, 'Date Continuity Check’. Validation_Status should have the column PASS/FAIL. Finally provide the overall score for the validations.

 

Unity Catalog: ‘indication_config','indication_level_metric ’