Total_CPD Anomaly Detection using PySpark

Requirement:

Develop a PySpark script that executes SQL query stored in c360_config table for the vale metric_name 'total_cpd'. before loading the result into .c360_s_field_total_cpd_report table check whether the total_cpd value for a given classification_value exceeds 60% greater than the corresponding value in the same c360_s_field_total_cpd_report table. overwrite the data back to c360_s_field_total_cpd_report and if it exceeds 60% of current value in the table, set the flag_anomaly column to 1; otherwise, set it to 0.

 

 

 

Unity Catalog: purgo_playground.c360_s_field_total_cpd_report

 

Expected Output: PySpark/Spark SQL