Anomaly analysis in claims Data

Information: Anomaly analysis focuses on identifying unusual patterns or outliers, such as claims that deviate from expected ranges or behaviors. Anomaly detection highlights unexpected or irregular patterns in the data.

 

Requirement: Create a Pyspark code to find anomaly for the table health_insurance_claimsunder different levels.

 

1. Year level: In a year (in `Service_Date`) if the count of Claim_ID ,sum of Billed_Amount and sum of Allowed_Amount  is 1.5 times preceding years values then flag it as anomaly
2. Provider level : for each provider (`Provider_ID` ) if the count of claim id ,sum of billed amount  and sum of Allowed Amount is 1.5 times preceding years values then flag it as anomaly
3. Medication level: for each Medication if the count of Claim_ID ,sum of Billed_Amount and sum of Allowed_Amount is 1.5 times preceding years values then flag it as anomaly

 

plot 3 line graphs with count of claim id ,sum of billed amount  and sum of Allowed Amount to study each of these anomalies

 

Unity catalog information: health_insurance_claims

 

Expected output: Databricks Pyspark code, Show the Result and Graphs.