Implement Hash-Based Data Sync for Total NBRx Indication Metrics

Requirements: Create the PySpark logic to execute the SQL query where metric_name = 'total_NBRx' from the indication_config table. From the SQL output of the indication_config table, concatenate all columns and generate a hash. Apply the same concatenation and hash generation to the records in the indication_level_metric table. Compare the hash values from both tables and perform the following operations:

 

If the hash matches in both sources, the record is considered unchanged*, and no update is required.

If the hash does not match, and the hash record appears in the SQL output but does not exist in the indication_level_metric table, that record must be inserted into indication_level_metric as a new entry*.

 

Expected Output:

Display the result with the status ‘unchanged record’ when the hash values match, and ‘appended record’ when the hash record is not present in the indication_level_metric table

 

Unity Catalog: ‘indication_config', ‘indication_level_metric’, ‘indication_level_table’.