Deduplication on C360 Cofig Target Tables

Requirement:

Develop a de-duplication PySpark script which take distinct target table from c360_config and remove the duplicates. Load the deduplicated records in separate table named “<target_table>_dedup” and duplicate records into <target_table>_reject”. if target table is “NA” then ignore it for de-duplication

 

Note: don’t disturb the actual target table

 

 

 

Unity Catalog: c360_config