Withdraw the data from facts table which is not qualified for KPI

Background: The purpose of withdrawal utility is to prevent sending the bad data to the business layer. The daily incremental data of fact tables are required to design the business KPIs on dashboard. As per DQ compliance, if facts data is not qualified then it should be cleaned up from each layers.

 

Requirement: Create a PySpark script to DELETE records from “f_item”, and “d_product” table where item_nbr is “52710c66-9928-4978-b4d7-bf817576842b“ and “e3a6583a-105c-4197-bcfc-f632915e7143“ respectively and then update the 'data_status' column as 'withdrawal' in 'master_log' table for the table_name “f_item”, and “d_product”.

 

Unity Catalog: f_item, d_product, master_log