Refactor 'Filler_1' to 'DELETE_FLAG' with Constraints in Outbound FIA ODS Tables

Description: As part of the updates for the Outbound FIA process, the fieldname needs to be renamed to 'DELETE_FLAG' in the following 3 extracted tables, the DELETE_FLAG field should have a NOT NULL constraint to ensure data integrity and consistency. This change ensures better clarity, compliance with database constraints, and alignment with the updated naming conventions.

 

Requirements: create a Pyspark logic with below conditions for 3 extracted tables.

 

Rename the field 'Filler_1' to 'DELETE_FLAG'* in the schema definition for the three Delta tables.

update NOT NULL constraint to the Filler_1 column in DDL script and also Filler_1 field by filtering Not NULL Value in 'DELETE_FLAG*'.

 

Prerequisite:

 

1. Drop the table ods_free_drug_clone, ods_non_contracting_clone, ods_copay_clone if exist.
2. Perform the requirement in replica of ods_free_drug table in ods_free_drug_clonetable, ods_non_contracting in ods_non_contracting_clonetable, ods_copay in ods_copay_clone table.

 

Unity Catalog details(tables): ‘purgo_playground.ods_free_drug’,'purgo_playground.ods_non_contracting','purgo_playground.ods_copay'