Automate Data Type Standardization for patient_data_bronze via S3 Mapping File

Requirement:

 

Develop a Databricks PySpark script to perform dynamic data type conversion on the patient_data_bronze table using a JSON file stored in S3. Read the JSON file follows in S3 which has the format → {data_type: column_name} (eg: STRING: <column_name>). Display the output after applying the conversions.

 

 

 

S3 path: s3://agilisium-playground-dev/filestore/mapping-files/patient_datatype_mapping.json.

 

Databricks Secret Information: The access_key and secret_key have been stored in Databricks secrets under the scope named aws_keys.