Automated DDL Synchronization for Evolving CSV Structures

Introduction: When enhancing a database schema to include new columns, it is critical to ensure that changes are systematically applied to both the schema definition and the associated Data Definition Language (DDL) scripts. Adding columns requires not only modifying the table structure but also updating scripts to reflect the revised schema, ensuring consistency across development, testing, and production environments. This process typically involves writing the existing schema definition to incorporate the new columns and revising DDL scripts.

 

Requirements:

 

Create a Databricks Pyspark code and read the file ‘/Volumes/agilisium_playground/purgo_playground/de_dq/product_temperature.csv'. create the table ‘product_temperature'. Column structure changes are expected in this file so create a script in such a way that it updates the product_temperature table’s DDL by inferring the columns in file.

 

Final Output: Only Show and write the results in the table.