Upgrading Scripts for Databricks Runtime 15.3 Compatibility

Introduction: To maintaining compatibility with newer versions of Databricks Runtime (DBR) is crucial for ensuring optimal performance, scalability, and reliability. Older scripts, often relying on RDD-based operations like zipWithIndex, may work but are less efficient and harder to maintain when compared to modern DataFrame-based APIs introduced in newer DBR versions, such as Databricks Runtime 15.3. Databricks, ensuring compatibility with Databricks Runtime 15.3 and beyond. It also simplifies the codebase, making it easier to read, debug, and maintain.

 

Requirements:

 

Read the DBR older script (old_script.txt). Use the data in the table('agilisium_playground.purgo_playground.actual_file'). Script rely on RDD-based operations, which are less efficient and harder to maintain compared to DataFrame APIs. Value for rows_skip is 3. Provide the updated new DBR version 15.3 and beyond.

 

Expected Output: Provide the actual script with upgrade of DBR version 15.3 and beyond

 

Prerequisites: Don't capture the log information.

 

Unity Catalog Details: 'purgo_playground.actual_file'