Renaming Files in S3 with Date-Based pattern in the Suffix

Introduction: This involves applying a dynamic renaming logic where the filenames are appended with the current processing date in the format YYYY-MM-DD (e.g., study_dosage_results_2025-04-02.csv). The renamed files will be saved to the designated target S3 path.

 

Requirements: Create a Pyspark databricks logic, Rename the file s3_file.csv located at the S3 path s3://agilisium-playground-dev/filestore/archived_data/s3_file.csv by appending the current datetime() in YYYYMMDD format, using an underscore (_) as the delimiter. Rename only the filename and it should be saved to the same S3 directory path.

 

Databricks Secret Information: “access_key” and “secret_key” are placed in Databricks secret under the scope “aws_keys”