Email Notifications for Failed or Incomplete Ingestions

Introduction: The data ingestion framework must proactively identify failed or incomplete ingestions based on missing files, outdated weeks of data, or inactive flags. Upon identification, the system should trigger email notifications to the configured recipients (from email_id column in config). This automation helps to prevent data loss in production reports, proactively alert the data operations team, reduce time to issue resolution.

 

Requirements: Read the table “purgo_playground.file_processing_log“. Filter failure or incomplete records, when Files where processing_status is 'FAILURE'Or record_count is null. Extract relevant columns, Selects only the columns we care about for the email report — file name, pattern, system name, status, and who to email. Check if any failures exist, If the result is not empty, then we build and send an email. Send an email using Outlook SMTP by Email is sent via Outlook (smtp.office365.com), Fromemail id :Dhinesh.Muthupandian@agilisium.com, To email_id: Dhinesh.Muthupandian@agilisium.com; Ganesh.Guruprasath@agilisium.com, You must use an App Password as “btjqllztcypgwfps“ instead of a regular password, The failed entries are formatted into an HTML table and added to the email body. If there are no failures in processing_status, the code simply prints a success message and exits.

 

Unity Catalog: “purgo_playground.file_processing_log“