Overview:

This project demonstrates a fully automated data pipeline using AWS S3 and Snowflake. Raw employee CSV files are ingested from S3 into Snowflake, processed to filter active employees in a dynamic table, and the cleaned data is exported back to S3. The pipeline leverages Snowflake pipes, dynamic tables, and storage integration to automate ingestion and processing.

Responsibilities:

• Designed and implemented the Snowflake schema including tables, dynamic tables, stages, and pipes.
• Configured AWS S3 buckets for raw and cleaned data storage.
• Created IAM roles and storage integration for secure Snowflake-S3 connectivity.
• Automated file ingestion using Snowflake pipes with auto_ingest from S3 event notifications.
• Wrote SQL scripts for data cleaning and transformation of employee records.
• Verified pipeline performance and ensured automated export of processed data back to S3.

Outcome:

• Raw employee data from S3 is automatically ingested and processed in Snowflake.
• Only active employee records are stored in the dynamic table and exported back to S3.
• Achieved a fully automated, end-to-end data pipeline with minimal manual intervention.

Conclusion:

This project demonstrates how Snowflake and AWS S3 can be integrated for automated ETL pipelines. It improved data processing efficiency, ensured accurate data handling for active employees, and provided a foundation for scaling to more complex pipelines or additional datasets.