Job Summary
- Develop automated scripts to validate source-to-target data movements, ensuring schema consistency, completeness, and timeliness.
- Build robust, reusable testing utilities using Python, native AWS services that integrate seamlessly into DataOps workflows.
- Set up continuous data drift monitoring, schema validation, and data quality dashboards to proactively catch pipeline failures.
- Partner with data engineers, BI teams, and software engineers to translate business rules into strict data quality KPIs and SLAs.
Key Responsibilities
- Develop automated scripts to validate source-to-target data movements, ensuring schema consistency, completeness, and timeliness.
- Build robust, reusable testing utilities using Python, native AWS services that integrate seamlessly into DataOps workflows.
- Set up continuous data drift monitoring, schema validation, and data quality dashboards to proactively catch pipeline failures.
- Partner with data engineers, BI teams, and software engineers to translate business rules into strict data quality KPIs and SLAs.
Skill Requirements
- Deep proficiency in AWS data and storage services like Amazon S3, AWS Glue, Amazon Redshift, and AWS Lambda.
- Advanced proficiency in Python for building automation frameworks and orchestrating data tests.
- Experience with orchestration tools (e.g., Apache Airflow), data transformation, and data quality monitoring frameworks (e.g., Amazon Deequ).
- Strong command of SQL and UNIX/Linux scripting for querying and analyzing large-scale, distributed datasets
Other Requirements
Certifications: Candidate with AWS Certified Data Engineer – Associate - Highly preferred