Job Summary

Key Responsibilities

• Design, develop and test scalable batch and streaming data pipelines.
• Validate end-to-end ETL and ELT workflows, including ingestion, curation, transformation and loading.
• Perform data reconciliation, database testing and data-warehouse validation across large datasets.
• Validate dashboards and BI reports against source data, transformation rules and business metrics.
• Develop reusable test-automation frameworks using Python or Java and SQL.
• Integrate automated data-quality tests into CI/CD pipelines.
• Build and validate scalable data models using star, snowflake, normalised and denormalised structures.
• Work with cloud data technologies such as BigQuery, Data Factory, Databricks, Spark, Kafka, Pub/Sub and Airflow.
• Implement monitoring, alerting, data-quality controls and secure data-access practices.
• Diagnose defects and data-quality issues, document findings and drive resolution with relevant teams.
• Contribute to data governance, metadata, catalogue and lineage processes.
• Collaborate with engineers, architects, analysts, data stewards and business stakeholders.
• Participate actively in Agile planning, refinement and retrospective ceremonies.

Required Skills and Experience

• Proven experience in data engineering, data-quality engineering or ETL testing.
• Strong SQL skills and proficiency in Python; Java or Scala experience is beneficial.
• Hands-on experience testing data ingestion, transformation, curation, loading and reporting processes.
• Experience with cloud data services, preferably GCP or Azure.
• Knowledge of distributed data-processing technologies such as Spark, Flink or Storm.
• Familiarity with messaging technologies such as Kafka, Pulsar or Pub/Sub.
• Strong understanding of database, data-warehouse and dimensional-modelling concepts.
• Experience developing or using test-automation frameworks.
• Familiarity with CI/CD tools such as Jenkins, GitHub Actions or Azure DevOps.
• Experience with orchestration tools such as Airflow or Cloud Composer.
• Ability to validate dashboards developed using Power BI, Tableau or similar BI tools.
• Understanding of cloud security, IAM, authentication, authorisation and encryption.
• Experience using JIRA or Azure DevOps for defect tracking and project management.
• Strong analytical, troubleshooting, communication and stakeholder-management skills.
• Ability to work independently, take ownership and deliver with minimal supervision.

Desirable Qualifications

• ISTQB or CSQE certification.
• Azure or GCP cloud certification.
• Experience implementing reusable data-test automation frameworks.
• Exposure to Kubernetes, containerisation, data governance, cataloguing and lineage tools.

In short :

• experience in testing ETL or ETL - so I need someone who cna test ingestion, curation, load and also validate dashboards
• someone who has automation mindset - any experience in having implemented automation framework is a plus
• familiar with cloud platform
• understanding of data models 
• is able to run on their own with minimum hand holding (experience would define this)
• collaborates with wider team and also communicates well

Key Responsibilities

Key Responsibilities

• Design, develop and test scalable batch and streaming data pipelines.
• Validate end-to-end ETL and ELT workflows, including ingestion, curation, transformation and loading.
• Perform data reconciliation, database testing and data-warehouse validation across large datasets.
• Validate dashboards and BI reports against source data, transformation rules and business metrics.
• Develop reusable test-automation frameworks using Python or Java and SQL.
• Integrate automated data-quality tests into CI/CD pipelines.
• Build and validate scalable data models using star, snowflake, normalised and denormalised structures.
• Work with cloud data technologies such as BigQuery, Data Factory, Databricks, Spark, Kafka, Pub/Sub and Airflow.
• Implement monitoring, alerting, data-quality controls and secure data-access practices.
• Diagnose defects and data-quality issues, document findings and drive resolution with relevant teams.
• Contribute to data governance, metadata, catalogue and lineage processes.
• Collaborate with engineers, architects, analysts, data stewards and business stakeholders.
• Participate actively in Agile planning, refinement and retrospective ceremonies.

Required Skills and Experience

• Proven experience in data engineering, data-quality engineering or ETL testing.
• Strong SQL skills and proficiency in Python; Java or Scala experience is beneficial.
• Hands-on experience testing data ingestion, transformation, curation, loading and reporting processes.
• Experience with cloud data services, preferably GCP or Azure.
• Knowledge of distributed data-processing technologies such as Spark, Flink or Storm.
• Familiarity with messaging technologies such as Kafka, Pulsar or Pub/Sub.
• Strong understanding of database, data-warehouse and dimensional-modelling concepts.
• Experience developing or using test-automation frameworks.
• Familiarity with CI/CD tools such as Jenkins, GitHub Actions or Azure DevOps.
• Experience with orchestration tools such as Airflow or Cloud Composer.
• Ability to validate dashboards developed using Power BI, Tableau or similar BI tools.
• Understanding of cloud security, IAM, authentication, authorisation and encryption.
• Experience using JIRA or Azure DevOps for defect tracking and project management.
• Strong analytical, troubleshooting, communication and stakeholder-management skills.
• Ability to work independently, take ownership and deliver with minimal supervision.

Desirable Qualifications

• ISTQB or CSQE certification.
• Azure or GCP cloud certification.
• Experience implementing reusable data-test automation frameworks.
• Exposure to Kubernetes, containerisation, data governance, cataloguing and lineage tools.

In short :

• experience in testing ETL or ETL - so I need someone who cna test ingestion, curation, load and also validate dashboards
• someone who has automation mindset - any experience in having implemented automation framework is a plus
• familiar with cloud platform
• understanding of data models 
• is able to run on their own with minimum hand holding (experience would define this)
• collaborates with wider team and also communicates well

Skill Requirements

Other Requirements