Job Summary
Design, develop, and maintain secure and scalable data pipelines and infrastructure using AWS services such as EMR, S3, Glue, and Lambda.
Implement Infrastructure as Code using Terraform to automate the provisioning and configuration of cloud resources, ensuring consistent and repeatable deployments.
Collaborate closely with FinOps teams to monitor, optimise, and control cloud costs while maintaining robust security postures.
Develop and enforce security best practices across the data lakehouse environment, including data encryption, access controls, and compliance with organisational policies.
Work in partnership with data engineering, data science, and security teams to integrate security and cost-efficiency into data workflows and infrastructure.
Monitor and troubleshoot performance, security, and cost issues across AWS and Snowflake environments.
Ensure Airflow workflows are optimised, secure, and maintainable, supporting critical data orchestration needs.
Key Responsibilities
1. Design, develop, and maintain data workflows and etl processes using amazon glue
2. Write complex sql queries to extract and manipulate data effectively
3. Develop python scripts for data processing and automation tasks
4. Implement devops solutions for continuous integration and deployment of data pipelines
5. Work with snowflake data warehouse for data storage and retrieval
6. Collaborate with cross functional teams to understand business requirements and deliver effective solutions
7. Troubleshoot and optimize existing data pipelines for performance and scalability
Skill Requirements
Proven experience as a Data Engineer or Cloud Engineer working extensively with AWS services, particularly AWS EMR, S3, Lambda, and Snowflake.
Strong hands-on experience with Terraform or similar IaC tools for automated infrastructure deployment and management.
Knowledge of FinOps principles and experience collaborating on cloud cost optimisation and governance.
Solid understanding of cloud security best practices, including identity and access management, network security, and data protection.
Experience with data orchestration tools such as Apache Airflow.
Proficient in scripting languages such as Python or Bash.
Excellent problem-solving skills and the ability to work effectively in cross-functional teams.
5. Familiarity with snowflake data warehouse for data storage and retrieval
6. Analytical mindset with problem-solving skills
7. Strong communication and teamwork abilities