Job Summary
Job Title
Senior Data Engineer / Cloud Data Engineer (Python, PySpark, AWS & Terraform)
Job Summary
We are seeking a highly skilled and motivated Data Engineer with strong expertise in Python, PySpark, AWS Cloud, and Terraform to design, develop, and maintain scalable cloud-based data platforms and ETL/ELT pipelines. The ideal candidate will have experience building data solutions on AWS, automating infrastructure using Infrastructure as Code (IaC), and processing large-scale datasets using distributed computing frameworks.
The role requires collaboration with architects, data scientists, business analysts, and DevOps teams to deliver secure, reliable, and high-performing data products.
Key Responsibilities
1. To maintain existing features, troubleshoot bugs , resolve ad-hoc requests and provide support for enhancement (major and minor) /new developments based on business/ client requirements as well as from the functional and technical team of HCL on the project.
2. To provide client support by presenting data, information, ticket resolution and day to day support activities like monitoring client requirements as well as keeping track of schedule for on time delivery of assigned tasks as per the defined quality standards.
3. To perform activities related to enhancement creation of documents for CMMi and client requirements
4. To provide technical guidance to junior developers
5. To interact with the customer and internal teams to gather requirements for development purposes.
Skill Requirements
Data Engineering & Development
- Design, develop, and optimize scalable data pipelines using Python and PySpark.
- Build and maintain batch and real-time data processing solutions.
- Develop reusable data ingestion, transformation, and validation frameworks.
- Perform data cleansing, enrichment, aggregation, and quality assurance.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
Cloud & AWS Platform Development
- Design and implement cloud-native data solutions on AWS.
- Work with AWS services such as:
- S3
- Glue
- EMR
- Lambda
- Athena
- Redshift
- RDS
- DynamoDB
- CloudWatch
- Step Functions
- SNS/SQS
- IAM
- KMS
- Implement secure data storage, access control, and governance practices.
- Monitor cloud infrastructure and optimize AWS resource utilization.
Infrastructure as Code (Terraform)
- Develop and maintain infrastructure provisioning using Terraform.
- Create reusable Terraform modules for AWS infrastructure deployment.
- Automate environment creation across Development, QA, UAT, and Production.
- Manage infrastructure versioning and state management.
Ensure compliance with DevOps and security standards.