Job Summary
Role Overview
We are hiring an experienced Data Engineer to design, build, and optimize scalable data solutions across AWS and Microsoft Azure. The role requires strong hands-on expertise in Databricks, data lakes and lakehouse architecture, ETL/ELT pipelines, data warehousing, and the delivery of business-ready reporting solutions using Power BI.
Key Responsibilities
Responsibilities (Brief)
- Design, develop, test, and maintain scalable batch and streaming data pipelines across AWS and Azure.
- Build and optimize Databricks lakehouse solutions using PySpark, Delta Lake, and medallion architecture.
- Integrate structured and unstructured data from multiple sources while ensuring reliability, scalability, and cost efficiency.
- Develop curated data models and semantic layers to support Power BI dashboards, KPIs, and self-service reporting.
- Apply data quality checks, security controls, governance standards, metadata management, and lineage across the data lifecycle.
- Monitor and troubleshoot pipeline failures, improve query and workload performance, and support production data operations.
- Collaborate with architects, analysts, BI developers, and business stakeholders to translate reporting needs into dependable data products.
Skill Requirements
Key Skills Required
- Programming & Querying: Advanced SQL, Python, and PySpark
- AWS Data Services: Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, IAM, and related data integration services
- Azure Data Services: Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, and Azure Key Vault
- Databricks: Apache Spark, Delta Lake, notebooks, jobs, workflows, performance optimization, and Unity Catalog
- Data Engineering: Batch and streaming ingestion, ETL/ELT development, orchestration, monitoring, and reusable data pipelines
- Data Architecture: Data lake, lakehouse, cloud data warehouse, and medallion architecture
- Data Modelling: Dimensional modelling, star and snowflake schemas, and semantic data models
- BI & Reporting: Power BI, DAX, Power Query, data modelling, dashboard development, and performance tuning
- Data Governance: Data quality, security, metadata management, lineage, access control, and cataloguing
- DevOps: Git, CI/CD, infrastructure deployment practices, and environment management for data solutions
Other Requirements
Preferred Qualifications
- 6–10+ years of data engineering experience, including hands-on delivery on AWS and Azure.
- Strong experience with Databricks and Power BI in enterprise-scale analytics environments.
- Exposure to Snowflake, Fivetran, HVR, Tableau, Atlan, or similar data integration and governance tools.
- Understanding of cloud security, cost optimization, deployment automation, and production support practices.
- Strong stakeholder communication, problem-solving, documentation, and cross-functional collaboration skills.