Job Summary
Senior Data Engineer – Azure Databricks & ADF
Senior data engineer responsible for designing, developing, and optimizing scalable data pipelines using Azure Databricks and Azure Data Factory, with strong ownership in delivery, transition, and stabilization as part of offshore ramp-up.
Key Responsibilities
Key Responsibilities
· Design and develop end-to-end data pipelines using ADF and Databricks
· Build scalable transformation logic using PySpark / Spark SQL
· Implement and maintain Delta Lake architecture (Bronze/Silver/Gold layers)
· Translate business requirements into efficient data models and pipeline designs
· Lead complex data ingestion scenarios (structured, semi-structured, APIs)
· Actively support knowledge transition (KT) from incumbent team
· Drive performance tuning and optimization of pipelines and clusters
· Ensure data quality, validation, and reconciliation checks
· Contribute to CI/CD setup, code reviews, and release processes
· Mentor junior engineers and ensure delivery quality
Skill Requirements
Mandatory Skills
· 8+ years of data engineering experience
· Strong hands-on expertise in:
o Azure Databricks (PySpark, Spark SQL, Delta Lake)
o Azure Data Factory (pipeline orchestration, triggers, monitoring)
· Deep understanding of:
o Data pipeline design patterns
o Data modelling concepts
o Large-scale data processing
· Strong SQL expertise
· Experience working in complex enterprise data environments
Preferred Skills
· Prior experience in transition / offshore backfill engagements
· Familiarity with:
o Medallion architecture
o Data lakehouse concepts
· Exposure to:
o Azure DevOps / CI-CD pipelines
o Monitoring and logging frameworks
· Domain experience in Insurance / Financial Services
Soft Skills
· Strong ownership and accountability (critical for incumbent replacement)
· Ability to work independently with minimal supervision
Other Requirements
Mandatory Skills
· 8+ years of data engineering experience
· Strong hands-on expertise in:
o Azure Databricks (PySpark, Spark SQL, Delta Lake)
o Azure Data Factory (pipeline orchestration, triggers, monitoring)
· Deep understanding of:
o Data pipeline design patterns
o Data modelling concepts
o Large-scale data processing
· Strong SQL expertise
· Experience working in complex enterprise data environments
Preferred Skills
· Prior experience in transition / offshore backfill engagements
· Familiarity with:
o Medallion architecture
o Data lakehouse concepts
· Exposure to:
o Azure DevOps / CI-CD pipelines
o Monitoring and logging frameworks
· Domain experience in Insurance / Financial Services
Soft Skills
· Strong ownership and accountability (critical for incumbent replacement)
· Ability to work independently with minimal supervision