Job Summary
We are looking for an experienced Azure Databricks Data Engineer to design, build, optimize, and support scalable enterprise data platforms. The professional will work across the complete data engineering lifecycle, including source onboarding, ingestion, transformation, data quality, lakehouse modelling, orchestration, security, monitoring, and production support. The role requires strong hands-on expertise in Databricks and the Azure data ecosystem, along with the ability to translate business requirements into reliable and cost-effective data solutions.
Key Responsibilities
Design and implement scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, and SQL. • Build lakehouse solutions using Delta Lake and medallion architecture patterns across Bronze, Silver, and Gold layers. • Develop reusable ingestion frameworks for structured, semi-structured, and unstructured data from databases, files, APIs, event streams, and SaaS platforms. • Implement orchestration and scheduling using Azure Data Factory, Databricks Workflows, and related Azure services. • Design dimensional models, curated data products, and consumption-ready datasets for analytics and reporting platforms such as Power BI. • Apply data quality checks, schema enforcement, lineage, metadata management, auditability, and error-handling standards. • Optimize Spark workloads, cluster configurations, storage layouts, SQL queries, and Delta tables for performance and cost. • Implement secure access using Microsoft Entra ID, managed identities, service principals, Azure Key Vault, Unity Catalog, and role-based access controls. • Build CI/CD pipelines and infrastructure automation using Azure DevOps or GitHub, Git, Databricks Asset Bundles, and Terraform where applicable. • Monitor production pipelines, troubleshoot failures and performance issues, perform root-cause analysis, and contribute to operational support and continuous improvement. • Collaborate with architects, analysts, data scientists, source-system owners, and business stakeholders to refine requirements and deliver fit-for-purpose solutions. • Prepare technical designs, mapping specifications, runbooks, deployment documentation, and knowledge-transfer material.
Skill Requirements
5–8 years of overall experience in data engineering, data warehousing, or large-scale data platform delivery. • At least 3 years of strong hands-on experience with Azure Databricks and Apache Spark. • Advanced proficiency in PySpark and SQL, with practical experience in performance tuning and debugging. • Strong understanding of Delta Lake, lakehouse architecture, medallion patterns, partitioning, file optimization, and incremental data processing. • Hands-on experience with Azure Data Lake Storage Gen2, Azure Data Factory, Azure Key Vault, and Azure DevOps or GitHub. • Experience implementing ETL/ELT pipelines, change data capture, slowly changing dimensions, data reconciliation, and rerun or recovery mechanisms. • Working knowledge of Databricks governance and security capabilities, preferably Unity Catalog. • Experience with REST APIs, JSON, Parquet, CSV, relational databases, and common enterprise integration patterns. • Good understanding of dimensional modelling, data warehouses, data marts, and analytics consumption requirements. • Experience working in Agile delivery teams and participating in design reviews, code reviews, testing, release, and production support.