Job Summary
We are seeking a highly experienced Senior Lead – Azure Databricks to drive enterprise-scale data engineering, advanced analytics, and AI/ML initiatives. The ideal candidate will have deep expertise in Azure Databricks, strong leadership skills, and the ability to architect and deliver end-to-end data solutions integrated with Azure AI services.
Key Responsibilities
Key ResponsibilitiesData Engineering Leadership: Lead design and development of large-scale data pipelines using Azure Databricks, Spark, and Delta Lake.
Solution Architecture: Architect scalable, secure, and high-performance data platforms on Azure.
AI/ML Integration: Collaborate with AI teams to integrate Databricks pipelines with Azure Machine Learning and Azure AI Foundry.
Data Governance: Implement governance, lineage, and compliance frameworks across data platforms.
Performance Optimization: Optimize Spark jobs, Delta tables, and cluster configurations for efficiency.
Team Mentorship: Mentor junior engineers and lead cross-functional teams in delivering enterprise data solutions.
Stakeholder Collaboration: Partner with business leaders, data scientists, and product teams to translate requirements into technical solutions.
Innovation: Explore emerging Databricks and Azure AI capabilities to enhance enterprise data strategy.
Skill Requirements
Azure Databricks: Deep expertise in Databricks, Spark, Delta Lake, and MLflow.
Data Platforms: Strong knowledge of Azure Data Lake, Synapse Analytics, and Data Factory
.Programming: Proficiency in Python, Scala, and SQL for data engineering.
AI Services: Experience integrating Databricks with Azure AI Foundry, Cognitive Services, and Machine Learning.
DevOps: Knowledge of CI/CD pipelines, DataOps, and MLOps practices.
Visualization: Ability to deliver insights using Power BI and other visualization tools
.Leadership: Proven track record of leading teams and managing enterprise projects.