Job Summary
Job Summary
We are seeking a highly experienced, strategic, and hands-on Lead Data Engineer to join our enterprise data engineering team. With over 12 years of professional expertise, you will act as a bridge between complex business requirements and high-performance technical solutions.
In this role, you will architect resilient, cloud-agnostic Databricks and Lakehouse environments, design enterprise-grade streaming and batch ingestion pipelines, and implement modern architectures such as Data Mesh. You will also drive innovation by integrating Generative AI capabilities (including RAG frameworks and LLM fine-tuning) into modern data platforms, while establishing robust, enterprise-wide data governance and compliance.
Key Responsibilities
1. Architecture & Lakehouse Design
- Architect and optimize highly scalable, resilient cloud-agnostic Databricks Lakehouse environments to convert multi-terabyte batch and streaming data into actionable intelligence.
- Lead complex, large-scale database and process migrations from legacy estates (e.g., SQL Server, on-premise systems) to modern cloud environments (Azure, AWS, Snowflake, MS Fabric), improving query performance and reducing infrastructure overhead.
- Design and deploy modern semantic data models and high-performance BI reporting solutions (such as Power BI or Tableau) to enable executives with real-time KPI insights.
2. Data Pipeline & API Engineering (ETL/ELT)
- Design, build, and dynamically parameterize reusable ETL/ELT pipelines using platforms like Azure Data Factory (ADF), AWS Glue/Lambda/Step Functions, Snaplogic, Ab Initio, or Microsoft Fabric.
- Develop robust stream-processing pipelines utilizing Apache Kafka, Azure Event Hubs, or Amazon Event Bridge alongside REST APIs to build synchronous and event-driven data workflows.
- Implement automated data-loading accelerators (such as Databricks Auto Loader) to ingest varied, heterogeneous data structures (structured, semi-structured, APIs, and flat files) with minimal manual overhead.
3. Data Governance, Compliance & Security
- Devise and enforce unified data governance frameworks incorporating data quality (DQ) metrics, Attribute-Based Access Control (ABAC), data lineage, and compliance tracking.
- Ensure rigorous protection of PII and adherence to international regulatory standards (e.g., ISO, GDPR, HIPAA) using tools like Microsoft Purview or Informatica MDM/IDD.
4. Generative AI & Innovation
- Lead the adoption of emerging AI technologies by designing RAG (Retrieval-Augmented Generation) virtual assistants and integrating them natively into Databricks platforms.
- Collaborate on fine-tuning Large Language Models (LLMs) on domain-specific datasets to deliver high-quality, context-aware responses in real time.
5. Leadership & Project Delivery
- Provide technical leadership and mentorship to cross-functional, onshore-offshore engineering teams.
- Manage stakeholder relationships, collaborating with business users, functional analysts, and executive teams to scope and prioritize key deliverables.
Enforce agile practices (Scrum/Kanban) and DevOps standards (CI/CD pipelines via Azure DevOps/GitHub) to maintain delivery excellence
Skill Requirements
Required Technical Skills:
- Cloud & ETL/ELT: Deep expertise in Azure (ADF, Azure Synapse, ADLS), Azure Databricks, MS Fabric, AWS (S3, EMR, DMS, Glue, Lambda), Snowflake, Snaplogic, or SSIS.
- Big Data & Databases: Strong experience in Postgres, Oracle, SQL Server, Snowflake, Teradata, MongoDB, and Hadoop/Hive ecosystems.
- Programming & Scripting: Mastery in Python, PySpark, T-SQL, and Unix Shell Scripting. Experience in C# or Java is highly desirable.
- Streaming & Integration: Apache Kafka, Azure Event Hubs, REST API integration, and Change Data Capture (CDC) technologies (e.g., Qlik Replicate).
- AI/BI Tools: Power BI, Tableau, RAG implementations, and LLM orchestration.
- DevOps & Tools: Azure DevOps, GitHub, Jira, and SFTP automation.
Qualifications & Education:
- Education: Bachelor’s Degree in Engineering (B.E./B.Tech) in Computer Science, Information Technology, or a closely related field.
- Certifications (Highly Valued):
- Databricks Certified Data Engineer Associate / Professional
- Databricks Generative AI Fundamentals or Lakehouse Fundamentals
- Scrum Fundamentals Certified (SFC)
- Snaplogic or cloud provider certifications (Azure/AWS)