Senior Data Lead - Azure Data Factory (ADF), Databricks
India
Apply now
Job Description
Senior Data Lead - Azure Data Factory (ADF), Databricks
Ahmadabad City, Gujarat

Job Summary

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Databricks.
  • Build and optimize ETL/ELT processes for structured and unstructured data.
  • Develop solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
  • Implement data ingestion frameworks from various source systems including APIs, databases, cloud storage, and streaming platforms.
  • Design and maintain Bronze, Silver, and Gold data layers following Medallion Architecture.
  • Optimize Spark jobs for performance, scalability, and cost efficiency.
  • Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.
  • Collaborate with business analysts, data scientists, architects, and stakeholders to gather requirements.
  • Ensure data quality, governance, security, and compliance standards.
  • Monitor and troubleshoot production data pipelines and workflows.
  • Participate in code reviews and implement data engineering best practices.

Key Responsibilities

Required Qualifications

 

  • 7+ years of experience in Data Engineering and Data Warehousing.
  • 4+ years of hands-on experience with Databricks.
  • Strong expertise in:
    • PySpark
    • Spark SQL
    • Python
    • SQL
    • Delta Lake
  • Experience building large-scale batch and streaming data pipelines.
  • Strong understanding of data modeling and data warehousing concepts.
  • Experience with cloud platforms (Azure, AWS, or GCP).
  • Knowledge of CI/CD and DevOps practices.

Skill Requirements

Technical Skills

Databricks & Big Data

  • Databricks Lakehouse Platform
  • Apache Spark
  • PySpark
  • Spark SQL
  • Delta Lake
  • Databricks Workflows
  • Unity Catalog

Programming

  • Python
  • SQL
  • Scala (Preferred)

Cloud Platforms

  • Azure Databricks
  • Azure Data Factory (ADF)
  • AWS EMR / Glue
  • Google Cloud Platform (GCP)

Data Storage

  • Azure Data Lake Storage (ADLS)
  • Amazon S3
  • Snowflake
  • Synapse Analytics
  • SQL Server
  • Oracle

Data Integration

  • Kafka
  • Event Hub
  • REST APIs
  • CDC Implementation

DevOps & Monitoring

  • Git
  • Azure DevOps
  • Jenkins
  • Terraform
  • Monitoring and Logging Tools

Other Requirements

Preferred Qualifications

  • Experience with Azure Data Factory (ADF), Synapse Analytics, or Snowflake.
  • Knowledge of Data Governance and Unity Catalog.
  • Experience building data lakes and lakehouse architectures.
  • Exposure to machine learning workflows and MLOps.
  • Databricks Certification:
    • Databricks Certified Data Engineer Associate
    • Databricks Certified Data Engineer Professional

Experience working in Agile/Scrum environments.
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 230,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending September 2026 totaled $14.8 billion.