Azure Senior Data Lead
India
Apply now
Job Description
Azure Senior Data Lead
Bangalore, Karnataka

Job Summary

We are seeking an experienced Senior Azure Databricks Data Engineer to lead the modernization and migration of existing Python Object-Oriented (OOP) applications into scalable PySpark and Spark SQL-based data processing solutions on Azure Databricks.

The ideal candidate will possess deep expertise in Python OOP, software engineering principles, distributed computing, PySpark, Spark SQL, and Azure Databricks. The primary responsibility will be to analyze existing Python applications, redesign processing logic for distributed execution, optimize performance, and build enterprise-grade data engineering solutions on Azure.

This role requires a strong blend of software engineering, data engineering, cloud architecture, and performance optimization skills.

Key Responsibilities

Python Application Modernization

  • Analyze existing Python OOP applications, libraries, and frameworks.
  • Refactor and convert object-oriented Python code into scalable PySpark-based distributed processing solutions.
  • Identify bottlenecks in single-node Python applications and redesign them for Spark execution.
  • Transform procedural and class-based business logic into DataFrame-based processing patterns.
  • Modernize legacy ETL frameworks into cloud-native Spark architectures.

Azure Databricks Development

  • Design, develop, and deploy enterprise-scale data pipelines using Azure Databricks.
  • Build reusable PySpark frameworks, libraries, and utility modules.
  • Develop Spark SQL transformations for large-scale analytics workloads.
  • Implement Delta Lake-based solutions using Bronze-Silver-Gold architecture.
  • Optimize Spark jobs for performance, scalability, and cost efficiency.

Data Engineering

  • Design robust ETL/ELT pipelines using:

    • Azure Databricks
    • Azure Data Factory
    • ADLS Gen2
    • Delta Lake
    • Azure Synapse Analytics

  • Process structured, semi-structured, and unstructured data from multiple enterprise sources.

  • Implement data quality, reconciliation, validation, and monitoring frameworks.

Performance Engineering

  • Optimize Spark jobs using:

    • Partitioning
    • Bucketing
    • Caching
    • Broadcast Joins
    • Adaptive Query Execution
    • Delta Optimization Techniques

  • Benchmark converted PySpark applications against original Python implementations.

  • Reduce execution time and improve scalability for large-volume data workloads.

Skill Requirements

Azure Stack

  • Azure Databricks
  • Azure Data Factory (ADF)
  • Azure Data Lake Storage Gen2 (ADLS)

Programming

  • Python (Expert Level)
  • Object-Oriented Programming (OOP)
  • Advanced Python Design Patterns
  • PySpark
  • Spark SQL
  • SQL

Big Data Technologies

  • Apache Spark
  • Delta Lake
  • Data Lakehouse Architecture
  • Distributed Computing Concepts

Other Requirements

1.Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus.
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 227,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending March 2026 totaled $14.7 billion.