Azure Senior Data Lead
India
Apply now
Job Description
Azure Senior Data Lead
Bangalore, Karnataka

Job Summary

Design and build optimization capabilities for Databricks - spanning Spark tuning, cluster 
right-sizing, job orchestration, DBU consumption, and Delta Lake storage.
• Translate platform expertise into product features - detection rules, recommendation 
engines, and safe automated actions for production environments.
• Build POCs to validate optimization ideas, demonstrate value, and support pre-sales 
engagements.
• Partner cross-functionally with backend, AI/ML, and data engineering teams to ship features 
end-to-end

Key Responsibilities

Design and build optimization capabilities for Databricks - spanning Spark tuning, cluster 
right-sizing, job orchestration, DBU consumption, and Delta Lake storage.
• Translate platform expertise into product features - detection rules, recommendation 
engines, and safe automated actions for production environments.
• Build POCs to validate optimization ideas, demonstrate value, and support pre-sales 
engagements.
• Partner cross-functionally with backend, AI/ML, and data engineering teams to ship features 
end-to-end

Skill Requirements

Engineering experience; hands-on exp in Databricks in production.
• Apache Spark internals - Catalyst optimizer, Tungsten engine, AQE, DAG scheduler, shuffle 
behavior, partitioning, broadcast/sort-merge joins, data skew handling, and Spark 4.0 
capabilities.
• Databricks platform depth - Delta Lake (transaction log, OPTIMIZE, ZORDER, vacuum, liquid 
clustering, schema evolution, time travel, CDC/merge), Lakeflow Declarative Pipelines, Unity 
Catalog (governance, lineage, fine-grained access), Photon engine, Databricks Workflows, 
Lakebase, and all cluster types (job, all-purpose, serverless SQL, serverless compute).
• Databricks REST API & SDK - programmatic management of clusters, jobs, permissions, and 
workspace configuration.
• Performance tuning - Spark UI interpretation, physical plans, shuffle/skew/spill diagnosis, 
join optimization, caching strategies, and Photon adoption decisions.
• Cost optimization - DBU forecasting, cluster sizing, autoscaling policies, spot vs. on-demand 
trade-offs, instance pools, job-vs-all-purpose decisions, predictive optimization, serverless 
economics (Performance vs. Standard mode, serverless GPU, egress, DBU trade-offs).
• Advanced Python & expert SQL; deep PySpark and Spark SQL internals.
• Cloud platforms (AWS/Azure/GCP) - IAM, networking, storage (S3/ADLS/GCS), and cloudnative services underpinning Databricks.
• Experience with Docker, Kubernetes, Terraform, and modern CI/CD pipelines.
• Strong fundamentals in data structures, algorithms, distributed systems, and large-scale 
system design

MLflow, Mosaic AI ecosystem (Agent Framework, Agent Bricks, AI Gateway, Vector Search), 
feature stores, Databricks SQL Warehouses, or Databricks Asset Bundles.
• FinOps practices and cost-attribution models for data platforms.
• Observability tools - Prometheus, Grafana, OpenTelemetry, Datadog.
• Contributions to open-source Spark/Delta/Databricks projects

Other Requirements

Databricks certifications a plus

BS/MS in Computer Science, Engineering, or related field
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 226,300 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending December 2025 totaled $14.5 billion.
23 Benefits At HCLTech, we believe in empowering our employees with comprehensive benefits that support their professional growth and enhance their well-being. When you sign up for a career with us, you gain access to: https://rmkcdn.successfactors.com/147eb21f/a701dca9-f32d-4fc9-9447-6.svg Industry-benchmarked compensation https://rmkcdn.successfactors.com/147eb21f/b0c54381-ddcc-4a33-9b35-9.svg Best-in-class healthcare benefits https://rmkcdn.successfactors.com/147eb21f/b73027be-7aae-4d36-a090-4.svg Personal time off https://rmkcdn.successfactors.com/147eb21f/d5b4fdfd-2e99-4e26-9878-9.svg Maternity and paternity benefits https://rmkcdn.successfactors.com/147eb21f/3d42b0fc-4652-435a-9ece-c.svg Access to skills / higher education programs/resources https://rmkcdn.successfactors.com/147eb21f/aeddeaf2-9e25-4584-ad11-d.svg Discounts on products and services via Benefit Box https://rmkcdn.successfactors.com/147eb21f/a9609a3b-2700-4b3c-9d90-a.svg Participate in CSR programs and live life with a purpose https://rmkcdn.successfactors.com/147eb21f/c6e33851-710f-4634-bd69-f.svg Opportunities to grow and advance your career Note: The benefits listed above vary depending on the nature of your employment and the country where you work. Some benefits may be available in some countries but not in all.