Senior Data Lead - Azure Data Factory (ADF), Databricks
United States
Apply now
Job Description
Senior Data Lead - Azure Data Factory (ADF), Databricks
Chicago, Illinois

Job Summary

We are seeking experienced Data Engineers to lead and support the migration of large-scale analytical and reporting workloads from Amazon Redshift to Databricks Lakehouse Platform. The ideal candidate will have hands-on experience in data warehouse modernization, ETL/ELT migration, performance optimization, cloud-native architectures, and big data processing.

Key Responsibilities

Lead the migration of data pipelines, data models, and reporting workloads from Redshift to Databricks.Perform assessment and discovery of existing Redshift environments, schemas, workloads, and dependencies.Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.Convert SQL-based ETL/ELT processes into Databricks notebooks, workflows, and pipelines.Optimize data ingestion, transformation, and storage for performance and cost efficiency.Develop batch and streaming data pipelines using PySpark and Spark SQL.Implement data quality, reconciliation, and validation frameworks during migration.Collaborate with business stakeholders, architects, and analytics teams to ensure successful migration and adoption.Troubleshoot performance bottlenecks and tune Spark jobs, Delta tables, and SQL workloads.Establish CI/CD processes and infrastructure-as-code practices for Databricks environments.Document migration strategies, architecture, and operational procedures.

Skill Requirements

Strong experience with large-scale data warehousing and data lake architectures.Hands-on experience with Amazon Redshift and Databricks.Expertise in PySpark, Spark SQL, and Python.Strong SQL development and query optimization skills.Experience with data modeling (Star Schema, Snowflake Schema, Dimensional Modeling).

Other Requirements

Migration ExperiencePrior experience migrating from:Redshift to DatabricksTeradata/Snowflake/Hadoop to Databricks (preferred)Data validation, reconciliation, and cutover planning.Performance tuning and workload modernization.
Maximum Salary (US): 
Minimum Salary (US): 
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026 totaled $14.8 billion. 

 

Compensation and Benefits

A candidate’s pay within the range will depend on their skills, experience, education, and other factors permitted by law. This role may also be eligible for performance-based bonuses subject to company policies. In addition, this role is eligible for the following benefits subject to company policies: medical, dental, vision, pharmacy, life, accidental death & dismemberment, and disability insurance; employee assistance program; 401(k) retirement plan; 10 days of paid time off per year (some positions are eligible for need-based leave with no designated number of leave days per year); and 10 paid holidays per year.