Senior Data Lead - Azure Data Factory (ADF), Databricks
Bulgaria
Apply now
Job Description
Senior Data Lead - Azure Data Factory (ADF), Databricks
Ablanica, Blagoevgrad

Job Summary

Position Overview:
The Data Engineer will be responsible for designing, implementing, and maintaining data pipelines and processes to support the organization's data needs. The ideal candidate should have expertise in Python/pyspark frameworks/libraries, Azure Data Factory (ADF), SQL, and various data transformation frameworks. They should also understand underlying systems, including storage, processing, and visualization, and be familiar with Airflow for workflow orchestration.
Key Responsibilities:
1.
Python/pyspark Expertise:
o
Proficiency in Python/pyspark frameworks/libraries for data processing and analysis.
2.
Azure Databricks (ADB) Expertise:
o
Strong knowledge and experience working with Azure Databricks for data engineering tasks.
3.
SQL Skills:
o
Expertise in SQL for querying and manipulating data in relational databases.
4.
ETL and Process Flow Design:
o
Design and implement complex ETL (Extract, Transform, Load) processes and workflows to ensure efficient data movement and transformation.
5.
Data Transformation Frameworks:
o
Proficiency across various data transformation frameworks to handle diverse data processing requirements.
6.
Understanding of Underlying Systems:
o
Familiarity with the underlying systems including storage, processing, and visualization technologies to optimize data workflows.

7.
Airflow Configuration and Orchestration:
o
Configure and orchestrate data pipelines using Apache Airflow for workflow automation and scheduling.
Good to Have:
1.
Azure Data Factory (ADF) Expertise:
o
Experience with Azure Data Factory for building, orchestrating, and monitoring data pipelines.
2.
Database Execution Plans:
o
Ability to read and understand database execution plans to optimize query performance.
3.
Azure/Azure DevOps Knowledge:
o
Familiarity with Azure cloud services and Azure DevOps for managing and deploying data solutions.
4.
Configuration Management, Continuous Integration, Continuous Deployment (CI/CD), and Cloud Implementations on Azure:
o
Understanding of configuration management practices, CI/CD pipelines, and cloud implementations on the Azure platform.
5.
Azure Certification:
o
Azure certification is a plus, demonstrating proficiency in Azure cloud technologies and services.
Qualifications:
•
Bachelor's degree in Computer Science, Information Technology, or related field.
•
Proven experience as a Data Engineer or similar role.
•
Strong proficiency in Python/pyspark, SQL, and Azure Data Factory.
•
Experience designing and implementing ETL processes and data pipelines.
•
Familiarity with data transformation frameworks and workflow orchestration tools like Apache Airflow.
•
Understanding of database systems, cloud platforms, and CI/CD practices.
•
Azure certification

Key Responsibilities

Position Overview:
The Data Engineer will be responsible for designing, implementing, and maintaining data pipelines and processes to support the organization's data needs. The ideal candidate should have expertise in Python/pyspark frameworks/libraries, Azure Data Factory (ADF), SQL, and various data transformation frameworks. They should also understand underlying systems, including storage, processing, and visualization, and be familiar with Airflow for workflow orchestration.
Key Responsibilities:
1.
Python/pyspark Expertise:
o
Proficiency in Python/pyspark frameworks/libraries for data processing and analysis.
2.
Azure Databricks (ADB) Expertise:
o
Strong knowledge and experience working with Azure Databricks for data engineering tasks.
3.
SQL Skills:
o
Expertise in SQL for querying and manipulating data in relational databases.
4.
ETL and Process Flow Design:
o
Design and implement complex ETL (Extract, Transform, Load) processes and workflows to ensure efficient data movement and transformation.
5.
Data Transformation Frameworks:
o
Proficiency across various data transformation frameworks to handle diverse data processing requirements.
6.
Understanding of Underlying Systems:
o
Familiarity with the underlying systems including storage, processing, and visualization technologies to optimize data workflows.

7.
Airflow Configuration and Orchestration:
o
Configure and orchestrate data pipelines using Apache Airflow for workflow automation and scheduling.
Good to Have:
1.
Azure Data Factory (ADF) Expertise:
o
Experience with Azure Data Factory for building, orchestrating, and monitoring data pipelines.
2.
Database Execution Plans:
o
Ability to read and understand database execution plans to optimize query performance.
3.
Azure/Azure DevOps Knowledge:
o
Familiarity with Azure cloud services and Azure DevOps for managing and deploying data solutions.
4.
Configuration Management, Continuous Integration, Continuous Deployment (CI/CD), and Cloud Implementations on Azure:
o
Understanding of configuration management practices, CI/CD pipelines, and cloud implementations on the Azure platform.
5.
Azure Certification:
o
Azure certification is a plus, demonstrating proficiency in Azure cloud technologies and services.
Qualifications:
•
Bachelor's degree in Computer Science, Information Technology, or related field.
•
Proven experience as a Data Engineer or similar role.
•
Strong proficiency in Python/pyspark, SQL, and Azure Data Factory.
•
Experience designing and implementing ETL processes and data pipelines.
•
Familiarity with data transformation frameworks and workflow orchestration tools like Apache Airflow.
•
Understanding of database systems, cloud platforms, and CI/CD practices.
•
Azure certification

Skill Requirements

Position Overview:
The Data Engineer will be responsible for designing, implementing, and maintaining data pipelines and processes to support the organization's data needs. The ideal candidate should have expertise in Python/pyspark frameworks/libraries, Azure Data Factory (ADF), SQL, and various data transformation frameworks. They should also understand underlying systems, including storage, processing, and visualization, and be familiar with Airflow for workflow orchestration.
Key Responsibilities:
1.
Python/pyspark Expertise:
o
Proficiency in Python/pyspark frameworks/libraries for data processing and analysis.
2.
Azure Databricks (ADB) Expertise:
o
Strong knowledge and experience working with Azure Databricks for data engineering tasks.
3.
SQL Skills:
o
Expertise in SQL for querying and manipulating data in relational databases.
4.
ETL and Process Flow Design:
o
Design and implement complex ETL (Extract, Transform, Load) processes and workflows to ensure efficient data movement and transformation.
5.
Data Transformation Frameworks:
o
Proficiency across various data transformation frameworks to handle diverse data processing requirements.
6.
Understanding of Underlying Systems:
o
Familiarity with the underlying systems including storage, processing, and visualization technologies to optimize data workflows.

7.
Airflow Configuration and Orchestration:
o
Configure and orchestrate data pipelines using Apache Airflow for workflow automation and scheduling.
Good to Have:
1.
Azure Data Factory (ADF) Expertise:
o
Experience with Azure Data Factory for building, orchestrating, and monitoring data pipelines.
2.
Database Execution Plans:
o
Ability to read and understand database execution plans to optimize query performance.
3.
Azure/Azure DevOps Knowledge:
o
Familiarity with Azure cloud services and Azure DevOps for managing and deploying data solutions.
4.
Configuration Management, Continuous Integration, Continuous Deployment (CI/CD), and Cloud Implementations on Azure:
o
Understanding of configuration management practices, CI/CD pipelines, and cloud implementations on the Azure platform.
5.
Azure Certification:
o
Azure certification is a plus, demonstrating proficiency in Azure cloud technologies and services.
Qualifications:
•
Bachelor's degree in Computer Science, Information Technology, or related field.
•
Proven experience as a Data Engineer or similar role.
•
Strong proficiency in Python/pyspark, SQL, and Azure Data Factory.
•
Experience designing and implementing ETL processes and data pipelines.
•
Familiarity with data transformation frameworks and workflow orchestration tools like Apache Airflow.
•
Understanding of database systems, cloud platforms, and CI/CD practices.
•
Azure certification

Other Requirements

Position Overview:
The Data Engineer will be responsible for designing, implementing, and maintaining data pipelines and processes to support the organization's data needs. The ideal candidate should have expertise in Python/pyspark frameworks/libraries, Azure Data Factory (ADF), SQL, and various data transformation frameworks. They should also understand underlying systems, including storage, processing, and visualization, and be familiar with Airflow for workflow orchestration.
Key Responsibilities:
1.
Python/pyspark Expertise:
o
Proficiency in Python/pyspark frameworks/libraries for data processing and analysis.
2.
Azure Databricks (ADB) Expertise:
o
Strong knowledge and experience working with Azure Databricks for data engineering tasks.
3.
SQL Skills:
o
Expertise in SQL for querying and manipulating data in relational databases.
4.
ETL and Process Flow Design:
o
Design and implement complex ETL (Extract, Transform, Load) processes and workflows to ensure efficient data movement and transformation.
5.
Data Transformation Frameworks:
o
Proficiency across various data transformation frameworks to handle diverse data processing requirements.
6.
Understanding of Underlying Systems:
o
Familiarity with the underlying systems including storage, processing, and visualization technologies to optimize data workflows.

7.
Airflow Configuration and Orchestration:
o
Configure and orchestrate data pipelines using Apache Airflow for workflow automation and scheduling.
Good to Have:
1.
Azure Data Factory (ADF) Expertise:
o
Experience with Azure Data Factory for building, orchestrating, and monitoring data pipelines.
2.
Database Execution Plans:
o
Ability to read and understand database execution plans to optimize query performance.
3.
Azure/Azure DevOps Knowledge:
o
Familiarity with Azure cloud services and Azure DevOps for managing and deploying data solutions.
4.
Configuration Management, Continuous Integration, Continuous Deployment (CI/CD), and Cloud Implementations on Azure:
o
Understanding of configuration management practices, CI/CD pipelines, and cloud implementations on the Azure platform.
5.
Azure Certification:
o
Azure certification is a plus, demonstrating proficiency in Azure cloud technologies and services.
Qualifications:
•
Bachelor's degree in Computer Science, Information Technology, or related field.
•
Proven experience as a Data Engineer or similar role.
•
Strong proficiency in Python/pyspark, SQL, and Azure Data Factory.
•
Experience designing and implementing ETL processes and data pipelines.
•
Familiarity with data transformation frameworks and workflow orchestration tools like Apache Airflow.
•
Understanding of database systems, cloud platforms, and CI/CD practices.
•
Azure certification
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 227,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending December 2026 totaled $14.7 billion.
23 Benefits At HCLTech, we believe in empowering our employees with comprehensive benefits that support their professional growth and enhance their well-being. When you sign up for a career with us, you gain access to: https://rmkcdn.successfactors.com/147eb21f/a701dca9-f32d-4fc9-9447-6.svg Industry-benchmarked compensation https://rmkcdn.successfactors.com/147eb21f/b0c54381-ddcc-4a33-9b35-9.svg Best-in-class healthcare benefits https://rmkcdn.successfactors.com/147eb21f/b73027be-7aae-4d36-a090-4.svg Personal time off https://rmkcdn.successfactors.com/147eb21f/d5b4fdfd-2e99-4e26-9878-9.svg Maternity and paternity benefits https://rmkcdn.successfactors.com/147eb21f/3d42b0fc-4652-435a-9ece-c.svg Access to skills / higher education programs/resources https://rmkcdn.successfactors.com/147eb21f/aeddeaf2-9e25-4584-ad11-d.svg Discounts on products and services via Benefit Box https://rmkcdn.successfactors.com/147eb21f/a9609a3b-2700-4b3c-9d90-a.svg Participate in CSR programs and live life with a purpose https://rmkcdn.successfactors.com/147eb21f/c6e33851-710f-4634-bd69-f.svg Opportunities to grow and advance your career Note: The benefits listed above vary depending on the nature of your employment and the country where you work. Some benefits may be available in some countries but not in all.