Job Summary
As a Microsoft Fabric Data Engineer at HCLTech, you will play a pivotal role in architecting and implementing advanced data solutions leveraging Microsoft Fabric. You will be responsible for designing robust data pipelines, ensuring the integrity and quality of data, and enabling actionable business intelligence across the organization. Your expertise will directly contribute to empowering data-driven decision-making and accelerating the company’s transformation initiatives.
Detailed Responsibilities
• Design, develop, and implement scalable ETL processes using Microsoft Fabric to extract, transform, and load data from diverse sources into cloud-based data warehouses or lakehouses.
• Collaborate closely with data analysts, engineers, and stakeholders to gather business requirements and translate them into effective data solutions.
• Build and maintain Power BI dashboards and reports, delivering actionable insights from complex datasets to drive informed business decisions.
• Enforce data quality, integrity, and accuracy throughout the ETL lifecycle, implementing industry best practices for data management and governance.
• Monitor ETL processes, proactively identifying and resolving issues or performance bottlenecks to maintain high system reliability.
• Document ETL workflows, data mappings, and process changes to ensure transparency and clear communication across technical and non-technical teams.
• Implement advanced data transformation and ingestion patterns, including CDC, schema evolution, and error handling for robust data operations.
Key Responsibilities
Detailed Responsibilities:
• Design, develop, and implement scalable ETL processes using Microsoft Fabric to extract, transform, and load data from diverse sources into cloud-based data warehouses or lakehouses.
• Collaborate closely with data analysts, engineers, and stakeholders to gather business requirements and translate them into effective data solutions.
• Build and maintain Power BI dashboards and reports, delivering actionable insights from complex datasets to drive informed business decisions.
• Enforce data quality, integrity, and accuracy throughout the ETL lifecycle, implementing industry best practices for data management and governance.
• Monitor ETL processes, proactively identifying and resolving issues or performance bottlenecks to maintain high system reliability.
• Document ETL workflows, data mappings, and process changes to ensure transparency and clear communication across technical and non-technical teams.
• Implement advanced data transformation and ingestion patterns, including CDC, schema evolution, and error handling for robust data operations.
Skill Requirements
Must Have Skills:
• 7+ years of hands-on experience in ETL development, specifically utilizing Microsoft Fabric in recent projects.
• Expertise in Microsoft Fabric Data Pipelines (activities, triggers, parameters) and Notebooks (PySpark/Spark SQL).
• Deep working knowledge of Delta Lake, including schema evolution, MERGE/UPSERT, OPTIMIZE/VACUUM, partitioning, and checkpointing.
• Proficiency in CDC ingestion patterns (watermarks, change tables, log-based ingestion).
• Strong SQL skills (window functions, CTEs) and experience with PySpark for data transformations and performance tuning.
• Experience with data quality implementation: rule patterns (null, range, referential, uniqueness), scorecards, and exception handling.
• Advanced error handling & observability: idempotency, retries, dead-letter handling, and alerting hooks.
Nice to Have Skills:
• Experience with additional ETL tools (e.g., SSIS) and data warehousing solutions.
• Understanding of data governance, data privacy, and compliance standards.
• Familiarity with programming languages such as Python or R for advanced data manipulation.
• Experience with stored procedures in Fabric transformations.
• Knowledge of metadata-driven pipelines and parameterization (YAML/JSON controlled).
• Awareness of Power BI model readiness (star schema, aggregations).
Other Requirements
Required Qualifications:
• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
• 7+ years of hands-on experience in ETL development, specifically utilizing Microsoft Fabric in recent projects.
• Solid understanding of Business Intelligence concepts and hands-on experience in Power BI for data visualization and reporting.
• Strong grasp of data warehousing concepts, architectures, and modeling techniques.
• Familiarity with major cloud platforms (Azure, AWS, Google Cloud) and their data storage and processing services.
• Proficiency in SQL and experience with relational databases such as SQL Server or Oracle.
• Knowledge of data modeling, normalization, and denormalization.
• Excellent analytical, troubleshooting, and problem-solving skills.
• Strong communication skills, capable of conveying complex technical concepts to non-technical audiences.