Job Summary

Hands-on build (the core of the role)

• Design and build data pipelines in Microsoft Fabric — Data Factory pipelines, Copy activities, and Spark notebooks (PySpark) — across a bronze/silver/gold medallion architecture.
• Write production-grade PySpark and SQL for ingestion, transformation, cleansing, and star-schema modelling in Delta Lake / OneLake.
• Build and optimise semantic models (Direct Lake), including relationships, DAX measures, and row-level security, tuned for performance against the gold layer.
• Design and implement metadata-driven ingestion frameworks — config-driven, parameterised pipelines that scale across many sources rather than one pipeline per table.
• Implement robust handling for incremental loading, schema drift/evolution, and data quality, including CDC/watermark/hash-based patterns as appropriate per source.
• Integrate and configure governance via Microsoft Purview — lineage, cataloguing, classification — as a first-class part of the build, not an afterthought.

Technical leadership

• Own the technical architecture and set the design patterns, coding standards, and best practices the team builds against.
• Lead code reviews, establish CI/CD for Fabric (Git integration, deployment pipelines, environment/workspace strategy), and enforce version control discipline.
• Make and document the key build-vs-buy and architectural trade-off decisions (lakehouse vs. warehouse, batch vs. streaming, load strategies).
• Partner with data governance, infrastructure, and security on workspace design, access, networking (e.g. Private Endpoints), and capacity/cost management.

Delivery & mentorship

• Translate business and analytical requirements into a scoped, sequenced technical roadmap.
• Mentor and grow data engineers — pairing, design guidance, and raising the team's Fabric capability.
• Manage delivery within a sprint/Agile cadence (Azure DevOps), balancing hands-on contribution with unblocking the team.
• Communicate status, risks, and trade-offs clearly to technical and non-technical stakeholders.

Key Responsibilities

• Strong hands-on experience building data platforms on Microsoft Fabric (or very recent Azure Synapse / Databricks + Power BI experience directly transferable to Fabric).
• Expert PySpark and strong SQL; production experience with Delta Lake and the medallion architecture.
• Proven experience designing and building data pipelines — both orchestration (Data Factory / pipelines) and transformation (Spark notebooks) — and knowing when to use which.
• Hands-on semantic modelling and DAX, ideally with Direct Lake, including performance optimisation.
• Solid grasp of data warehousing and dimensional modelling (star schema, SCD, surrogate keys).
• Experience with incremental load patterns, schema drift handling, and data quality frameworks.
• CI/CD and Git for data platforms; workspace/environment management.
• A track record of technical leadership — setting standards, reviewing code, mentoring — while remaining hands-on.

Skill Requirements

• Python, PySpark, Meddalion Architect, Microsoft Fabric, Data Pipeline, Data Engg.
• Azure ecosystem (ADLS, Key Vault, networking/Private Endpoints, Azure DevOps).
• Real-Time Intelligence / streaming (Eventstream, KQL) exposure.
• Domain experience in [your industry — e.g. financial services / BFSI].
• Relevant certification (e.g. DP-600 / DP-700 Fabric, or Azure Data Engineer).

Other Requirements