Job Summary
|
Reports to |
Programme Tech Lead |
|
Capabilities covered |
Standardized Data Platform Architecture; Data Toolkit Design; Data Observability, Data Validation, Data Contracts, Data Transformation, Platform Observability modules; Common CLI/SDK; Synthetic Data Generator |
|
Location |
Prudential VCC Bangalore (preferred) |
Role summary
Own the common Azure Databricks and GCP architecture and the data toolkit architecture (modules, SDK, observability) and guide its deployment across LBUs.
Key Responsibilities
- Maintain the standardised cross-cloud architecture covering ingestion, processing, consumption and BI serving layers; no redesign without an approved gap.
- Maintain toolkit design (e.g. dbt, Airflow) and the common SDK architecture integrating observability, validation, transformation and contract modules.
- Guide integration of Synthetic Data Generator on UDP and deployment in PACS.
- Define versioning, backward-compatibility and release approach for the CLI/SDK and toolkit modules.
- Review LBU-specific rollout designs for conformance; document approved deviations.
- Mentor Data Engineers and platform engineers on patterns and quality gates.
Skill Requirements
Must-have skills and experience
- 7–10 years in data architecture with hands-on delivery on both Azure Databricks and GCP (BigQuery).
- Experience designing reusable engineering frameworks (SDKs, CLIs, modules, data contracts, DQ/observability).
- Strong knowledge of dbt, Airflow or equivalent orchestration and transformation tooling.
- Strong Python and SQL; solid understanding of SCD patterns and DDL generation.
Good to have
- Experience with data observability tooling and metrics APIs.
- Experience with synthetic data platforms for privacy-safe testing.
Certifications (within 90 days of assignment)
- Databricks Data Engineer Professional
- GCP Professional Data Engineer
- Azure Data Engineer / Solutions Architect
- HashiCorp Terraform Associate