Job Summary
Location: Hybrid / Global Platform: Databricks on AWS/GCP Role Summary Lead cloud data migration and build governed, enterprise-scale Lakehouse solutions for analytics and AI. Key Skills Strong expertise in Databricks (Spark, PySpark, Spark SQL) Experience deploying Databricks on AWS/GCP Unity Catalog (governance, lineage, access control) Apache Iceberg (schema evolution, optimization) Strong experience in large-scale data migration Responsibilities Lead migration to Databricks Lakehouse architecture Design and implement data governance frameworks Build and optimize large-scale Iceberg tables Integrate Databricks with cloud storage and downstream tools Ensure compliance in regulated environments Ideal Background 10+ years in data engineering Experience in financial services / regulated domains Proven experience in cloud migration programs One-Line Summa
Key Responsibilities
1. Design, develop, and implement data pipelines using azure data factory (adf) to move data between various sources and data warehouses.
2. Utilize azure databricks for data transformations and analytics to derive valuable insights from data.
3. Write and optimize sql queries to extract, manipulate, and analyze data from multiple sources.
4. Develop and maintain python scripts for data processing and automation tasks.
5. Collaborate with cross functional teams to understand data requirements and ensure data solutions meet business needs.
6. Monitor data pipeline performance and troubleshoot issues to ensure data accuracy and reliability.
7. Stay updated on industry trends and best practices related to azure data factory, azure databricks, sql, and python.
Skill Requirements
2. Experience working with azure databricks for data engineering and analytics.
3. Strong sql skills for data querying and manipulation.
4. Proficient in python programming for data processing and automation.
5. Strong analytical and problem-solving skills with a high attention to detail.
6. Excellent communication and collaboration abilities to work effectively in a team environment.
7. Ability to adapt to changing priorities and handle multiple tasks simultaneously.