Job Summary
Job Description : Data Engineer
Key Responsibilities
Must-Have Skills
Databricks: Expert Databricks Engineer, Delta Lake, DLT / Lakeflow, Streaming Pipelines, Workflow Orchestration, PySpark
Advanced skills: Structured Streaming, Window Functions, Partitioning, Broadcast joins, Adaptive Query Execution, Performance Tuning
SQL Expert level: Plain Text, Advanced Joins, CTEs, Window Functions, MERGE, Subqueries, Recursive CTEs, Query Optimization, Execution Plans
AWS: S3, Glue, IAM, Lambda, CloudWatch, Kinesis
Data Quality: DQX, Great Expectations, Delta Expectations, Reconciliation Frameworks
AI Data Engineering: Chunking methods, Document ingestion, Embedding pipelines, Vector Search implementation, Metadata enrichment
CICD: Git, Azure DevOps / GitHub Actions, Terraform, Databricks Asset Bundles
Required Skills & Qualifications:
- Bachelor’s or Master’s degree in computer science, Information Systems, or a related field.
- 7-10 years of experience in data governance, data management, or a related field.
- Proven experience in data engineering and data governance roles.
- Proficiency in SQL, Python, and ETL tools. Experience with cloud data platforms. Familiarity with data governance tools.
- Strong understanding of Databricks, data modeling, data warehousing, and data architecture.
- Experience with data quality management, metadata management, and data lineage practices.
- Excellent analytical, problem-solving, and decision-making skills.
- Strong communication and interpersonal skills, with the ability to work effectively with cross-functional teams.
- Ability to manage multiple projects and priorities in a fast-paced environment.
Skill Requirements
Must-Have Skills
Databricks: Expert Databricks Engineer, Delta Lake, DLT / Lakeflow, Streaming Pipelines, Workflow Orchestration, PySpark
Advanced skills: Structured Streaming, Window Functions, Partitioning, Broadcast joins, Adaptive Query Execution, Performance Tuning
SQL Expert level: Plain Text, Advanced Joins, CTEs, Window Functions, MERGE, Subqueries, Recursive CTEs, Query Optimization, Execution Plans
AWS: S3, Glue, IAM, Lambda, CloudWatch, Kinesis
Data Quality: DQX, Great Expectations, Delta Expectations, Reconciliation Frameworks
AI Data Engineering: Chunking methods, Document ingestion, Embedding pipelines, Vector Search implementation, Metadata enrichment
CICD: Git, Azure DevOps / GitHub Actions, Terraform, Databricks Asset Bundles
Required Skills & Qualifications:
- Bachelor’s or Master’s degree in computer science, Information Systems, or a related field.
- 7-10 years of experience in data governance, data management, or a related field.
- Proven experience in data engineering and data governance roles.
- Proficiency in SQL, Python, and ETL tools. Experience with cloud data platforms. Familiarity with data governance tools.
- Strong understanding of Databricks, data modeling, data warehousing, and data architecture.
- Experience with data quality management, metadata management, and data lineage practices.
- Excellent analytical, problem-solving, and decision-making skills.
- Strong communication and interpersonal skills, with the ability to work effectively with cross-functional teams.
- Ability to manage multiple projects and priorities in a fast-paced environment.