Job Summary
Data Engineer – Data Migration & Modern Data Platform
Role Overview
We are looking for an experienced Data Engineer to support a large-scale data migration and modern data platform transformation program.
The role will initially have a strong focus on the Discovery and Assessment phase, where the Data Engineer will work with business stakeholders, application teams, database administrators, architects, and subject matter experts to understand the existing data landscape.
The successful candidate will be responsible for profiling source databases, assessing data quality and complexity, conducting stakeholder interviews and technical discovery workshops, documenting source-to-target requirements, and gathering the information required to develop a robust data migration strategy and execution plan.
Following discovery, the Data Engineer will become an integral member of the migration delivery team, responsible for designing, developing, testing, and executing data migration pipelines.
The role will also contribute to the development of the organization's modern data platform, including implementation of data ingestion, transformation, data-quality, and consumption layers based on a Medallion Architecture (Bronze, Silver, Gold).
Key Responsibilities
Key Responsibilities
1. Data Discovery & Migration Assessment
-
Perform detailed discovery and assessment of existing databases, applications, data stores, interfaces, and data flows.
-
Profile source databases to understand:
-
Data volumes and growth patterns
-
Table structures and relationships
-
Primary and foreign keys
-
Data types and formats
-
Null values and completeness
-
Duplicate records
-
Data-quality issues
-
Referential integrity
-
Historical data requirements
-
Archival and retention considerations
-
Develop automated or reusable data-profiling approaches where appropriate.
-
Identify data anomalies, inconsistencies, migration risks, and remediation requirements.
-
Analyze dependencies between applications, databases, interfaces, downstream systems, and reporting platforms.
-
Assess the complexity and migration readiness of source datasets.
Stakeholder & Requirements Discovery
-
Conduct interviews and workshops with business users, application owners, SMEs, database administrators, architects, and technical teams.
-
Understand the business meaning, usage, ownership, and criticality of data.
-
Capture data retention, reconciliation, compliance, security, and historical data requirements.
-
Identify business rules and transformation logic currently embedded within applications, databases, ETL processes, reports, or operational procedures.
-
Translate business and technical discussions into clearly documented data requirements.
-
Support identification of authoritative data sources and systems of record.
Data Migration Planning
-
Produce technical inputs required for migration planning
-
Work with project leads to define migration waves and migration strategies.
-
Identify datasets requiring cleansing, enrichment, transformation, consolidation, or archival prior to migration.
Data Migration Engineering
-
Design, build, test, and execute scalable data migration pipelines.
Skill Requirements
Data Engineering
-
Strong experience as a Data Engineer working with complex enterprise datasets.
-
Strong SQL skills with the ability to analyze and profile large relational databases.
-
Experience developing ETL/ELT and data integration pipelines.
-
Experience working with relational databases such as SQL Server, Oracle, PostgreSQL, MySQL, or equivalent.
-
Experience working with large-volume data extraction, transformation, and loading.
-
Strong understanding of data modeling, relational data structures, and database design.
-
Experience developing data-quality and data-validation frameworks.
-
Experience with Python, PySpark, SQL