Job Summary
Design and deliver end‑to‑end data pipelines on cloud platforms (GCP preferred).Build scalable data ingestion, transformation, and processing workflows using distributed technologies such as Spark, Flink, Storm, or similar.Develop robust ELT/ETL processing and migration pipelines, including support for legacy Datastage decommissioning and modernisation.Work with a variety of database technologies including relational, NoSQL, MPP and columnar stores (BigQuery, Redshift, Azure SQLDW, HBase, MongoDB).Implement streaming and messaging‑based pipelines using Kafka, Pulsar or Pub/Sub.Build optimised, scalable data models to support diverse consumption patterns, applying partitioning, sharding, bucketing and aggregation strategies.Apply performance tuning and optimisation across storage, compute and query layers.
Key Responsibilities
Design and deliver end‑to‑end data pipelines on cloud platforms (GCP preferred).Build scalable data ingestion, transformation, and processing workflows using distributed technologies such as Spark, Flink, Storm, or similar.Develop robust ELT/ETL processing and migration pipelines, including support for legacy Datastage decommissioning and modernisation.Work with a variety of database technologies including relational, NoSQL, MPP and columnar stores (BigQuery, Redshift, Azure SQLDW, HBase, MongoDB).Implement streaming and messaging‑based pipelines using Kafka, Pulsar or Pub/Sub.Build optimised, scalable data models to support diverse consumption patterns, applying partitioning, sharding, bucketing and aggregation strategies.Apply performance tuning and optimisation across storage, compute and query layers.