Job Summary
Must be a UK National with minimum 5 years residency **
Mandatory:
Building integrations against source-system APIs (webhooks, REST, event streams) — connecting CDS estate systems to the platform
Event-driven ingestion patterns — API Gateway/Lambda receivers, EventBridge, SQS per-consumer buffering
State/watermark tracking (DynamoDB) and reconciliation to catch delivery gaps
Python for pipeline/integration development
Data migration experience (e.g. AWS DMS) for onboarding new sources"
"The Data Engineer is responsible for integrating CDS source systems into the AI platform — building the connectors and ingestion pipelines that get source-system data flowing into the platform reliably.
Key Responsibilities:
Build source-system integrations (webhooks → API Gateway/Lambda → EventBridge → SQS)
Implement watermark/state tracking (DynamoDB) and periodic reconciliation to catch webhook delivery gaps without placing polling load on source systems
Onboard new CDS source systems onto the platform's ingestion pattern as scope expands (e.g. Confluence, JIRA, and others across the CDS estate)
Handle data migration activities where a source system needs backfilling into the platform (e.g. AWS DMS)
Work with Knowledge Engineers to ensure ingested data meets the quality/structure ne"
Key Responsibilities
Must be a UK National with minimum 5 years residency **
Mandatory:
Building integrations against source-system APIs (webhooks, REST, event streams) — connecting CDS estate systems to the platform
Event-driven ingestion patterns — API Gateway/Lambda receivers, EventBridge, SQS per-consumer buffering
State/watermark tracking (DynamoDB) and reconciliation to catch delivery gaps
Python for pipeline/integration development
Data migration experience (e.g. AWS DMS) for onboarding new sources"
"The Data Engineer is responsible for integrating CDS source systems into the AI platform — building the connectors and ingestion pipelines that get source-system data flowing into the platform reliably.
Key Responsibilities:
Build source-system integrations (webhooks → API Gateway/Lambda → EventBridge → SQS)
Implement watermark/state tracking (DynamoDB) and periodic reconciliation to catch webhook delivery gaps without placing polling load on source systems
Onboard new CDS source systems onto the platform's ingestion pattern as scope expands (e.g. Confluence, JIRA, and others across the CDS estate)
Handle data migration activities where a source system needs backfilling into the platform (e.g. AWS DMS)
Work with Knowledge Engineers to ensure ingested data meets the quality/structure ne"
Skill Requirements
Must be a UK National with minimum 5 years residency **
Mandatory:
Building integrations against source-system APIs (webhooks, REST, event streams) — connecting CDS estate systems to the platform
Event-driven ingestion patterns — API Gateway/Lambda receivers, EventBridge, SQS per-consumer buffering
State/watermark tracking (DynamoDB) and reconciliation to catch delivery gaps
Python for pipeline/integration development
Data migration experience (e.g. AWS DMS) for onboarding new sources"
"The Data Engineer is responsible for integrating CDS source systems into the AI platform — building the connectors and ingestion pipelines that get source-system data flowing into the platform reliably.
Key Responsibilities:
Build source-system integrations (webhooks → API Gateway/Lambda → EventBridge → SQS)
Implement watermark/state tracking (DynamoDB) and periodic reconciliation to catch webhook delivery gaps without placing polling load on source systems
Onboard new CDS source systems onto the platform's ingestion pattern as scope expands (e.g. Confluence, JIRA, and others across the CDS estate)
Handle data migration activities where a source system needs backfilling into the platform (e.g. AWS DMS)
Work with Knowledge Engineers to ensure ingested data meets the quality/structure ne"
Other Requirements
Remote within UK - Hybrid in office London/Manchester