Job Summary

To design, build, and maintain scalable, reliable, and secure data pipelines that enable seamless integration of diverse global data sources for Generative AI solutions. The Data Engineer ensures high-quality, timely, and accessible data for AI models by managing RAG indexes, monitoring pipeline health, and continuously onboarding new data sources. This mission supports the organisation’s goal of delivering intelligent, data-driven experiences across enterprise applications.

Key Responsibilities

Key Responsibilities:

• Design, implement, and maintain robust data pipelines and ETL processes to support business and product requirements.
• Monitor the performance and reliability of data pipelines, proactively addressing issues and optimizing workflows.
• Connect and integrate diverse data sources (structured and unstructured) across global systems.
• Identify and integrate new potential data sources to enhance the breadth and quality of available data.
• Build and maintain Retrieval-Augmented Generation (RAG) indexes for efficient data retrieval and model performance.
• Optimise indexing strategies for large-scale, multi-domain datasets.
• Ensure data quality, consistency, and compliance with enterprise standards.
• Implement monitoring and alerting for data pipeline health and SLA adherence.
• Work closely with AI/ML engineers, data scientists, and product teams to deliver end-to-end solutions.
• Collaborate with cross-functional teams to align data engineering efforts with project roadmaps and timelines.

Skill Requirements

Qualification and Experience

• Bachelor’s or Master’s degree in computer science engineering with 12+ years of experience
• Proficiency in programming languages: Python, PySpark, and SQL.
• Experience with data engineering tools, including Azure DataBricks, Azure DevOps, and VS Code.
• Strong understanding of working with REST APIs for data integration and automation.
• Hands-on experience with RAG architecture and vector databases
• Familiarity with Generative AI concepts and LLM integration.
• Familiarity with cloud-based data platforms and scalable data architecture.

• Personality requirement:

   

  Ability to adapt to new working environments and technologies is crucial, as are your outstanding written and verbal communication skills. As a team player, you'll maintain transparent communication, providing daily updates within the team and towards stakeholders. Your customer-focused approach, coupled with an adaptable and flexible mindset, will be key in handling the varied work environment. A collaborative, good-natured, and helpful attitude, along with a transparent mindset in all situations, sets the tone for success. Your ability to handle critical and urgent situations effectively and calmly rounds out the qualities we value

Other Requirements