Job Summary
We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way.
Key Responsibilities
Key Responsibilities: • Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient data pipelines to collect, process, and store large volumes of data from various sources. • Data Integration: Integrate data from multiple sources, including APIs, databases, and external datasets, ensuring data consistency and reliability. • Data Modelling: Develop and maintain data models and schemas that support efficient data storage, retrieval, and analytics. • Database Management: Manage and optimize databases, ensuring their performance, availability, and security • Data Quality: Implement and monitor data quality checks to ensure the accuracy, completeness, and consistency of data. Perform analysis required to troubleshoot data related issues and assist in the resolution of data issues. • Automation: configure data extraction and load jobs and automate repetitive tasks and processes to improve efficiency and reduce errors. Improve performances of jobs and pipelines by applying optimisation and automation techniques. • Documentation: Maintain clear and comprehensive documentation of data pipelines, data models, and data integration processes. Maintenance of time allocation reports. • Develop understanding of business/ processes and high-level understanding of high-quality digital product delivery • Ways of working: Follow Agile and SDLC processes including the creation of data related user stories. Perform code peer reviews and testing as per SDLC or industry standards. • Collaboration: Work closely with internal customers, product owners, various core team members, and other stakeholders to understand their data needs and provide appropriate solutions. Technical collaboration and oversight with Vendors as required. Create SOWs and work packets with vendors as required.
Skill Requirements
Education: · Bachelor’s or Master’s degree in computer science, data science, software engineering, information systems or a similar field or equivalent experience. Experience and skills: · Over 6 years’ experience as a Data Engineer or in a similar role. · Experience with data pipeline, ETL and workflow management tools (e.g., Databricks, Data Factory). · Proficiency in programming languages such as SQL, Python, R, or Scala. · Strong experience with SQL and database management (e.g., MySQL, PostgreSQL, SQL Server). · Excellent problem-solving and analytical skills. · Strong understanding of data architecture, data modelling, and ETL processes. · Ability to work in a fast-paced, dynamic environment and manage multiple tasks simultaneously. · Strong communication and collaboration skills. · Experience with best practice UX / UI design & development of Business Intelligence data visualisation / exploration tools (Power Bi, Qlik etc) · Familiarity with SDLC and Agile ways of working. · Familiarity with DevOps tools and practices (e.g. Jenkins, Azure DevOps, etc). · Familiarity with Atlassian tools like Bitbucket, JIRA, Confluence. Preferred: · GxP and Non GxP SDLC experience · Knowledge of data governance and data security best practices. · MS Power Platform / power apps experience · Cloud DevOps knowledge (e.g. Terraform) · Certifications in cloud data technologies (e.g. Azure Data Fundamentals) · Basic data science experience and knowledge.
Other Requirements
Other: · English language required. · May require travel 10-20% of time based on project locations.