Job Summary
To be responsible for managing technology in complex projects ,providing technical guidance and ensuring successful delivery of solutions.
Data Engineer Job Description
We are seeking a skilled and motivated Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, data architecture, and data integration. This role involves building and maintaining scalable data pipelines, ensuring data quality and integrity, and supporting data analytics initiatives. You will work closely with internal customers, product owners, core team members, and other stakeholders to ensure that quality data is available, reliable, and easily accessible in a cost effective efficient, scalable way.
Key Responsibilities:
•
Data Pipeline Development: Design, build, and maintain robust, scalable, and efficient data pipelines to collect, process, and store large volumes of data from various sources.
•
Data Integration: Integrate data from multiple sources, including APIs, databases, and external datasets, ensuring data consistency and reliability.
•
Data Modelling: Develop and maintain data models and schemas that support efficient data storage, retrieval, and analytics.
•
Database Management: Manage and optimize databases, ensuring their performance, availability, and security
•
Data Quality: Implement and monitor data quality checks to ensure the accuracy, completeness, and consistency of data. Perform analysis required to troubleshoot data related issues and assist in the resolution of data issues.
•
Automation: configure data extraction and load jobs and automate repetitive tasks and processes to improve efficiency and reduce errors. Improve performances of jobs and pipelines by applying optimisation and automation techniques.
•
Documentation: Maintain clear and comprehensive documentation of data pipelines, data models, and data integration processes. Maintenance of time allocation reports.
•
Develop understanding of business/ processes and high-level understanding of high-quality digital product delivery
•
Ways of working: Follow Agile and SDLC processes including the creation of data related user stories. Perform code peer reviews and testing as per SDLC or industry standards.
•
Collaboration: Work closely with internal customers, product owners, various core team members, and other stakeholders to understand their data needs and provide appropriate solutions. Technical collaboration and oversight with Vendors as required. Create SOWs and work packets with vendors as required.
Experience and skills:
· Over 6 years’ experience as a Data Engineer or in a similar role.
· Experience with data pipeline, ETL and workflow management tools (e.g., Databricks, Data Factory).
· Proficiency in programming languages such as SQL, Python, R, or Scala.
· Strong experience with SQL and database management (e.g., MySQL, PostgreSQL, SQL Server).
· Excellent problem-solving and analytical skills.
Key Responsibilities
2. To conduct comprehensive code reviews, establish and oversee quality assurance processes, performance optimization , implementation of best practices and coding standards to ensure successful delivery of complex projects.
3. To ensure process compliance in the assigned module| and participate in technical discussions/review as a technical consultant for feasibility study (technical alternatives, best packages, supporting architecture best practices, technical risks, breakdown into components, estimations).
4. To collaborate with stakeholders to define project scope, objectives, deliverables and accordingly prepare and submit status reports for minimizing exposure & closure of escalations.