Job Summary
Data Engineers with engineering skills to build pipelines within the analytics platforms, moving ingested data from raw to modelled and ultimately to dimensional models that can be accessed by BI tools or ML models. Primary duties include:
Build data pipelines for complex sized Data and Analytics solutions (e.g. Data Lake, Data Warehouse, Business Intelligence Dashboards) in a team. Translate functional and non-functional data and analytics requirements into fit for purpose technical design. Ensure solution performance, business edge cases and security related issues are addressed while developing software.
Debug issues of complexity, resolve blockers and follow design documents with minimal supervision. This includes Production incidents.
Complete data engineering coding tasks on problems of moderate to high scope and complexity. Demonstrate good coding principles. Conduct code review for peers. Ensure solutions adhere to published data privacy and cybersecurity principles.
Work with senior and lead data engineers in the technical design process by contributing to the analysis of data application, data integration, large data storage or data pipeline requirements.
Update data inventories and registries as required to keep metadata and data lineage up to date, following agreed Data Governance standards, guidelines and principles.
Carry out unit testing independently. Troubleshoot issues, and fix defects that are of moderate to high complexity.
Shadow lead data engineers on design and architecture components and collaborate with members of the cross-functional team to identify areas of inefficiency and propose solutions.
Adhere to published coding standards, guidelines and best practices and contribute to Data Engineering Playbooks and other data technology blueprints.
Required technical skills: Microsoft Azure and Cloudera technology stacks, Hadoop, Spark, Scala, Hive, H-Base, Sqoop, Oozie, Apache Nifi, Airflow, HDFS, ADLS (Gen 2), Azure Data Factory (ADF), DataBricks, Kafka, Elasticsearch, AVRO / PARQUET file formats, Streaming (e.g. Spark Streaming), SnapLogic, TIBCO, Kafka, Snowflake, SQL, MicroStrategy, Power BI, GIT Bitbucket, Azure DevOps. Note - stronger Data Engineers will have some expertise with all these tools, but even familiarity with some is good to build parts of the pipeline (e.g. Data Engineers with Snowflake/Power BI skills can focus on DW/BI portions, while those with Spark/Scala skills can focus on data ingestion and data product transformations).
Team player: good attitude and willingness to learn and adapt.
Familiar with Agile and Waterfall methodologies.
>8 years of focused experience on end-to-end data engineering.
Key Responsibilities
2. Design, develop, and maintain data pipelines using snowflake, azure data factory (adf), and data bricks to support business requirements.
3. Work closely with stakeholders to gather requirements, identify opportunities for data analytics, and propose data driven solutions.
4. Monitor and troubleshoot data pipelines, ensuring data quality, reliability, and performance.
5. Stay updated with the latest trends and technologies in data management and analytics, incorporating best practices into projects.
6. Collaborate with cross functional teams to integrate data solutions with existing systems and applications.
7. Provide technical expertise and mentorship to team members, promoting a culture of learning and innovation.
Skill Requirements
2. Strong experience in azure data factory (adf) for data integration and orchestration.
3. Handson knowledge of data bricks for data engineering, data processing, and machine learning.
4. Ability to design, develop, and optimize complex data pipelines for etl processes.
5. Strong problem-solving skills and the ability to troubleshoot data pipeline issues.
6. Excellent communication skills to interact with technical and nontechnical stakeholders effectively.
7. Strong leadership skills to guide and motivate technical teams towards project delivery and success.
8. Experience in agile methodologies and project management practices for efficient project execution.