Job Summary

Data Engineering

Coding:

Coding/scripting experience developed in a commercial/industry setting (Python, Java, Scala or Go and SQL).

Databases & frameworks:

Working experience with operational data stores, data warehouse, big data technologies and data lakes.

Experience working with relational and non-relational databases to build data solutions, such as SQL Server/Oracle, experience with relational and dimensional data structures.

Experience in using distributed frameworks (Spark, Flink, Beam, Hadoop)

Strong experience working with Kafka technologies

Containerisation:

Good knowledge of containers (Docker, Kubernetes etc)

Cloud:

Experience with cloud platforms such as GCP, Azure or AWS.

Good understating of cloud storage, networking and resource provisioning

 

Data Ops

Cloud Skills: Advanced hands-on experience with major cloud platforms (Azure, AWS, GCP), including cloud-native data services, orchestration, monitoring, and automation. Proven experience in GCP-based data platforms, including BigQuery, Dataflow, CloudSQL, Spanner, Pub Sub, Cloud composer and GKE.

Data Engineering: Proficient in modern data tools and frameworks for building, operating, and optimising ETL/ELT pipelines, streaming data, and batch processing.

SRE & Automation: Strong understanding of SRE principles, CI/CD, infrastructure as code, and operational automation.

Data Literacy: Ability to design, develop, and apply data and analytics experience to drive effective decision-making and foster a data-driven culture.

Values & Behaviours: Demonstrates innovation, challenges existing processes, adopts new technologies, inspires others, embraces diverse viewpoints, and maintains a growth learning mindset.

 

 

Data Governance & Visualisation: Looker, Ataccama, Dataplex, Collibra, PowerBI

AI/ Analytics: Exposure to TensorFlow, PyTorch, Scikit-learn, OpenCV, LangChain and GenAI tools

Experience of designing & executing complex automation testing strategies and frameworks for Functional and Non-Functional requirements of Data and AI platforms

Create or accurately request complex test data, taking into account referential integrity, data quality and governance

Incorporate automated tests into the CI/CD pipeline and DevOps tooling like Jenkins, Harness, Terraform, Dynatrace etc

Strong programming proficiency in Python(preferable), Java or JavaScript

Experience with Agile tools (Jira, Confluence, Xray)

Experience on Source code management tools (Git, GitHub)

 

 

Key Responsibilities

1. Design, develop, and maintain google bigquery data pipelines for efficient data processing.
2. Utilize sql for querying and analyzing large datasets in google bigquery.
3. Develop and implement python scripts for automation and data manipulation tasks.
4. Create and optimize data processing pipelines using google dataflow.
5. Collaborate with stakeholders to understand data modeling requirements and utilize ca erwin data modeler for effective data modeling.
6. Monitor data quality, troubleshoot issues, and ensure data integrity across systems.
7. Stay updated with the latest technological advancements in data analytics and contribute to process improvements.

Skill Requirements

1. Proficiency in google bigquery for data warehousing and processing.
2. Strong command over sql for querying and analyzing datasets.
3. Advanced knowledge of python programming for data manipulation and automation.
4. Experience with google dataflow for building and optimizing data pipelines.
5. Familiarity with ca erwin data modeler for effective data modeling and database design.
6. Strong analytical and problem-solving skills to handle complex data related challenges.
7. Excellent communication skills to collaborate with cross functional teams and stakeholders effectively.
8. Ability to work in a fast paced environment and manage multiple priorities simultaneously.

Other Requirements

1.Relevant certifications in Google BigQuery, SQL, Python, Google Dataflow, or CA Erwin Data Modeler would be a plus.