Job Summary
Job Code
15182
Locations
Bengaluru
Minimum Experience
4
Maximum Experience
8
Mandatory Skills
Pyspark
Skill to Evaluate
Pyspark
Experience
4 to 8 Years
Location
Bengaluru
Job Description
· 4-6 years of experience in data engineering tools and technologies.
· Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure/AWS Databricks
· Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained
· Willing to work in 2nd shift
· Willing to work in Hybrid mode
· Strong knowledge of Data Management principles
· Good Experience in SQL or any native SQL like query languages.
· Experience in ETL / data warehouse transformation processes
· Direct experience of building data pipelines using Apache Spark (preferably Databricks).
· Good to have experience in writing code in python.
· Good to have experience in Power BI.
· Working with other members of the project team to support delivery of additional project components.
· Evaluating the performance and applicability of multiple tools against customer requirements.
· Working within an Agile delivery / DevOps methodology to deliver proof of concept and production implementation in iterative sprints.
· Able to Work Independently
· Good oral and written communication skill
Key Responsibilities
· Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure/AWS Databricks
· Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained
· Support Biz for adhoc issues
Create and maintain optimal data pipeline architecture
Assemble large, complex data sets that meet business requirements
Identify, design, and implement internal process improvements
Optimize data delivery and re-design infrastructure for greater scalability
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS technologies
Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics
Work with internal and external stakeholders to assist with data-related technical issues and support data infrastructure needs
Skill Requirements
Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Azure/AWS Databricks
· Integrating the end-to-end data pipeline to take data from source systems to target data repositories ensuring the quality and consistency of data is always maintained