Job Summary
Job Description – GCP Data Engineer
Position
GCP Data Engineer – 6+ Years Experience
Role Overview
We are looking for an experienced GCP Data Engineer with 6+ years of experience in designing, developing, implementing, and supporting scalable data engineering solutions on Google Cloud Platform. The candidate should have strong hands-on experience with BigQuery, GCS, Dataflow, Cloud Composer/Airflow, Pub/Sub, Cloud Functions/Cloud Run, and Python/SQL.
The role involves developing and supporting batch and real-time data pipelines, data ingestion, transformation, monitoring, troubleshooting, and performance optimization.
Key Responsibilities
ob Description – GCP Data Engineer Position GCP Data Engineer – 6+ Years Experience Role Overview We are looking for an experienced GCP Data Engineer with 6+ years of experience in designing, developing, implementing, and supporting scalable data engineering solutions on Google Cloud Platform. The candidate should have strong hands-on experience with BigQuery, GCS, Dataflow, Cloud Composer/Airflow, Pub/Sub, Cloud Functions/Cloud Run, and Python/SQL. The role involves developing and supporting batch and real-time data pipelines, data ingestion, transformation, monitoring, troubleshooting, and performance optimization. Key Responsibilities Design, develop, and maintain scalable data pipelines on GCP. Develop batch and streaming pipelines using Google Cloud Dataflow / Apache Beam. Develop complex SQL queries, stored procedures, views, and data transformation logic in BigQuery. Work with Cloud Storage (GCS) for data ingestion, staging, archival, and file-based processing. Develop and manage workflows using Cloud Composer / Apache Airflow. Work with Pub/Sub for event-driven and real-time data processing. Develop and support Cloud Functions and Cloud Run based integrations and services. Perform data ingestion from databases, APIs, files, and other source systems into GCP. Implement data quality, validation, reconciliation, and error-handling mechanisms. Monitor production pipelines and troubleshoot failures, performance issues, and data discrepancies. Optimize BigQuery queries, partitioning, clustering, and storage/compute usage. Implement appropriate logging, alerting, and operational monitoring for data pipelines. Participate in production deployment, incident management, root-cause analysis, and problem management. Collaborate with application, infrastructure, cloud, database, and business teams. Follow CI/CD, source-control, coding, security, and data governance standards. Create technical documentation, support procedures, and operational runbooks. Required Technical Skills Technology Expected Experience GCP 4+ years BigQuery 4+ years Dataflow / Apache Beam 3+ years Python 4+ years SQL 5+ years Cloud Storage (GCS) 4+ years Cloud Composer / Airflow 2+ years Pub/Sub 2+ years Cloud Functions / Cloud Run 2+ years Git 3+ years Linux/Unix Good working knowledge Preferred Skills Experience with Cloud Scheduler, Secret Manager, IAM and service accounts. Experience with Dataproc / Spark. Experience with Terraform or Infrastructure as Code. Experience with Cloud Logging and Cloud Monitoring. Knowledge of Docker and CI/CD pipelines. Experience with CDC and real-time data technologies such as Striim, Kafka, or Debezium. Experience working with Oracle, SQL Server, MySQL, MongoDB, or PostgreSQL. Knowledge of data warehouse and dimensional modeling concepts. Understanding of data governance, security, encryption, and PII handling. Healthcare data experience is a plus. Production Support Responsibilities The candidate should be comfortable providing L2 production support, including: Monitoring GCP data pipelines and applications. Troubleshooting failed Dataflow jobs and Airflow DAGs. Investigating BigQuery load and transformation failures. Troubleshooting Pub/Sub message processing issues. Analyzing application and GCP logs. Performing data reconciliation and identifying data gaps/duplicates. Managing production incidents and participating in RCA. Coordinating with development and cloud infrastructure teams for complex issues. Supporting planned deployments, maintenance activities, and DR exercises. Required Soft Skills Strong analytical and problem-solving skills. Good communication and stakeholder-management skills. Ability to work independently in a production environment. Ability to troubleshoot issues under time constraints. Strong documentation and knowledge-sharing skills. Ability to collaborate with offshore/onshore teams. Good underst understanding of ITIL/production-support processes. Experience 6+ years of overall experience in d
Skill Requirements
Required Technical Skills
|
Technology |
Expected Experience |
|
GCP |
4+ years |
|
BigQuery |
4+ years |
|
Dataflow / Apache Beam |
3+ years |
|
Python |
4+ years |
|
SQL |
5+ years |
|
Cloud Storage (GCS) |
4+ years |
|
Cloud Composer / Airflow |
2+ years |
|
Pub/Sub |
2+ years |
|
Cloud Functions / Cloud Run |
2+ years |
|
Git |
3+ years |
|
Linux/Unix |
Good working knowledge |
Preferred Skills
- Experience with Cloud Scheduler, Secret Manager, IAM and service accounts.
- Experience with Dataproc / Spark.
- Experience with Terraform or Infrastructure as Code.
- Experience with Cloud Logging and Cloud Monitoring.
- Knowledge of Docker and CI/CD pipelines.
- Experience with CDC and real-time data technologies such as Striim, Kafka, or Debezium.
- Experience working with Oracle, SQL Server, MySQL, MongoDB, or PostgreSQL.
- Knowledge of data warehouse and dimensional modeling concepts.
- Understanding of data governance, security, encryption, and PII handling.
-
Production Support Responsibilities
The candidate should be comfortable providing L2 production support, including:
- Monitoring GCP data pipelines and applications.
- Troubleshooting failed Dataflow jobs and Airflow DAGs.
- Investigating BigQuery load and transformation failures.
- Troubleshooting Pub/Sub message processing issues.
- Analyzing application and GCP logs.
- Performing data reconciliation and identifying data gaps/duplicates.
- Managing production incidents and participating in RCA.
- Coordinating with development and cloud infrastructure teams for complex issues.
- Supporting planned deployments, maintenance activities, and DR exercises.
Required Soft Skills
- Strong analytical and problem-solving skills.
- Good communication and stakeholder-management skills.
- Ability to work independently in a production environment.
- Ability to troubleshoot issues under time constraints.
- Strong documentation and knowledge-sharing skills.
- Ability to collaborate with offshore/onshore teams.
- Good understanding of ITIL/production-support processes.
Experience
6+ years of overall experience in data engineering, with significant hands-on experience designing, developing, and supporting data solutions on Google Cloud Platform.
Other Requirements
Production Support Responsibilities
The candidate should be comfortable providing L2 production support, including:
- Monitoring GCP data pipelines and applications.
- Troubleshooting failed Dataflow jobs and Airflow DAGs.
- Investigating BigQuery load and transformation failures.
- Troubleshooting Pub/Sub message processing issues.
- Analyzing application and GCP logs.
- Performing data reconciliation and identifying data gaps/duplicates.
- Managing production incidents and participating in RCA.
- Coordinating with development and cloud infrastructure teams for complex issues.
- Supporting planned deployments, maintenance activities, and DR exercises.
Required Soft Skills
- Strong analytical and problem-solving skills.
- Good communication and stakeholder-management skills.
- Ability to work independently in a production environment.
- Ability to troubleshoot issues under time constraints.
- Strong documentation and knowledge-sharing skills.
- Ability to collaborate with offshore/onshore teams.
- Good understanding of ITIL/production-support processes.
Experience
6+ years of overall experience in data engineering, with significant hands-on experience designing, developing, and supporting data solutions on Google Cloud Platform.