Job Summary
We are seeking a highly skilled Data Integration Engineer (Level 2) to design, build, and manage enterprise-scale data integration solutions on Azure. This role requires strong expertise in cloud-native data services, integration patterns, DevOps practices, and L2 production support to ensure reliability, performance, and scalability of data platforms.
Key Responsibilities
-
Design, develop, and maintain end-to-end data integration solutions across enterprise systems.
-
Build scalable ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Data Flows.
-
Develop high-performance data processing solutions using Azure Databricks (ADB) and Python.
-
Implement event-driven architectures using Azure Service Bus.
-
Develop and manage serverless applications using Azure Functions.
-
Design and expose APIs using Azure API Management (APIM).
-
Build and maintain NoSQL data solutions using Cosmos DB.
-
Manage secure access to secrets and credentials using Azure Key Vault.
-
Monitor and troubleshoot applications using:
-
Azure Application Insights
-
Log Analytics Workspace
-
Deploy and manage containerized applications using Azure Kubernetes Service (AKS).
-
Implement CI/CD pipelines using Azure DevOps and GitOps practices via ArgoCD.
-
Automate infrastructure provisioning and configuration using Terraform.
-
Ensure effective use of Azure services for integration (eventing, messaging, API, compute).
-
Provide Level 2 (L2) operational support, including:
-
Incident management and resolution
-
Root cause analysis (RCA)
-
Performance optimization and system tuning
-
Collaborate with architects, data engineers, and business stakeholders to deliver robust solutions.
-
Implement best practices for security, monitoring, governance, and reliability.
Skill Requirements
-
Strong experience in Data Integration / Data Engineering.
-
Hands-on expertise in:
-
Azure Data Factory (ADF), Azure Databricks
-
Confluent Kafka, Azure Event Hubs, Azure Service Bus Explorer
-
Real-time data processing, Event-Driven Architecture, CDC, Micro-batch & Continuous Streaming
-
Cosmos DB, Azure Blob Storage, Azure SQL, SQL
-
PySpark, Python
-
CI/CD Pipelines, Azure DevOps, Git, ETL Deployments
-
Azure Monitor, Application Insights, Log Analytics, Alerts creation, Metrics analysis
-
Azure Kubernetes Service (AKS)
-
Control-M, Postman, Datadog
-
Experience with:
-
Azure Functions
-
Azure Service Bus
-
Azure API Management (APIM)
-
Cosmos DB
-
Monitoring and observability tools:
-
Azure Application Insights
-
Log Analytics Workspace
-
Security:
-
Azure Key Vault
-
Containerization and orchestration:
-
Azure Kubernetes Service (AKS)
-
DevOps & deployment:
-
Azure DevOps
-
ArgoCD
-
Infrastructure as Code:
-
Terraform
-
Strong understanding of Azure services related to integration (messaging, APIs, serverless, data pipelines).
-
Experience in performance tuning and cost optimization on Azure.
-
Exposure to API-led and microservices-based architectures.
-
Proven operations/support (L2) experience in production environments.
-
Good understanding of:
-
Distributed systems
-
Data modelling & data warehousing
-
Cloud architecture and scalability patterns
Other Requirements
-
Experience with legacy or enterprise ETL tools:
-
IBM DataStage
-
Advanced monitoring/observability tools:
-
Datadog
-
Splunk
-
Knowledge of hybrid integration patterns.
Qualifications
-
Bachelor’s or Master’s degree in Computer Science, IT, or related field.
-
Total integration experience of 12+ years with 6 to 8+ years of relevant experience in data integration, data engineering, or related roles.
-
Azure certifications (preferred):
-
Azure Data Engineer Associate
-
Azure DevOps Engineer Expert