Job Summary
Databricks Data Engineer responsible for developing and maintaining scalable data pipelines that support the RCIS Crop Insurance Data Modernization program. Design and implement metadata-driven ingestion, CDC, transformation, and reconciliation frameworks to integrate Guidewire CDA, CIMax/DB2, reporting databases, and legacy systems into a unified Silver Layer data foundation. Build trusted Policy, Claims, Billing, APH Yield, Acreage, Premium, Subsidy, and Indemnity data products that establish a Single Source of Truth (SSOT). Collaborate with Data Architects, Data Analysts, Guidewire SMEs, and BI teams to deliver governed, audit-ready, high-quality datasets supporting regulatory reporting, analytics, dashboards, and future AI-driven capabilities.
Key Responsibilities
Develop and optimize Databricks, PySpark, and Delta Lake pipelines for batch and CDC-based ingestion from Guidewire CDA, CIMax, DB2, and external crop insurance data sources. Build Bronze and Silver layer data products, implement source-to-target mappings, and perform data transformations aligned to canonical Policy and Claims models. Embed data quality, reconciliation, lineage, observability, and audit controls within all pipelines. Support data profiling, testing, SIT/UAT validation, and report reconciliation activities. Develop automated monitoring, alerting, and performance optimization solutions. Collaborate with business SMEs and architects to ensure accurate, trusted, regulatory-compliant data supporting RCIS reporting and analytics initiatives.
Skill Requirements
Top 6 Key Skillsets – Databricks Data Engineer
1. Databricks & Spark Development
Azure/AWS Databricks
PySpark
Spark SQL
Delta Lake
Delta Live Tables (DLT)
Databricks Workflows
2. Data Engineering & ETL/ELT
Metadata-Driven Ingestion
CDC (Change Data Capture)
Batch & Incremental Processing
Data Transformation Frameworks
Source-to-Target Mapping (STTM)
3. Data Modeling & Lakehouse Architecture
Bronze/Silver Layer Development
Canonical Data Models
Medallion Architecture
Data Product Development
SSOT Implementation
4. Data Quality & Reconciliation
Data Profiling
Data Validation
Data Reconciliation
Audit Controls
Data Observability
Lineage & Traceability
5. Cloud & DevOps
AWS / Azure
ADF / AWS Glue
Git
CI/CD Pipelines
Airflow
Infrastructure Automation
6. Crop Insurance & Guidewire Data Expertise
Guidewire CDA (PC, CC, BC)
CIMax / DB2
Policy & Claims Data
APH Yield & Acreage Reporting
Premium, Subsidy & Indemnity Data
RMA Regulatory Reporting
Resume-Friendly Skill Summary
Databricks, PySpark, Delta Lake, ETL/ELT Development, Guidewire CDA Integration, Data Quality & Reconciliation, CDC Pipelines, Medallion Architecture, Data Governance, AWS/Azure Cloud, Crop Insurance Data Domains, Regulatory Reporting Support.