Job Summary

The Databricks Data Arch Lead is responsible for providing architectural leadership in designing and implementing enterprise-scale data solutions using Databricks, Apache Spark, and Python. This role drives the adoption of modern data engineering practices, ensures alignment with industry standards, and shapes the technical vision for advanced analytics platforms, directly impacting the organization’s ability to deliver scalable, high-quality data solutions.

Key Responsibilities

1. Architect And Design Advanced Data Processing Solutions Using Databricks, Apache Spark, And Python, Ensuring Scalability, Performance, And Alignment With Business Objectives.
2. Define And Enforce Architectural Standards And Governance For Databricks-Based Data Platforms, Ensuring Compliance With Industry Best Practices And Organizational Policies.
3. Lead Requirement Analysis And Solution Blueprinting By Leveraging Deep Expertise In Databricks And Spark To Translate Complex Business Needs Into Robust Technical Architectures.
4. Guide The Implementation Of Data Pipelines And Workflows On Databricks, Optimizing For Reliability, Security, And Cost-Effectiveness.
5. Mentor And Develop Team Members In Databricks, Spark, And Python Technologies, Fostering A Culture Of Continuous Learning And Technical Excellence.
6. Review And Validate Architectural Deliverables, Providing Expert Feedback To Ensure Solutions Are Innovative, Maintainable, And Future-Proof.
7. Drive Knowledge Sharing By Submitting Whitepapers, Participating In Industry Forums, And Contributing To Intellectual Property Initiatives Related To Databricks And Big Data Architectures.

Skill Requirements

1. Expert Proficiency In Databricks Platform Architecture, Including Cluster Management, Data Lake Integration, And Security Frameworks.
2. Excellent Knowledge Of Apache Spark For LargeScale Data Processing And Advanced Analytics.
3. Advanced Proficiency In Python For Data Engineering And Scripting Within Databricks Environments.
4. Strong Understanding Of Data Modeling, Etl Pipeline Design, And Performance Optimization In CloudBased Data Platforms.
5. Solid Experience With Governance, Compliance, And Best Practices In Enterprise Data Solutions.
6. Architectural Leadership In Designing And Scaling Complex Data Solutions.

Other Requirements

1. Databricks Certified Data Engineer Professional (Optional But Valuable)
2. Apache Spark Developer Certification (Optional But Valuable)
3. Microsoft Azure Solutions Architect Expert Or Aws Certified Solutions Architect (Optional But Valuable, Based On Cloud Platform Used