Job Summary
Key Responsibilities
2. Design and develop efficient and reliable etl processes for large datasets.
3. Collaborate with cross functional teams to understand business requirements and translate them into technical solutions.
4. Optimize data workflows, troubleshoot issues, and ensure data quality and integrity.
5. Implement best practices for data security, governance, and compliance.
6. Provide technical guidance, mentoring, and support to junior team members.
7. Stay uptodate with the latest trends and technologies in data engineering and analytics.
Skill Requirements
1. Proficiency in azure data factory (adf) and azure databricks for data integration and processing.
3. Experience with oracle pl/sql for database development and management.
5. Solid understanding of data warehousing concepts.
- Extensive professional experience building and operating production-grade systems in Python (required).
- Advanced SQL expertise, including query optimization and modeling for very large datasets (required).
- Proven experience designing and delivering large-scale data pipelines or data platform components.
- Strong working experience with Databricks or similar Spark-based data processing platforms.
- Demonstrated ability to own systems in production, including observability, troubleshooting, and incident response.
- Deep understanding of software design principles, data modeling, and engineering best practices.
- Comfortable using LLM-based coding tools responsibly, with full ownership for correctness, testing, and maintainability of resulting code.
- Excellent written and verbal English communication skills, with the ability to clearly explain complex technical topics to varied audiences.
Nice to have:
- Experience with workflow orchestration tools (e.g., Airflow).
- Prior experience working in analytics, economic data, or large-scale data products.