Senior Technical Lead
United Kingdom
Apply now
Job Description
Senior Technical Lead
City Of London, England

Job Summary

Role:

  • Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
  • Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
  • Carry out data modelling in Databricks.
  • Expose the Delta tables to the visualisation layer.
  • Ensure that the data pipelines are optimised.

Skillsets:

  • Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
  • Fluent in PySpark and Databricks SQL.
  • Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Key Responsibilities

Role:

  • Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
  • Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
  • Carry out data modelling in Databricks.
  • Expose the Delta tables to the visualisation layer.
  • Ensure that the data pipelines are optimised.

Skillsets:

  • Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
  • Fluent in PySpark and Databricks SQL.
  • Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Skill Requirements

Role:

  • Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
  • Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
  • Carry out data modelling in Databricks.
  • Expose the Delta tables to the visualisation layer.
  • Ensure that the data pipelines are optimised.

Skillsets:

  • Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
  • Fluent in PySpark and Databricks SQL.
  • Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.

Other Requirements

Role:

  • Configure Databricks Auto Loader or Apache Spark Structured Streaming to securely subscribe to the relevant SAP AEM topics.
  • Develop the PySpark/SQL data pipelines to process the incoming JSON payloads through the Delta Lake layers.
  • Carry out data modelling in Databricks.
  • Expose the Delta tables to the visualisation layer.
  • Ensure that the data pipelines are optimised.

Skillsets:

  • Expert in Apache Spark Structured Streaming and Databricks Auto Loader.
  • Fluent in PySpark and Databricks SQL.
  • Proven experience building Databricks architecture patterns.

Familiarity with Unity Catalog for data governance and access control.
Apply now
Information at a Glance

Why HCLTech?

At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first.

HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026 totaled $14.8 billion.