Data lead

Databricks Data Lead

Role Overview

We are seeking an experienced Databricks Data Lead to provide technical leadership in the design, development, and delivery of modern cloud-based data platforms using Databricks. The role will be responsible for establishing data engineering standards, leading technical delivery, and ensuring that data solutions are scalable, secure, reliable, and aligned with business objectives.

The Databricks Data Lead will work closely with data architects, data engineers, analysts, data scientists, business stakeholders, and cloud engineering teams to deliver enterprise data and analytics capabilities.

Key Responsibilities

• Lead the design and implementation of scalable enterprise data platforms using Databricks.
• Provide technical leadership across data engineering, data integration, data transformation, and analytics initiatives.
• Define and implement modern data architecture patterns using the Databricks Lakehouse architecture.
• Design and implement data pipelines using Apache Spark, PySpark, SQL, and Databricks.
• Establish standards and best practices for data ingestion, transformation, processing, storage, and consumption.
• Lead the development of batch and real-time/streaming data pipelines.
• Design and implement data lake and lakehouse solutions using technologies such as Delta Lake.
• Define approaches for data modelling, data partitioning, optimisation, and storage management.
• Lead the implementation of data governance, security, access control, data quality, lineage, and metadata management.
• Work with architects and engineering teams to establish appropriate integration patterns between Databricks and enterprise data sources and platforms.
• Drive performance optimisation and cost management across Databricks workloads.
• Establish development standards covering coding, testing, deployment, monitoring, and operational support.
• Implement and promote CI/CD and DevOps practices for data engineering solutions.
• Support Infrastructure as Code and automated deployment approaches where appropriate.
• Lead technical design reviews and provide guidance on complex data engineering challenges.
• Mentor and support data engineers, helping establish consistent engineering practices and technical standards.
• Work with delivery managers and product owners to translate business requirements into technical solutions.
• Estimate technical effort, identify dependencies and risks, and contribute to delivery planning.
• Troubleshoot and resolve complex data pipeline, performance, reliability, and integration issues.
• Produce and maintain technical documentation, architecture diagrams, design specifications, and engineering standards.
• Evaluate new Databricks capabilities and emerging data technologies and recommend opportunities for adoption.

Required Skills and Experience

• Significant experience in data engineering, data architecture, or cloud data platforms.
• Strong hands‑on experience with Databricks and the Lakehouse architecture.
• Strong knowledge of Apache Spark, PySpark, and SQL.
• Experience designing and implementing enterprise‑scale data pipelines.
• Strong understanding of Delta Lake and modern data lake/lakehouse architectures.
• Experience working with cloud platforms such as Azure, AWS, or Google Cloud Platform.
• Strong understanding of data warehousing, data lakes, data modelling, and data integration.
• Experience developing both batch and streaming data solutions.
• Strong understanding of data quality, governance, security, and access‑control principles.
• Experience with source control, CI/CD, automated testing, and DevOps practices.
• Experience with data orchestration technologies such as Databricks Workflows, Apache Airflow, Azure Data Factory, or equivalent.
• Experience integrating data from relational databases, APIs, files, event streams, and enterprise applications.
• Strong programming experience in Python and/or Scala.
• Experience leading technical teams or providing technical leadership to data engineering teams.
• Strong problem‑solving, communication, and stakeholder‑management skills.

Desirable Skills

• Experience with Unity Catalog and Databricks governance capabilities.
• Experience with Delta Live Tables / Lakeflow Declarative Pipelines.
• Experience with Databricks SQL and SQL Warehouses.
• Experience with Structured Streaming and technologies such as Kafka.
• Experience with dbt or other modern data transformation frameworks.
• Experience with Terraform or other Infrastructure as C