Databricks Data Platform Lead

ph3Databricks Data Lead /h3h3Role Overview /h3pWe are seeking an experienced strongDatabricks Data Lead /strong to provide technical leadership in the design, development, and delivery of modern cloud-based data platforms using Databricks. The role will be responsible for establishing data engineering standards, leading technical delivery, and ensuring that data solutions are scalable, secure, reliable, and aligned with business objectives. /ppThe Databricks Data Lead will work closely with data architects, data engineers, analysts, data scientists, business stakeholders, and cloud engineering teams to deliver enterprise data and analytics capabilities. /ph3Key Responsibilities /h3ulliLead the design and implementation of scalable enterprise data platforms using strongDatabricks /strong. /liliProvide technical leadership across data engineering, data integration, data transformation, and analytics initiatives. /liliDefine and implement modern data architecture patterns using the Databricks Lakehouse architecture. /liliDesign and implement data pipelines using strongApache Spark, PySpark, SQL, and Databricks /strong. /liliEstablish standards and best practices for data ingestion, transformation, processing, storage, and consumption. /liliLead the development of batch and real-time/streaming data pipelines. /liliDesign and implement data lake and lakehouse solutions using technologies such as strongDelta Lake /strong. /liliDefine approaches for data modelling, data partitioning, optimisation, and storage management. /liliLead the implementation of data governance, security, access control, data quality, lineage, and metadata management. /liliWork with architects and engineering teams to establish appropriate integration patterns between Databricks and enterprise data sources and platforms. /liliDrive performance optimisation and cost management across Databricks workloads. /liliEstablish development standards covering coding, testing, deployment, monitoring, and operational support. /liliImplement and promote CI/CD and DevOps practices for data engineering solutions. /liliSupport Infrastructure as Code and automated deployment approaches where appropriate. /liliLead technical design reviews and provide guidance on complex data engineering challenges. /liliMentor and support data engineers, helping establish consistent engineering practices and technical standards. /liliWork with delivery managers and product owners to translate business requirements into technical solutions. /liliEstimate technical effort, identify dependencies and risks, and contribute to delivery planning. /liliTroubleshoot and resolve complex data pipeline, performance, reliability, and integration issues. /liliProduce and maintain technical documentation, architecture diagrams, design specifications, and engineering standards. /liliEvaluate new Databricks capabilities and emerging data technologies and recommend opportunities for adoption. /li /ulh3Required Skills and Experience /h3ulliSignificant experience in strongdata engineering, data architecture, or cloud data platforms /strong. /liliStrong hands-on experience with strongDatabricks /strong and the Lakehouse architecture. /liliStrong knowledge of strongApache Spark, PySpark, and SQL /strong. /liliExperience designing and implementing enterprise-scale data pipelines. /liliStrong understanding of strongDelta Lake /strong and modern data lake/lakehouse architectures. /liliExperience working with cloud platforms such as strongAzure, AWS, or Google Cloud Platform /strong. /liliStrong understanding of data warehousing, data lakes, data modelling, and data integration. /liliExperience developing both batch and streaming data solutions. /liliStrong understanding of data quality, governance, security, and access-control principles. /liliExperience with source control, CI/CD, automated testing, and DevOps practices. /liliExperience with data orchestration technologies such as strongDatabricks Workflows, Apache Airflow, Azure Data Factory, or equivalent /strong. /liliExperience integrating data from relational databases, APIs, files, event streams, and enterprise applications. /liliStrong programming experience in Python and/or Scala. /liliExperience leading technical teams or providing technical leadership to data engineering teams. /liliStrong problem-solving, communication, and stakeholder-management skills. /li /ulh3Desirable Skills /h3ulliExperience with strongUnity Catalog /strong and Databricks governance capabilities. /liliExperience with strongDelta Live Tables / Lakeflow Declarative Pipelines /strong. /liliExperience with Databricks SQL and SQL Warehouses. /liliExperience with Structured Streaming and technologies such as Kafka. /liliExperience with dbt or other modern data transformation frameworks. /liliExperience with Terraform or other Infrastructure as Code technologies. /liliExperience implementing data mesh or domain-oriented data architectures. /liliExperience with data catal