Senior Azure Databricks Data Engineer

Senior Azure Databricks Data Engineer - Banking Client

Location: Brussels, Belgium – Hybrid

Rate: Flexible

Onsite: 8 days per month in Brussels

Remote: Remaining days can be worked remotely

Engagement: Umbrella preferred; Belgian company/BV can also be considered

Duration: 6 months

The Opportunity

We are looking for an experienced Senior Azure Databricks Data Engineer to join a large-scale enterprise data transformation programme within a complex financial services environment.

This is a hands-on engineering position for someone who enjoys designing, developing and optimising production-grade data solutions rather than operating purely at architecture or management level.

You will work within a modern Microsoft Azure and Databricks ecosystem, building scalable data pipelines, Lakehouse solutions and reusable engineering components that support analytics, applications and emerging AI/ML use cases.

The role combines Data Engineering, Databricks/Spark development and software engineering, so we are particularly interested in engineers with strong coding skills and experience taking data solutions from design through to production.

What You'll Be Doing:

• Design, build, test and maintain scalable data engineering solutions using Microsoft Azure and Azure Databricks.
• Develop production-grade ETL/ELT pipelines using Python, PySpark, Scala and SQL.
• Build data-processing applications covering ingestion, transformation, enrichment, validation and serving.
• Develop Delta Lake / Lakehouse solutions using modern data engineering patterns.
• Design and implement Bronze, Silver and Gold / Medallion architectures where appropriate.
• Build reliable batch and streaming data-processing workflows.
• Develop curated datasets and transformation layers supporting analytics, reporting, applications and AI/ML use cases.

Databricks & Apache Spark

• Develop and maintain Databricks notebooks, jobs and workflows.
• Build and optimise Apache Spark / PySpark workloads operating across large datasets.
• Improve Spark performance through appropriate partitioning, caching, cluster configuration and query optimisation.
• Implement schema evolution, incremental processing and robust data-quality controls.
• Develop reusable Spark/Python components, libraries and engineering frameworks.
• Apply appropriate Delta Lake optimisation and data-management techniques.
• Troubleshoot and optimise production data pipelines for performance, scalability, reliability and cost.

Azure Data Platform

Work across a modern Azure data ecosystem including:

• Azure Databricks
• Apache Spark / PySpark
• Delta Lake
• Azure Data Lake Storage (ADLS)
• Azure Data Factory
• Azure Synapse
• Azure Event Hubs / streaming patterns
• Azure Key Vault
• Azure DevOps
• Azure monitoring and logging capabilities

You will work closely with Cloud, Architecture and Platform teams to ensure solutions are secure, scalable, observable and aligned with enterprise standards.

Software Engineering & Application Development

This role goes beyond traditional ETL development.

You will apply strong software engineering practices to data applications, including:

• Modular and reusable development
• Clean, maintainable code
• Automated testing and validation
• Error handling and logging
• Code reviews
• Git/version control
• Technical documentation
• Reusable libraries and frameworks

You will be expected to contribute to the overall quality of the engineering environment rather than simply delivering individual pipelines.

DevOps & CI/CD

• Build and maintain CI/CD processes for data applications.
• Deploy Databricks and data-engineering code across development, test and production environments.
• Work with Azure DevOps and YAML pipelines.
• Collaborate with DevOps and Cloud teams on environment configuration and deployment.
• Apply release-management and environment-promotion best practices.
• Work with Terraform / Infrastructure as Code where required.

Terraform expertise is beneficial, but this is primarily a Data Engineering and application-development role rather than an Infrastructure Engineering position.

Data Quality, Security & Governance

• Build data-quality controls and validation into engineering pipelines.
• Implement appropriate logging, monitoring and operational alerti