Senior Databricks Data Engineer

Il y a 4 heures

Zaventem, Flanders, Belgique MPower Plus Temps plein

Mandatory Skills: Databricks, Data Engineering, Lakeflow, Unity Catalogue, Python, PySpark, ETL.

About the Role:

We are looking for an experienced Senior Databricks Data Engineer with strong expertise in Databricks, SQL, Lakeflow Jobs, PySpark notebooks, Python, and Unity Catalogue. This role requires hands-on experience in designing scalable data pipelines and delivering reliable data solutions on the Databricks Lakehouse platform.

Key Responsibilities:

  • Design, develop, and optimize data pipelines using Databricks, Python, PySpark, and Spark SQL.
  • Orchestrate and schedule workflow jobs that invoke notebooks, Python wheel packages, and Python notebooks.
  • Work with Unity Catalog, manage tables, and streaming jobs.
  • Implement and manage data ingestion, transformation, and loading processes from various sources into the Databricks Lakehouse.
  • Build and maintain efficient and reliable ETL/ELT processes to ensure data quality and availability.
  • Optimise Databricks workloads for performance, cost efficiency, and scalability.
  • Troubleshoot and resolve data-related issues while ensuring data integrity and accuracy.
  • Stay current with the latest Databricks features, best practices, and industry trends.

Required skills & Qualifications:

  • At least 4 years of proven experience as a Data Engineer with a strong focus on Databricks (mandatory).
  • Strong knowledge of Databricks architecture, Lakeflow Jobs, and data modelling.
  • At least 10 years of experience in the data domain, particularly in data warehousing and ETL (mandatory).
  • Expertise in PySpark, Spark SQL, and Python programming (mandatory).
  • Hands-on experience with Git, including code check-in and check-out processes (mandatory).
  • Strong understanding of data warehousing, data lakes, and Lakehouse architecture.
  • Databricks Certified Data Engineer Associate or Professional certification is an advantage.

Soft Skills:

  • Strong communication and stakeholder collaboration skills.
  • Proactive, self-driven, and accountable mindset.
  • Excellent analytical and problem-solving abilities.
  • Ability to work effectively in Agile and cross-functional teams.
  • Customer-focused with a commitment to quality delivery.
  • Ability to converse seamlessly in English and one of either Dutch or French or both