Senior Data Engineer

Applying for this role is straight forward Scroll down and click on Apply to be considered for this position.

Role Overview

We are looking for a Senior Data Engineer with strong experience in Python, PySpark, Databricks, AWS, and SQL to join the Downstream Data & IT team in Brussels.

The consultant will be responsible for developing, integrating, processing, and maintaining data solutions across multiple data sources and platforms. The role will focus on ensuring data quality, data integration, ETL processes, Data Lake management, and production stability.

The candidate will work closely with Data Analysts, Data Scientists, IT teams, business stakeholders, and the Chief Data Officer to ensure reliable and high-quality data delivery.

Key Responsibilities

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Technical Skills

Mandatory

• Python 3 – Strong experience
• PySpark – Strong experience
• Databricks – Strong experience
• AWS – Strong experience
• SQL / Relational Databases / Data Warehousing – Strong experience

Good to Have

• Apache Airflow
• Azure DevOps / TFS / VSTS
• Terraform
• Postman / Insomnia
• CI/CD
• C#

Databricks Experience

Candidates should have experience with:

• Creating and executing ETL tasks
• Managing Spark clusters
• Developing data pipelines
• Data transformation and processing
• SQL query development and optimization
• Data Lake environments
• Performance tuning

Functional Skills

• Strong understanding of relational databases, SQL, and data warehouses.
• Experience working with Databricks data engineering platforms.
• Ability to collaborate with technical and business teams.
• Strong analytical and problem-solving skills.
• Ability to work independently and proactively.
• Good understanding of data management and data quality.
• Ability to understand business requirements and deliver suitable technical solutions.
• Experience working in Agile environments is a plus.

Nice to Have – Domain Experience

Experience in any of the following will be an advantage:

• Energy / Utilities industry
• Energy trading
• Portfolio management
• Risk management
• Energy forecasting
• Gas and power markets

Key Responsibilities in the Data Management Area

The consultant will help ensure the quality and usability of enterprise data by:

• Collecting data from different applications and external sources.
• Integrating data from multiple systems.
• Structuring and standardizing data.
• Mapping available data elements.
• Cleaning and removing duplicate data.
• Validating incoming data.
• Supporting the creation and maintenance of data repositories.
• Ensuring reliable data availability for Data Analysts and Data Scientists.

Soft Skills & Behavioral Competencies

The candidate should demonstrate:

• Strong analytical and problem-solving abilities.
• Excellent communication and interpersonal skills.
• Ability to work with multiple stakeholders.
• Strong attention to detail and quality.
• Ability to work in a fast-paced, multidisciplinary environment.
• Ability to handle confidential and sensitive customer data.
• Ability t