Senior Data Engineer

Role Overview

Skills, Experience, Qualifications, If you have the right match for this opportunity, then make sure to apply today.

We are looking for a Senior Data Engineer with strong experience in Python, PySpark, Databricks, AWS, and SQL to join the Downstream Data & IT team in Brussels.

The consultant will be responsible for developing, integrating, processing, and maintaining data solutions across multiple data sources and platforms. The role will focus on ensuring data quality, data integration, ETL processes, Data Lake management, and production stability.

The candidate will work closely with Data Analysts, Data Scientists, IT teams, business stakeholders, and the Chief Data Officer to ensure reliable and high-quality data delivery.

Key Responsibilities

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Technical Skills

Mandatory

• Python 3 – Strong experience
• PySpark – Strong experience
• Databricks – Strong experience
• AWS – Strong experience
• SQL / Relational Databases / Data Warehousing – Strong experience

Good to Have

• Apache Airflow
• Azure DevOps / TFS / VSTS
• Terraform
• Postman / Insomnia
• CI/CD
• C#

Databricks Experience

Candidates should have experience with:

• Creating and executing ETL tasks
• Managing Spark clusters
• Developing data pipelines
• Data transformation and processing
• SQL query development and optimization
• Data Lake environments
• Performance tuning

Functional Skills

• Strong understanding of relational databases, SQL, and data warehouses.
• Experience working with Databricks data engineering platforms.
• Ability to collaborate with technical and business teams.
• Strong analytical and problem-solving skills.
• Ability to work independently and proactively.
• Good understanding of data management and data quality.
• Ability to understand business requirements and deliver suitable technical solutions.
• Experience working in Agile environments is a plus.

Nice to Have – Domain Experience

Experience in any of the following will be an advantage:

• Energy / Utilities industry
• Energy trading
• Portfolio management
• Risk management
• Energy forecasting
• Gas and power markets

Key Responsibilities in the Data Management Area

The consultant will help ensure the quality and usability of enterprise data by:

• Collecting data from different applications and external sources.
• Integrating data from multiple systems.
• Structuring and standardizing data.
• Mapping available data elements.
• Cleaning and removing duplicate data.
• Validating incoming data.
• Supporting the creation and maintenance of data repositories.
• Ensuring reliable data availability for Data Analysts and Data Scientists.

Soft Skills & Behavioral Competencies

The candidate should demonstrate:

• Strong analytical and problem-solving abilities.
• Excellent communication and interpersonal skills.
• Ability to work with multiple stakeholders.
• Strong attention to detail and quality.
• Ability to work in a fast-paced, multidisciplinary environment.
• Ability to handle confidential and sensitive customer data.
• Ability to simplify and abstract complex technical topics.
• Creative and entrepreneurial approach to problem-solving.
• Strong ownership and accountability.
• High level of integrity and ethical standards. xlxgzvr

Deliverables

The consultant will be expected to provide:

• Monthly time tracking
• Activity reports
• KPI dashboards
• Workload estimations