Senior Data Engineer/Architect

Il y a 3 jours

Brussels, Brussels-Capital, Belgique Virtual Connect Solutions Temps plein 60 € - 70 € CDI

Gratuit avec email ou Google

Enregistrez cette offre et organisez votre recherche

Créez un compte gratuit pour enregistrer des offres d'emploi, créer des alertes et revenir à cette liste depuis votre tableau de bord.

Enregistrer le travail Enregistrer et créer une alerte

Gratuit avec email ou Google

Design and deploy ingestion pipelines from heterogeneous sources: files, APIs, relational databases. Implement pipelines for incremental and streaming ingestion. Ensure the reliability, idempotency and traceability of each pipeline. Transformation & Semantic Layer Develop the transformation models on the Bronze, Silver and Gold layers. (Medallion Architecture) Implement good practices for data historization (e.g. SCD2, upsert, append only,...). Expanding exports to Delta Lake Transactions ACID & Time Travel Manage and maintain Delta Lake tables. Ensure transactional consistency of data in a multi-domain environment. Orchestration Implement and maintain the assests to be orchestrated. Monitor runs, handle errors, and automate alerts. Governance & Lineage Push technical and business metadata to the data platform. Maintain lineage from start to finish. Contribute to the drafting of DUAs, SLAs and associated governance documents. Data exposure Develop endpoints to expose the Gold layer to BI Dashboards. Collaborate with reporting teams to feed BI dashboards Desired profile Technical Skills Required Proficiency in Python Proficiency in SQL Proven experience with dbt-core (templates, snapshots, macros, tests, profiles). Good knowledge of DuckDB as an embedded analytical query engine. Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM, ...). Knowledge of relational basics: MSSQL, PostgreSQL (Advanced SQL, CDC). Experience with a data orchestrator: Dagster, Airflow or equivalent. Comfortable with Linux/OpenShift and PowerShell (Windows dev environment). Familiarity with governance tools: DataHub, OpenMetadata, or equivalent. Experience in BI development. Skills Valued Kafka/Debezium experience for real-time data capture. Connaissance de Spark (Spark SQL, Thrift Server, Beeline). Experience with Power BI Report Server (PBIRS) or Apache Superset. Sensitivity to data security in restricted environments. Have knowledge of Lakehouse (Medallion architecture), DWH and Datalake concepts

Skills:
data,lake,dagster

Senior Data Engineer/Architect

Enregistrez cette offre et organisez votre recherche

Emplois similaires à Brussels

Offres similaires

Amériques

Europe

Asie / Océanie

Afrique

Senior Data Engineer/Architect

Enregistrez cette offre et organisez votre recherche

Emplois similaires à Brussels

Offres similaires