QA & AI Evaluation Engineer - Antwerp office

Delen Private Bank is a private wealth manager with a well-established IT team of 200 colleagues within an organisation of 800 employees. We support our clients through discretionary wealth management and wealth planning, and are strongly committed to automation through our strategically developed in-house core banking software.

Role and context

As QA & AI Evaluation Engineer, you safeguard the quality, reliability and safety of the AIHub Platform. This is Delen Private Bank’s central in-house AI platform, integrated into the bank’s processes and systems.

AI solutions do not always behave predictably. That is why quality here goes beyond "does the functionality work as intended?" You define how we measure whether an AI application is correct, consistent, safe and useful. You also check that it stays that way after changes to models, prompts or data.

You work closely with AI engineers, developers, business analysts, domain experts and stakeholders. You combine strong QA expertise with analytical insight into AI systems. You translate business expectations into measurable evaluation criteria. You build and maintain evaluation datasets and embed automated evaluations in our delivery pipelines.

Responsibilities

Analysis & Evaluation Design

• Work with business analysts and domain experts to translate expectations about AI behaviour into measurable acceptance and evaluation criteria.
• Define evaluation datasets (golden datasets) that represent real-world usage scenarios. Include edge cases and failure scenarios.
• deterministic: e.g. format validation, schema checks, exact or structured matches, mandatory source citation, correct tool calls;
• non-deterministic: e.g. LLM-as-a-judge, semantic similarity, groundedness and relevance.
• Calibrate LLM-as-a-judge evaluations against human judgements, so that automated scores stay reliable.
• Help shape the evaluation strategy, coverage and quality thresholds per use case and for the platform as a whole.

Test Execution & Validation

• Run functional and non-functional tests on platform components: APIs, integrations, data processing and user interfaces.
• Evaluate RAG applications for retrieval quality, groundedness and correct source citation.
• Test agentic flows: correct tool selection, parameters, multi-turn behaviour and error handling.
• Perform adversarial testing and red teaming: prompt injection, jailbreaks, leakage of confidential or personal data, unwanted or misleading output.
• Validate that guardrails and content filters work.
• Log bugs in Azure DevOps, actively follow up on their resolution and validate fixed issues.
• Facilitate UAT sessions with end users and collect structured feedback.
• Monitor regression on model upgrades and prompt changes, and compare versions objectively.
• Maintain evaluation datasets based on online evaluations and production feedback. You turn unforeseen edge cases into permanent test cases within the standard eval sets.
• Track quality in production through monitoring and tracing (quality scores, drift, user feedback) and flag degradation early.
• Use AI tooling yourself to generate test cases and synthetic test data and to analyse results faster.
• Help build reusable evaluation standards, templates and guidelines, so that teams building on AIHub can measure the quality of their own use cases.
• Write annotation guidelines and involve domain experts in reviewing and labelling output.
• Communicate findings clearly to technical and non-technical stakeholders.

Profile and competencies

• Excellent Dutch and good English. French is a plus (multilingual evaluation).
• Experience as a QA Engineer in an agile IT environment, with strong analytical skills.
• You have hands-on experience testing or evaluating LLM applications, or you show clear maturity in this domain.
• You understand the challenges of non-deterministic systems and report on them reliably and with statistical rigour.
• You have a critical and creative "break it" mindset to find edge cases and abuse scenarios.
• Accurate, structured and quality-driven.
• Team player with excellent communication skills.

Basic knowledge of or experience with:

• Python for test automation and evaluation scripts (e.g. pytest)
• LLM evaluation frameworks (e.g. LangSmith, Azure AI Foundry evaluations, promptfoo, DeepEval, Ragas, Langfuse)
• Concepts such as RAG, embeddings, prompt engineering, agents and tool calling
• API testing (e.g. Postman, REST, Open